From the Tv Tropes website:
Regardless of how frail, small or seemingly underpowered the Action Girl might look, when the chips are down she is the embodiment of badassery in a small packet, whether for good or evil. Normally used with a lot of acrobatics and turns while fighting, probably implying some kind of mystic martial arts that focuses on agility and precise hits.
http://tvtropes.org/pmwiki/pmwiki.php/Main/WaifFu
Now I can understand that for some people certain things break their suspension of disbelief. But it seems that this issue of "no one that small could take on someone that big" is only ever a problem when it comes to female characters. I certainly don't recall any complaints about Spider-Man, Aang from A:TLA, pre-DBZ Goku or Dre Parker (Jaden Smith's character in the 2010 Karate Kid film) being too skinny to be effective fighters.