I get your point, I really do but sub-classification and genrefication of music, animation and comics helps. Somebody might like rock music only, another one likes pop music only. Same with comics and animation, somebody likes bugs bunny cartoons but not anime, somebody likes mecha anime but not slice of life anime. That's why these terms are easier and help people to describe in short what kind of music/animation/comics they like. I don't like all kinds of music or everything animated or every comic but I still consider myself a fan of all of these. So I don't see genrefication being the reason for dividing fans into different camps, we mostly do it ourselves. And even from marketing/promotion point it helps fans also to classify these things because when I hear about a comic it helps me to estimate whether I might enjoy it or not and whether I should spent money on it when I hear if it's an American superhero comic or a Japanese high school drama comic. Because I don't usually care for the latter type while I usually love the former.
All in all I don't have any problem to call manga comics and anime cartoons and often I do use them that way because it's a shorter and quicker way to describe my interest by saying that I'm a fan of animation and comics than I'm a fan of American and European comics, manga, anime and western animation/cartoons. However when I discuss with someone online for example about our interests then it's easier to use terms like anime and manga to separate Japanese cartoons and comics so when they say they read manga then I immediately know what they are talking about.
There's also the fact that anime and manga are usually made in Japan by Japanese and mainly for the Japanese first and foremost so cultural differences affect the product and make it very different from its western counterparts both stylistically and thematically. Some people that like western animation and comics don't care about Japanese because they can't get over the style differences. And I have sometimes had trouble watching anime series because I didn't know all of the Japanese mythology or all the folk tales involved or referenced in them. So Japanese products getting their own terms in US/Europe while European/US products don't get them is more justified because there are more differences when you compare American comics/animation to Japanese comics/animation than if you compare them to European comics/animation.
Of course it varies with some anime being very western influenced (Cowboy Bebop) while some western cartoons are very anime influenced (Teen Titans, Avatar) but usually there are some distinct differences. So separate terms like manga and anime aren't that bad but they're still not that important to me and I could honestly go either way with it and can't see why anyone would care about it that much because at the end of the day it is all comics and animation after all.
I do however think that words manga and anime have now become so associated as the terms for Japanese comics and cartoons in western world that it would probably be kind of difficult to get people to start talking about them under the umbrella of comics and cartoons. For example in my mind at least, if somebody mentions the word cartoon I first think of funny animals and slapstick, Bugs Bunny, Garfield, Daffy Duck, Woody Woodpecker, Flintstones or Scooby-Doo or something classic like that but when someone mentions the word anime I think of practically noseless, big eyed girls in school girl uniforms with rainbow colored hairs, giant robots and maybe martial arts, some weird and violent/gorey **** (and tentacles too sometimes, lol), you know, Haruhi Suzumiya, Sailor Moon, DBZ, Gundam, Miyazaki's work, NGE, and what have we.
Incidentally if somebody adds Saturday and morning in front of the word cartoons I start thinking about TMNT, Transformers, Thundercats and before or after school (cartoons) make me think of Ducktales, Goof Troop, Darkwing Duck etc.