For me, it's important for immersion, specially in an anime-esque RPG. When the voices are good, then it's not much of a deal (it does not add a huge value to the game), but when it's bad, the "flow" of the game is broken every time i'm immersed in story or gameplay and have to hear that out-of-place voice, so it's a big minus for me.
Regarding turning the voices off... perhaps that can be a solution. I'll still feel like something is wrong, but it can make it playable.
Regarding gibberish, i understand almost all of spoken japanese (but can't read kanji).
Regarding the proud language of nation of birth joke, English is not my first language, so it does not apply.
Those seem like good reasons, but i can't help but feel that Atlus is losing a portion of possible buyers.