You're not alone there, Klyde. I've never been crazy about gaming on handhelds. My PSP and DS libraries are filled with unfinished games. I think it's the ergonomics (or lack thereof) that really gets to me. Leaning over and squinting at some tiny screen for long sessions is not comfortable, and these systems don't exactly feel good in your hands either.
But oddly enough I don't have any aversion to gaming on my Vita. I seriously love the thing. I dismissed that as new gadget syndrome at first, but it's been quite a while and I still feel that way. Must be that big, beautiful screen I guess. In a lot of ways I prefer it to console gaming since it's much faster to get into the games. (Instead of having to turn on my TV and sound system, boot up the console, select a game and go through all the loading screens and stuff everytime...) Now if only it had more games...well, I still play PSP games on it and there are a ton of those.
I haven't gotten a 3DS yet, but I've tried one before and wasn't impressed. Ergonomically it's pretty much just the same as the DS, and the 3D affect gives me headaches. Can't deny that the library is looking pretty impressive, though...