Probably because the game hasn't really opened up yet, but immediately afterwards you get magic. All the 2300 dungeons containing some kind of puzzle or gimmick doesn't really help either.
Some belated additions:
The entire Fourth Stratum in the first EO. This is due mostly to the game's limitations in mapping compared to the newer titles, but it makes the Stratum an absolute nightmare and utterly unfun.
Air's rock in Golden Sun 2.
Kalm in FFVII. I admit this is probably because I think Midgar is the best part of the game, but Kalm is always, always where I lose interest when I attempt to replay the game.
Just about every ghost ship ever. I don't think I've ever found a game I enjoyed a ghost shop segment in. I'd welcome recommendations, though.
Again, Golden Sun 2 isn't really a great game or anything above mediocre. And if I had to name the most annoying part in any Golden Sun game it would be anytime a text box appeared and heads started bobbin'.
FFVII: Coral Prison. How many screens must a man walk down?
SD3's Ghost Ship is kinda annoying if only because its also part of that Volcano Island business and the boss can be a bitch with the wrong party setup but that's more dependent on your choices rather than anything the game does.
If I remember Grandia's Ghost Ship correctly, its the only time in the plot where Feena isn't portrayed as incompetent or a damsel in distress (though I could be wrong).
Ironically, Fire Emblem 8's Ghost Ship is probably the only really challenging map in the main game. Especially on Hard Mode and without grinding.
Wind Waker's Ghost Ship is only kinda annoying due to being impregnable without the Ghost Ship Chart which only really becomes available after you get the Hookshot at the end of the game (and holds one, of like two, Triumph Forks charts that can only be gotten by the end game).
And speaking of Wind Waker: The Stealth bullshit.