I think that, first, there are a lot of flaws in iOS device syncing (as many readers have shared similar problems with me). But I also believe that some sort of database file on iOS devices can get corrupted if you interrupt syncs before they've finished. I restored my iPhone a few weeks ago and I now make sure to always let the sync finish before unplugging it. In the past I would sometimes click the X icon in the iTunes LCD to stop syncs, and I suspect that caused some corruption.
I suggest you restore the iPod touch, then sync all your music, and try to make sure to never stop the sync when it's in progress. It will still take a while--it will never be as fast as syncing an iPod classic--but this may prevent the very long syncs you've been experiencing.
Sign up for CIO Asia eNewsletters.