Hey Mr. Nervous, you really are, hey? I'm not an experienced person in terms of moving jobs but my advice first off is to think positively about it all! Presumably you're changing jobs because you are looking for something more challenging or different. Sounds like a great idea. If that is genuine, I don't think any employer would think badly of you, they'll probably just be sad to see you go.. (And by the way, I don't think there's much incentive for giving bad references, so don't worry about that too much.)
I think any move will be a lot easier if it makes sense for you both professionally and on a personal level, so take time to find the right job. Make sure that you meet as many of the people you will be working with first and get a really good idea of what it is you'll be doing (preferably from someone with no vested interest in giving a biased opinion.) That should minimise your "what if" scenarios.