A few days ago, my agency mate Natalie had a great discussion on her blog about how the middle of a book is often the toughest part to write.
For me the problem is that I know what's going to happen (um, presumably), and I sometimes forget that the reader doesn't. So while the late-middle may feel to me like I'm dragging things out along the only possible path, I have to make sure that path isn't obvious to anyone else; for the reader, it should feel like the story could go in any direction.
Even if you're not writing a mystery, you're still working backwards from a set conclusion, opening doorways for characters to peek through en route to their predestination. It's my job to make sure the ending is both surprising and inevitable.
But when I'm in the middle, argh, that seems like a daunting challenge indeed.