The short version is simply that there's a huge amount of data being sent across the network and then processed and finally rendered by your computer. We've looked at the logs for the machine hosting prettymaps and it seems happy enough serving all the requests that people are sending it.
The other reason is that the data hasn't been compressed or optimized as much it could be. For example, the four separate image layers could be collapsed in to a single layer. Likewise the data layers, although we've found a few more places to tighten things up since we first launched.
We haven't done a lot of optimizations by design because we wanted to leave all the plumbing exposed so that people could look at it and learn from it and, hopefully, build something new.
Which is cold comfort if you're one of those people waiting on the ghosts in the Internets to hurry up and show you what all the fuss is about so we've also created a static, non-interactive, version of the map so you can at least get a flavour for the work.