Tom (giblet) and I spent the afternoon working on image and profile caching in Feh. In the process we did some profiling and ran across a few bottlenecks. At the moment the image and pixmap caching isn't stable at all, but if you want to play with it, here's the diff. It's against the latest CVS (which has the other speed improvements already).
Update:
You'll also need to drop these
files in feh/src in order to get the beast to compile.