I may have just found the first half-decent, quick and small image viewer that suits my expectations.
JustLooking

It can be configured as default viewer for all your image files and when you open an image it has a nice interface that lets you skip forward and back in the directory from where you opened the file. JustLooking also has a nice fullscreen option and can rotate images (90° only).

I think this app may have a good chance of replacing Preview as my default image viewer.