I'd agree. The best portfolio sites are like that, but personally I prefer sites with a clean and simple layout to those with a rough look. A subtle use of colour, borders and so on can be stylish and still effectively emphasise the work, but without being 'wizzy'.
There are various ways of doing this, but for one example, I like this (by Dave):

http://www.4160.co.uk/