I wonder why we all accept that photos must be rectangular? Won't things be much more interesting if we have parallelogram shaped pictures? Or circular, keyhole shaped, or blobby? Are we so fixed in our thinking that we can't handle other shapes and sizes? How will the rule of thirds fit in? I think you could use different shapes to emphasise different parts of the image as well - such as if you have a cross shaped photo, then the intersection would have all the attention.
These days, windows in microsoft windows can be any shape and size, and I'm sure some smart cookie can make some software that makes managing these odd shapes a breeze. Did you know that lenses are circular and we chop off great chunks of the usable image to make it rectangular?
Anyway, just some food for thought.
If you play around too much with the shape of the photo it distracts from the photo subject. Instead of being a picture of something with the thing of the photo playing a subordinate role, it becomes an art object and the subject plays a subordinate role.
Actually, I think photos can be keyhole-shaped or circular. Like, if you take a shot which really fits in a keyhole-shaped or circular frame, it'd be wonderful. This kind of frame would increase the emotion you get from looking at the shot. But like Don said, it'd be more an art piece than a photography.
Anyway, Shuttertalk, you can give it a try and then we'll comment your invention.
EDIT:
I did some research in the Internet today and found two photos from an Estonian photo critique website which umn.. let's say 'are not the shape we are used to see photos'
http://hagar.nomad.ee/pildid/images/1003485782.jpg A fish-eye shot, looks circular if you look it on a black background.
http://hagar.nomad.ee/pildid/images/1011482706.jpg A rounded rectangular photo, works on a black background.
There was a time when lots and lots of portraits were cut and framed in oval shapes, wasn't there? actually, I kind of like that!
or of course and amulet (charm), who would wear a square amulet to fit a picture?
I guess we are used to looking through squares and rectangles. Windows, books, tv's, cinema - it is all the same. I think other shapes are nice once in a while - but ultimately the rectangle/square is best.
You could always set up an action in Photoshop to create a layer mask to give you a keyhole look.
That's a really good point there Chris. I suppose historically you could add to that things like paintings and icons, so that the westernised expected "view" in terms of pictorial representation is rectangular. I'd agree with Don too; I don't suppose we'd want the frame pulling the eye from the actual subject.