Colourising Stereoscopic Glass Plates with Deep Learning
13 years ago, in 2006, I was commissioned to produce a stereoscopic movie by the Tasmanian Museum and Art Gallery (TMAG) for their then-new ‘Islands to Ice‘ Antarctic exhibition, based upon stereoscopic glass plates taken during the 1911-14 Australasian Antarctic Expedition. It arose from research I was doing at the time at the University of Melbourne.
Time to freshen it up with a new approach, using some new techniques from Deep Learning (Machine Learning) and much better software and hardware available these days. It was pretty much on the bleeding-edge at the time, but now, of course, could be remade in glorious Stereoscopic 4K High-Dynamic Range (HDR), surround sound – and, crucially, colour.
It’s just an experiment at this stage, as I develop a workflow.
Here is a low-rez (1920x720px) version suitable for viewing on mobile VR (of course, the iPhone didn’t even exist in 2006). I’ll plug away at a completely re-authored high-rez version when I have time or money – or preferably both. It probably needs a voice-over as well.
As this movie was intended for projection rather than VR, it needs a bit of work on the interocular separation and convergence (much easier now than in 2006), but for those of you with athletic eyeballs it should work fine. Switch to fullscreen or view in the vimeo app.
Hurley used a variety of large format cameras by Newman and Guardia (wooden-box construction), as well as a 1908 Folmer and Schwing (“Graflex”) stereo camera – perhaps one of the models listed here. These were large and difficult to operate (very un-ergonomic by today’s standards) – especially in sub-zero conditions. Parts of these remain in his darkroom in Mawson’s Huts to this day, amazingly enough.
Above is an example of one of the original stereoscopic glass plates. These were referred to as “half-plate” – which means a range of sizes. According to this source, the ‘1891 Photographic Congress’ adopted 6 ¾” x 3 ¼ ” plates (171.45 mm x 82.55mm) as a stereoscopic standard. This is a near-match to the glass plates held in the Mawson Collection. They’re not very large, but they contain extraordinary detail.
When using glass plate negatives, the plates had first to be loaded into wooden plateholders (some carried two plates, back to back). Each plate was covered by a dark-slide to prevent accidental exposure. To take a picture, the plate-holder had to be inserted into the camera (in some cases, after the camera had been focussed), then the dark-slide withdrawn. Once the plate had been exposed the dark-slide had to be re-inserted and the plate-holder removed. Removing the exposed plates and repacking them into negative boxes allowed the plateholders to be reloaded. When reloading, the photographer had to be careful not to let any light leak past the dark-slide This presented a problem in the continuous daylight of the Antarctic Summer.
The emulsion on plates and films was highly sensitive to the blue end of the spectrum but insensitive to the red. This made necessary the use of a yellow ‘screen’ (or filter) over the lens to balance out the colour response of the emulsion. This darkened the image and meant that a longer exposure time was necessary. Although this was not necessarily a problem, the ‘speed’ of the plate could limit the choice of subject. Quickly moving subjects became blurred.
Today ISO (‘speed’ or sensitivity) ratings of films tell photographers the speed of their films. But in Hurley’s day a multitude of manufacturers produced an array of plates and films that could not be directly compared objectively as to their relative sensitivity. To further complicate things, ‘Orthochromatic’ and ‘Panchromatic’ emulsions varied in their ability to correctly reproduce colours in their correct tonal relationship (greys) on ‘black and white’ film.Wilson-Roberts, C. (2004) Photography in the Mawson Gallery. Mawson Collection, South Australian Museum
For a restoration and colourisation workflow to succeed, it’s obviously best to work with high-resolution scans. I have 50+ original scans at ~15840 x 11695px resolution (made in 2003). There is a lot that can be done with them. Obviously it means a great deal of work removing scratches, spotting and other damage (most of which I have already done). Stereo is very tricky in this regard, as you have to match left and right views. Then there is the matter of geometric correction of the images, if there are optical artefacts from lens alignment, lens physics, shift of the film emulsion and so on. I have workflows for these that I’ve developed over the years, and new tools using adaptive grids that should enable better left/right registration.
I wonder if there’s a way of training an artificial neural network to align stereo pairs? It would be nice to semi-automate the process. Other approaches that come to mind: extract the 3D geometry using direct volumetric regression or photogrammetric approaches, then reshoot using virtual cameras in a 2.5D animation setup. Intriguing, but probably more trouble than its worth.
The key issue with Deep Learning (DL) colourisation is obvious: colour flicker. The ANN (GAN/CNN) does a pretty good job, but has occasional difficulty in maintaining temporal consistency/object recognition across frames – so we have the occasional ‘grey face’ effect. Some artefacts, some shifting colours, are aesthetically pleasant and give the movie an ‘old movie’ quality, others are obviously processing artefacts. The fact that DeOldify recognises clouds in blue sky and colourises them nicely is simply amazing (given the tonal range in the images – they’re virtually all white). No doubt there are many parameters to explore in order to improve the output and I will certainly be following DeOldify with great interest.
I wouldn’t like it to look like digital video, but I would like it to be crisp and distinct whilst retaining its photographic feel and colour dynamic. Obviously it would be better to colourise the stereo pairs first, and then re-shoot the movie with a virtual stereo camera – pretty easy to do these days using something like Blackmagic Fusion.
For colourisation the main obstacle is the GPU limits one encounters running DL workflows. I used an NVIDIA GTX 1080 (8GB) as the source material is low resolution (merely 960x360px), but will do future work using a GP100 (16GB). The key limit here will be the ability to process high-resolution images, something the software repertoire is slowly but surely getting better at. Alternatively, lower resolution colour maps from low resolution sources could be applied to the high resolution luminance information, as it is from this that the human eye derives detail. Useful alternatives to DeOldify include ideepcolor, which uses a manual human-intervention hinting system and operates in CIELAB colourspace, which is appealing. I suspect a strategy using multiple approaches is optimal. I can slice the images up for processing and reassemble them for final high-resolution output. Whatever works, basically.
VR, cultural heritage and ‘immersive restoration’ has been a long-term interest of mine, and I’d love to work with other stereoscopic sources – such as imagery from the First World War or other significant events and adventures. If only I owned a movie studio. Museum collections around the world are full of amazing stereoscopic resources – time-machines awaiting to be built.