AAE: Digital Ghosts Project

The AAE Digital Ghosts project develops and implements novel techniques in Deep Learning (DL) in application to Antarctic Cultural Heritage. Recent developments in DL facilitate the automated colourisation and extraction (or recreation) of three-dimensional data from black and white source photographs. In concert with conventional 3D-modelling and animation techniques, these reveal possibilities for the data-driven recreation of historical actors within simulated historical environments, based upon historical imagery, audiovisual recordings, contemporary GIS datasets and procedural modelling.

Hurley – Bage – Mawson 3D Model

The above image demonstrates the applicability of two recent DL applications for the construction of animated digital avatars (‘Digital Ghosts’) of AAE Expeditioners: three black and white portrait images have been interactively colourised using a convolutional neural net (CNN) model using learned deep priors, generating the following colourised portraits (with varying success):

Single image Direct Volume CNN Regression is used to create high-density 3D polygon manifolds of the faces, exported as .obj files. Cinema4D (R19) has been used to optimise and polygon-reduce mesh complexity. The colourised photographs have been used to create frontally-projected texture maps, with several interacting layers (colour, diffuse, luminance (with sub-surface scattering model) and bump). Finally, the three head models are rendered with a physical renderer, using ambient occlusion and global illumination (this reduces some render artefacts but substantially increases render time.) Two light sources have been used (a fixed spotlight and moving omnidirectional light) for this initial visualisation test.

The object of this work so far has been to determine the speed with which these processes can be undertaken by one person (myself) and a few computers (running MacOS 10.14, Windows 10, Ubuntu 18) and the satisfactoriness of the ‘realism’ of the output. It’s a qualitative measure – and at this point I can say I’m happy with the results for a day’s work. 

The textures currently demonstrate artifacting (e.g. tearing, smearing on edges) due to projection issues on the derived 3D models: these can be rectified via UV unwrap and in-painting. Further texture development involves de-lighting and shadow removal from the source photographs – several recent machine learning approaches show promise in this direction. The models require geometry correction and manual refinement (sculpting). Further work upon the eyes, a mouth model and possibly facial musculatures (IK or node-based) will add realism and flexibility – not to mention adding a body (I will probably attempt this using Houdini 17). The 3D models will be animated via DL(CNN)/mocap animation transfer as work progresses, including lip-synch with historical recordings of the actors where available, or re-enactments using CNN trained voice models I am working on.

Hurley – Bage – Mawson, 3D Reconstruction

More to come.