Using Camera Projection on Draft Matte Painting Scene

Proof of Concept

Draft matte painting photoshop

Simplified scene with Unreal Engine foreground removed.

Split the scene into four layers; sky replacement, the city, and close/far spaceship.

---

COMPOSITING IN AFTER EFFECTS

Rotoscoped foreground

I didn't spend too long trying to create a beautiful shot in this case. I rotoscoped out the background from the Unreal render with minimal adjustment.

The projection mapped render needed retiming to match the Unreal Engine footage.

I also added in a few subtle layers of smoke/mist atmospheric effects to test how they might work.

Rough Composition Output

Above is the rendered composition rendered as a highly compressed .gif for purposes of displaying in this blog. There's been little to no post production effects added to this as it's mostly just a proof of concept render to expose issues within the process.

IMPROVEMENTS

There are some obvious improvements that can be made to this image.

Stuttering - The camera movement of the Maya projection and Unreal Engine output did not match timings, resulting in time remapping of the projection layer. This has resulted in that layer having an inconsistent framerate which is visible as juttering as the pan is not smooth. A better or perfect camera match will eliminate this.

Atmospheric Effects - The semi transparent clouds around the space ships is an obvious issue. Removing this in the photoshop layers would clean up this issue, or projecting the spaceship with the clouds onto a plane instead of a cylinder would clean up the image too - the ship is far in the distance so parallax would essentially be a non issue.

Other atmospheric effects such as the smoke layers added require further rotoscoping or masking to stack/increase in intensity as the depth of the image increases. An easy solution to this would be to export/render the different planes as separate passes, allowing any adjustment layers/effects to be layered between the planes.

Visual Artefacts - As a result of projecting the buildings onto the greeble meshes, they don't match the silhouettes of the buildings properly, this has created to some gaps to appear, and also for some noticeable duplication as the camera pans.

I think that there is merit to using a mesh like this to project onto, but is this case it's unnecessary, and a simple plane or cubes would probably yield better results. I'll note the issues with the spaceships here too, again, using a plane would be easier and remove the backface projection issues here.

Foreground - The rotoscoping is not perfect - this was expected. The person is intended to be replaced by a greenscreen or simple 2d cutout element anyway, but it does showcase how rotoscoping is difficult with something that's got a lot of motion or lens blur.

Keying the background out of the shot would yield a more consistent result that rotoscoping, so rendering the Unreal Engine camera onto a green/blue plate should make that easier.