Creating ‘Qalam’

A technical deep-dive on how Stink Studios worked with Google to deliver high-end Artworks created in VR to mobile devices

Matt Greenhalgh, Technical Director

Image for post
Image for post
  • Files up to 42MB in size
  • Complex, custom shaders for unusual brush strokes like Fire, Electricity and ‘Chromatic Wave’

Anatomy of a Tilt Brush brush stroke

Tilt Brush is an amazing VR painting application which, conveniently for us, offers an export path from its proprietary .tilt file format to the common .fbx format as well as Base64 encoded JSON.

Image for post
Image for post
One of the more complex artworks, with it’s interesting UV unwrap of the ‘Light’ brush

Recreating Brush Shaders in WebGL

Ultimately we intended to deliver the Artwork customisation experience as a web page using WebGL with THREE.js to display the models and a React front-end providing the UI layer. This meant we had to find a way to recreate the particular look of the brushstrokes in WebGL shaders.

Image for post
Image for post
Early WIP on our attempts to recreate Tilt Brush Shader properties

Compressing geometry

Having recreated the look of the brush shaders, the next big challenge we faced — and by big I mean HUGE — was file size. The raw JSON exports from Tilt Brush were anything between 1 or 2MB to just over 40MB. There were around 50 artworks in total that we needed to make available through the website. Expecting users to wait and pay for ~1GB data over a mobile connection wasn’t exactly reasonable. We set to work looking at ways to reduce this massive payload.

Shared Geometry, GLB and Draco compression

The first thing we looked at was the source JSON file. By parsing the Base64 encoded data we were able to see that the brush geometries could be combined into single geometries for brush strokes that shared the same brush type.

Image for post
Image for post
Website Designers are FURIOUS at how you can create your own WebGL scene with this simple dat.gui CMS from Stink Studios

GZip and MIME Type trickery

The combination of GLB with Draco compression brought significant file savings. Many of our model files were 10% of their original size. But we still had a non-trivial initial load time of 11Mb.

Deferred Loading

Although we’d met with some considerable success reducing the size of individual files, the net load for the initial page view was still larger than we would like.

Image for post
Image for post
Our hemisphere environment used a cylinder projection of temporarily extruded geometry to allow a simple linear gradient mapping

Web Workers & Transferable Objects

Draco compression had been instrumental in reducing file download size and times. However it proved something of a double-edged sword. It had introduced a new overhead in the asset display pipeline: geometry decompression and upload to GPU memory. This task was now the single biggest contributor to the perceived ‘download’ time and, worse still, it introduced a small but significant lock-up of the browser while the geometry was decompressed.

Image for post
Image for post
Tiling texture with Vector Warp displacement created in Substance Designer for the scene transition effect

Conclusion

Our files had been on quite a journey! From 42MB JSON source though geometry re-encoding, GLB conversion, Draco compression, Base64 encoding and gZip compression we were able to get the same file down to 1.9MB without any visible compression artefacts. The brush strokes were near-perfect recreations of their Tilt Brush counterparts, and we hadn’t had to compromise on the artists’ ideas as we brought them to mobile devices.

Written by

A creative advertising and digital experience company.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store