— Kyle

Promethean AI Creation Engine

This is a bit of a follow-up to the concept of “Assisted Creation” I brought up in the latest newsletter. From their website:

Promethean AI is world's first Artificial Intelligence that works together with Artists, assists them in the process of building virtual worlds, helps creative problem solving by suggesting ideas and takes on a lot of mundane and non-creative work, so You can focus on what's important. All while learning from and adapting to individual tastes of every single Artist.

Typical ridiculous startup claims aside, what Promethean AI aims to offer is something like auto-complete or Gmail’s “smart compose” for the process of environmental art design. It’s very much that case that every project can be its own special flower that needs specific tweaks, but it’s also true that many projects contain easily automatable redundant work.

Tools like Houdini aim to solve a bit of this, but only at a a per-asset level (mostly). Houdini can help you generate millions of buildings that all look unique while still looking stylistically similar, but the act of actually arranging those buildings (or trees or chairs or whatever else) in a game world remains somewhat tedious work. Promethean AI aims to solve at least some of that, giving artists and creators a tool that ideally works with them to produce better environments, faster. It’s sort of like SpeedTree, but for whole environments and hand waves AI.

The whole keynote is definitely worth a watch, and hats off to the team for really showing a lot here, not just talking about dreams or ambitions.

NVIDIA Omniverse

Speaking of tools, Nvidia recently unveiled “Omniverse”. Nvidia press strategy is often “if you know you know” so at times it can be hard to parse if something they are showing off is a piece of hardware, software, plugin, etc.

From their own site:

Omniverse enables universal interoperability across different applications and 3D ecosystem vendors. It provides efficient real-time scene updates and is based on open-standards and protocols. The Omniverse Platform is designed to act as a hub, enabling new capabilities to be exposed as microservices to any connected clients and applications.

As far as I understand it, Omniverse leverages Pixar’s USD format (basically — a “scene description” file) in the cloud, and the disparate applications used in creative production essentially “commit” those changes to a cloud-hosted USD representation of the scene. Those changes then propagate to other connected client machines. RTX is involved via “views” on the content, generated by RTX machines (in the cloud), providing you high resolution renders without needing the hardware itself.

If it’s a bit confusing, it’s because it doesn’t seem like all parts of Omniverse are meant for all projects. VFX people can benefit from the RTX rendering, while game engine work is mostly concerned with asset synchronization.

Creative project collaboration is a notoriously sticky problem to solve, so it’s exciting to see someone tackle the issue in what feels like “the right way,” and with many vendors offering plugins and compatibility patches it seems like it will take.

Enjoying Rendered so far? Subscribe below to be notified when the next issue goes out.

Subscribe now

Google’s Project Starline

Google strapped “more than a dozen different depth sensors and cameras” to a TV and made a 3D telepresence demo. I think people care? But reading the Wired article that covered the project, it’s clear they have the same sort of ¯\_(ツ)_/¯ reaction I did:

Google’s Project Starline seems especially overengineered, an amalgamation of accessible tech (Google Meet), nerd tech (computer vision! compression algorithms!), and an intricately constructed, unmovable mini studio, all for the sake of … more video meetings.

3D telepresence demos are nothing new. We even did one with Depthkit a few years ago, and Or Fleisher of this very newsletter did one before that. Here’s the thing about them: the novelty of them for the user wears off quickly, and no matter how high-res your screen is it will still feel like looking at a screen. Yes yes they are doing sensor fusion and streaming the data over the internet and reconstructing it at a different location, but what is never talked about is why.

It’s presupposed that people want this, that higher fidelity remote conversations improve the human experience or something. I recognize this is a step of many towards something that may actually achieve those aims, but for now it’s hard to see this and not feel like I’m being asked to sit at attention and talk to someone in the most unnatural way possible.

That aside, the fact Google was cagey about the actual depth sensors used is interesting — is Google making a standalone depth sensor? We’ll see!

Volumetric Format Association Founded

From the press release:

The first industry association dedicated to ensuring interoperability across the volumetric video ecosystem has launched. Seven companies have joined forces on the association, including Verizon, ZEISS, RED Digital Cinema, Unity, Intel, NVIDIA, and Canon. The aim of the Volumetric Format Association is to establish a collection of specifications driving adoption of volumetric capture, processing, encoding, delivery, and playback.

This is a Big Deal. Some of the largest players in camera tech, game engines, and realtime technology have collectively decided to form an association dedicated to standardizing volumetric video. This is notable for a lot of reasons, as the volumetric video “format” right now is essentially whatever data container the programmer your company hired thinks it should be.

Of course this could go horribly wrong and they collectively decide to develop a standard that only makes sense when paired with their technology stack (5G, I’m looking at you), but I’m hopeful. The diversity of companies represented makes me think it’s an actual good faith effort to standardize an ecosystem that has suffered from fragmentation, so I look forward to seeing what comes of it. There’s also a legitimizing factor here, as these companies have all collectively acknowledged that volumetric video is really a thing worth caring about.

Worth noting: Microsoft and Epic are notably absent from the list. Maybe they weren’t at the right party at the right time, but Epic’s game engine tech stack and investment in virtual production is far more suited to volumetric than Unity’s current position, and Microsoft operates many capture stages worldwide. Hopefully they will both join soon enough.

Note: No papers this week! We’ll be back next month to share the latest in research!

Resources

Amazon Nimble Studio seems to be a full cloud based VFX suite

It’s hard to parse exactly what is on offer here, but it essentially sounds like access to scalable compute through Amazon, with a flavor suited for creative work (read: creators need GPUs!). Seems like a nice solution for VFX studios that don’t want to invest in workstation machines.

Large list of “Foundational or significant books or talks compiled by people who teach, study, or work around the edges of “immersive experiences”, “themed entertainment”, and XR”

Marie Foulston asked her followers on Twitter the above, and there’s a great list of stuff here for anyone looking to get involved in this space.

Epic has published Volume 2 of the Virtual Production Field Guide

Didn’t realize we were going to get a second volume! This (and the first) are essential reading for anyone looking to get involved with Virtual Production.

Briefly Noted

ARCore gets a RawDepth API - Expect every more selfie apps

Lightform Project LFX experiments with using a mapped space to show you custom information on any surface - Seems pretty cool!

VTuber (kinda) CodeMiko gets profiled in The Verge - if you don’t know who CodeMiko is and you read Rendered, you are contractually obligated to read this

Google Earth adds in Time-lapse features - Watch climate change happen at the global scale. The video even seems like an intro to a dystopian sci-fi movie.

Epic hosts roundtable discussion on Virtual Humans - it’s largely about Metahumans, but there’s some good discussion about the adjacent tech and implications as well

And that’s a wrap! As always, thanks so much for reading this newsletter. Feedback is always appreciated, and you can email me directly at kyle.kukshtel@gmail.com or simply respond to this email.

Additionally, if you enjoy reading Rendered, please share this newsletter around! We thrive off of our community of readers, so the larger we can grow that pot of people, the better. Thanks for reading!

— Kyle

News

Epic Games Acquires Capturing Reality

Epic Games has acquired Capturing Reality, makers of the (best-in-class) photogrammetry software RealityCapture. This move is… interesting, maybe even more so than the fact they just had a billion dollar raise.

Many of Epic’s acquisitions have focused on horizontal integration, buying up middleware companies like RAD Game Tools, Kamu, and Quixel (amongst others) to bolster Unreal’s feature set. As far as I know, this is the first time they’ve bought a company that make standalone software that’s more targeted at vertical integration, attempting to own the whole stack of photoreal capture.

Unity made a similar acquisition recently with their purchase of PlasticSCM, a source control tool used by many major game studios.

A way to read it is that both of these companies are starting to peek outside of their standard wheelhouse, snatching up companies that exist below and above Unreal in a project lifecycle (Epic's own move to build out the Epic Games Store is at the top of the stack).

Epic maintains that RealityCapture will still be available as a standalone product:

Epic plans to integrate Capturing Reality’s powerful photogrammetry software into the Unreal Engine ecosystem, making it even easier for developers to upload images and create photorealistic 3D models in instants. Capturing Reality will continue support and development for partners across industries like gaming, visual effects, film, surveying, architecture, engineering, construction and cultural heritage. This also includes companies that do not use Unreal Engine.

Again similar to PlasticSCM here — Epic can build platform affinity over time, providing a route to Unreal from RealityCapture, even if you used RealityCapture and weren’t an Unreal user in the first place.

This is also a bit of the dirty laundry of photogrammetry — generating geometry is [relatively] easy, but finding a distribution platform to share that with others is nigh impossible without game engine knowledge. Unreal 5 + Nanite is a perfect match.

For more thoughts on this, I recommend checking out this article by Mosaic51 that gives a good overview the wider ecosystem of photogrammetry, or this video from GameFromScratch that’s a great intro to the most common photogrammetry tools.

Microsoft Strikes $22 Billion Hololens Deal With US Army

From Techcrunch:

Microsoft will be fulfilling an order for 120,000 AR headsets for the Army based on their Integrated Visual Augmentation System (IVAS) design. The modified design upgrades the capabilities of the HoloLens 2 for the needs of soldiers in the field.

File this a bit under “Rendered #2 Follow Up” (along with NYPD robot police dog with a LiDAR sensor attached to its head).

The thing that most bums me about this news, is that this will likely never be brought up again. It won’t make press conferences, it won’t be in the video montages where Microsoft sells its vision of a better future, it won’t be listed as use case on the Hololens website, etc. The official Microsoft blog post even eschews the insane size of the deal, instead making it sound mostly like a few emails are going back and forth.

The size of the deal is notable, as the stated number of 120,000 is more than double the total number of Hololens headsets sold. As the only real MR headset in circulation (sorry Magic Leap), this effectively means that the market for these headsets is now warfare. Not medical applications, not factory work, but bold-faced violent warfare.

ML Assisted Creativity

There’s no lead article for this, but it’s a trend I’ve been tracking a bit of that I thought would be worth calling out. People are starting to feel out the edges of what easy-access ML (Machine Learning) can mean for creativity, and are starting to sketch out a few options.

The first “generation” of creations, like the image above, relied largely on applying some technique directly to some other piece of art (or picture to whatever) and seeing what the output is. The above is an example of “style transfer”, but many other ideas in similar form appeared — basically taking the direct output of the process as the end goal.

However, what’s been emerging recently is the what I think is the second generation of this sort of tech, defined mostly by its situation in with other tools. Or more that the technique is not an end in and of itself, but is more of a means to an end for another result.

This manifests in a few ways. One is that major tools are simply getting ML functionality baked into them. Unity has a new tool called “ArtEngine” that promises “AI Assisted Artisty” through a suite of tools that help you develop game art assets (though worth pointing out AI is not necessarily ML!). Nuke recently announced its latest version would also have ML-powered tools:

The Nuke example gets closer to the next video, where ML-based tools (like EbSynth) are being sort of “hacked” to assist traditional video workflow tasks like removal, rotoscoping, etc.

And lastly, this great thread by James Shedden goes through using EbSyth as part of a workflow in a way that makes it almost invisible (but also crucial!). He does an initial “style sketch” for the piece, and then creates a 3D model in Blender that acts as the target for a style transfer frame. Those are all then comped together for the final output!

Beautiful! I think tools like this will slowly worm their way into most (if not all) visual media creation tools, as the upsides are immense. They are definitely not without downsides (many introduce unwanted artifacts that need to be cleaned up), but what’s cool is that its helping to create a new tool in creator toolboxes. Not the tool as the “first generation” operated, but one that helps people find new modes and ways of artistic expression.

Papers Review

Or’s back! Read on for the latest!

In every issue, Or Fleisher digs through arXiv and random Github Pages sites to surface state-of-the-art research likely to affect the future of volumetric capture, rendering, and 3D storytelling.

Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos (CVPR 2021)

Paper | Code

3D reconstruction of humans from monocular video has always been challenging. This paper presents a model architecture for predicting the depth and surface normals of humans in video from RGB video inputs.The results are quite impressive and alongside the code the authors also released a TikTok based dataset of humans used to train the model.

Why it matters

We have seen 3D human reconstruction from a single view improving over the past couple of years, and this is another step at improving the fine shape detail (and ability to reconstruct clothing).

Neural 3D Video Synthesis

Paper | Project

Neural radiance fields are the hot new research topic on the block. Simply put, NeRFs are an implicit way to represent 3-D information inside of a neural network, which can then be used to synthesize novel views of the scene. This paper takes that same approach up to the next level, by allowing the model to learn temporal data and generate new views of the scene over time.

Why this matters

Neural radiance fields have proven to be able to cope with variance in changing lighting, transparencies and detail. This paper however, is the first demonstration of using that idea to encode multiview data over time.

High-Fidelity 3D Digital Human Creation from RGB-D Selfies

Paper | Code

This paper proposes a network architecture that is able to take in a couple of RGB and depth inputs, and produce a detailed, textured face reconstruction. The results border uncanny, but the research proves quite capable at tasks such as an ability to synthesize and fuse textures, normal maps and fine detail representations.

Why this matters

RGB-D face facing cameras are becoming increasingly common in mobile devices, and this paper shows stunning results using only a few selfies from such devices. The ability to reconstruct high quality 3D facial assets is promising.

Resources

tinytools.directory is a massive directory of “small, free, or experimental tools that might be useful in building your game / website / interactive project.”

Everest Pipkin made an appearance in Rendered 2 with their Anti-Capitalist software license, and is back again with a gold mine of tools for game development and software creation worth checking out.

Meshlete is an open-source library for generating meshlets

“Meshlets are small chunks of 3D geometry consisting some small number of vertices and triangles…Splitting the geometry processing to meshlets instead of processing the mesh as a whole has various benefits and has better fit with batch-based geometry processing (vs post-transform cache model) in modern GPU architectures as well.”

Meshlets are a hot topic right now in 3D graphics/rendering, and having an open source library to generate/play with them is great!

AIAS Game Maker’s Notebook Podcast Episode with Head of Microsoft Flight Simulator, Jörg Neuman‪n‬

This is a SUPER interesting podcast listen that goes in depth on the creating of the latest Microsoft Flight Simulator game. Jörg talks deeply about their tech-stack, and how MFS is essentially doing “sensor-fusion” across the entire planet, combining photogrammetry, satellite imaging, predictive modeling, tidal patterns, etc. This game flys (hah) under the radar due to it’s niche market, but the work being done here is truly ground-breaking and widely applicable to anyone building 3D words (or the world, in MFS’ case) at scale.

Briefly Noted

Playstation VR getting revamped for PS5

Balenciaga “Afterworld” Unreal Featurette

HTC Vive Pro Lip and Body Trackers

“Hex” is SideFX’s Youtube show highlighting Houdini creators

Inside Facebook Reality Labs: The Next Era of Human-Computer Interaction (also 1/5th of Facebook is working on AR/VR)

Waymo offers license for its LiDAR Sensors

And that’s a wrap! As always, thanks so much for reading this newsletter. Feedback is always appreciated, and you can email me directly at kyle.kukshtel@gmail.com or simply respond to this email.

Additionally, if you enjoy reading Rendered, please share this newsletter around! We thrive off of our community of readers, so the larger we can grow that pot of people, the better. Thanks for reading!

Subscribe now

— Kyle

News

Unreal Announces Metahumans

Two issues ago we talked about the then-just-announced Unreal Engine 5, and some of its aims to smooth out production workflows around high density meshes. Since then, Epic has been on a kick building out a case for itself and that goal, releasing various videos about virtual production and cinematic asset creation/lighting on their Youtube channel.

Epic has effectively no competition here on the software side outside of people’s willingness to switch over from offline tooling to Unreal, so their non-stop PR blitz about virtual production is commendable. A general theme of these videos is talking about how Unreal fits in all aspects of the production stack, but the elephant in the room has (and will likely always be), you know, humans.

Humans, the part of films we probably most watch films for, have been cut out of the conversation (or obscured) when talking about virtual production. Virtual production is largely about everything besides humans.

Virtual production, by Epic’s reasoning, is largely an efficiency mechanism. Not “cheaper” or “easier,” but efficient. Unreal’s own “Virtual Production Field Guide” goes through great pains to describe how Virtual Production’s primary goal is to “reduce uncertainty”:

All of these efficiencies and increased image quality offer a trickle-down effect to more modest and tightly scheduled productions. By leveraging virtual production techniques with a real-time engine, network series, streaming productions, and indies can all achieve very high quality imagery and epic scope. A real-time engine has the potential to eliminate many of the bottlenecks of budgeting, schedule, and development time that can prohibit smaller-scale productions from producing imagery on par with blockbusters.

This is at odds with the fact that humans are, and have always been, and will always be, the least efficient part of movie production. They run late, have to eat, sleep, be famous, etc. What if you could make a movie using humans, but… not really.

This, in my mind, is the ultimate promise of the recently announced Metahumans. Epic is, in part, saying, “Listen, everything you already do is in Unreal… why not have your humans in Unreal as well?”

Digital humans aren’t an idea that’s particularly new, but the the fact Metahumans work in a realtime context is what differentiates them. We’ve had really great digital humans for the past few decades through robust offline rendering, but offline rendering doesn’t mesh with the virtual production ethos. All the same live, micro-tweaking talked about with virtual production can now too be applied to human performances.

The pitch is salient, but at the same time you’re making a game engine the central nucleus of a film production. Show of hands: how many people feel comfortable debugging UASSET errors in Unreal while talent is staring at you from the stage and a producer is whispering in your ear that this bug is costing them thousands and the studio manager just walked in to say the next production has arrived and… oops you missed your shot. Every virtual production shoot day is essentially live coding a movie.

Don’t get me wrong, I’m a fan of virtual production and do think it’s what’s next, but I also think all talk of it, especially from Epic, often obscures the fact that you’re running a game engine in the background, which has all the issues game engines normally have. Not only this, but because you’re likely renting out space in someone else’s studio to shoot (you didn’t build your own LED wall right?), you’re relying on the past production to have cleaned up their Unreal workspace (lol) or the studio technical manager to have done it (lol). What happens when you boot up Unreal on shoot day and a Windows update broke Unreal (it happens)?

Matt Workman on Twitter

I’m being a bit facetious here too — as good as Metahumans look (what a name too right?), nobody right now will assume these are actual humans… yet. Most of the copy on the main Metahumans page talks about using them in games, with virtual production getting only a passing mention:

Imagine […] digital doubles on the latest virtual production set that will stand up to close-up shots, virtual participants in immersive training scenarios you can’t tell from the real thing: the possibilities for creators are limitless.

These avatars are definitely AAA games quality, a massive leap forward for any indie, but they are also a step forward towards realtime digital humans that are indistinguishable from the “real” thing.

Here we go again

Are we surprised that the most prominent Metahuman demos are an asian man and a black woman, and that most of the people demoing and puppeting their faces (in the closed beta of Metahumans — this tech isn’t publicly accessible right now) are largely men who don’t share their race?

Christian Andorade on Twitter

Rich Hemming on Twitter

StretchSense on Twitter

I spent all of the last issue of Rendered talking about this, saying “emerging technology can and will be used in ways that go beyond whatever cute scenario you’ve imagined for it, and as such we should be taking an active role in preventing these outcomes instead of accepting them as an inevitability.”

The fact the #metahumans tag on Twitter is full of men trying on blackness is a massive issue in the same strain. I’m not a diversity and inclusion expert by any means, nor do I propose to have solutions — but what I have to ask is… what the hell? What sort of precedent are we setting here by giving groundbreaking technology to, what seems to be, the same class of people that has abused technology in the past to reinforce the same social norms. We have to be better.

If you’re reading this and work at Epic (judging from subscriber emails, I know some of you do) — advocate for diversity and inclusion in even the smallest tests like these. Consider what message you’re sending to your prospective users and other non-white creators that rely on your tools to create their work. Bake this into your process.

Be better.

Microsoft Mesh Announced

At the annual IGNITE conference, Microsoft announced a new cloud service, Mesh:

Microsoft Mesh enables presence and shared experiences from anywhere – on any device – through mixed reality applications.

That’s a pretty big pitch. A clickbait-y Youtube title might be something like “MICROSOFT ANNOUNCES API FOR THE METAVERSE??!?!?!”. But we’re far from that (like and subscribe anyways).

Subscribe now

One thing to keep in mind right now, especially as this space (gestures at XR, spatial computing, game engines, volumetric capture, etc.) gets more mainstream traction, is that there is very little unique and new about it from an asset perspective. Acquisition, via tracking and depth capture, are the “new” things, but most software built in the space lives downstream of that process and assumes a fairly standard input of either 3D meshes or videos.

This is to say that most “new” technologies in the space are able to leverage pre-exisiting infrastructure to interact with these new mediums. Microsoft Mesh, put simply, is Microsoft (likely off some of the cloud work they were already doing with Simplygon) expanding Azure’s offerings to be able to upload/download 3D data. That’s less sexy than “Microsoft Mesh”, but ¯\_(ツ)_/¯.

Obviously Microsoft Mesh is more than just a branding exercise, but part of the point of this newsletter is to cut through the relentless optimism and history re-writing of these spaces.

What is interesting about Mesh is less the actual technical side (we had our own realtime volumetric streaming demo with Depthkit at VFNYC nearly two years ago) and more their pitch on “immersive presence”:

A fundamental aspect of multiuser scenarios is to be able to represent participants in distinct forms depending on the device that they’re joining from. Mesh delivers the most accessible 3D presence with representative avatars via inside-out sensors of the devices. The Mesh platform comes with an avatar rig and a customization studio so you can use the out-of-the-box avatars. The platform is capable of powering existing avatar rigs too with its AI-powered motion models to capture accurate motions and expressions consistent with the user's action.

Alongside avatars, Mesh also enables the most photorealistic 360 holoportation with outside-in sensors. These outside-in sensors can be a custom camera setup like the Mixed Reality Capture Studio, which helps capture in 3D with full fidelity or it could be Azure Kinect that captures depth-sensed images to assist in producing the holographic representations.

Microsoft’s pitch is that avatars, driven by inside-out sensors on devices like the Holoens, and volumetric captures (Microsoft calls it “holoportation”), driven by outside-in sensors on Azure Kinects, can exist side-by-side.

Again, the technicals of this aren’t what’s interesting — what is interesting is that almost every other pitch at some idea of embodied digital collaboration has been one of a totalizing vision — i.e. either everyone is avatars or everyone is a volumetric capture. That “purity” has been a major thorn for everyone involved because buying into either vision meant alienating the other side.

The vision of Mesh is one where those captures exist side-by-side. Obviously this is in Microsoft’s interest as well, as they own solutions to do both avatars (they acquired Altspace in 2017) and volumetric capture (via their Mixed Reality Studio program or aforementioned Azure Kinect), but I do hope other “metaverse”-esque solutions/platforms adopt the same thinking.

One other interesting part of this announcement is that it implies Microsoft will deploy its own solution for volumetric capture/streaming based off the Azure Kinect. Despite ignoring volumetric capture as a first-class use-case for the Kinect family for a long time (the Azure Kinect page now lists it in passing under a “Media” section but doesn’t provide further info), the announcement of Mesh now lights a bit of a fire under that use-case to provide a Microsoft-sanctioned option. Whether this is a proper Microsoft product, or instead a list of “Mesh compatible” vendors, we’ll have to wait and see.

Rendered #2 Follow-up

Since sending out Rendered #2, a few things have happened that are relevant to the discussions there.

The feature-length documentary film Coded Bias was released to a few film festivals (and reviewed in The New York Times)

CODED BIAS explores the fallout of MIT Media Lab researcher Joy Buolamwini’s discovery that facial recognition does not see dark-skinned faces accurately, and her journey to push for the first-ever legislation in the U.S. to govern against bias in the algorithms that impact us all.

The book Technocrats of the Imagination was published in early 2020 (missed it!) and talks about the relationship between emerging art movements and the military industrial complex:

John Beck and Ryan Bishop explore the collaborations between the American avant-garde art world and the military-industrial complex during the 1960s, in which artists worked with scientists and engineers in universities, private labs, and museums […] Beck and Bishop reveal the connections between the contemporary art world and the militarized lab model of research that has dominated the sciences since the 1950s.

I found some more Military Industrial Complex <-> HoloLens stuff:

Resources

Naughty Dog’s SIGGRAPH 2020 video presentations are live.

Naughty Dog has been pushing the boundaries of realtime graphics for as long as they’ve been around. These videos are a bit of a swan song for the standard model of 3D rendering as raytracing seems on track to subsume these old methods.

Epic Games’ Crash Course in Unreal

Great overview here of Unreal for anyone interested in getting started with the tool. Walks through the engine front to back to show you the general workflow and pipeline concepts. Need some assets? What Remains of Edith Finch's assets have been added to the Unreal marketplace.

Valve adds in Developer Commentary to Half-Life: Alyx

Alyx is one of the biggest (the biggest?) full throated attempts at making a AAA VR game by one of the best studios around, so a developer commentary on that project is like a master class in VR interaction best practices.

Game Development in 2021 Report Released by Unity

Interesting insight on budgets, projects, platforms, etc. Also generally good information on player trends. It also has startling revelations like:

Creating new content to delight and engage players should still be a top priority for game developers.

To think that these insights are free!

Mobile Computational Photography: A Tour

Really great overview here of different methods of mobile computational photography. I know it’s technically a paper but I think it’s use case is more broadly applicable as a general resource for people looking to get caught up on the space. One of the authors also has a video version of the paper that covers similar topics.

Unity Lighting Pipeline Guide

Unity’s documentation has always been hit or miss, so you’d be forgiven for not knowing there is now a giant guide on setting up render and lighting pipelines in the manual. This thing is full of amazing information, and should be required reading for anyone working in rendering or lighting in Unity.

Briefly Noted

Shutdowns

WaveVR is shutting down it’s app and focusing on “non-VR objectives”. Poly is shutting down. Carboard is dead.

Releases and Briefly Noted (link dump)

Davinci Resolve 17 is out. Houdini Engine now free in Unreal and Unity (!!!). Balenciaga made a giant volumetric game. EF EVE volumetric captures work in Notch. Create your own neural radiance field image with NERFIES. Record3D added a live-streaming volumetric capture (and export) solution. Adobe releases a computational geometry library for free (thanks Dimitri!). Omnivor raises nearly 3m. IO Industries and 8i strike a partnership.

Fun

Thanks for much for reading the newsletter. Feedback is always appreciated, and you can email me directly at kyle.kukshtel@gmail.com or simply respond to this email.

Or’s presence for the papers section on both this issue and last was sorely missed, but he’ll be back for #4 with some HOT PAPERS, and hopefully on every issue after.

Additionally, if you enjoy reading Rendered, please share this newsletter around! We thrive off of our community of readers, so the larger we can grow that pot of people, the better. Thanks for reading!

Subscribe now

NOTE: This is a long newsletter, and may get clipped in your mail client. To view the full newsletter, click here.

Subscribe now

Towards Accountability in the Development of Computing Technologies and Creative Media

Rendered #1 launched a few weeks before the death toll from COVID-19 in America hit 100,000. We’ve recently more than doubled that, and are showing no signs of slowing down. Like everyone else in the world, I’ve been watching this acutely American pandemic unfold from a computer screen, becoming a pseudo-expert in a domain that now dominates every moment of our waking life.

However, I’ve continued to compile articles and papers on all the latest advances and news I consider the purview of this newsletter. A week after the first issue was published the second issue was already drafted, and as I was putting the final touches on the piece I watched the police officer Derek Chauvin kill George Floyd.

I watched the protests afterwards, starting in Minneapolis and quickly making their way to NYC. I watched protesters march outside my window, and joined them a few days later. On Twitter I watched peaceful protests I attended during the day turn vicious at night, with police attacking non-violent protestors in protests against police violence. I waited for what I thought the “right time” would be to publish my newsletter of 3D technology. But the terrible reality is that I was waiting for a time that would never come, and never existed in the first place.

The violence perpetrated by the state against people of color and disenfranchised minorities in America has a history that spans the lifetime of this country, and continues to find new ways to manifest itself with each new generation. Outright slavery turns into Jim Crow, which turns into implicit segregation, which, aided by the GI bill, cuts anyone but white people out of the accumulation of generational wealth, which turns into police force to maintain the newly established suburbs, which turns into modern day militarized police, which turns into surveillance capitalism, and so on.

The violence continues, and, as a now nearly 500 person strong mailing list of people invested in pushing the technologies of digital representation and capture forward, I wanted to use the issue to highlight our past that places us squarely in the line of culpability for helping to perpetrate and enable systemic racism and discrimination across time, in the hopes that we can learn from our past actions to pave a more radically anti-racist future for ourselves and those that follow.

As someone who reads Rendered, it’s perhaps easy to see this issue as “over there,” where “politics happen.” However, to think that the creative/technical industries are not instruments of systemic racism and sexism is to be complicit in the exact structures that create the conditions people are on the street protesting against.

If this is news to you, I hope this newsletter can be a wakeup call. That you can understand how the technologies of computer vision, 3D rendering, and filmmaking are some of the biggest, most implicit perpetrators and profiteers of discrimination around.

This is also to say that, if you do not believe that Black Lives Matter, you will find no quarter here. Our industries should not be places shielded from the nature of race and politics. Further, you should recognize that every line of code you write is political, that every volumetric capture you take is political, that every render, compile, export, etc. takes place in a vast cultural network and contains within it all your own biases and opinions. To even claim that these complex cultural and technical systems are, or even can be, “without politics” is a racist/classist/sexist position.

For those on the fence, or for those who want more insight into this, I’m dedicating the rest of this newsletter to compiling some instances where our collective industries have perpetrated, enabled, and profited off of racism, even as recently as weeks ago. This is to show that these incidents are not merely one-offs, but part of a pattern the creative and technical community has been complicit in perpetrating throughout time, suggesting that a fundamental shift in how our industries operate is needed.

If you take nothing else from this newsletter, I’m calling for us to divest from military and police contracts, to divest from ICE, to divest from technologies that, even if we intend good, have the potential to be abused and maintain the abusive status of the hegemon. These are pro-active stances, that require an active conversation and reflection on not only what we are making, but how what we are making can (and will) be used to perpetrate racism and discrimination in America and beyond.

We need a Murphy’s Law of Systemic Discrimination - Everything that can be used to reinforce systemic discrimination and racism will be used to do so.

Exploring Systemic Racism and Discrimination through the Technical & Creative Community

Below I’ve compiled a (far from complete) list of specific incidents where the technical and creative community has not only been complicit in supporting discriminatory structures in society, but have also stuck to their guns when questioned about the potential impact of their work.

An IBM “Counting Machine”

IBM’s Enablement of Nazism

In their book IBM and the Holocaust, author Edwin Black breaks down the (very open and not hidden) connection between IBM and Nazi Germany. Though IBM regards some of the claims in the book as dubious, it has not denied the facts that IBM sold tons of their counting machines to the National Socialist German Workers Party (NSDAP), crucial in establishing the work of dehumanizing a populace and setting a precedent and pattern for the methods that would become core to the operation of the Nazi party itself.

IBM largely profited off of this endeavor (as did many other automobile manufactures like Mercedes-Benz), and in many ways is in the position it is today due to the lucrative work of selling technology to autocrats.

Microsoft, Amazon, Palantir, and ICE

This practice continues today. Companies like Microsoft, Amazon (via AWS), and Palantir sell software deals to government agencies like the United State’s Immigration and Customs Enforcement agency (ICE). These software deals, worth tens to hundreds of millions of dollars, provide crucial technical infrastructure that allows the organization to continue conducting its abhorrent behavior at the USA/Mexico border where it separates young children from their parents and keeps children in cages. Even as recently as a few months ago, when pressured to end Github’s contract with ICE, GitHub’s chief executive, Nat Friedman, said:

“Thank you for the question. Respectfully, we’re not going to be reconsidering this…Picking and choosing customers is not the approach that we take to these types of questions when it comes to influencing government policy”

The work of mass subjugation requires robust technological solutions and infrastructure, a need that all of the largest tech companies are gleefully ready to fill so long as the government continues helps to line their pockets.

Celluloid’s Bias Towards Whiteness

The chemically engineered light response of celluloid was, from the beginning, created with white people in mind as the “reference” or “normal” scenario. This assertion of whiteness-as-default will come up a few more times in this newsletter, but is exceptional in the context of celluloid as it is literally chemically engineered racism.

Film, as a medium and technical process, was developed as a way to make white or light skin not only look better, but to be able to “appear” at all as part of the development process. When color film is unable to accurately capture darker skin tones (or darker tones next to lighter tones, as the above video points out), erasure takes place. The whiteness of the image is centered, with other races pushed to the margins or disappearing from the developed image entirely.

Digital Sensor’s Bias towards Whiteness

Even as we move beyond celluloid and to digital cameras, the issues of cameras “seeing” other races (or multiple races in the same images), remains an issue. We gave up celluloid color chemistry, but in its wake rose imperfect digital sensors who, without the ability to capture the full color spectrum, relied on color LUTs to take the full range of possible colors and bake it down into a more manageable gamut.

Oftentimes standard LUTs favor white skin, and the process of photographing darker skin tones as a result needs extra consideration in a way white skin doesn’t.

Ava Duvernay, director of Selma and When They See Us (among other films), has spoke about this at length. In this video she goes over the easily skipped over nuances of shooting black skin in dark rooms and the steps she and her cinematographer have taken in order to do it right.

It also seems a cinematographic reckoning is happening right now, with the Guardian declaring a few years ago that “film finally learned to light black skin”.

Facial Recognition’s Bias

As we move beyond 2D images and into the realm of 3D, one would think that our capture technology would learn from past mistakes. However, the problem seems to never go away — It simply refocuses around newer technologies.

When Apple introduced Face ID, a Chinese woman found that it was unable to distinguish between her and her colleague. Black people were sometimes categorized as gorillas with Google’s facial recognition. This past month saw people experimenting with Twitter’s “content-aware cropping” to see how Twitter often favored the display of white people in cropped images instead of people of other races.

Facial Recognition in general is also a huge topic that I may cover in a future newsletter, but for others interested in some of the discourse occurring around the ethics of facial recognition, I encourage you to check out some of the pieces below:

When the Police Treat Software Like Magic

A Case for Banning Facial Recognition

Against Face Analysis

It’s also worth saying that IBM and Amazon have made at least symbolic gestures towards ceasing Facial Recognition tech, but IBM is a bit of a non-player in the space, and Amazon’s tech is its whole stack, not just the cameras. The ACM itself has also called for “an immediate suspension of the current and future
private and governmental use of [facial recognition] technologies in all circumstances known or reasonably foreseeable to be prejudicial to established human and legal rights.” (thanks @schock).

The “Lena” Photo

Baking in Bias with “Test” Data

Lena

You would think after the sunsetting of the aforementioned Shirley card for 2D image making that the image making industry would have learned its lesson, opting instead for a more diverse and robust test battery for their developing technologies.

Nope.

Around the same time the Shirley card began its road to deprecation, the burgeoning field of computer vision/digital image processing was starting to find its legs. And how would this new research field test its image processing algorithms and cameras?

By choosing a photo a naked white woman from Playboy of course!

Not only this, but the use of “Lena” as a test image for image processing algorithms exists to this day, once again forward whiteness-as-default. The documentary Losing Lena goes into more detail on the history of the image, and burgeoning efforts to replace it.

Volumetric Capture

If 2D image making is biased by design, its hard to overstate how the 3D image amplifies the problem, both in the terms of the circumstance of its creation and what it is purporting to “capture”.

In his book 3D: History, Theory and Aesthetics of the Transplane Image, author Jens Schröter makes a distinction between 2D media having the ability to control only time (they are single moments, or series of single moments played in rapid succession to show a single thing as captured), and 3D capture/representation offering the ability to exert “spatial control” in addition to the “control of time”.

Which is to say that 3D image making is totalizing in its capture of time and space. However when this ideal meets reality, this “perfect image” inevitably falls over as we project our own biases and opinions onto the act of image creation. And yet, even with its obvious metaphorical (and sometimes literal) holes, we act as if each 3D image is not only a truth but the truth.

More tangibly, many depth capture solutions are impossibly bad at not only capturing a variety of different hairstyles, but are particularly bad at capturing the hair of African-American people (and people with similarly textured hair). The texture of this hair tends to reflect IR light back at the camera’s sensor, rendering a subject’s hair not just invisible, but as if it never existed at all. The data is simply not there.

Instead of trying to fix the problem, many volumetric capture companies instead dodge these constraints by choosing to showcase captures of bald Caucasian males or those with short cut hair, or Caucasian females with hair pulled back into a ponytail.

The thinking here is that, in order to show off your volumetric capture system, you need to provide demo data that represents the ideal type of subject for capture. So just as with celluloid, we’ve again arrived at the how “whiteness” and “white characteristics” are the ideal for capture. This implicitly pushes other races and human characteristics into a philosophically “non-ideal” zone.

I’ll also take this as a moment to say:

Stop using white women in bikinis to showcase your volumetric capture technology

This practice, as well as dodging the portrayal of BIPOC, is once again forwarding the notion of whiteness-as-default, or stereotypical “beautiful” bodies as the “ideal”. If we are making progressive technology, so should we be pursuing progressive social depictions in our technologies of representation. Instead, many companies pursue pin-up types of depictions of (white) women as their data, distributed to thousands of people in order to judge the capacity of their technology.

I don’t even need to name names here — these data sets can be publicly found at nearly every volumetric capture company’s downloads page.

Military Industrial Complex

Underpinning so many “emerging” technologies is the military industrial complex. Governments are often the entities that can fund years long research to develop new technologies of representation and control that can then be redeployed and used for military operations (read: wars).

A simple example of this is the famous Institute for Creative Technologies (ICT) at USC. Home to such luminous researchers as Paul Debevec, Hao Li, Skip Rizzo, the ICT was founded as a result of talks between high ranking Army officials and Disney executives where they explored how the Army could leverage Hollywood’s expertise in the fields of simulation technology and 3D rendering to produce better training and simulation software for the Army.

This also isn’t dead history. The ICT is, to this day, a “UARC” — A “University Affiliated Research Center,” which directly receives annual funding from the Department of Defense in order to continue its operations. As recently as 2017, the ICT received more than 10 million from the DoD (pictured below) — a small amount in comparison to some other institutions, but notably the only institution of the list that self-describes as a place for creative technology development. (In 2015 they received 26m).

It’s perhaps unsurprising then that ICT alum Palmer Luckey has now moved directly into the Defense sector after selling Oculus to Facebook.

This extends beyond software as well. Our dearly beloved depth sensors are largely outgrowths of research done at leading Israeli universities to forward surveillance technologies for the Israeli Defense Forces. As much as we may think of them as creative tools, depth sensors are fundamentally surveillance technologies meant to serve those who wish to surveil.

Digital Blackface

Screenshot of Fortnite

Fortnite

Even though we still catch our politicians in blackface in their college photos, the idea of someone openly blackfacing in public in 2020 seems unlikely. However, as new technology gives us new ways of expression, old tropes remediate themselves in the new mediums.

Fortnite came under fire a few years back in one of the most high-profile cases of digital blackface, where the creators of the Carlton and Milly Rock dances (as well as the creator of Flossing), sued Epic Games for stealing their dances, repackaging them for Fortnite, and selling them to their players without any royalties to the creators.

Though the creator of Flossing is white, for the Carlton and Milly Rock, created by former Fresh Prince of Bel-Air star Alfonso Ribeiro and rapper 2 Milly, respectively, Epic’s stealing of their dances was seen as the next incident in a long history of black culture being pilfered from and repackaged in more “palatable” versions for white audiences (see… Elvis). Players are able to pantomime blackness and black culture without taking on any of the risks of what it means to actually be part of those cultures in “real” life. See also, people using reaction gifs of black people because of their perceived outsized emotional performance, forward a narrative of black emotion as “excessive”.

FaceApp / Face Filters

“Trying on blackness” is the real underpinning of digital blackface in that it allows people to, risk-free, dip in and out of black culture without any negative side effects. It builds up incomplete models of understanding of the culture they are lifting from, and can result in a false sense of understanding in an area that has historically been disenfranchised and ignored, then working to repeat the same cycle of ignorance.

FaceApp came under rightful fire then when it allowed users to select a “Black,” “Indian,” “Asian,” or “Caucasian” filter for your face to see what you look like “as another race”. As impossibly on the nose bad this is, it also has the additional terrible-ness of needing to pick out individual features from those races that FaceApp deems intrinsic to that race. Which then begs the question, what if you’re someone from that race that doesn’t have those features? Does that mean you aren’t your own race? This also comes after FaceApp made a “Hot” filter that basically lightened your skin to make you look more white.

I bring this instance up as well because face replacement technology is non-trivially difficult (for now). It requires domain-experts in the field in order to implement as well as teams of people designing the app itself. These aren’t silly one-off apps made by students somewhere (though that would still be an issue), these are real apps made by real funded teams.

From Face Depixelizer

Conclusion

It’s no longer good enough to be “well-meaning”. Emerging technology can and will be used in ways that go beyond whatever cute scenario you’ve imagined for it, and as such we should be taking an active role in preventing these outcomes instead of accepting them as an inevitability.

I’m not talking about symbolic gestures or forming committees to form plans to form committees. I’m talking about baking these considerations into the act of software development, rendering, and creative production itself. Having product people actively considering this when planning new features, have developers talk about this as part of SCRUM, part of sprint scoping, part of QA testing, etc. I’m talking about radically adopting new norms around the culture of production that actively disavow anything that gives off even the slightest smell of discrimination, reinforcement of the hegemon, racism, etc.

Symbolic gestures fail to lead to meaningful change and instead act as smokescreens to try and pacify dissent in hopes that causes can be forgotten. The fact it’s been a few months since Floyd’s death already gives us examples of this, as police reform legislation introduced in the wake of protests in California is now being curtailed. New York passed monumental legislation to repeal 50-A (a bill that essentially blocked public access to police misconduct records), but police unions temporarily blocked the release of records (which were then released in part by ProPublica). De Blasio also publicly pledged to defund some part of the NYPD, but in reality simply reclassified some NYPD officers as being under the Department of Education, virtually maintaining the status quo. This is after repeated issues of misconduct that represent pattern behavior, not “a few bad apples”. And then, after cutting virtually no money on police spending, De Blasio announced that he was unable to payback the nearly one billion dollars owed to furloughed teachers. Only after pressure from the largest teacher union in NYC did they announce that they would pay back some of it.

The work needs to be tangible, and invested in an ongoing commitment to change. The work should question the norms of development, and find ways to avoid pathways that lead us back to where we started. Everest Pipkin and Ramsey Nasser approximate this by baking the consideration directly into software use through their recently written Anti-Capitalist software license, meant to “consider the organization licensing the software, how they operate in the world, and how the people involved relate to one another” and “empower individuals, collectives, worker-owned cooperatives, and nonprofits, while denying usage to those that exploit labor for profit.”

Groups like The Algorithmic Justice League are committed to longterm work to “raise public awareness about the impacts of AI, equip advocates with empirical research to bolster campaigns, build the voice and choice of most impacted communities, and galvanize researchers, policymakers, and industry practitioners to mitigate AI bias and harms.”

The status quo that has brought us here doesn’t have to continue to be the world that we live in. As the people making the technology, films, and games that the rest of the world sees and interacts with, we should hold ourselves to higher standards and find new models for development that center the concerns and needs of the disenfranchised and systemically discriminated and set a new tone for the dawning age of AI, XR, 3D Capture, ML, etc. The work is just beginning.

Subscribe now

Welcome to the very first edition of Rendered!

Thanks to everyone who shared the initial announcement and got the word out! Since that initial announcement, it seems like everyday something exciting is happening, making it hard to publish because more exciting things always seem right around the corner. But! We’ve got to start somewhere, and to that end I’m happy to bring you the first newsletter! As a reminder, Rendered won’t just be collecting news that happens between newsletters, but is meant to capture both recent happenings as well as anything from the past few months/year that are relevant to what’s going on now.

Additionally, I’m happy to announce that the indefatigable Or Fleisher (of Volume.gl and so many others) is joining me for this newsletter to bring some insight into a selection of research papers we think are exciting that hint at the future of where our collective industries are heading. So let’s get into it!

— Kyle

To get in contact about sponsoring a future issue, please reach out here.

News

Unreal Engine 5 Announced

“Hundreds of Billions of Triangles”

Speaking of perpetually-occurring exciting news, this huge announcement hit right as I was about to publish this newsletter! As such, it’s fresh off the press, and we don’t have a ton of details, but there are two major announced features:

Lumen is a fully dynamic global illumination solution that immediately reacts to scene and light changes.

To quote an Epic employee from the announcement stream, “[light] baking is a thing of the past”. This is definitely exciting, but what has more of my interest, especially for its potential implications for volumetric rendering, is the engine’s second flagship feature, Nanite.

Nanite virtualized micropolygon geometry frees artists to create as much geometric detail as the eye can see. Nanite virtualized geometry means that film-quality source art comprising hundreds of millions or billions of polygons can be imported directly into Unreal Engine—anything from ZBrush sculpts to photogrammetry scans to CAD data—and it just works. Nanite geometry is streamed and scaled in real time so there are no more polygon count budgets, polygon memory budgets, or draw count budgets; there is no need to bake details to normal maps or manually author LODs; and there is no loss in quality.

I expect we’ll hear more details at SIGGRAPH this year, but I have one major thought. Volumetric Capture Rendering (different from volumetric rendering), in its most naïve implementations, is expensive. I mean this in all interpretations of the word, but want to narrow in on its implications for the computer that is trying to render the capture, specifically as its rendering relates to the budget (basically, how much stuff you can have in a game before running into performance issues) typically allotted for game assets.

Oculus recommends that, for any given frame of your application, your total number of polygons in that scene is somewhere between 50,000 (Quest/Go) - 2,000,000 (Rift) for VR development. A naïve, static 3D scan of a human can easily be 250K polygons alone. This means, if you want to try use this asset in a game engine, that character can easily take up a significant, if not all, of your frame budget.

But imagine you want more than a static capture and want to have a proper sequence of scans so your character can move/talk/etc. (what we think of when we say “volumetric capture” or “volumetric playback”). You’re playing back maybe only a single asset per frame, so your frame budget is unchanged (though still too high), but, assuming the above file size, just five seconds of capture inside of your project can bloat the size of your executable substantially. You aren’t just storing a single 250K poly capture, you’re storing 150 individual captures (30 (fps) * 5 (seconds)) meant to be played back in sequence. At a lowball estimate of 200MB per capture frame, and you’re looking at an executable size of ~30GB (150 * 200MB) for 5 seconds of an empty scene with only a single capture in it. On Oculus Quest, the file size for your whole application is limited to less than 1GB. But obviously you want to have the character move around/speak etc., and that will take far more than just five seconds. And if they are talking to someone else? Double that. 60GB just to have two people talk to each other for five seconds. There’s clearly a problem.

However, we still see volumetric content in VR. What’s going on? A few things.

1. The problem above can be broken down into two major parts, asset size and asset playback cost.

2. Most volumetric companies are focused on the first part of this problem, which is trying to bring down the asset size. As much as people want to tout how their stage is able to output 600GB/s, everybody knows nobody can use that much data in a production context, so the general idea is to capture the most data and then compress that into the smallest size (“compression ratio”).

3. Some companies generate smaller meshes, some companies generate binary files, and some (and this seems to be the emerging way), forgo using meshes as the “transfer format” and opt to use something much more transferrable, video.

4. The key innovation here is that, what really matters isn’t how the asset is transferred, but how it is displayed. Volumetric captures need a format that is based around time. Mesh sequences obviously work for this, but are giant. Video, however, is highly transferrable, and very small. A frame of video is a few MB tops. Compare this to our lowball mesh size, and this means video is 200x efficient at encoding.

   As an example, Metastage, a Microsoft Mixed Reality Capture Studio partner, recently uploaded a lot of their captured assets to the Unity Asset Store, and thankfully included some public tech specs. Their asset “Jim - Serious Talking”, is only 90MB for ~20 seconds of capture.

5. This method has its downsides though, as it places the emphasis on the encoding/rendering of the asset to portray an accurate version of what was captured. This is complicated. Meshes are “easier” in that you know exactly what something is supposed to look like because the format is literally meant to represent the model of something.

6. Notably, neither of these methods really change the issue of frame budget. If you use a video or a mesh, you still need to show the information stored in that, which means still rendering a potentially very dense mesh.

   I.e. The Jim asset linked above is still at a 20K polycount per frame. It has a significantly smaller filesize than using a mesh as the transfer format, but its polycount, while lower, is still substantial.

Why use meshes at all then? If video is smaller, more transferrable, etc., and has the same render cost, why do people still uses meshes? It’s because of one very important fact:

Meshes are very easy to generate.

Encoding a volumetric capture into a video is an advanced skillset. But anyone with a camera and a copy of Metashape/RealityCapture can produce a 3D scan of an object. Add in a few more cameras, and now they can make high-detailed, suboptimal volumetric captures! People are doing this now, but inevitably get stuck when they realize they can’t import a 500GB capture into Unreal/Unity.

Until now…maybe. If what Unreal says about Nanite is true, it could be a major boon for not just all the DIY capture stages that are unable to invest in refinement/compression/etc. technology, but means that large stages have an easier path to publishing because they can just use their large meshes instead of doing a lot of work to bring them down to a more reasonable size (and risk quality in the process). Said differently, playback (at least in Unreal + PS5), may be a bit of a non-issue now.

This also means quality capture is now within everyone’s grasp. So many large volumetric companies whole model is based around the pursuit of quality at the expense of file size, and if file size doesn’t matter anymore, quality becomes a lot easier to achieve for everyone. It’s no secret that Metashape/RealityCapture can produce high quality videogrammetry assets, and if you no longer have to effectively/intelligently decimate that to make it game-ready, you just cut out a lot of the space that typically stood between garage volumetric stages and the biggest ones in the world.

Additionally, because we can now use giant meshes, I think videogames in general will now be much more interested in volumetric capture. Before, mesh sequences were too big to really fit in any game (and games that do use scans use a singe scan and re-rig it for animation), and video as a sequential format actually doesn’t play that nicely with the realtime context of a game engine (and hence turned people away). But if a game studio can “just use meshes”, and studios can just “give them meshes”, then we can finally get some volcap in games. (Side note: I think this also represents a big opportunity for companies investing in volumetric retargeting functionality).

I’m dreaming big here and in full conjecture mode, but I’m excited about what this may unlock for people. We obviously still don’t know a lot about the details (does it compress imported meshes?? What about the web??), but I’m excited.

iPad Pro Gets a LiDAR Sensor

New depth frameworks in iPadOS combine depth points measured by the LiDAR Scanner, data from both cameras and motion sensors, and is enhanced by computer vision algorithms on the A12Z Bionic for a more detailed understanding of a scene. 

If there was ever a doubt that “spatial computing” wasn’t yet here, let this be the ringing bell to announce that it indeed has arrived. I think everyone recognized that the sensor-fusion based approaches for 3D scanning with an iPad were acceptable at best (why else make an attachable depth sensor?), but I don’t think anyone actually thought Apple would take the next step to give their flagship device a proper LiDAR sensor. But here we are.

Opinions seem mixed, but mostly from the “It’s not that great!” angle. Which, obviously. Anyone looking to this to replace a Velodyne or something comparable would be looking in the wrong direction. Instead, I think this should be seen as an indicator of what’s to come, not just taken at face value. Besides, people are getting cool stuff out of this, they just aren’t mounting them to the tops of their cars for DIY autonomous vehicles… yet.

What’s also interesting about this is how it reconfigures some of the reality capture market on iOS/iPad OS. Before, companies needed to do a lot of their own sensor fusion to make similar functionality work. With past releases of ARKit and now the LiDAR sensor, Apple seems to be slowly eating the bottom rung of these apps. The hope here is that it raises the ceiling of what the apps that leverage these features can accomplish, but given that we’re still seemingly stuck in a place where the best (read: profitable) uses of facial tracking + scanning are just placing a fake couch in your room to see how it looks next to your ficus or trying out different Instagram filters, it’s easy to see a future where Apple throws up its hands and just offers these things natively.

Magic Leap Lays Off Significant % of Employees

For a while now people have been talking about the post-hype slump of VR. We’re beyond grand talk of the “empathy machine” and are in the process of crashing into a few harsh realities that come from no longer being the golden child of tech:

1. Headsets are expensive in the best of times

2. Premium experiences can require a lot of computer power, so the ask for a casual observer is to buy not only a headset but also an expensive gaming PC (“Which one? How? Do I build it? What’s a driver?”)

3. Nobody knows what “content” is supposed to look like.

4. Revenue is rising but it’s not hockey-sticking like a lot of people who invested in VR would like to see.

I think that the amount of hype generated by people with the will to endure steep headset prices, buggy software, sub-par experiences, etc., was enough to make the world turn to look at VR for a bit and be interested, but the world now seems to have mostly turned and looked in a different direction.

The revenue for all XR headsets combined since 2018 is 3.2b. Magic Leap raised, in total, around 2.6b. The idea that the Magic Leap headset was somehow going to be bigger than the market it existed in is the sort of sideways group-think that leads people to continually invest in a moonshot idea in hopes that it becomes too big to fail. Add in an over-promised and under-delivered core product, a global pandemic, and an an industry not trending up as fast as expected, and something like this is bound to happen sooner or later.

We all knew it wasn’t going to work out, but I think we also all wanted to believe it would. Not just from a company or financial angle, but the promise (or better, the dream) of what was being offered was, and still is, compelling. It’s also why Magic Leap was able to continually bring in great people to work at the company despite the looming reality of the situation, missed deadlines, and bad signs. Magic Leap cultivated a brilliant group of people working in XR space, and a lot of these people are now without jobs. Luckily for us, they’ve organized all of themselves into a massive spreadsheet! There’s nearly 250 people in the spreadsheet below, all experienced in the XR space and all looking to be hired (and many willing to relocate!) after being laid off from Magic Leap. If you’re hiring, definitely take a look at the list and try to bring some of these wonderful people into your fold.

Magic Leap Employees Spreadsheet

Huge Virtual Production Set Used for The Mandalorian

One of the biggest things to happen in the virtual production space in the past year was the release of Disney’s The Mandalorian and subsequent press circuit touting how it was made.

There’s more detail in the linked article above, but I think a key thing to note here is that the principles of what this set is have been around for a while. The primary innovation of virtual production (in my eyes) is only one thing: tracking a camera’s position in space. The effusiveness of the press circuit around the tech here may make you think that The Mandalorian stage is offering more than that, but that’s essentially it: a tracked camera and some giant LED screens.

When you know where your camera is at in space, you can do a lot of things. The main thrust of virtual production right now is to pipe that camera position into Unreal (as done on The Mandalorian), and then display what the virtual camera in Unreal sees somewhere, be it a directors monitor, the physical camera’s monitor, or in The Mandalorian’s case, a giant screen.

This used to be an incredibly hard/expensive process, in part because there wasn’t really any affordable middleware that abstracted the hard parts away from a production. You couldn’t “just use Unreal”, you would have to build your own realtime-ish asset viewer (making game engines seems to be a theme for Rendered #1) from scratch.

Avatar did this though a giant Vicon-style camera tracking array that was able to see where the camera was and stream in proxies to a directors monitor. They developed their own streaming asset viewer.

Oblivion used an approach very approximate to the Mandalorian, but didn’t need to rely on a tracked camera position because the “background” was depicting a scene far enough away that you didn’t need to worry about parallax. So instead they rendered out a high quality back plate and played that back behind the actors.

Gravity pre-programmed a video scene inside a LED cube to match pre-programmed camera patterns. A downside of this is that the motion in the LED wall is “baked” - you can’t really improvise anything and all the motion has to be perfectly choreographed, otherwise the background and the lighting on a character’s face won’t light up.

The most recent Star Wars films used an approach similar to Oblivion in that they used projectors get “real” lighting for inside the Millennium Falcon.

What The Mandalorian does is bundle up all of the above, and then throw a ton of money at the problem. The reason nobody has done what they did before wasn’t because the ideas weren’t there, but mainly because it takes a studio like Disney to build four massive, high-resolution video walls and a company like Epic to leverage decades of game engine development to be able to deliver a robust third-party solution for realtime asset viewing. A lot of people have been approximating the same experience, but Disney was the one to finally put the pieces together and just do it.

It’s not a holodeck, it’s not a revolution, it’s just the next permutation of an idea that has been steadily evolving in film over the past decade.

Robert De Niro Get De-Aged for The Irishman

Speaking of evolving ideas — the recently-ish released film The Irishman used some pretty insane techniques to portray Robert De Niro’s character aging over time. When I say insane here, I also don’t mean “awe inspiring” — I think I’d more describe it as masochistic, and in a way that speaks to the divide that exists between the emerging spheres of virtual production and traditional production, mainly in how The Irishman approaches the problem it is trying to solve.

Problem: Robert De Niro in real life is older than the character he is meant to portray. How can we de-age him to make him look younger?

Traditionally, you do it all by hand. Yes you can guide it with some reference 3D scans, but at the end of the day someone is tweaking sliders around lip creases to try and get each frame to look right. I’m being a bit facetious here, but this is the general thrust of how you get an uncanny resemblance to Rachel in Blade Runner 2049, Leia in Star Wars: The Last Jedi, and Benjamin Button in The Curious Case of Benjamin Button.

From the FXGuide post on Rogue One:

A blendshape rig is the dominant approach to facial animation used throughout the industry.

The perpetual issue is that film is a 2D medium, but people (and faces) are 3D. To really get the representation right, you need to somehow derive a 3D model of a face from a 2D reference. The most advanced approaches to this problem will have actors sit in for a face scan in order to generate a 3D dataset of their face, and then VFX artists will use that asset library and build up a face that most closely matches the face/light/etc. that needs to be replaced in the corresponding shot. Custom tools exist at each step of this process to make it easier as well (Flux/Medusa/etc. mentioned in the video).

For deceased people or ones that otherwise can’t sit in a capture stage, you have to look at archival footage/video, and then uses those images as part of a very rough photogrammetry project to reconstruct their face from scratch.

Also from FX Guide post:

For Leia, with Carrie Fisher’s blessing, the team sculpted a digital Leia solely from reference.

For The Irishman, the production company did… everything. The took De Niro and other cast members in for a scan at Light Stage. Did the archival work, and custom built their own IR stereo camera to get the lighting information (via disparity) from a scene.

Notably, through the use of their tool Flux, “The new Flux pipeline is designed to produce results without an explicit animation stage.” It sounds like they mostly automated the actual process of generating the 3D face from the 2D data.

What I think these approaches still miss though is that they still treat the act of filming as a fundamentally 2D process. You are likely not surprised to find that I think we should start treating filmmaking as a 3D process.

If the film set used an actual RGBD sensor (via Depthkit, or even a custom one), they would have been able to capture 3D data as a first priority, instead of having to derive it later. The IR sensor they built was also only really used to capture 3D lighting information! They derived the 3D from the 2D image! This is a huge inefficiency.

The issue is that 3D capture is seen as a mutually exclusive process from the 2D capture. If you’re on set, you’re doing normal 2D capture. If you’re off set, you’re doing 3D in a big capture dome, devoid of context. But, moving forward and in my mind, these will no longer be mutually exclusive processes. However, it will require DPs and other techs on set to understand the 3D capture process just as much as they understand 2D capture.

It’s also funny because, even considering all The Irishman did, one of the techs on the project, in the video linked above, states:

“The achievement is not wearing markers.”

All this work to get around not having to wear markers… c’mon. If the VFX industry is able to brute force their way into getting a 3D model from a 2D capture, I think that’s fine, but if all we’re trying to do to not get people to wear markers, I do wish some productions would try out using depth sensors more prominently for key capture — not just for reference, compositing and camera tracking.

I have more to say on face replacement in Hollywood (small tease), but I’ll have to save it for the next newsletter, as this one is already getting too long.

Custom Game Engines Guide

In the time before Unity/Unreal, to make a game meant to build an engine to make a game and then make the game with that engine. This meant that games creation was far less accessible than it is now. I think often of Chris Wade’s post in Gamasutra about why it’s hard to make a game, primarily the “inverse pyramid of complexity” that anyone who has made a game can understand:

Making you own engine is like its own mini pyramid inside that foundation section, but one just as wide as the top of the main pyramid. You just wanted to make a simple platformer, but ended up needing to learn about cross-platform graphics APIs. It’s easy to see how this was prohibitive.

However, custom engines still persist! In part because of the ways some studios operate/like to make games, it can be easier to build a tool that is custom made to create the game you want to make, instead of needing to bend a more general tool like Unity/Unreal to be the engine you want it to be.

Raysan5, the creator of the fantastic game framework raylib, put together a giant list/compendium of known custom game engines and the games they are used on, linking where possible to information about the engines themselves.

For anyone interested in game engines/game engine development, I highly recommend checking this out!

C# Source Generators

It’s not often a programming language gets a feature added to it that is worth talking about in a newsletter, but the announcement of C# getting metaprogramming/macros/whatever you want to call it through their introduction of “Source Generators” is big.

I recognize that other programming languages have very similar, and even more robust, permutations of this same idea, but why this is big in the context of C# is that C# is one of the most widely used languages for game development. This is largely due to the fact Unity is a behemoth in the space, but, if you look at the list of custom game engines above, many use some version of Monogame (a C# game framework), and other custom ones may use C# as well.

There’s a whole newsletter’s worth of reasons I think this matters, but I want to focus on a specific feature of Source Generators and shine a light on what I think its implications are.

Part of the Source Generator feature set is that they have the ability to load in arbitrary files before your code is complied. They can then parse that file and turn its contents into C# code. This means I could pass in a JSON file, and that gets converted to some C# representation of that JSON file.

One of the largest barriers to entry for people learning Unity is that, much to their chagrin and despite how much they were told “Unity is easy!”, they still need to write C# code. Imagine instead if that person was able to author a simple JSON file to describe what they wanted in their scene, feed it to some plugin, and then have a working scene in front of them, complete with logic, lighting, etc., all without needing to touch C#. You’d be able to author games/interactive experience/whatever without needing to code, but could have all the affordances of code at the same time.

This is the promise here. It’s a half step between needing to have a robust plugin that allows for realtime authoring (where logic has always been an issue), and writing pure C# code. I think, once Unity gets access to this feature of C#, we’ll see a lot of plugins utilizing something similar to the above scenario to do a variety of different things, from populating scenes, to setting up build pipelines, to managing serialization, etc.

Papers Review

In every issue, we hope to surface state-of-the-art research in adjacent fields that is likely to affect the future of volumetric capture, rendering, and 3D storytelling. To do this, I’m happy to have Or Fleisher on board to give us an overview of some really cool research that has emerged over the past few months. Take it away Or!

Thanks Kyle! Conferences around the globe are hit hard by the Coronavirus pandemic, many of which have been canceled or postponed (SXSW, GDC, E3, and more). However, with SIGGRAPH (July 19) and CVPR (June 13) going virtual we're seeing increasing amount of incredible research being published online — some of which actually tackles the inherent issues imposed on research by the pandemic.

NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

B. Mildenhall, P. Srinivasan, M. Tancik, J. Barron, R. Ramamoorthi, R. Ng

Project | Paper | Code

What are neural representations of scenes? Good question! This research presents a method for synthesizing new views given a set of input images with known cameras (Sound familiar? I am thinking about Photogrammetry too). The research explores the idea of modeling a scene as a radiance field, or, put simply, a function that takes in the position and direction of a view and returns the color that (imaginary) ray hits.

To do that, the research trains on a set of input images of a scene which embeds and optimizes the radiance fields. Upon training for a very long time you can then sample new views by providing a position and direction coordinates, which is nonetheless, mind-blowing 🤯.

Why this matters

Right now, this research could help reduce the amount of images needed for phtogrammetric reconstruction by synthesizing new views based on a small collection of input images and known cameras. Down the road, this is one of a few examples that suggest a radically different approach to rendering - instead of a traditional rendering pipeline that draws pixels on the screen based on known geometry, pixels here are the result of an optimization algorithm - meaning there is no sense of geometry in the rendering process.

3D Photography Using Context-aware Layered Depth Inpainting

M. Shih, S. Su, J. Kopf, J. Huang

Project | Paper | Code

I’ve seen this parallax effect before and wondered, how many hours of compositing were spent creating it? Although the results of this research are presented in somewhat cliché ways, the potential uses and implications of what it’s solving are big. The research presents a way to segment and in-paint occluded areas in RGBD captures. In some ways this work relates perfectly to NeRF (above) as it also uses view synthesis to fill occluded areas.

Why this matters

Many of today’s volumetric capture systems are limited by line of sight occlusions. For example, if someone holds up their hand in front of their face, you’re unable to see their face from that capture. You could use a different capture, but the issue is often much more subtle. The camera that is being occluded will often have the best view for some number of arbitrary viewing angles, but if I’m looking at that capture from a ten degree offset, even if it’s the best angle, I’m going to see missing data where I expect a face to be. While not the original intention of this research, in theory, it could be used to in-paint these missing parts. Or if nothing else, at least you can make cool GIFs?

AdversarialTexture

J. Huang, J. Thies, A. Dai, A. Kundu, C. Jiang, L. Guibas, M. Nießner, T. Funkhouser

Project | Paper | Code

Texture fusion from multiple views has been one of those longstanding research topics, and while there are a lot of approaches out there that work - I found this recent paper to provide a fresh take on using machine learning to try and tackle that problem. By employing a discriminator network that trains on pairs of “real” and “fake” (re-projected) images this approach is able to achieve good looking results for texture optimizations from RGB-D scans.

Why this matters

Blurry/”melting” textures have always been an issue with 3D scans. Especially if you happen to catch a shear angle, your texture can look spread like cream cheese. To anyone that knows this problem, this this paper suggest ways to finally (and consistently) get around this problem.

Resources

Kyle here again. In the sprit of trying to collectively grow the knowledge of everyone in this space, I want to share a few resources for people to check out each newsletter. These will take various forms, but ultimately should be something you can directly engage with now, without needing to set up a camera, program, etc.

Arcturus Worldwide Volumetric Studio Map

Arcturus, creators of Holosuite, have updated their map of the volumetric studios around the world! The map breaks down where studios are at globally, breaking them down by region and type (4DViews, Microsoft, etc.). Check it out here.

Scatter Resource Collection for Depthkit on Are.na

Scatter maintains a public collection of a ton of interesting links that are related to volumetric filmmaking, VR, and 3D scanning on Are.na. I’m sure I’ll highlight some of the resources linked here in future newsletters, but for now just dive in here!

Open Source Full Body Volumetric Capture Software

This has been on my radar for a bit but I’ve never known anybody to use it. They recently added in Azure Kinect support, which seems like a good as time as any to try it out.

Briefly Noted

These are smaller news items that I didn’t want to primarily feature but should still be just as interesting as some of the news featured above.

TetaVi raises $4m, partners with 8th wall to scan president of Israel, and partners with Crescent to build a volumetric capture stage in Japan. Occipital, creators of the Structure Sensor, are teasing something new. Intel released a few demos that show the Realsense LiDAR camera in action. The Realsense docs also have a surprisingly robust breakdown of depth image compression via colorization. Promising game engine The Machinery (made by ex Bitsquid/Stingray developers) released a new version of its beta. Paul DeBevec did a big breakdown of the most recent iteration of Light Stage for FXGuide. Ncam releases virtual production software for Unreal.

And with that we close out the first issue of Rendered! Thanks for much for reading the newsletter. Or and I both hope it was everything you hoped for, and if not, please send us feedback (and go ahead and send some feedback if you liked it too)! You can reply to this newsletter directly, or reach out to me at kyle.kukshtel@gmail.com.

Additionally, please share this newsletter around! Rendered thrives off of its community of readers, so the larger we can grow that pot of people, the better. Thanks for reading!

Subscribe now

Rendering…

I’ve wanted to start this newsletter for a while. The feedback on my news digests at Volumetric Filmmakers NYC has always been overwhelmingly positive, and I’ve been thinking about how to break that section of the programming out into something more regular.

This newsletter, Rendered, is that and more. I’m expanding the scope to include related happenings in virtual production, game engine technology, 3D rendering, reconstruction, synthetic media, etc.

For some examples on the types of content to expect, check out some old slides from a previous meetup here, a record of topics from our first meetup, and a record of topics from our second meetup.

I’m also hoping (especially in these time of quarantine) to feature some small interviews or blurbs from people in the space on what they are working on or discussion around a certain topic, maybe in audio form. Please reach out if you’re interested in sharing your work.

I’m still prepping the first real issue of the newsletter, but if you’re interested in:

• Volumetric Filmmaking

• Synthetic Media & Deep Fakes

• 3D Rendering & Reconstruction

• Telepresence

• Virtual & Remote Production

• Videogames & Game Engines

• Creative Tools

Subscribe now to make sure you get every issue of the newsletter, delivered monthly.

Subscribe now

Header photo by Clark Van Der Beken on Unsplash