How MSNBC used tech from ‘The Mandalorian’ to bring election night to life

It's 6:55 on November 8th and on the 3rd floor of 30 Rockefeller Center, just beyond a silent sea of empty desks, is the beating heart of election night in America. In Studio 3A, Rachel Maddow, Chris Hayes, Joy Reid, Nicolle Wallace and Ari Melber sit at a curved desk, addressing the nation. Opposite them, Steve Kornacki checks a computer in front of his Big Board. Outside the studio, hallways are bustling with staff coming and going. Jen Psaki floats by in white slacks, on her way to join an expert panel analyzing the results.

While Kornacki and his khakis are always a major pull for the network, the star of the evening is an impressive bit of TV magic. As one of the first results of the night is called, the camera swoops from Kornacki into an adjoining open room just behind him, 15-feet deep and swathed in green.

But viewers at home don't see the green room. From their perspective, the camera zooms from Kornacki, out into the night air, and on to the grounds of the United States Capitol in Washington, D.C. As the camera rests on the Capitol walk, faces of candidates pop up as races are called.

SEE ALSO: Go on a spacewalk with ISS astronauts in virtual reality

A sample of the Capitol illusion in action during election night 2022.

The illusion was dreamed up by MSNBC's real time graphics team, who worked for months to create it. To do it, they called on Unreal Engine, a tool originally developed by Epic Games to create gaming environments. Unreal has recently been used by the production teams on television shows the Mandalorianand House of Dragonto project a virtual world onto a physical set. Those projections enabled directors and actors to see, in real time, what would be later added by CGI.

While Unreal Engines may be the new darling of televisions, its application to news and sports is inchoate. Marc Greenstein, SVP of Design & Production at NBC News and MSNBC, says his team is the first to seamlessly transition from real life space into a digital one on live television.

The hosts' view of Kornacki and his khakis at his board. Beyond him to the right is the green room. Credit: MSNBC

Studio 3A is small, but it's packed with tech: more than 500 LED lighting units and 40 million pixels of video walls make it possible to change the entire theme of the space in minutes. Hanging over the center of the room is a first-of-its-kind video archway made of a single, 30-foot video panel. Cameras automatically orient themselves within the space by measuring their distance from hundreds of randomly-placed, infrared-reflecting stickers on the ceiling.

Mashable Light Speed Want more out-of-this world tech, space and science stories? Sign up for Mashable's weekly Light Speed newsletter. By clicking Sign Me Up, you confirm you are 16+ and agree to our Terms of Use and Privacy Policy. Thanks for signing up!

A swath of studio lights crowd a square white sticker that helps studio cameras track their location. Credit: Elizabeth de Luna

Alex Bassett, Senior Director of Production Technology, points out one clearly visible sticker, crowded by a dozen studio lights. The other squares are invisible on purpose, made to blend in with the wooden panels on the ceiling and the video arch.

Chris Hayes, Joy Reid, Rachel Maddow, Nicolle Wallace, and Ari Melber at the desk on election night. Credit: Elizabeth de Luna

Just down the hall, a control room of campaigning alums and politics wonks direct the flow of the show and focus of coverage. Fourteen people sit in darkness in front of double monitors, flanked by another handful of staff members standing around the room. On the wall in front of them are more than 100 live video feeds playing at once across at least 15 feet of screen. A race is called and the camera swoops into the Capitol again. A staff member in the back of the room tuts in awe of the technology, "It's so good!"

A staffer watches Steve Kornacki from the control room. Credit: MSNBC

It took months of careful work to bring the Capitol illusion to life, and Greenstein's team also prepped the Big Board for its big night. His team gleans data from MSNBC's Decision Desk, which calls the races internally. As they import that data, Greenstein's team takes care to "respect it in a way that we won't possibly manipulate, change or alter it. It's actually harder than you think," he says, because "the data can show up in 14 different ways. We have to go through every variation of those 14 different ways, times the 1000s of [versions of the] board. And then you have the design part of the data: Can I read it?"

When asked if Kornacki may ever step away from his board and into the green screen room to interact with virtual data there, Greenstein says it's not likely. The direction of MSNBC's VR development will be dictated by the storytelling needs of its on screen talent. "Steve Kornacki is one of the greatest examples in the world" of talent that can work with data live on air, Greenstein says, "No one's better than him in the business and we stillgive him a physical glass screen. Just because the technology can do it doesn't mean it should.

"For Steve, rather than investing in AR and VR, we really made a huge investment in what the board could do. He's not a technologist, he's a great storyteller," Greenstein continues. "Conversations with him never start with 'Here's this great new technology, what can we do with it?' It starts with 'What type of stories do we want to tell with it?' We're never going to put him in front of something that he thinks is gonna get in the way of what he's trying to explain to the viewer."

James Matarese, Managing Art Director of Real-Time Graphics at MSNBC, watches the fruits of his labor on live television. Credit: Elizabeth de Luna In a mock control room, two screens display two realities. On the bottom, the green screen room at it appears on camera. At the top, what the nation is seeing in its place on television. Credit: Elizabeth de Luna

It's hard not to imagine the Capitol illusion being expanded upon to deliver information for the Presidential race in 2024, along the National Mall perhaps. And Greenstein thinks NBC may eventually be able to leverage the metaverse to enhance training for NBCU Academy or to allow viewers at home to try on shoes virtually while watching hosts model them on the Today Show. To explore these possibilities, NBCUniversal has convened a Virtual Production Council with members across its news, sports and entertainment properties.

But when it comes to the most practical application of virtual reality, James Matarese, Managing Art Director of Real-Time Graphics, says it's daily production. Right now, using the Capitol illusion takes a team. He's working towards "Making [this tech] as scalable and accessible as possible" so that any producer can use it, without the help of a technical team. "To have this as a living tool as part of a sustained workflow, that's my finish line," he says. "And I think we're close."

"What we've learned from this is going to pay dividends as we move forward," says Greenstein, "I compare [our VR tech] to the space program internally...we wouldn't have a microwave oven if we didn't try to send someone to the moon."