THE WORLD OF EXTENDED REALITY

Hello, this is Les Goldberg, and welcome to The Road Ahead. The Road Ahead podcast is dedicated to the future of the live events business, bringing together industry experts. Hello, production world, this is Les Goldberg and The Road Ahead. Today I have three amazing guests. I have Danny for who is the co-founder of All Of It Now, I have Nick Rivero who's the co-founder of MEPTIK, and I have I have Val Syganevich, who is the producer and founder of Freckled Sky. Hey guys, welcome to the show. Thank you, thank you. Hi guys. We are going to talk about extended reality today. And extended reality is something that's relatively new in the production world. And I'd like to kind of get you all to help us understand it and really what you can do with it. And the world of so we're going to start with Nick. Nick, if I asked you what is XR? Maybe you could give me your simple layman's definition. Yeah. I think the way we look at it is it's the idea of combining the physical live world with the virtual world to create something new, something entirely new in the physical world. I think, simply put. Hey, Val, could you add to that? Maybe so, as we see it is, XR is a tool that gives a possibility to create live shows on a par with the Hollywood movies. When the artists or a live real person are inside an environment and interacts with it. But environments can be like realistic or fiction. And in the both ways, this live person looks inside of them. So this is how we see this. It sounds like we're creating illusions, things that maybe the illusion. Danny, could you opine on the simple definition of Z. Yeah, I mean, at this point, it's an evolving kind of concept. The shortest answer is magic at this point, because in many ways it is exactly that. It is that sufficiently advanced technology that is indistinguishable from magic. It is changing on a daily basis, and it is something that is an incredible creative tool for people in our industry. So, Val, I'm going to throw this question to you first. Where have you found the best use of XR? I believe there are a lot of different areas, but let's start from music first. If even if you months or a few like a couple of years ago for for a country, you would need to rent a big stadium, right. And build a big stage. So right now the artist can just come into pretty small space with LED stage, XR led stage, and artists can perform just one concert on a few different virtual stadiums or for corporate, for example. It's just a new possibility to level up keynote presentation on a new level speaker, not just on a stage and with LED wall behind him and PDF or PPT presentation on it, but he is inside an environment and he can just operate with holographic data around him. Or it can be a new level of audience engagement, where audience can be integrated into environments with speakers or artists and they can like, have live dialogs with them. So product launches, fashion shows, TV productions. Sounds like there's a lot of possibilities. So, Nick, can you tell me where you have found the best use of - A - of - augmented - of - V - of XR? I'm sorry. I got my terms mixed up. Yeah. What we've seen is really in, say, the film, TV commercial and then also music video production world. The biggest pro we see is the idea of locations is now we can go anywhere and for say a TV commercial, you might want to shoot in three, 4 or 5 different locations. But now we're able to actually bring that into one place. And we worked on a piece just a few weeks ago that was a tabletop production, so tables with bottles and classes on them, and we're actually able to make that into so many different places, all while staying in the same room. So something that might have taken a week or more even to get to these places set up for production, shoot a scene we literally were able to accomplish entirely in a day, in one place. We don't have to pack our stuff. We don't have to get on airplanes. We don't have to move trucks. We avoid all that cost and we just create, as you were saying, the magic, I think Danny said, it's just magic. We create this magic and we save a lot of money and the audience can't tell the difference. Is that true? Yeah. Yeah, definitely. We use the same quote Danny said is, you know, any good technologies indistinguishable from magic? We say the same thing all the time. And I think it's true. That's awesome. So I'm going to throw this to Danny and Danny. When you're designing extended reality environments, maybe you could share with us the considerations you think about. And maybe it would be a checklist of these are the kind of things that are things we want to deal with and make sure we understand. These are our criteria. What would your criteria be in the world? Yeah, I mean, one of the first ones is kind of how many people are on the are on the stage in the first place. And something that we found is it's easy for that person to get lost in the environment, especially if there's a lot of things happening that if you have, you know, because the the possibility of infinite space and infinite possibilities in terms of what this environment is and can do, it is very easy for the performer or the on stage talent to get lost in that environment. And so one of the considerations is to kind of actually tone it back, make it more minimalistic, make it more simplistic sometimes so that you you don't overwhelm the viewer with this environment. And that's something that a lot of that we are creatively having to, to do something with where normally our, our content, our, our, our designs fit into boxes and fit into predefined shapes on a stage. And they stayed there. But now that there is this kind of infinite possibility for, for content, for motion, for interaction, it changes the nature of the presenters relationship to the stage and to the content and to the into the screen, the presenters or the people on on stage. Is there an uncomfortable feeling or something to get used to? Danny when, when when I'm sitting on a stage, is that is that something they have to adjust to? Yeah, absolutely. I think one of the challenges as well with, with I mean, even even VR or AR, well, not in a VR as much, but AR is eyeline is that oftentimes, especially for those who are new to presenting, even on a green screen or any sort of virtual environment is it's easy to look off camera and to look at a monitor to see what you look like on camera. Because even even if the environment is enveloping you, you still kind of want to see that. And it's not as viscerally visceral or tangible as an actual stage, right? When you're looking to the right, you know that you can put your hand and touch this thing. But in an environment, that thing is an illusion. And so when you try and reach your hand over, it doesn't actually exist. So it takes a little bit of time to get familiar with virtual sets, virtual environments, and that kind of changing spatial relationship that you would have compared to a physical set. And then that changing relationship with performing, because you kind of have to be very conscious about where you're looking. And that is also for the technical crew, making sure that if you are going to have a kind of a confidence monitor, making sure that it's it's where the camera is. And given that cameras often move that, that it can can move with them, otherwise their eyeline is always going to be a little bit off and it's going to it's going to bring pull some of that that performance out, you know, away from their their focus. You know it's interesting I was just thinking about I'd like to do a production on top of Mount Everest and, and I think if we could all just get our bags and pack and climb and go up there, it would probably take a few months. We could, you know, the idea of being able to do that and not really go to Mount Everest is very appealing to me. And Mars would be the same. We'd get on a spaceship and we'd be gone for a while. So the question to Val, when, when you're designing for XR, what are the considerations? Do you have anything to add to what Danny has shared, and what do you think about when you're coming up with the design? I think two major things I would like to mention. The first one that because of using real time graphics AR in life motion tracking, nothing is defined by pre-rendered content or choreography, so it makes appearance this. At this point it makes appearance of an artist or presenter inside this environment easier. So if you if you know that. Yes. At this moment you would have this kind of effect. But if you late just a little bit with your movement it's, it's, it's everything's fine. If you're not in an exact mark for example. So it makes it easier. And the other thing is again, you don't need to have a huge stage because the the technology extend extends your, your environment. So it's also very helpful. So when you when you need someone just to walk somewhere, it doesn't mean that he literally needs to walk somewhere because you have an environment moving around this person. It sounds like a lot of things are moving and it sounds like the, the perspective. And there's there's a lot of pieces to this puzzle. Nick, could you just maybe explain when I say I want to build an XR stage, what are the pieces to an XR stage? Maybe the audience might want to hear what you think are the components that make up an stage. Yeah, well, it really breaks down into two areas, which are the the technical, the actual physical pieces. And then there's the creative side to it. And with the technical, there's really kind of two different ways you can approach XR. We look at is one is use of LED and LED screens. And that's where we've seen things such as The Mandalorian happen, where LED is really promising. And what we're using for is a lot in the television and film world, because you get those global illumination, you get reflections and things of that nature. But on the flip side, we have tried and true green screen technology, which we've seen all the way through for many years on the nightly news with weather casts. Exactly. We've seen it forever, but it still has so many great uses because it's it's very portable and it's very scalable technology. You can go into so many existing green screen studios and create really compelling virtual productions in a green space when you don't have the needs for LED. But as I said, the other half comes down to really the creative and as we say, quite a bit internally, as with Shure, it's actually less of a technical problem and more of a creative one because the creative is actually what drives the story. It's actually what drives the narrative of what these things happen and do. So you have to really create compelling content. That is what is actually sharp. It's actually what's creating the experience. Very interesting. I can tell you the stages that we have built are typically, you know, two LED walls at a 45 degree angle with an LED floor. We put in a Stipe camera or multiple Stipe cameras, and or we might have camera tracking and we may have cameras on on rails motion tracking. So all of those are the pieces. And then lots of media servers to to kind of put all the pixels in the right place. So this question is to Danny. Danny, what are the greatest challenges in making the best XR experience? In our experience, it's mostly been time. There have been situations where we have been, you know, we get asked to kind of create something that are these large scale environments. And unlike a standard content treatment where you have a pixel map, you only create content within the confines of these boxes. You now have to create content that is a a world of 360 environment. And there have been cases where we say, oh well, as long as the camera stays in this area, we've created an environment to match this. And it's kind of like the situation where in a film set where the, you know, set team builds only part of the house and they don't have to build the entire house because the camera doesn't move that far, but they just kind of run out of time to do that, or they run out of budget to build the entire house. And we kind of have that similar issue of, there's never enough time to do this thing, to do what we need to do, because ultimately it's it's a tremendous stack of technology, hardware and software that all needs to kind of work in tandem with, with each other. And that all costs money then many clients have right now. And so the way to kind of to, to pull it off at all is ultimately to sacrifice the amount of time that that we can spend on the creative, we can spend the technical. And that time is always kind of the biggest enemy to to pulling off a, you know, an extra production. But it's something that will get better over time as, as we can kind of educate clients on what they need to know so that there's a little bit less time spent on the discovery process and education process, and more time spent in the collaboration process. Sounds like all I need is some time. Time is the enemy or time is our friend. So. So, Val, I'd like you to maybe to touch on one thing. He mentioned a lot of different kinds of AR and VR and mixed realities and all these Rs. Maybe you could just give us a very simple explanation of the difference between the Rs and how is different than the other Rs. So I love AR and X are more than others, but I would say actually extended reality is just a collective term combining all of other R's relative virtual reality and mixed reality elements to extend the reality. That's the thing. So our mental reality is just virtual objects that you can see around and all around real people or within real environments, virtual reality. This is where the entire reality is virtual. So you don't see real people there. They're all the people are fictional or looks like fictional and mixed. Reality is something when you have a real set or something real place with some additional virtual or AR elements. Sounds like each reality is a little bit connected to the other realities in this virtual world we're building. So, Nick, where are we in the lifecycle? Like if you know how things, you know, have a life cycle, where would you put us in that life cycle? At the tip of the iceberg? I think we're just at the beginning, and I think we have so far ahead of us when we kind of zoom out and look at the big picture of society and history, is we're really just at the beginning. And I think, you know, in 20 years, these types of technologies are going to be commonplace, like light bulbs and refrigerators. These are going to be things that are woven into every event we do in some way, shape or form every technology. We already see this in the phones in our pockets. So, you know, even just five years from now, we're going to see things like more prevalent, just in commonality of how we use it in a show or how we can do webcast in the future. These things are going to be so much more normalized. And right now, we're just at the tip of the iceberg with what the technology is, what it allows us to do, how far we can push the creative. So I think we have a long road ahead of us. My question is how has the production industry been impacted by? Are people not going on location anymore? They're just changing the content and the backgrounds and and you can take us there and we can all stay in the same city. No airplanes. I mean, I think that in many ways, XR has definitely helped the the live events industry in these really, really difficult times. It has been a kind of opportunity to put a lot of the extremely talented people who work in this industry back to work in ways that they couldn't work normally, and that's just tremendous power in that, and that one of the really great things about the live industry is the kind of collaborative nature of their work in the first place, but also the fact that in many, many cases, they are the most kind of apt people to tackle working in because they're working in a collaborative and most often 3D previous environment to begin with. And that understanding of 3D space is a truly crucial component to how XR works, and that that understanding of 3D space is really, really important. And so working with lighting designers, working with content designers, working with creative directors and Nvidia designers who understand the 3D nature of the content that they're building and have built in the past, is, is, is extremely crucial. And the kind of willingness to collaborate and and oftentimes even even before COVID, the kind of willingness to collaborate through remote sessions, calls, etc. are what make this industry great. And what how many, many of us in that industry have risen to the challenge. You know what it sounds like to do what you three guys do is you have to understand your 3D spatial worlds and use the technology to produce really, really cool outcomes. The magic, which I find is probably such a simple concept. You're all really magicians, you're just masked as XR engineers and creative people. So this is my last question, and I'm going to throw it to Val and I'll throw it to all of you before we end. Val, take me into the future. What is the future of czar look like over the next few years? How do you see the future and what you know? We got a little taste of what Nick said, but take me into the future. First of all, I would say we also always believed in magic. So we started doing magic years ago without knowing it all on live stage. So we believe we want to. We wanted to and we we want to bring this magic to as many places as possible, as many areas as possible. And this is the one of the tools that can help us doing this, for example, of our business to areas we didn't see ourselves before the pandemic at all. So we started we started working on next level TV shows. These days, I believe the ethnic set is just very, very beginning and the future will bring us to the touch of bring us to so many different options, specific TV shows they based only on extra technologies, or live streaming concerts based on technologies. They can change movie industry as well because it can, like digital industry. A change in the TV industry like Netflix, Amazon. Right. So the same thing will happen with when it comes to to TV industry as well. So Danny, tell me what the future is. Take me into the future. Yeah I think the future of X is mobile and interactive. At the at the core of our technologies, we're dealing with game game engines or virtual production engines. These are inherently interactive pieces of the puzzle. And that as we can get audience interaction and and performer interaction more dialed in over time, we'll see kind of more engaging and dynamic environments that actually respond to what performers are doing and that also what the audience is doing. And I think that is the really exciting kind of like what Nick said is being the tip of the iceberg is that I think we'll see more of that over the course of life cycle. So, Nick, your final thought about the future? Well, will, will our worlds just be completely XR in the future? Is that just where we're going? I think to some extent we will. And I think kind of echoing with a somewhat, Danny said, is, is the technology now is very large, very bulky. It takes a lot of effort, a lot of energy. But just like anything, it's only a matter of time before the power of this technology is things that are completely in our hands and much more mobile, much more interactive. I think on a another thought is that, you know what this technology has taught us in this year and in this season is what types of new permutations are possible in the digital space. I think before this year, when we thought of something such as a webcast, that was, I think, in all of our minds, just like a Zoom screen, as we all just looked at a computer and talked. But now we know that there's such a larger world of possibilities out there that I think that's going to reshape the face of, of every digital event moving forward. So when we go back to doing real events five years from now, the virtual component, the baseline of what that virtual component is, is going to be so much higher, I think, moving forward, because really set that expectation for these new possibilities. You know, I think all three of you guys are simply amazing in your own way and how you approach actor. And as anyone listening to this podcast, to hear your perspectives, the futuristic nature of what's coming and how we prepare for it. And I think all of you have that, that tricked out amazing. You're on a journey and we're all going to have to jump on that journey or be lost as we move into the future. So I want to thank Nick, Val and Danny for joining me today. I hope all of you, I get to work on some type of cool project in some capacity, and if you're doing something on Mars or on top of Everest, I'm coming. I just want to stand on the stage. We can make a short video and then I can tell everybody I was there. Guys, thank you again. And this is Les Goldberg and The Road Ahead.