Transcript

Rachid Finge 00:00:04 Today we're talking to Lu Silverstein, a senior product manager who works on Pixel Audio, and Lillian Chen, a product manager for the Google Photos experience on Pixel.

Rachid Finge 00:00:20 Lu, Lillian, please tell me about the first time you used Audio Magic Eraser and, and what you used it for. Lu, maybe we can start with you.

Lu Silverstein 00:00:37 Yeah, sure. Thanks. I think, I think the first time, like I remember using it was, we were messing around with some of the videos and some of the research work that has been done at Google on removing certain sounds or at least reducing them. And it was on a video that I had of my kids. They were on the beach and it was really windy, and there was kind of, the wind was moving back and forth and you could kind of hear them chattering a little bit, the kids, and it, it reduced the winds so significantly that you could actually hear a little bit of their chatter and, and what they were saying. And so it was a pretty cool moment. It really helped you focus on what was going on in the video as opposed to, you know, if you ever got that wind noise on your mic kind of thing, that's annoying. It was nice to get rid of it. So it was, it was kind of like a little delight moment.

Rachid Finge 00:01:27 Amazing. How long ago was it that you realized that something like this could actually be made into a product?

Lu Silverstein 00:01:35 To be honest, when it started, we see a lot of, I should say, we hear a lot of demos and demonstrations from research. And so it's not always clear actually when something could be taken into a product. But when you start, playing with a little bit more in thinking about where it could be applied, then it becomes real. We've been looking at, we know in general, right, that noise in any sort of audio or video is not a good thing and people like to get rid of those distracting noises. So it seemed like a good match between what the needs of our users were and the technology and development that we saw within the research team.

So I'll ask you the same Lillian. Tell me about the first time that you used Audio Magic Eraser and, and sort of where were you when that happened?

Lillian Chen 00:04:15 Uh, yeah. So I've been working for a few months and we have got some initial results ready for testing. So I was really excited to try it out. I happened to be in New York City for work and I passed by this super busy restaurant that my coworker had highly recommended to me, but it was really loud, just like lots of tourists talking, the kitchen is going. Um, so I just took a little video saying I was in front of her favorite restaurant. And then when I tried Audio Magic Eraser, it worked really well in being able to make my voice message clearer and reduce the sounds of the busy restaurant to just sort of ambiance noise.

Rachid Finge 00:04:49 So Lillian, you're a product manager on Google Photos, but specifically for Pixel, and Lu, you make audio work on Pixel. In fact, we had you on the podcast before when we were talking about Spatial Audio in Pixel Buds Pro. It seems to me that Audio Magic Eraser can only exist because your teams came together. So what is sort of that story of matching up on working on this?

Lu Silverstein 00:05:12 Yeah, that is one of the things that's actually really fun at Google is that we have a lot of different teams that we can collaborate with. And the way that these come together, this particular project is sort of prototypical in some ways, is that we, we search around or shop around within the research teams for different technologies that they're they're developing, A lot of 'em obviously today have to do with AI or ML machine learning mm-hmm. . And so we started with watching some of the demos that the research team had done, and then the product team got interested because it matched with some of the needs that we knew from users. And we put together some demos that we then collaborated with other teams to figure out, well, where should we actually take this to market? It could go in different applications and different experiences.

So, Lillian, I guess maybe Lu's team came up with, Hey, we have this technology now that could erase audio. How did you sort of feel that that could benefit Google Photos users?

Lillian Chen 00:07:09 Something that we were really wanting to do was make 2023 the year of video across Pixel. And that's a strategy that we cared about both on Pixel camera, but even in the Google photo space. So I was keen to look for those types of innovations we could bring to video. So when I first heard about itI thought it was really interesting. It also relates a lot to what we do for users, which is we take things that, how they remembered them and make them feel, feel true and authentic that moment. So, you never remember that the wind was really loud when you're at the beach, right? That's not part of what you intend to capture in a video. So I thought that this really resonated well with the types of editing and sort of corrective applications of AI that we offer in Google Photos.

Lillian Chen 00:07:57 And one thing about this feature in particular is we really needed to hear from the voice of our user. 'cause sometimes you shop an idea around, and depending on who the person is, they might say, I can't think of a video where I needed to edit my audio. And then we have to remind them, look, let's actually see what users think. And so, Lu and I worked together with different members of our Google Photos user research team, and we put together kind of a little demonstration of what this feature might be like and included it in a round of concept testing. And that's when we really heard from users that this was a really relatable use case where a video would've been perfect except for somebody talking over it or someone in the background, or a weird noise went by. And that's why when we really knew we had something special.

Rachid Finge 00:08:39 So let's go back to that Chelsea market example. Let's say I have a Pixel 8 or Pixel 8 Pro. What would I do with the recorded video to actually get that audio fixed? What do I gotta do in the app?

Lillian Chen 00:08:52 Yeah, so it's pretty simple. First you go to Google Photos, and you tap on the video that you want to edit, make your way into the editing menu. It's gonna be under the audio tab called Audio Eraser. And when the tool loads, what we wanted to do was make something really easy available for the user right away. So we have a button labeled Auto, and so I would start there, just tap Auto and see if it creates a better audio experience for a video. And then if we don't get it quite right, or you're someone who likes to kind of have more control, get into the weeds, you're a perfectionist, you can go and tap specific buttons, like in this case it could be the crowd noise button and slide it even further down to reduce that background noise even further.

Rachid Finge 00:09:38 Amazing. Well, I love live demo, so I was like, you know, since this is an audio platform in a podcast, we should maybe give that a little try. So, I was looking through my Google Photos library and I found this video from October 4th, 2016. Now, real Pixel Superfans will recognize that date as the launch of the very first Pixel phone. And I was at the launch in London, where none other than Craig David was performing. And, it was an acoustic performance. And I was filming that with not a Pixel phone, but a Nexus phone. I still had to get my hands on a Pixel phone, recorded that from the audio, uh, audience. And I thought maybe we can play that and see how much of the noise, we can maybe clear in that concert, with the Audio Magic Eraser. So this is the original and I'll just at some point slide down the noise, put the noise back in, and we will make it silent again. I thought that made my video a lot better actually hearing it like that and, actually being able to make out a song like that. And of course, it's annoying for many people that maybe it's not completely silent during a concert. So, Lillian just explained how you use this as a user, but Lu, how do you, for example, use AI to distinguish between the wanted and unwanted sound? So what's happening on my phone in the background is actually what I'm asking.

Lu Silverstein 00:11:40 Yeah. There's actually a lot going on in the background. First of all, everything that you process there that we do is done on the phone. Mm-hmm. , it's not sent to, to the cloud or anything. And what we've done there is we have a model or an algorithm, right, that runs when the audio you have on that video that you took, comes in. What we do is, first thing is we identify the sounds that are in it and we can identify up to 527 different sounds. We then separate them all out and determine a ranking score of which ones are the most prominent mm-hmm. and we've also trained them. So we actually have a lot of videos that we run through our training process to, and then determine, we have, you know, people actually tagging and saying like, this is this type of noise and this is, you know, could be wind or traffic or in the case of your video, it was a crowd Yep.

Lu Silverstein 00:12:34 Was the noise that was in the background. But we also identified music and speech. And then we allow you to separately control those by separating those sounds out and then giving you the controls from the most prominent noises that are distracting that you probably want to reduce. In that case, it was the crowd. You could, I'm guessing what you did just 'cause we can't see is you used auto, but you could also go and further reduce the crowd if you wanted to by going into the manual editing.

Rachid Finge 00:14:25 So Lillian, you mentioned Pixel 8, 2023, the year of honing in on video. It sort of sounds counterintuitive to then come up with a video feature that's focused on audio. So, how did that happen and why was that important?

Lillian Chen 00:14:41 I think for audio. I think it's one of those things that you forget how important it is as a part of video until you really hear about people's experiences where there's a perfect video of your kid's first birthday, but then, you know, the sound just isn't quite right or you're, what often happens too is, someone's talking in it or you're really close to the mic so the balance isn't right. So if you could balance the music in the video differently from the speakers in the video, that helps make that recreate that perfect experience in memory. But I guess my answer is just that I do think audio is a big part of video. Lu knows that well since he's worked on audio, for a lot of his career, but it's one of those things that people almost forget until they hear it and then they realize how valuable it is.

Rachid Finge 00:15:42 Now we love to offer the listeners of the Made by Google Podcast a top tip with the feature you worked on. So, I'm wondering what would you as a top tip give our users when it comes to also capturing great audio on the video that they're recording?

Lu Silverstein 00:16:02 Yeah, I can give a tip. There are two tips that I would give with Audio Magic Eraser. One of the tricks you can do is when we recognize different sounds, if you're taking a video of somebody speaking if you want to hear them better, one trick is, you can reduce the other sounds and leave the speech alone. So it highlights the speech a bit better. That's not perfectly intuitive because if you get three buttons, you know, one of 'em says speech and, and you can reduce the speech, your natural instinct is probably to reduce it. But if you just leave that alone and reduce the other ones, it actually highlights the speech. And then I think the other one is one of the goals of having this feature is that when somebody's recording a video, they don't have to move now to a different location that has less distracting noises. And I think that's a really big advantage and tip that people have naturallylearned unfortunately because the technology is not there. They've learned to move location when they're shooting a video to get to an acquired spot, but now you can actually do that and that, then you can edit it later, which is a nice benefit.

Rachid Finge 00:17:09 Lillian you mentioned the Chelsea market example. I'm sure there are probably more places where you've experienced that Audio Magic Eraser is really an awesome tool to have on your Pixel 8 or Pixel 8 Pro. Any examples of that to add?

Lillian Chen 00:17:23 So one example was I was filming my friend's traditional Nigerian wedding and there was just music cheering, clapping, people talking, there's an MC on a mic. It was just lots of things happening, which is really fun. But when you listen to it later, there might be certain parts that you want to kind of balance out to sound more prominent than others. In that case, I felt like the music and the MC, having that come through more cleanly was more ideal, and so I was able to use Audio Magic Eraser to help with that video.

Rachid Finge 00:18:21 Amazing. Well, if you as a listener want to try Audio Magic Eraser, you'll find it in the Google Photos app on your Pixel 8 and Pixel 8 Pro. And may all your videos sound just as nice as they look. Lu, Lillian, thank you so much for joining the Made by Google podcast and I hope to have you back some other time.

Lu Silverstein 00:18:40 Thanks for having us, Rashid.

Lillian Chen 00:18:42 It's great talking to you.