NotebookLM elevates your understanding of anything

Transcript - Season 7 Episode 6 - NotebookLM

Steven 00:00:00 One of our slogans is ‘understand anything’ like that's, you know, whatever you give it NotebookLM will help you make sense of that material in whatever modality you want.

Voiceover 00:00:09 Welcome to the Made by Google Podcast, where we meet the people who work on the Google products you love. Here's your host, Rachid Finge.

Rachid 00:00:16 Welcome to another deep dive. That's right, we're talking NotebookLM today, together with Steven Johnson, the editorial Director of Notebook. And Simon Tokumine, Director of Product Management at Google Labs.

Voiceover 00:00:29 This is the Made by Google Podcast,

Rachid 00:00:32 Steven, Simon. Welcome to the Made by Google Podcast. Steven, you studied semiotics at Brown University. You wrote extensively about innovation networks. So how does an author end up at Google pioneering a tool like NotebookLM?

Steven 00:00:46 I was writing about technology. I've always been an enthusiast with technology. I did a couple of tech startups when I was younger, so I was just always very interested in how I could use software to help me think and organize my ideas and to write the books. And so I had, I'd been an early adopter of a lot of different kind of note taking tools and writing tools and I'd written about the way I was using the tools and things like that. And in the early days of Google Labs, just almost exactly three years ago, Clay Bavor, who was running labs back then and Josh Woodward, who now runs labs. They'd been reading my stuff for years, actually. Um, read a couple of my books and they knew about this obsession of mine. And I, I guess at some point Clay said to Josh like, we should get Steven to come do a talk at Labs.

Steven 00:01:27 You know, and his, his whole worldview is kind of an inspiration for what labs should be. And you know, he's interested in software and, and apparently Josh was like, what if we got him to come and actually join Google Wow. And build something? 'cause he is obsessed with, you know, writing and research tools and we've got these new language models that are just so amazing, you know, maybe we can do something. And so they cold called me out of the blue and said, would you have any interest in coming? And we have a small team that can help you prototype some ideas. And because I'm not an idiot, I said, yes, I would like to do that. And I think I met Simon maybe like three, four weeks into my tenure there. I was initially part-time.

Simon 00:02:04 By the way. It's so funny. It's so funny, Steven, by the way, like I think I was just new to Labs and when I joined Labs, you know, to me you'd been in Labs forever, , so, so

Steven 00:02:15 Yeah, I was an old timer.

Simon 00:02:17 This guy's been here forever. Wow. He knows everyone. But it's funny to hear it's only three weeks. That's the first time I've learned that. That's awesome. Yeah.

Steven 00:02:24 Yeah.

Rachid 00:02:25 And then Simon, you joined Google 10 years ago. So you saw that transition from sort of mobile first to AI first and now working on large language models. So what is your driving force and how does that apply to, you know, working on NotebookLM?

Simon 00:02:38 Yeah. Yeah, so I joined Google, like you said, it was 2013, so a bit over 10 years ago now. But I joined just as Android was really getting off the ground, really getting traction. I worked on the Google search app. So you know, my journey there kind of went from standing up in brand new mobile app and having a ton of fun on that to really kind of getting deep into almost like the origins of curiosity. You know, where does the desire to learn something new come from? And so when I moved to labs, Josh and I were all friends from back in my search days. And at the time we were building, you know, there was an idea that we'd build a large language model, API. And then following that, I was like, I talked to Josh a bit and, and was sort of planning my next move.

Simon 00:03:19 You know, I kind of do things in two year increments. And we sort of realized, well, hey, you know, the models look like they're on track, they look good, you know, that we, we sort of know roughly where they're going, but where are all the products? So I started thinking through what are the opportunities for information to come at people that are different from just a simple chat bot? And that led to a ton of investigations and lots of failed prototypes that I won't bore you with here. But one of them that made it through was this notion of what we now call audio overviews. You know, can we make kind of engaging dialogue between multiple AI hosts to really guide you the listener through an information space. And so the idea was that one of the hosts is almost you, you know, so they're the one that are asking the questions, like automatically prompting the model in a sense. And the expert is guiding you through the information.

Steven 00:04:10 I'm just concerned about this news that Simon works only for two years on a project. So I just wanna know, when do you consider your clock starting on notebook? Because I need to start planning if you're moving on man.

Simon 00:04:24 Maximum. I've been on it for a year. I think it's probably fair to say at this point. Okay,

Steven 00:04:28 Good to know. I'm gonna add to this in my calendar right now.

Rachid 00:04:31 Still some runway left, fortunately. And we'll talk about audio overviews in a minute for sure. But let's go back to NotebookLM and its essence itself because everyone that I know who uses it has a slightly different description of what the product is. So just wondering, Stephen Simon, if someone asks you what is NotebookLM, what is your sort of one or two sentence answer? Yeah,

Steven 00:04:52 I mean I think one of the really nice things is that we had just hit on when we started really talking in earnest about integrating audio overviews and, and working really with Simon and that team is, we'd hit on this framing of Notebook as a tool for understanding things like that was kind of clarifying because it was big enough to describe the ambition we had for this project. I mean, I think in the early days there sometimes people would be like, is this for students? Is this for journalists? Is this for, you know, they, people wanted it to have this very narrow focus. And I think maybe naively, because I hadn't been at Google for very long, I was like, I think it's bigger than those things. Like I think we should not narrow the focus on it. We should be ambitious with this project.

Steven 00:05:35 And then eventually we hit on this idea that like, look, there's so many different user journeys that involve trying to understand complex material in whatever form it is. So if you're a writer and you're trying to understand the research material for the book you're writing, sure, that's great. If you're a student trying to understand the material for a course you're taking, if you're a marketer trying to digest information about the business that you're promoting to turn it into a press release or an ad campaign, that's another kind of understanding. When you looked at the market, there was a big opening there. We felt like there were a lot of tools that helped you kinda specifically create in specific formats. Like you need to make a slide deck. Well, we have a tool for that, but a tool that was really oriented from the ground up around, you know, finding insights and going deeper and exploring and making new connections in material like that seemed like a kind of a blank spot on the map that was nonetheless really important for people. And that language models we could sense were gonna be very, very good as guides for that journey. When we first started thinking about this, the models weren't quite up to speed in a way. I mean there's a great moment when we rolled out Gemini 1.5, which was right before we announced audio overviews. And I remember seeing Simon, I don't know if you remember this in the micro kitchen and you came up to me and you said Notebook finally works

Rachid 00:06:47 .

Steven 00:06:50 It was just like we'd been kind of building ahead of the models a little bit. Like we've been building the application around the idea that the models were gonna catch up in their ability to analyze and interpret and make sense of things for the user. So that's, you know, one of our slogans is understand anything like that's, you know, whatever you give it notebook will help you make sense of that material in whatever modality you want.

Rachid 00:07:11 Simon, do you sort of remember the first thing that you asked within NotebookLM, where you got an answer that was like, yes it works as Steven just described. Do you remember what kind of things you were trying out? I

Simon 00:07:22 Do remember the first time I used it, it was when Steven showed me the very early prototype that we had that didn't look anything like the notebook of today, when it was, it was much more of a, for some reason, Steven, it was more of a writer's tool. Yeah. And I'm not entirely sure why , why, yeah. I wonder why that works.

Rachid 00:07:39 Suspicious. Suspicious. Yeah.

Simon 00:07:40 No, but in addition to what Steven just said, I think the one thing that really sets Notebook apart is its ability to work over material that you provide it. So for example, like as you can tell I'm not a US citizen, but things relating to the US tax code for example, that are important for me to get exactly right or that are related to the fact that I am both, you know, I have to think about the US and the UK at the same time and that type of thing. So I think anytime you have something that needs to be grounded on information that's maybe not necessarily broadly available or anytime you're working with information and knowledge in a way where you need to accumulate more of that information over time, notebook is just a perfect fit.

Steven 00:08:19 I had a really interesting one, I haven't even told you this Simon, because it happened basically last night, but I had a just a wonderful notebook moment. So there's a very talented screenwriter who's trying to adapt one of my books to turn it into a TV show. And he had sent me the script for the first episode and so I was giving him kind of notes on the script and I had this idea for an opening scene for the kind of next episode. And I just wanted to kind of like give him a quick sense of what I didn't wanna do his work for him 'cause I'm not a screenwriter, I don't know how to do that stuff, but I wanted to give him a sense of what it could be. And so I just loaded into Notebook, like his draft of episode one, the entire text of my book, and then I just said, Hey notebook, like write the opening scene of the next episode in the style of that episode that I've just given you based on the facts in my book and here's a, you know, paragraph long description of what I want in this scene.

Steven 00:09:09 And it was like, Oop, here you go. And it was, it was a little bit off. So I was like, that's good, but change the ending a little bit. Whoop, here you go. And I just pasted that in an email and I said, Hey, I'm not trying to do your job for you. I wrote this with an AI, but this gives you a sense of what I'm thinking here and the speed with which I was able to get to that vision and convey it to someone else versus trying to write out that scene myself. I don't know, it was, it was 10 times faster and I got to basically like as good a result as I think I would've gotten if I'd done it myself.

Rachid 00:09:40 Right? So it's getting more insights, learn faster based on your own sources. I remember when NotebookLM got sort of announced when it was still called Project Tailwind at IO in 2023. I think that was sort of what people thought, like, oh, it's like Gemini, but it's only for my own documents. I think that's sort of the idea that that caught fire. So Project Tailwind, why was it called that? Who came up with that? What does it even mean?

Steven 00:10:04 That was Adam, the engineer I mentioned. We had an early kind of like, what are we gonna call this? And Adam, it was exactly as what someone was saying that it had a little bit more of a writing component, which maybe someday we will get back to, but whatever we'll see about that. And the idea was that you would use it and feel as if there was kind of like the wind was at your back and you know, the software was just helping you write and think in a very fluid way without having to kind of fight against it. And it was a great code name for it.

Rachid 00:10:29 Amazing. And that started in 2022, right? Like sort of only one year before it got some sort of public appearance?

Steven 00:10:36 Yeah, we built the original prototype we started sharing with folks in October of 2022.

Rachid 00:10:44 I guess with any product you could build, but especially NotebookLM, you sort of have, and Steven you alluded to it as a sort of an audience in mind. You probably have some use cases in mind, but then of course you put it out in the real world and people do completely different things with them. Do you both have some sort of a favorite example of what, what you've seen people do with NotebookLM where you thought like, gosh, I never considered that could be useful. Our product could be useful in that context.

Steven 00:11:09 I mean we could do a whole podcast on

Steven 00:11:12 On unexpected uses. I'll give one Simon, I'm curious what yours would be. I mean, I think probably my favorite is the role playing game enthusiasts like Dungeons and Dragons. So it turns out that like all those role playing games are very like textual, right? They, you know, you can write a campaign for your D and D game that's, you know, 200 pages long and it has this complex backstory and so on. And so people early on were like, oh, this is great. I can manage my campaigns and I can actually be the dungeon master for my game with like notebook open next to me. And if I'm trying to remember like what was the palace, where the orc was found and how many hit points did they have or whatever is happening in the fantasy role playing game, they're able to just conjure up that information.

Steven 00:11:56 And in fact, there was a group of folks in our Discord, which we have a wonderful Discord community that were actually trying to hack notebook and to turn it actually into a game platform so that you could actually play the games. And Simon and I have gone down a couple of like rabbit holes over the last year and try to figure out like, is there a playable version of NotebookLM that someday may happen? You can kind of make it work already now as it is. But yeah, that was, that one was really fun to see.

Rachid 00:12:23 Amazing. Simon, what is your favorite example?

Simon 00:12:26 I mean, you know, a lot of mine are from the very latest things that we've shipped, but you know, we, we've just shipped, um, international audio overviews. Yes. So audio overviews in over 75 languages. So one of the best ones that I've seen recently is a user in Japan actually uploaded, they're a knowledge worker. They were interested in finance, but they were also learning English. And so they loaded into notebook, a English language blog post maybe about the financial report from company X, Y, Z, whatever that happened to be. And they asked the Japanese audio overview hosts to host a language lesson using key phrases from the English language, the financial report, so that they could basically bone up on the financial English financial language. And so when you listen to when they play, you know, they shared it on Twitter and all this kind of stuff or on X rather.

Simon 00:13:19 And, and it was this like amazing Japanese language audio overview where these hosts were actually kind of like debating, you know, these very, very, very, very specific and esoteric financial terms, you know, and kind of, and having a very engaged conversation about it. And I just sat there thinking, you know what, like you're never gonna go to, uh, this is never, this lesson is never gonna happen. This lesson would never have happened. Language lessons, if you've ever done them, and I'm sure you have, you know, they're, they're pretty dry, right? And they don't connect with you and your own interests. And this user was able to basically craft their own almost like professional grade language lesson based on a blog post that they'd found within their, you know, their their specialist domain. I just thought that was amazing. And we certainly didn't plan for that.

Simon 00:14:06 So a lot of people dunno this, but, but if you load sources into notebook in other languages to the notebook that, you know, you are, your system language notebook will just Gemini under the hood, it just sees all languages as one language. You can have sources from France, from Germany in English, uh, in Japanese, Chinese, whatever, and you can be conversing or you can be generating mind maps or whatever in your language. And that's the thing that, I mean, Steven, I dunno, we don't, I always feel like we don't make enough of a big deal about this thing, but whenever people find it that are always like, this is insane. This is, this blows my mind. And that was a really good example where somebody had found it and connected it to the audio overview feature to get something pretty magical.

Rachid 00:14:46 Well, I'm tempted to say welcome to another deep dive, but this time about audio overviews themselves because I guess that's the one that made notebook limb go viral across many, many audiences, right? People hearing them first, not even understanding what's so remarkable about it until they learned that those weren't actual people speaking. So, you know, computers have been able to generate voices for decades actually, but this hits really different. So what is sort of, well there's not one magic sauce I guess, but what are the ingredients to, to create something like audio overviews that's so compelling and real and, and speaks to people in a very human way?

Simon 00:15:23 I actually think the first thing is a mindset Google has, and most companies have created audio products that are very crisp and very efficient and they're, they're very exacting. And I've never been particularly happy with them. You know, I just don't really like them personally. Like personally they don't do it for me. You know, what we really wanted to do is we really wanted to, to make something that just felt very natural and very approachable. I dunno if you've ever, you know, interacted with any of the voice assistants, but I've always felt very kind of tense using them. Like I'm sort of, they're very needy, you know, and you've got be feeding them with quite precise information and the information they give back, it feels quite on brand, let's put it that way, you know, whereas, you know, we wanted to explore, you know, I think, I think a, I think a lot of the, the kinda the insights with audio overviews is really this notion that people learn best when they're feeling comfortable and relaxed and when they can actually connect in a more emotional way, rather than in a purely, you know, kinda logical way to the information.

Simon 00:16:21 So that was one thing. And, then the second thing that I think makes them really magical is, is an observation that especially if you're trying to understand something, having information just come at you like a fire hose, you know, if you've ever used, I don't know, text to speech to read a document or something like that, you'll find that the text to speech kind of runs ahead of how quickly you can process the information, or at least it does For me. One of the nice things about the way audio reviews comes together is it paces the information. There are these natural breaks in, in the information as it comes at you. And that I think gives you, and you know, folks like me who can't quite keep up with the TTS machines, it gives us chance to just catch up with the information itself and, and kind of stay with it. And so, you know, while it may appear superficially inefficient to introduce ums and ahs and some of the more natural parts of how we all talk, and, and that's probably a reason that this is probably one of the reasons why we talk like we do.

Rachid 00:17:14 You're saying you made it better by making it less perfect in a way.

Steven 00:17:18 Yeah. It's one of the things that is so magical about the way that they were designed is if you listen to a script of two people talking to each other without those exchanges and ums and talking each

Rachid 00:17:29 Other, call 'em each other disfluencies, I think, right?

Steven 00:17:31 Disfluencies, you know, you can kind of tolerate it a little bit in a single assistant talking to you. Although I agree exactly with Simon that there's a limit to that. But if you listen to two people talking that way to each other, like nobody wants to listen to two robots talk to each other. That's not interesting . So, so you need, you need the disfluencies and, and every now and then we'll hear people say like, I love it, but can you turn that stuff down? I don't need all the ums. And I was like, you do, I'm telling you, if you tried to listen to it, the human ear will reject it. Like it's just not, it's not right. The other thing that I would add to that is, I mean there's just a, there's a beautiful prompt behind the scenes there that elicits this stuff almost on the content side and less on the audio side, which is before audio overviews came out.

Steven 00:18:13 I would often tell people when I would do demos, I would say, listen, one of the amazing things you can do with these models now is you can ask for the most interesting things in this material. Like ask the model to go through all the documents and pull out what is the most surprising or interesting. And that was like a search query that you just could not do before. Like you couldn't just say like, find the most interesting things. It would be like, I'm looking for the keyword interesting. And, you know, that just wouldn't work. But because these models have a much more nuanced kind of sense of that, it used to work in text chat before we brought in audio overviews. But I think what audio overviews really did was bring that experience to a much larger audience because it was so viral as a phenomenon. And so many people, you know, heard these virtual hosts go through their documents and pull out the interesting bits, build a show around the most interesting things. And Simon had this slogan also, uh, in addition to understanding anything it was make anything interesting.

Rachid 00:19:08 There's been some examples of sort of mundane things that were actually pretty exciting when they were an audio overview, right?

Steven 00:19:14 Well we did Hard Fork, uh, Kevin Roose put his, uh, his credit card statement right into it. And it was like, wow, uh, Kevin is taking a lot of Ubers, he really should be using more public transportation in San Francisco. I mean, there was perfectly good Muni and Bart, I don't understand what his problem is. ,

Rachid 00:19:32 You're right, these are also pretty funny scripts. I guess they have a lot of real world analogies. How do you sort of create that out of the content that is in the user's notebook?

Simon 00:19:43 Yeah, the first thing to say is that it's not only the content in the user's notebook, you know, one of the insights is really that the hosts themselves, they create the experience around the information. And so every host has got a very rich and detailed persona. They're actually role playing almost as real people. And we found that unless we did that, unless we gave them a very realistic persona, they would fall back on just sounding like the average model. You know. And so we, we want, we kind of, we intentionally break them out of post trained persona, you know, the kind of the, the Gemini persona and give them these very specific complimentary personas that make for an interesting to and fro of information. So that's kind of one component that's almost like the negative space that people don't see. And it's probably the case in podcasts overall, right?

Simon 00:20:32 Actually, like, you know, we should, you, yourself or a personality and, and you are giving this podcast its own specific flavor. And so we sort of recognized that was a really key component of it. And then the second thing is, is probably it's close to what Steven was saying, you know, the process of writing a really interesting show or a really interesting script, it's not, you don't just kind of come up with it in one go. You have to read this, that then you have to make a plan. Then you're like, well, is it all good? You critique it, you know, you're like, well what's, what can I improve about it? You know, you get feedback on it and then you maybe go back and, and address that feedback in another way. So, you know, when they take five minutes to generate it, that's not five minutes spent on the audio synthesis. That's actually five minutes spent on the agents working together to create the script itself. When you push go and you generate one of these things, it's not that you're waiting for it to get queued up and executed. We get to work straight away and it's, it's literally Gemini working for five minutes, turning this thing over, trying to find the right interesting dimensions to really engage you.

Rachid 00:21:38 So there's basically almost like editors, writers sort of conjured up to, to make that script,

Steven 00:21:42 Rachid, that's exactly right. And, and it's, it's one of the ways where there was just such a beautiful kind of continuity for the team when the Audio overviews team came in because they were, they were so literary in a way, in what they were doing. It was really an edit cycle that they were had developed for this thing, which is like, come up with a vision, modify the vision, edit the vision, all that stuff that you learn like as a journalist to do or as a writer to do. You know, we were doing on the level of prompts, basically, you know, there's an amazing technical achievement that is happening with audio overviews and the quality and the conversational model and what's actually happening with the audio. But there's a kind of classic almost journalistic edit cycle, uh, innovation that's happening there as well. And that, you know, the intersection of those two things. That's the sweet spot of NotebookLM.

Rachid 00:22:32 So let's talk quickly about two other features that were recently added that, uh, were appreciated by a lot of people: Mind maps and the ability to discover new sources where Notebook now goes on the hunt for you to find sources. Where did that idea originate?

Steven 00:22:47 I mean, mind Maps is one of my favorite stories about the team. I mean, it, it was a weekend project of a new engineer Chang, who just joined our team who was a big tool for Thought Maven, really interested in software that augments your ability to remember and organize. And he had this idea of like, maybe we could create this visual representation of all the concepts in your sources and you could then kind of explore through this kind of map of the ideas, uh, rather than just listening to an audio overview or, uh, having a chat conversation. So the funny, I mean, this is an amazing thing. You built this prototype like over the weekend, came in and showed it to everybody. And just to show that like, while I am one of the original members of the team, I am often wrong about things .

Steven 00:23:34 I had always, I'd followed mind maps over the years because I'm also interested in that same space. And I always thought they were kind of limited because you were just seeing this visualization of the terms, but there wasn't anything to do with it. So you're like, okay, that's a nice map, but like, how do I really engage with this material? And when I first saw it, I was like, oh, that's nice, but that's not gonna be a really mainstream use. I mean, we should do it. And I'm supportive of, you know, this wonderful new colleague who's gone outta the way to do this passion project. But then I realized that the way he designed it was that each of those little nodes on the map you can click on, right? And it takes you directly back to the chat and you get a detailed conversation about that particular topic with citations and all the notebook details and stuff like that.

Steven 00:24:17 And I'd never seen a mind map that was capable of doing that before. 'cause it was kind of technically impossible to do before you had grounded language models. And once I saw that, I was like, oh, this is genius. Like this is fantastic. Well, you have to do it. And it's become a big hit. I mean, it really, people have absolutely embraced it. It's again, it has that same magical feeling of like, I uploaded my journals and I see this completely different representation of that information and it gives you just a new way in a new way to explore. So we've been really happy with that one.

Simon 00:24:48 Yeah, that's, that's true. And I think Steven, me and you were the big cheerleaders for it, right? I think as I seem to remember for Mind Maps, that was such a good feature. I mean, Discover Sources was, you know, similar in, in terms of it being a single engineer that really kind of made it happen. But it came from the insight of, hey, notebook's great if you come at it with the sources ready to hand. But if you come at Notebook with a desire to understand something, but you don't have the sources yet, what do you do? You know, it's actually really hard to find them. And so Discover Sources was our, is our V1, and actually, you know, really interesting roadmap in ways that Notebook can, you know, also help you when you're in that, in that state as well, actually discover really high quality sources from across the, the broader internet, I would say. And, you know, the digital world to base your session off of.

Rachid 00:25:35 So a longtime listeners of the Made by Google Podcast know that we have some sort of a closing tradition where we ask our guests to give the listeners a top tip while they tried a product that you've been working on. So what is the one thing you feel people, whether they're new to Notebook or maybe advanced users, what is one thing they maybe sort of underappreciate that they really should try and make notebook, limb even more useful to them?

Steven 00:25:59 I would give a tip to new users, actually. Mm-hmm . Um, which is when you're trying to figure out what, what this is capable of, one of the quickest ways to do that is to just open up notebook, go to add sources, just go add docs or slides from your Google Drive and don't even think about organizing it. Just grab the last 15 docs that you have and pull it into the notebook and then just start having conversations, generate an audio overview, generate a mind map, and you will see that just by grabbing the last 15 things that you've touched, the AI inside of notebooks suddenly has this very nuanced understanding of where your head has been for like the last two weeks and whatever projects you've been working on. And then you can craft your own notebooks on specific projects and cultivate that. But it's a really amazing way to just get a sense of what it's capable of.

Simon 00:26:48 Yeah, and I suppose I'll do the other side of that, if that's okay, Steven. So the other way that you can do this is pick any topic in the news, anything, any, any event that's happening around the world. Doesn't matter what it is, doesn't matter what country it is, what language it is, go and hit discover sources, type in whatever it is that's on your mind, don't even bother checking you know, where they're from, just yolo all of the sources in and do it for a couple of them. So you've got like 40 things in there. And then generate a mind map, especially if it's like a sporting event, like I did it for the Formula One recently. You'll just get this huge view of everything that's going on and it's, it really starts to show you the power of the tool. So that would be my, my kind of alternative tip to Stevens.

Rachid 00:27:31 Amazing. Those are two great tips for, for users starting out, or even people who have been using Notebook for a longer time. Great way to export a product even further. Steven, Simon, thank you so much for joining the Made by Google podcast. I can see you're having a lot of fun, Simon. I hope you're gonna stay a lot longer than just a year . Um, and, uh, and wish you all the best with the upcoming innovations that you'll undoubtedly bring to amp. Thank you so much. Cool.

Simon 00:27:55 Pleasure. Thank you. See you.

Voiceover 00:27:57 Thank you for listening to The Made by Google podcast. Don't miss out on new episodes. Subscribe now wherever you get your podcasts to be the first to listen.