In this episode of the
The
Paras and Edward talk about how the team has trained the Fall Detection algorithm using extensive datasets of real and simulated falls. Data is safely collected through partnerships with university labs, other external labs, and even professional stunt doubles to validate real-world scenarios.
While discussing crash detection and sharing personal experiences, the team emphasizes that Fall Detection is designed to keep everyone’s safety in mind. Tune in to this episode of the Made by Google Podcast to learn more about how
Transcript
Rachid (00:01): Thank you for coming back to the Made by Google Podcast. I'm your host, Rachid Finch, and today is all about falling happens to everyone. I fell off the stairs actually last December and found out I broke my foot. And still, I gotta say I got lucky because I was able to get up myself. And of course, that's not a given. What if you're alone or cannot get up? Wouldn't it be great if you could get help in such a desperate situation? Well, the good news is that Pixel Watch now can help, thanks to a new feature called Fall Detection. Pixel Watch can detect when you fall and get help, it could actually save lives. So let's talk more about Fall Detection in today's episode of the Made by Google Podcast. Our guests know everything about it. I mean, they made it. So with us are Paras Unadkat from the Fitbit team and Edward Chi from our safety teams, Paras, Edward, welcome to the Made by Google Podcast. Please tell us a little bit about your roles at Google and how you ended up here. Paras maybe we can start with you.
Paras (01:09): Yeah, so I'm a product manager on the Fitbit team. And I was a product lead for the Fall Detection feature. I work on a number of our sensors and algorithms based features, basically using the sensing technology on wearables along with our amazing machine learning capabilities at Google to create great features for our users.
Rachid (01:31): Amazing. And what about you, Edward?
Edward (01:33): Yeah, I'm also a product manager on our safety team on the Android and Pixel safety team. And I've been working on products that aim to help users feel and be safer such as Car Crash Detection and in this case Fall Detection. And when I heard about the feature with Fall Detection Paras and I basically thought it'd be a great way to collaborate given the cross section between the algorithm and the safety themes of the feature.
Rachid (01:56): Today's guests work together to improve your personal safety using the latest in technology. Paras Unadkat joined Google six years ago working on machine learning in the Google Ad team, but then joined Google Earth and now Fitbit, where he works on the cross section between sensors and health, making sure those sensors do helpful things. If Paris's work is making sensors detect false, then it's Edward's work to make useful things happen when a fall is detected. Edward Chi works on our safety teams. You might have seen the personal safety app on your Pixel phone. That's Edward's team. So often the whole is greater than the sum of its parts and the work of Paras and Edward is one of the most powerful examples. Let's hear from them how Fall Detection was developed and how Edward went above and beyond taking one for the team. I hope you'll enjoy our conversation. Let's, let's bring this a little bit into reality. I actually fell off the stairs last December. Now let's say that would've happened somewhere this week. I was wearing my, my Pixel watch and I have Fall Detection enabled. Edward, what happens, you know, when I fall and I'm wearing my Pixel watch?
Edward (03:07): Well, first off, I hope you're okay. I'm glad, I think you're okay and I'm glad to see that.
Rachid (03:10): Yeah, pretty much
Edward (03:11): When Pixel watch first detects a hard fall, it'll wait and see if it detects you moving. Now, if you don't move for about 30 seconds, it'll start to display a fall detected on-screen notification to check in and see if you need help. Now, during this time, it'll also vibrate and sound an alarm, so you know that Pixel watch is detected a potential fall, and you can of course respond accordingly. So in the case that you need help, you can tap, I fell and need help to be connected immediately to emergency services or if you're okay and you don't need help, you can tap I'm okay on your watch face to dismiss that notification. Now, otherwise, if you're unable to respond for about a minute, Pixel watch will automatically attempt to call emergency services and play an automated message that lets them know that the watch detected a possible fall, you didn't respond and also share your location. So emergency services can know where to send help during this time. If you recover or you're able to speak, you can also speak to an emergency operator yourself, and if possible, let them know that you need help.
Rachid (04:07): Amazing. Okay. And we'll get back to that 30 seconds and additional 30 seconds a little bit later. Now, Paras I guess, your job to make sure that the watch detects the fall in the first place. So how does it do that?
Paras (04:21): So the fall detection feature uses an algorithm that works based on the motion sensors on your pixel watch, and it uses machine learning to differentiate between the motion signatures of fall events like you were saying, falling down a flight of stairs compared to things throughout your day-to-day life that might look like falls. Right? So things like exercising, doing burpees, kettlebell swings, and things like that. There's no way to build a perfect system. But by collecting a lot of data and trading our models across all these different data sets of configurations, our model does a pretty good job of differentiating between different types of motion patterns.
Rachid (04:56): I find it interesting because like you mentioned burpees, which give me a sort of headache, I guess for you as an engineer that gives you a completely different kind of headache where you need to be able to distinguish it from an actual fall. So how do you keep the two apart?
Paras (05:10): So it really just comes down to the data sets that we've collected, and by collecting a large dataset of real and simulated falls along with a huge data set of real world activities, like things like burpees, people driving in cars, different types of vehicles, breaking hard all sorts of things like that, we've been able to build a model that does a really good job of being able to differentiate and learn the difference between those different motion censures.
Rachid (05:34): Okay. And we'll get back to, to how we train the model a little bit later, but Edward, we we're just discussing the 30 seconds notification and the 60 seconds where actually emergency services will be called if you're unresponsive after a fall. So how do you decide that it's 30 seconds at first and then an additional minute to have these things kick in?
Edward (05:54): Yeah, and to start, I did want to say, you know, know at any time when a fall is detected, users can immediately contact emergency services by tapping the I need help button on the in-screen notification. And that's really important to us but that being said, you know, we wanted to ensure that users have the opportunity to respond if they are able to, after a fall, for instance, we know sometimes when a user falls, they're able to recover themselves and they don't actually need to call emergency services. So we wanted to give them that opportunity to dismiss that call during that 60 seconds, essentially, if help isn't needed. We also know at the same time, unintentionally calling emergency services can be really stressful experience for both the user and it can be burdensome to emergency call centers. So with the 30 and the 60 seconds, this gives us an opportunity for users to indicate that they're okay and cancel and stop automatic calls while algorithmically at the same time, if we detect their significant movement, we can also stop that automated call from going through. So this is our approach in essence, to balance getting users help quickly while minimizing stress cause by unintentional calls.
Rachid (06:57): And, and how does that go, like during the development of the product? So at some point you're gonna say it's not 25 seconds, not 35, it's gonna be 30. So how does that come about? Yeah,
Paras (07:07): So I think one of the big things that we were trying to do with, with our feature was sort of make sure it's providing as much value as possible, being able to detect when our users actually need help and, you know, when they're just doing burpees or when they're just getting up really quickly after an event happens. So because of that, we kind of built in the stillness threshold where we sort of make sure that, hey, before we place any emergency calls or do any of that, or even kind of just bother the user and say that, Hey, it looks like you fell, we just make sure that they're actually motionless or they're, they're underneath a particular stillness threshold after, after that fall event and the idea is that if the user actually needs help and they're actually unable to make a call for themselves, we'll be there for them in that situation, but we won't spam them with any alerts or anything like that otherwise.
Rachid (07:52): Right. So let's say I'm playing football and I actually mean soccer, so that's football in here in Europe and I'm wearing a Pixel watch and I fall, but I stand up immediately to continue the game. Then I will not get any notification because Pixel watch note, it's like, okay, it seems to be you're fine. You're moving again.
Paras (08:09): We know you're okay. Exactly. We know you're okay. We don't wanna take you outta your game. We don't wanna cause a bunch of chaos. So we just won't trigger at that point.
Rachid (08:16): Perfect. Now for the people who don't know, within Google, we have this tradition of what we call dog food, which is where we test products that are about to be launched, you know, with the whole Google population. So the product teams get as much feedback as we can. Now, I'm just wondering, Paras, did I miss any like dog food invitation for maybe, I don't know, volunteering to fall and train your, your model? I'm just so curious how you gather data about falling likely without, hopefully without hurting people.
Paras (08:45): Yeah. No, you definitely didn't miss any dog food communications about us collecting data from, from Googlers. We wanted to collect, just make sure we collected all of our data in a way that was, that was safe. And nobody got hurt. So no Googlers were harmed in the creation of this product
Rachid (09:06): All right. And what was in the actual way of doing it? How do you gather that data?
Paras (09:10): Yeah, it's a really good question. So, you know, obviously the problem of gathering fall data is, is quite difficult. You can't exactly ask people to, you know, just go and take falls for you. And there are a lot of different things that impact how a fall looks to a set of sensors. Right. So if you ask somebody to go on like a crash pad and fall there. That doesn't look like a real fall because there's a crash pad there, right. The level of impact is significantly attenuated. The thing that causes harm is a thing that you're actually trying to detect. Yeah. which means that it's really difficult to get people to actually do that. So we had a few different ways of going about this. Initially early on in the program, we worked with a number of different university labs and external fall labs to really just understand what a human fall looks like.
Paras (09:58): Collected a ton of data that way. A lot of harness arrested Falls, had people come into the lab, set them up on a bunch of different elaborate rigs to kind of, you know, have them lose balance, see how they react to that, figure out what that, what that all looked like really understand like what the different components were to, to a fall and different types of falls and, and that sort of thing. So we collected a lot of data in that way. We also had other kind of mechanisms for collecting real world and simulated data. And, you know, one of the pro biggest problems that you run into in building a model like this is the problem of data set variability, right? Which is you collect a number of falls, but you can't collect every single type of fall in the world. And there's so many different ways and configurations in which people might fall and might be harmed. So one of the ways that we, we approached this was that we started a partnership with the Google research team to use AI and computer vision to take videos that we'd collected of people falling and use those videos to map them into sort of a simulation space and simulate the physics of different fall types to really augment our data set of real world falls. So we still had that big data set of real falls, but to make it even bigger and just collect this, you know, really massive variety of different falls to really make sure that our algorithm was picking up on all the different varieties across different scenarios we were able to tune things like, you know, different lengths of limbs, different body types, different weights, different parts of the body, different surfaces of people falling on slopes, you know, just different ways of falling to really make sure that we had broad coverage.
Paras (11:30): And then one of the more interesting ways I think that we were able to validate that our algorithm was actually working on real-world data. We worked with a team of stunt doubles for, for a week. Basically just directing this team of stunt doubles to say, Hey, here's like different types of falls that we wanna validate against. And you know, it was like, it was incredible. But I think sort of a combination of all these things help to give us a lot of confidence that our, our algorithm and our feature is gonna be working really well and actually being able to detect these types of events when they happen in the real world.
Rachid (12:02): I think we can hopefully safely add a disclaimer that no people were harmed during the development of this feature. Right? Maybe that's a good thing to say.
Paras (12:10): Done very safely and you know, with all the right kinda safety precautions in place, but really with an eye towards collecting as much real world realistic data on this as possible in, in the safest
Rachid (12:21): Possible way. Excellent. Now, Edward, you also worked with the safety team on something like crash detection in a car. I'm wondering like, you know, from a product perspective, when do you decide that the system is good enough to go to real world people? Is there something you can say about that?
Edward (12:36): Truthfully, the balance is a little bit nuanced and we're always looking at the data in particular. And so a lot of the different criteria that parse across look at for fall detection also applies in similar ways to crash detection. We really look at, okay, what are the different scenarios that we want to capture? You know, we wanna make sure we capture or we optimize for crashes above a certain a certain speed, for example, et cetera. And so when we have our data and we look at those particular scenarios that we're really optimizing for, we wanna make sure both ends, you know, of course an actual crash is detected, and then we also at the same time wanna make sure anything that may look like a crash but isn't a crash is also mitigated. So it's striking a good balance between those two. And that's when we, and when we look at the data, we wanna see that before we actually release to the public.
Rachid (13:22): The public. Amazing. And of course, Fall Detection new on Pixel watch for people who, who owned the watch. I'm just wondering from maybe your personal experience testing it, have you already experienced the benefit of it or have you seen people maybe testing it that already were happy that they had the feature on the device?
Edward (13:40): It's funny that you mentioned it. When Paras said that no one was harmed, you know, I did take some voluntary falls just, just because out of my curiosity and I may have had like a bruise elbow but it was worth it in that sense. I've been fortunate that I haven't had an actual fall, but I'm definitely relieved and I have that extra peace of mind to know that if I did take a hard fall, I trust and believe in our product to, to help me if I need it.
Rachid (14:05): Amazing. Definitely. You took one for the team. So that's absolutely great. Now we close every episode of the Made by Google podcast with a top tip for our listeners. How can they get the most out of fall detection?
Paras (14:22): I think one, one big thing with, with Fall Detection in general is that, you know, a lot of times people have this image of falling as a thing that, you know, is really built for people at risk of falling, right? So you kind of imagine like those old life alert ads like that have fallen and can't get up, type type thing but really falls are a much broader widespread problem. Kind to your point about tripping and falling down a flight of stairs or doing, you know, household chores and you're climbing up on a stool for something and the stool tips over and you fall and hurt yourself. So there are all sorts of situations that you can get yourself in that I think a future like this could be really valuable for. We're, we're very focused on, you know, making sure we kept all that in mind, developing this feature. So it really, I think is a feature that's built for everyone and can really add this element of peace of mind and safety no matter who you are.
Rachid (15:13): Amazing. And indeed, I think especially useful for people who might have had a harder time getting up definitely the case and Edward to use fault detection. Anything you need to be mindful of. Do I need to enable anything on the device or, or in settings? Yeah.
Edward (15:29): Tactically, I'd say for Fall Detection specifically, you wanna make sure you've granted personal safety location permission on your Pixel watch. We can still operate without that, but in the case that you do fall, we wanna make sure we can share your location with emergency services so they know precisely where to send help. So that's kinda my top tip for Fall Detection. And then separately, at the same time, I know we talked about crash detection here. You know, please feel free to check out our other safety features available as well. We have a number of things to, you know, help give you peace of mind from crash detection to emergency location sharing, Emergency SOS. So all to hopefully help everyone be a little safer.
Rachid (16:05): Excellent. Paras and Edward, thank you so much for your time. Hope you stay safe out there while you undoubtedly test new personal safety features for Pixel Watch. Thank you so much for talking to us.
Edward (16:16): Thank you.
Paras (16:17): Thank you.
Rachid (16:19): A big thank you to Paras and Edwards. Check out the personal safety apps on your Pixel phone and Pixel watch. And yes, great advice to grant, a personal safety app, a location permission so that if it has to call Emergency services on your behalf, it can tell them where you are. We have some more great episodes of the Made by Google podcast coming up, so if you don't want to miss those, best to subscribe to our podcast next time we're talking about talking on the phone. Catch you next time. Take care. Talk soon.
Fall Detection is not available in all countries and depends on network connectivity and other factors. Your watch may not be reliable for emergency communications. Fall Detection may not detect all falls. To call emergency services on a Google Pixel Watch without 4G LTE, your paired phone must be nearby. Users must grant location permission for Fall Detection to share location externally. Data rates may apply. See
g.co/pixelwatch/personalsafety for more details.Works with most phones running Android 9.0 or newer and requires a Google Account, Google Pixel Watch app and internet access. Some features require a Fitbit mobile app and/or a paid subscription. Google apps and services are not available in all countries or languages. See
g.co/pixelwatch/specs for technical and device specifications.