Android 17 & “Personal Intelligence” Should Scare You

Last week Google announced Android 17 at their “The Android Show”, proclaiming it the “biggest Android update ever”, mostly thanks to “Gemini Personal Intelligence” being intertwined so heavily throughout the operating system. This should horrify you. 

The vice president for Google Labs, Gemini app and AI Studio, Josh Woodward, describes Gemini Personal intelligence as something that can “save[d] me from a headache at the tire shop”, and as he tells it (and as Google’s AI voice speaks it), “Since connecting my apps through Personal Intelligence, my daily life has gotten easier. For example, we needed new tires for our 2019 Honda minivan two weeks ago. Standing in line at the shop, I realized I didn’t know the tire size. I asked Gemini. These days any chatbot can find these tire specs, but Gemini went further. It suggested different options: one for daily driving and another for all-weather conditions, referencing our family road trips to Oklahoma found in Google Photos. It then neatly pulled ratings and prices for each. As I got to the counter, I needed our license plate. Instead of searching for it or losing my spot in line to walk back to the parking lot, I asked Gemini. It pulled the seven-digit number from a picture in Photos and also helped me identify the van’s specific trim by searching Gmail. Just like that, we were set.” That’s quite the story! I find it a bit of an infantilizing story, as it’s clear that Josh is trading his own planning ability for reliance on an LLM searching all his pictures just to find his licence plate and car make and model – like imagine you know you need new tyres. Like you have the presence of mind to check your tyre tread and think, ‘huh, that’s low, let me go get new ones’, but you don’t bother to take a picture of the side of the tyre that tells you what tyres you need, and your plate number, so when you’re literally in the line at tyre shop you have to ask the Google AI for the most basic information? Doesn’t that seem ridiculous to you? But we’ll get back to turning your brain off later, for now let’s talk about the implications of this story. 

“Since connecting my apps through Personal Intelligence” is a wild sentence. What that means is, “I gave Google’s LLM free reign to access my pictures, emails, calendar and YouTube history”. This means Gemini, in the background, can and will trawl through all your emails, pictures and calendar appointments to ‘find relevant data’. I don’t know about you, but letting the same bots that are known for deleting entire production databases because they are black boxes that do whatever the hell they want loose on your emails and photos feels… diabolical. This is also Google nakedly demonstrating just how much information they have of yours. Their chat bot can search all your images to find your car’s licence plate, make, model and even trim level, it knows you take recurring family trips to Oklahoma! Josh, if anyone else had this level of information on you, you’d call them a stalker! And yet, this is information many of us give willingly. By ‘backing up’ your photos to Google Drive, you are GIVING Google those photos do with as they please. Same with your emails. These features are Google flaunting their incursion into your life, and that alone is terrifying. 

Then there’s the meat of this story, in the lines, “These days any chatbot can find these tire specs, but Gemini went further. It suggested different options: one for daily driving and another for all-weather conditions, referencing our family road trips to Oklahoma found in Google Photos. It then neatly pulled ratings and prices for each.” It suggested different options. That is a very key message. IT suggested. Not a comparison website (which has to disclose sponsorship arrangements), not an impartial reviews website where you can make your own mind up about what factors matter the most to you, Gemini. This system that they are so explicitly designing to make you implicitly trust is the one whispering in your ear telling you what to buy. How long will it be before those ‘suggestions’ are bought and paid for? Maybe they won’t be direct adverts, maybe they just pay to only suggest theirs and a sub-brand they own. But if you implicitly trust Gemini and have already offset your critical thinking skills in favour of ‘do as the AI says’, well that makes you a ripe mark. An advertiser’s wet dream, built by the largest advertising agency in the world. Remember, if the product is free to use, you are the product. 

A little further down the press release details their approach to privacy. “We built Personal Intelligence with privacy at the center.” Hahahahahahahahahahaahahahahahahahahaha! Oh man, that’s a great joke. First, “Gemini will try to reference or explain the information it used from your connected sources so you can verify it. If it doesn’t, you can ask it for more information. And if a response feels off, just correct it on the spot (“Remember, I prefer window seats”)” is a wild thing to say, as even Google doesn’t hide that this system WILL hallucinate and need correcting, but more so the later statement, “We train on limited info, like specific prompts in Gemini and the model’s responses, to improve functionality over time.” means that they are using you as the beta tester to improve their product, and are actively training their product on your data. They say that Gemini isn’t trained on your Gmail inbox or photos, but if it was used in a prompt, that becomes fair game, no? 

This is even more terrifying through another of their examples, where Dietrich shows us that Gemini Intelligence can scour all the pictures ‘on his phone’ and auto-fill his passport information to this airline booking form. He says specifically, “My phone can securely pull that information right into the app to fill out the whole form with a single tap”, which implies this is entirely local processing with none of this being stored, but, one, I don’t trust that without independent verification, two, that is insanely wasteful – having an LLM scan all your photos every time you need to fill in some information is either drinking water in a datacentre, or decimating your phone’s battery life – and three, even if this is local now, this is something they absolutely could decide is information that is better stored on their servers. Having Google know everything about you, including key identifying pieces of information, means when ICE comes knocking, Google can just hand everything over. That should terrify you. 

The true pièce de résistance though is this, the ‘agentic workflows’. For those that don’t know, “agentic” means give the bot a task, and let it go away and work out what actions to take to do it, then do them. This is the reason bots delete entire companies, they are given free reign to act on anything the word-prediction-bot thinks should come next, which seems to always lead back to destroying everything. Entropy is a beautiful thing. Google manifests this agentic-ness as “custom widgets”, and as “task automation”. In their example, someone takes what sure looks like an AI generated video of a leaflet of a coffee and chocolate tour, then asks Gemini to use Expedia to find something similar, then it goes away, does it, and brings you not to the listing so you can see if that is actually what you want, but to the checkout page, with all your information – including your Google Pay card – ready to go. Or, to use their other example, “On Chewy find a top-rated personalized dog bowl. Customize it for Stella and add it to my cart.” and it does. It picks a bowl, customises it, and adds it to the cart. Neither of these examples make sense, not one bit. Both are deeply personal choices being callously offloaded to a faceless corporation’s robot – who as I’ve already said is a PRIME candidate for advertising manipulation, or even AI-engine-optimisation. If you want a nice dog bowl for your dog, or worse if it’s a present for someone else, you aren’t even going to look at what you’re buying? And the tour, FOR SIX PEOPLE I might add, you’re just gonna trust Gemini picked the best one for you and your group? Ah, but that’s the thing isn’t it? Trust. Google has very, very carefully designed these features – and these examples – to have you implicitly trust this system. To make you not question the decisions you are offloading to it. To quash your critical thinking skills and just go with the flow. Just do what Gemini tells you, it knows best! Just tap to pay. Just do it. Don’t check it. Don’t worry that we picked the options the companies paid us to use. Don’t worry about it. Pay the money we get a kickback from. Pay it. C’mon. Do it. Do it. DO IT! Oh, and don’t worry that as we increase these models’ ability to reason, they intrinsically hallucinate more, and use tools less. Don’t worry about it. 

All of these systems are designed to be as convenient as possible, to remove as much effort and thought as it can from a task, which, according to every study I can find, shows they wear you down. An analysis paper said: “Overuse of AI decreased cognitive engagement and decreased memory retention”. A psychology paper said: “Results indicated that greater AI dependence was associated with lower levels of critical thinking, with cognitive fatigue partially mediating this relationship”. And most damningly, an incredibly thorough MIT paper said: “ChatGPT users had the lowest brain engagement and consistently underperformed at neural, linguistic and behavioral levels. Over the course of several months, ChatGPT users got lazier with each subsequent essay, often resorting to copy-and-paste by the end of the study.” This is literally a brainrot machine. Is this what you want literally running your life? Scanning your emails, picking your presents, holidays and tyres, and slowly chipping away at your ability to think clearly. To reason yourself. That sounds like the worst dystopian novels I can think of. 

To be clear, I do actually like AI, but there are some caveats to that statement. First, it has to be local. I can’t fathom purposefully giving one corporation – who removed “Don’t be evil” from their company motto for god’s sake – that much information and control over my life. The irony of me writing this script using Google Docs and posting it to YouTube isn’t lost on me though. The other caveat is that it has to be actually useful, not just convenient. Tools like Frigate that do local computer vision through my local CCTV cameras is useful. Home Assistant’s local Whisper and Piper for voice commands to control my smart house is useful. Hell, even local coding assistants to help me do grunt work or learn new stuff is useful. Even some of the stuff Google showed off, like your messaging app having access to your calendar so when you’re texting about making plans it can check if you’ve got conflicts and then add it to your calendar is really cool, I just don’t trust Google to be the one to do that. It must be local for me to use it with any reliability. 

In short then, Google is, I think anyway, being much more overt in showing just how much of your life they have access to, which is terrifying in and of itself, but it’s even worse when tied in with the notion that these systems are specifically engineered to wear down your critical thinking skills and make you the easiest mark around. A sheep, if you will. Genuinely, if your partner had this level of decision making control over your life, you’d want to run for the hills and get a specialist lawyer to file for your divorce, so why is it okay that a trillion dollar corporation does? Sure, a lot of these features are opt-in now, but that’s how this works. You make it an ‘appealing option’, then it becomes ‘highly recommended’, to ‘necessary’. You don’t back your photos up with us? Oh, but that’s necessary so Gemini can verify your age and prove you aren’t hiding anything criminal! You turned location services off? What are you hiding? Gemini needs to be able to keep track of you for your own safety! Yeahhhhhh, no thanks. I self-host as much of my own tech as I can, and I’ve got a bunch of videos showing you exactly how to do it yourself with hardware you might already have, so if you want to stop giving the trillion dollar spyware company your information – and money – maybe check ‘em out. 

I am scared of Android 17, and what it means not just for privacy, but people’s ability to function independently of their phone, and an internet connection. Especially young people. This seems like a step towards dystopian convenience, and I’m not here for it, and I hope you aren’t either.