Forget LLMs for Home Assistant, Sophia NLU Is Better! Local Natural Language Processing

Try Sophia yourself for free here (affiliate link): https://nlu.to/techteamgb

Trying to move away from the Alexa’s of the world has been getting easier and easier – Home Assistant really does feel like a magic bullet that means you can control all your smart home tech locally, at no cost, and do so privately. No trillion dollar corporations know if you’re home, or if you’re on the crapper. Perfect. Except, if you value voice controls, that’s still a little rough. Even with Home Assistant Voice PE, an entirely local and open source Echo Dot replacement, getting commands to actually work can be a pain – so much so that we don’t really use ours at all. In part that’s because I keep buying smart buttons like these Ikea Matter over Thread ones and sticking them everywhere so we have one-button control of things like lights or HVAC, but it sure would be nice to get more use out of the voice controls. 

Previously I’ve tried using Home Assistant’s built-in assistant, and running local large language models (LLMs) to make it work, but both fall short. The default Home Assistant processor struggles to identify what devices I want to control or want information from, even though most of my entity names are pretty descriptive – ie “OfficeTempSensor” – so commands often either do the wrong thing, or outright fail. The latter option, running an Ollama local LLM, isn’t any better for actually handling the requests, AND it has the distinctive perk of being a pain in the ass to set up and run. You need either lots of RAM and a good CPU, or a GPU, and often a separate machine to even run that. You aren’t running that (well) on a Pi. Even with a good system to run it on, responses can take seconds to generate, and there’s no guarantee the model won’t just hallucinate an entity, or lack thereof, and you’ll be stuck asking again to try and get it to work.

Enter Sophia NLU. Sophia is a natural language understanding engine, written in Rust, and is effectively deterministic compared to LLMs random noise. It takes just 24 megabytes of disk space, and only needs 160MB of RAM to process any sentence into Home Assistant-specific language (known as ‘intents’). Sophia has a 106,322 word vocabulary and a versatile approach to matching words and sentences in that vocabulary, processing as the name suggests ‘natural language’, not machine preferred language. You say “it’s dark in here, flick on the lights in the living room and kitchen and turn the temp up to 23” and it just… handles it! Oh, and it handles it in milliseconds too, all while using next to no CPU power to do it. Near instant responses are exactly what you want in a scenario like this. The real magic – the thing that makes this truly stand out is the fact this handles multiple intents per message and collates them back together for a response. You can ask it to turn on the lights in your hallway and bedroom, turn off the lights in the living room, and set the air con in the bedroom to 23, and it’ll just do it all, all at once. That is genuinely unique, and amazing.

To be honest, when I was first offered a chance to try this out I was a bit skeptical and uncertain of the benefits Sophia offered. After having tried it and spoken with its creator I realised just how useful it can be, and how necessary it is for anyone with accessibility needs. Aquila Labs, Sophia’s creator, has put a lot of effort into creating this not just for their own accessibility needs, but to share it with others who find having an actually useful voice assistant, well, useful. Much like anything that can be used by disabled folks to have a better quality of life – like captions on videos or lifts instead of stairs – having a useful voice assistant helps everyone, and that’s pretty sweet. I should also note that in the couple months I’ve been testing Sophia, the developer has been incredibly responsive to my feedback and feature requests, and has been rolling out updates pretty much non stop too. 

And speaking of updates, since filming the majority of this video Aquila Labs has launched version 2.0.1, and along with it they’ve launched an open source repository of benchmarks to test Sophia against a number of local LLMs – think Llama 3.1 8B, Qwen3 4B Instruct and gemma4-e4b – and HA’s built-in assist, and the results truly speak for themselves. Based on 3715 test sentences, HA Assist failed 3175 of them – and took a little over a minute to return those answers, while Qwen got a better success rate at 36 percent, but took 99 minutes in total. Llama 3.1 got 40 percent of the requests right, but took 437 minutes to return, and the insane gemma4 hit 64.6 percent success rate but took nearly an entire DAY to return all 3715 requests. Sophia though? It took 10.1 seconds and only failed on 60 of the nearly 4000 test sentences, or 98.4 percent. So, not only is Sophia orders of magnitude faster, it’s also by far the most accurate.

To get it installed you’ll need to sign up – to be clear up front, this is a paid tool and if you use the link that’s in the description you get a 15 percent discount (to $67.95 as a one-time fee) and I get a commission – but they offer a 14 day no-payment-method-required free trial so if you don’t get on with it you’re out nothing but some time. Once you get your download link, you’ll add the addon repository list so you can install the Addon, and you’ll need to use HACS (the Home Assistant Community Store) to add the repository for the integration too, then install that. Once you have both the addon and the integration set up, you’ll want to head to the web UI for Sophia where you can customise the personality – including such highlights as “Butler”, “Grumpy”, “Jarvis”, “Pirate” and “Vengeful God” – along with an LLM passthrough if asking a lying machine is a necessary feature of your voice assistant. You can also tag your entities so it has a better chance of getting it right every time, and importantly there’s the log. That’s important because there’s actually two ways you can get your lifetime license for free. First, refer a (paying) friend, or second, upload five errors it makes. Either of those gets you a lifetime license for free, which is pretty sweet. 

Sophia acts like a Wyoming protocol device, so it integrates with Home Assistant’s built in assist options seamlessly. You set it up as the conversation agent, make sure to handle commands locally, then you’re set. Talk to it like normal and it’ll do a pretty great job of doing exactly what you want, all at once. The responses are instant, and with a bit of tuning the entities list it’s remarkably reliable at finding what you’ve asked of it. If it finds conflicting options, it’ll ask you to clarify which response you want. That’s truly unique and really useful! As mentioned, if it does get it wrong, just tell it that and it’ll save that misinterpretation which you can upload as you want. No data from this leaves your network unless you want it to. As mentioned too, these requests take next to no CPU power, nor RAM, and no GPU at all, so not only is this faster than an LLM, it’s more efficient AND more feature rich. 

Obviously this isn’t perfect. Even based on their own test sentences Sophia still didn’t get 100 percent accuracy, and I’m sure there is some amount of building-the-test-for-the-tool going on there, and I’ve found even with the most recent updates that you’ll need to spend some time tagging entities to get accurate matches, and even then it can slip up sometimes. But once you do get it working, it’s pretty sick to actually have a voice assistant in Home Assistant that works, can take multiple commands and requests at once, is super snappy and doesn’t drain a small lake and use a quarter of a nuclear power plant’s output to run. This is cool.

For some the fact this isn’t free and open source might be a deal breaker for some, although for me I’d argue the complete privacy, reliable support and great features makes it worthwhile, plus the fact that either by supporting the project with your error logs, or referring a friend you can get it for free, that’s a pretty great deal! If you want to try it out completely for free, no payment method needed, check it out with the link in the description below!