Home Assistant Voice Commands Are a MESS (feat Sophia NLU & LLMs)
I’ve been trying for two straight weeks now to write a video comparing Home Assistant’s built-in Assist, using an LLM, or using Sophia NLU, but it has been absolutely impossible to get consistent results. So this is a cry for help, and a rant, rather than the level-headed comparison I intended on making. While I still have some sanity left I’ll say this – Sophia NLU is absolutely the best tool for this job, and you should give it a (free) try with the link in the description. They aren’t paying me to say that, that is a genuine belief, although if you do sign up with the link below I get a commission. Right, let’s dive into the insanity.
Until this week, HA Assist has been rock solid. Limited, for sure, but reliable. Today I tried to get it to tell me any temperature, and all I get is a red “Sorry, I couldn’t understand that” response. If I check the debug log it correctly reads as a “HassClimateGetTemperature” intent, but the tool now fails, with the “no_valid_targets” error. I have no idea why this has only just started happening – I have to assume the core update I did in the last week did it – but to confirm there are three exposed temperature sensors in the bedroom, including a TRV that is a full climate/thermostat entity. Any request for temperature now fails, no matter the request. Even asking for a specific entity, it still fails. It still works for turning lights on and off though, and same for climate change requests. It’s only getting temperatures that are broken. Awesome.
Using Ollama running on a separate server is just dreadful. The base experience is pretty awful anyway. While it does understand natural language pretty well, it takes FOREVER to do anything. It took 23 seconds to turn the hallway lights on. 16 seconds to get the temperature in the office. And another 18 seconds to turn the hallway lights off and get the temperature in the bedroom. That’s insanely slow. Annoyingly slow. Enough that we didn’t use the voice commands at all. But it gets worse. If you try and do multiple things at once, say turning the hallway and bedroom lights off, the LLM will either hallucinate an ability to request multiple areas in one request, or it’ll return the json request as text instead of sending it to HA. At one point it took several minutes to get a response – I think one request, a simple request, took 11 minutes total. Oh, and sometimes it says it did something like turning the lights off, but it actually fails to do it. Awesome.
Sophia NLU is by far the most useful, but even it has been struggling for me. Sophia is written in Rust, is entirely deterministic, takes just 160MB of RAM to run and no GPU and basically no CPU, and is instantaneous. While basic requests are often handled by HA Assist – assuming you leave the “Prefer handling commands locally” on, which for Sophia I think is better to turn off honestly – anything HA Assist can’t do, Sophia can. Mostly. And instantly. Turning the hallway lights on takes, according to HA’s debug menu, 0.32 seconds. Ollama took 23.48 seconds to do the same command. Getting the temperature in the office took 0.02 seconds. Ollama took 15.51 seconds. Insane.
Sometimes though Sophia gets it wrong. Even though Sophia has access to every entity, and I’ve even manually tagged my main bedroom temperature sensor as bed, bedroom, temperature, temp, it is incapable of telling me the temperature in the bedroom right now. Although as I’ve found out, that’s because Sophia thinks asking for the temperature in the bedroom is a HassTurnOn request, instead of a get climate request. It doesn’t do this for any other room I ask for. Luckily Sophia is the only one of the three where you can just reply “incorrect”, it’ll mark it as wrong, and with one click you can send that failed request to the developer who will actually fix it. Doing this five times by the way gets you Sophia for free, so it’s in everyone’s best interests to report errors. Honestly this is my favourite feature. It’s never going to get everything perfect first time, but the ability to report an error and have it fixed is amazing. You can’t do that with Ollama!
Sophia’s ability to handle multiple requests borders on miraculous. You can say, “turn the lights in the hallway and office off, and turn the living room hue bulb on” and while it takes a second because a couple of my lights are off at the switch and therefore can’t be reached, it just does it. One thing I would love is the option to exclude entities from Sophia. I don’t care about the adaptive lighting entity being voice controlled, and it seems to pick that over actual lights sometimes, so removing that from its list would be great.
Don’t get me wrong, Sophia is worlds better than HA Assist in terms of functionality, reliability and usability, and while LLMs generally offer similar features – plus the ability to wax philosophically about the best home automation system – they take absolutely forever to work, need decently powerful hardware on hand to run it and more power, and often just don’t work, either because they’ve hallucinated functionality, or because they send their intent to the chat window instead of to the HA system. If you want to use voice commands, use Sophia. I just hope it can get a little more reliable and add a few nice-to-have features too. Again, link in the description if you want to try it yourself.
