alt Hacker NewsFunny result from the web demo. I'm well aware that it's an extremely small and, well, stupid, model, but even so:
Query: HN
Result:
{ "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.", "confidence": 0 }
I'd expect it to at least ignore (call no tools) for the queries that it doesn't understand. And it seems like it does do that, just not consistently.
Replies
It seems to love locking doors. I was hoping this would dim the lights.
> I'm hungover
{ "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. 'hungover' implies a security door. No specific door named, so use 'front door' as default.", "confidence": 0 }
no, this is the appropriate response to hearing the words "HN" :P
I've got an identical output with the prompt "do not lock the door".
yeah I got the same, almost like its biased heavily towards that as the 0 ranking -- my prompt was just the word 'potato'
Was that the first message you sent it?
This is exactly why the confidence feature was introduced, the model knows when its wrong, we could hide that part and return a placeholder "sorry I only do function calls", would that be better or you prefer to see everything?
[dead]
The website says the model is for "tool calling, device use, and structured extraction". Your example just doesn't seem to be very relevant. FWIW, it did a pretty good job for tool calling when I tried it, and I think it could be pretty nice to have this running on locally and integrate with Home Assistant.