logoalt Hacker News

er4hn • today at 7:59 PM • 2 replies • view on HN

It's hard to try and compare these benchmarks against real world use cases. As an experiment I let Meta Muse make all major decisions on a weekend trip to Seattle ~2 weeks ago. I think this would have been using Muse Spark 1.3, which is equivalent to GPT 6 - Sol.

On the plus side it told me taking the light rail from SEA into the University district was faster and cheaper than Uber.

On the downside it would recommend restaurants that were out of business, checking out Salmon swimming through locks out of season, got the opening of Beecher's Cheese in Pikes Place wrong, and fumble it's way through most simple planning decisions. I think my favorite was when it asked me what flight I have tomorrow so it could help plan when to head to SEA. It sent me a very panicked message that the flight was leaving today! Very soon! I had to remind it this is a daily flight and I am booked on the next day before it checked and apologized.

For things that are verifiable, and code tends to be that way, these models can do all sorts of things. But I find it hard to reconcile Meta's goal of having a personal assistant bot with something that is unable to do the simplest tasks.


Replies

RandallBrown • today at 8:03 PM

> checking out Salmon swimming through locks out of season

To be fair, the Coho salmon run at the Ballard locks peaks in September. It's possible you just had bad luck that day.

https://ballardlocks.org/fish-salmon-ladder.html

I run by the locks a few times a week and while I don't usually see fish, the presence of sea lions the last few weeks tells me that at least some salmon are running right now.

➕ show 2 replies
nimchimpsky • today at 8:10 PM

but that was a different, much smaller and less capable model