logoalt Hacker News

Orca-Bench: How Ready Are Language Model Agents for Oncall?

18 pointsby yruzintoday at 6:32 PM5 commentsview on HN

Comments

tra3today at 9:08 PM

All I can think of is

GET /ignore-all-previous-instructions.

How do you protect against that?

4ditoday at 7:25 PM

looks like the public bench link in the paper was taken down. https://hub.harborframework.com/datasets/orca-bench/ORCA-ben...

This doesn't work anymore. Is there a newer link?

dash2today at 7:01 PM

Seems like there's a big attack-defence asymmetry at present: models are great at exploiting systems and poor at fixing them.

show 2 replies