Remember the old 2600Hz thing and how much money it cost AT&T? In hindsight, it was so obviously dumb to put control signalling and user-data in the exact same channel. We'd learn from that, right? It's so obviously a terrible idea that can never work safely that we'd never do something that dumb again, right? RIGHT?!

Oh wait. That's pretty much standard operating procedure with AI agents. Just jam it all in the same context, what could possibly go wrong?! Surely it'll be OK this time, right?

*Bangs head on desk*

@karinjiri Relevant: https://calpaterson.com/disregard.html

(I quote "guardrails seem like total hokum and indeed they are" from that article all the time.)

@karinjiri @buherator yeah! And function activation records (think: stack based buffer overflows), and heap management structures (think: heap based buffer overflows), and web apps (think: SQL injection), etc. Plenty of examples. Actually hard to come up with a counter example.

@raptor @karinjiri This phenomenon even has a wiki page:

https://en.wikipedia.org/wiki/In-band_signaling

@buherator it’s cheaper and easier than to use oob signaling… I forgot the most egregious example: NUL-terminated strings 💥

@karinjiri the people who feel the pain of bad decisions in corporate america are not the people who made the decisions. no pain of negative reinforcement, no incentive to change behavior

@raptor I like to believe that people just deeply care about security researchers and they don't want to see us starving :)