somehow I only recently learned that LLMs need the whole conversation fed back to them on each prompt so their i/o cost scales as O(n^2), something that would be considered completely unacceptable in almost any other production network-accessible software

they ran into this and then didn't figure out a way to externalise the state in some fixed-size form so the LLM can resume from there. you just have to restart the computer and redo everything

in any other program this would be considered a DoS vector and have CVEs issued

gosh I wonder why their costs are completely unmanageable

this also makes it hard to take seriously claims that the marginal cost of inference is not a serious contributor to the resource use of data centres. your program has polynomial running cost and people run it for hours on end

SAMA ET AL: why can't we make a profit with this revolutionary technology
ME: your operating cost scales super-linearly with the value it delivers to users
SAMA ET AL: better tell the government we have invented the angels from evangelion or something

they invented a technology business with super-linear marginal cost of production I am losing my *entire mind*

@jcoglan @buherator This also means you can “gaslight” the LLM by manipulating the conversation before it’s sent back to the LLM

@schrotthaufen @jcoglan And coding agents gaslight the user by injecting "system reminders" prefixes that are invisible to the user but the LLM sees them as user messages and regularly refer back to those as user requirements.

@jcoglan the main reason that the longer you use them, the crazier they get is that at some point it will resort to resummarizing the prompt to reduce how much of the context window it’s eating; a machine that destroys its own nuance to live

@jcoglan There is prompt caching, although I'm not entirely sure how it works.

Basically everyone wants to solve that problem, but it's still easier to manage state by just shoving the whole conversation again down that pipe.

@jcoglan
Ok ok, it gets worse faster than it gets better, but have you considered they make up for it in volume? 🫠

@jcoglan I think part of the reason is that injecting randomness into each step is crucial to the way they are marketed and to many of their use-cases. Without it you'd get predictable outputs which would destroy the illusion of a "natural" conversation with the user.

@jcoglan this goes hand in hand with the obsession of getting better results by running these things in a loop (aka "agentic" stuff). You're exploring a statistical space through sheer brute force and randomness in the hope that you'll encounter something resembling the right answer

@jcoglan Yes they did - the K/V cache. And prices are cheaper if your subsequent queries with the same prefix are within time that that stays in memory on the server.

Agreed it is extremely inefficient, and mostly LLMs are better if you do controlled prompts with JSON output, rather than long conversation threads. But people are pushed into the long threads by the standard harnesses. That is a waste!

@jcoglan @ozzelot yeah it took us a few years of pointedly not learning this before we figured it out and had one of those "it can't be that stupid" moments

@gabrielesvelto @jcoglan They don't need to inject randomness, it's a more fundamental feature:

https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

@buherator @jcoglan having spent a lot of time looking at hardware faults of all sorts - many extremely subtle - I wouldn't be surprised if there's even more factors involved given the scale of the compuation