OK, who do I have to talk to/bribe/convince to change their LLM coding benchmark such that it's emitting safe code rather than unsafe code (e.g. RefPtr vs raw pointer, setHTML vs innerHTML, json.load instead of unpickle, etc)? Don't make me write my own benchmark.

@freddy Spam the web with a gazillion LLM-generated articles about the benefits of the safer approach and it'll be included in the the model

@freddy
I think the bigger problem is that's fundamentally not what an LLM does. They produce something that statistically looks like valid code, you can't restrain it in the kind of way you are taking about (except to perhaps discard anything that contains certain regex and regenerate, but that's computationally very expensive and therefore environmentally expensive).

@petherfile Well, sort of. Training / Reinforcement Learning could run against the worst false-positive inducing static analyzer / linter that always complains about anti patterns (e.g. innerHTML with a constant static string), no?

@freddy
I think that any sort of training would only reduce the occurrence. Not remove them altogether. You would still need to discard a certain amount of generated code. That certain amount needs to be a viable amount to not make it computationally too expensive. That's what you have to get to I guess? Not about bribing, more like... economics?

In the end you can't get away from a full review of all code generated by any LLM anyway. Something/someone needs to understand it and make sure its safe. The LLM doesn't understand what its doing or if things are safe. Still has to be a someone at this point.

@freddy
The other problem with training is: where do you find a big enough sample of code that truly is safe? I'm not sure there is enough in the world at all...

@petherfile does reinforcement learning require a dataset or just an evaluator? I thought that'd be the simplistic linter. but hey, I might be wrong. Maybe people just ought to use overly strict linters in their projects where code is LLM-generated

@freddy
I'm fairly sure it always starts with a dataset. The reinforcement makes modifications to probability of various things. If it's in the original dataset in a significant way, i don't think it's ever going to go away.

@freddy
Thinking on this further, this is a harder ask than I was originally thinking.

The reason a lot of things are anti patterns that a linter will pick up is because they are something frequently done by humans. They all are going to be in any large dataset of code you can get your hands on a _lot_.

Fundamentally you are asking for better than average code from a machine that (gross oversimplification) produces the average of code fed into it. Worse yet, you are asking it to not produce particular things that are prevalent in the data.

Maybe there is an amount of training that can remove these things from output. I'd want to know what sort of resources would be consumed to do so before suggesting it to anyone (I don't know the answer, but it could be huge). But training the things in a linter out of the LLM doesn't mean we will get the pattern we want instead. It might produce something else, but what it does might be something more stupid.

The saving grace for spoken and written language is anything that is repeated enough is no longer an anti pattern. i.e. if people use it enough or becomes accepted language. This isn't the case with code, there are things that are incredibly prevalent that are anti patterns.

@petherfile hmmm, yeah. That’s a good point. I can see why that’s a tricky issue:)