I know Mastodon hates LLM's and AI. So here goes!

I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.

The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.

I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.

I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.

The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.

I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.

I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.

Please run an LLM over your code base if it's internet facing or something critical, we thank you!

Can't wait for the discussions on this!

@drwhax Maybe the advice of the wiser among us will be heard after all these decades: focus on robust mitigations and attack surface reduction, because the moles can't be whacked anymore.

@buherator 100% this is the way, defense in depth helps kill bugs way faster. Prime examples, suhosin and grsecurity!

@drwhax I think we need a bit of nuance here.

As much as I am no fan of LLMs, I can see the value of running them over one's own codebase in order to find vulnerabilities.

But I remain unconvinced that vibe-coded fixes are a good idea, or ever will be. I think this needs to be clearly said.

The reason is that these are fundamentally different problems.

@drwhax do you have any pointers on *how* to run an LLM over the code base? Just tell Claude to "look for vulnerabilities" or is the process more elaborate?

@can yes for sure!

There's a couple things you can do, one of them is indeed as basic as, this is app XYZ, written in PHP, review the codebase for security problems, then adversarially review your findings before writing them down in findings/ per bug in markdown. Also make a root cause and look for variants of the same class of issues across the code base.

What are some good skills to run is from Trail of Bits which are open-source: https://github.com/trailofbits/skills

If it's C/C++, LLM's are pretty good at making fuzzers and seeding the dictionary needed to fuzz them effectively.

If it's open-source you can apply for some openai codex api grant: https://openai.com/form/codex-open-source-fund/

@drwhax finding vulnerabilities can be stochastic because it has a very clear and effective verification function: either the exploit works or it does not. Exploit code can be messy and convoluted, as it is not going to be maintained after the vulnerability is fixed.

Vibe-coding fixes does not have that kind of verification function: the fix must not only close the specific vulnerability, but *also* not introduce new ones or re-introduce old ones, and it has to be maintainable in the future.

@rysiek Human in the loop is important here, vibe-coded patches are most of the time not very good. However, the models do tend to improve, but what really helps in this case in lets say an C/C++ code base, alright parrot, make me an fuzzer that fuzzes these paths and we'll let it run on every commit so we can catch things.

Last night, well, eherm,... 2am, my parrot was adding a fuzzer to code that hadn't been fuzzed and it showed, bugs came falling out.

The human needs to stay in the loop with all these things of course, i'm not advocating for a pure AI/LLM/Parrot solution. But oh boy, are they good at pattern recognition.

@drwhax and in the end this will come down to managing limited resources and incentives, as always.

If a company can spend $1m in tokens on their codebase, are they incentivized to spend it on finding vulnerabilities, or churning out new vibe-coded features? I know what my guess is.

@rysiek For sure and the other part is guardrails that currently aren't aligned with the maintainer from these parrots either. Patching is hard or it'll fail because "CYBER"

@drwhax yup. All I am saying is that this distinction is important and needs to be constantly stressed, because people will end up creating more problems by vibe-coding.

@drwhax The bottleneck seems to be a lack of maintainers, so I don't really see how running LLMs on codebases will fix that. It'll take a maintainer to make the right decision based on LLM output and we're clearly running out of those :P

@png sadly, you're right.

@drwhax and just to drop some data on vibe-coded software quality:
https://www.faros.ai/blog/ai-acceleration-whiplash-takeaways

The report tries to be very "AI"-positive. But the numbers don't lie: +861% code churn, +242% incidents per PR, +54% bugs per developer.

@buherator @drwhax I sometimes advise investors (poor life choices I guess) on how to deal with the increased risk of Mythos & others. And this was the bottom line.

I don't even need AI specifically to argue for that. The time-to-exploit has been dropping forever. We've built better catalogs and more versatile infrastructure and complex automations. It's very clear that this includes vuln research as well.

We had this already with static analysis hitting the field. We answered with "shift left": Building security into the dev process, since the earlier you do it, the less issues you'll end up with. Every step earlier has more downstream impact.

We've finally reached the stage where security as part of your design & architecture is not only best practice but becoming a hard requirement. Our automations (including LLMs) pushed us there and we need to deal with the outcome.

Or should. Because I'm a cynic and I've seen what's happened, I doubt that engineering will follow through with this. I see two paths and we'll move along both, and I don't like either: Acceptance and few centralized Oligopols.

1/2

@rysiek human in the loop etc :)

@drwhax i have personally helped 3 different fediverse server projects by sharing exploits with them that LLMs found. Most software is full of holes. Running an LLM to hunt for bugs is good advice. If you're not doing it, someone else will.

@lain amen

@buherator @drwhax

1. Acceptance. There's the saying "you just need to be more secure than your neighbor". It's not fully true but biz people hold it so. It's flip side is a cynic commentary on security, and a true one at that: If your neighbor is not secure, you don't need to be either. This is still true if you're getting pwned. When everyone's getting hacked regularly, society will accept this new reality. Only the defense sector will still care for security. We've seen similar developments in different industries: Fast fashion isn't an adequate replacement for garment that stands the test of time. But it's getting impossible to buy good clothes, even if you want to. But also with other goods: If everyone produces low quality items, it's financially more reasonable to do the same.

2. Centralization. Coz we won't do the right thing and reduce complexity, we need to find a way to make complexity "secure": Cloud, Big5, etc. Corps that have the resources to host an Exchange server securely... or in compliance with regulations. Every tool you do not let a big house host is a liability. Not coz they're necessarily more secure, but because nobody will blame you if something gets wrong. We had that with "buying IBM" once.

I hate this outlook but I seriously doubt that we will correct course.

@drwhax Using the stones to destroy the stones seems an unavoidable task

@decapitae Ideally we get more Suhosin's projects for all these different languages, but I doubt it'll happen at this point.

@drwhax the problem is this works better for certain tasks (finding vulnerabilities) and much worse for other tasks (vibe-coding) because of the shape of these tasks.

"Human in the loop" is not the get-out-of-LLM-problems-free card people try to pretend it is.

Human in the loop works for vulnerability findings because there is a reliable way of verifying the finding. It clearly does not work well for vibe-coding at all because there is no such reliable way of verifying code correctness.

@rysiek I think we'll get there in a number of years, the way the field is developing now we got all these super fast interconnects and HBM memory and not to mention advancements in the machine learning field.

I almost puke writing this lol

@drwhax a lot of us at Red Hat have been working at 100% for the last 6 months on basically this. I personally spend ~2h/day on AISLE and Mythos reports and getting fixes upstream.

@hughsie I'd love to hear more on the maintainer side of this. What i'm ultimately scared of is that everyone will burn out from the immense amount of stuff maintainers have to clean up.

@drwhax @can great pointer, thanks. Going to try this tomorrow on a couple of our codebases (one 100% human coded, one originally human but with progressive LMM development over the last couple of months) - might be adjusting some development schedules depending on what I find!

@Offbeatmammal @can I'm very curious to hear if it helps! If you need help i'm here!

@rysiek @drwhax no we don't need nuance here, get those issues fixed.

@lain @rysiek nuance on social media????

@drwhax @rysiek that's not what i voted for

@lain @rysiek wait, we can vote for this?

@drwhax @can thanks! I am.bothbterrified and excited what I will find!

@Offbeatmammal @can I think that's accurately describing my view on AI for the past couple months now hehe

@rysiek @drwhax I remember reading an article about shit hospital admins wanting to ditch radiologists all but one who would check the AI dx...and if anything went wrong, can blame the hoo-man! And fire that one, hire another at an even LOWER rate! All for your Hellth.

@FIAR_Light @rysiek yeah that's pretty bad....

@drwhax It's not that you're wrong but you're not specific enough. Which AI, in what way? Plus, though I see you have enough technical expertise you come out with that OpenAI/Claude vibe that I detest. We should use LLMs, but in our own way, after stripping away the hype. All the LLM technobros have been stealing our code and other people's copyrighted work, and now want to sell it back to us with added bias, lack of provenance and old data. Excuse me if I'm not wildly enthusiastic about their for-profit products.

@ljrk @buherator @drwhax

The nice thing about open source is that there's always *someone* doing whatever obscure thing the mainstream finds too niche. I'm sure there are people right now who have for decades been building smaller software, and maybe for a certain non-corporate group there can be a pivot to that stuff. To avoid an endless edge of new code to harden, you need to forgo compatibility with new software & hardware, which has until now meant relative uselessness. But given how the AI bubble is driving up hardware prices, and the fact that we've seen standards like USB mature to a point that feels like it could be stable for a decade or more, maybe the endless cycle of new hardware stuff just slows down in certain backwaters. Of course this is anathema to corporate strategies, but I wonder if given the fact that the churning edge will be nearly impossible to secure, maybe a chunk of consumers will find ways to be satisfied with a smaller more stable core.

@drwhax
isn't this a proof that humans are not better at writing software than SOTA models? In general, of course.

@fnrd you're right, but I think if we take this defeatist stance we're not going to improve things for the better.

Unfortunately, we'll not be able to make our own models, we're compute starved, our best bet is maybe open-weights models.

It's all a mess though, I do agree with that.

@tiotasram @buherator @drwhax Absolutely. I believe that, for personal use, low-power devices, cloud-less with P2P collab features are a viable route. I fear however, that using such devices becomes ever harder w/o locking yourself out of the rest of the world: Requiring apps for concert tickets, public transports, online banking, or event finding events through instagram. The latter can possibly be fixed through better communities, the former is hard to address, except by having a personal device and one just for carrying such credentials.

@hajovonta I don't think so, that's how LLM's trick you. LLM's are just really good at pattern recognition and have been trained on an endless stream of CTF writeups on how to exploit vulnerabilities. This is why they shine in finding vulnerabilities generally.

@drwhax The laws of diminishing returns will kick in at some point. In the meantime, it is unlikely to be a smooth ride.

@whvholst it's one crazy rollercoaster right, including after the subsidy for compute ends

@ljrk @drwhax Re: 1., the usual question we got from CISO and above on our reports is "how does it look compared to similar companies?". I agree that this is in part the "you don't have to outrun the bear" logic, but also that people in position don't want to look incompetent in front of their peers.

Another thing is that you don't get fired if you didn't follow the hackers advice, but you do get fired if you don't pass compliance which is one part BS, and the other part is easy to cheat.

@drwhax On the flip side, when the subsidy for compute ends, edge computing will become affordable again. And given that things like rudimentary voice recognition now has been reduced to models that fit in an ESP32, I am inclined to think that special-purpose LLMs may be a cat that has left the bag or at least to be in a different league than frontier models.

@whvholst maybe things will turn cheap(er) again? I have some doubts about this and there's all kinds of ways to keep prices for components high by the companies and they've done this before and will do it again and they seem to get away with it mostly.

@drwhax Look, many people have different reasons to be against AI.

Mine is a very specific position against vibe coding, generative AI and ghoulish tech bros and their wannabes wanting to actively destroy excellence, art, nature and human work.

If you have a tool that does not contradict these specifics, then I have no issue. A static analyser ML-like code tool that works on premises, doesn't make a tech bro richer and dry a lake? This is OK for me (and I suspect, the future or much of this)

@drwhax But are you maling a tech bro richer to vibe code everything you do and just pressing Y without thinking, wasting tokens as much as possible because who cares? I have beef.

Are you actively (and disgustingly so), trying to replace every human in the loop just to create ghibli styled art slop or disgusting political videos so you can further your agenda? I have beef.

Are you actively numbing your brain, stop thinking about anything and becoming a reverse centaur on purpose? I have beef.

@drwhax I personally dont and will never vibe code because, in a very personal position, I love my brain and I like to think, and a brain without friction will cease to function (use it or lose it).

I love people making art (and yes, this includes code). And I deeply hate the contempt for people paying a tech bro to use genAI to dismiss and degrade those artists with a sort of vindictive glee, especially since most artists are already treated so badly. It is disgusting.

@glitchypixel I think art should be made by artists and not AI fwiw!

@drwhax

> I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure

You're missing the point. Even strong "AI critics" have already been seen to use these tools for these scans themselves. Just like other static code analysis tools and so on.

The main point and what people despise the most is when randoms use it and file bug reports they don't understand causing high workload validating their crap.

@agowa338 I agree and that's unfortunately the shitty side of it.

@drwhax
do two differemt LLMs find the same bugs?

@buherator @drwhax Yup same. It's what we do at $dayjob to motivated customers, partially. I'm waiting for a cohort to realize that, if nobody does anything, nobody will look too bad.

And yup. It's all a bit fuked up. :3

@rysiek @drwhax there is nothing magical about an AI fix, just look at it and either you can tell it actually fixes it, or you can't.

@sun @drwhax and how do you know it doesn't introduce more issues? It's not about "magic" nor "AI", it's about the complexity of creating software.

@wolf480pl sometimes, sometimes they don't, sometimes one finds a better way to chain vulnerabilities to achieve a certain objective. It all depends a bit on the harness as well, lots of small knobs to twist.

Some benchmarks are available on: https://exploitbench.ai/

@drwhax
my point is that fixing vulns only works if your enemy finds the same vulns as you found

@drwhax I am very very doubtful we will in any meaningful way.

In the end coding is an exercise in translating intentions into machine-readable code, and also an exercise in communication between those whose intentions are enshrined in code, and those who then need to maintain it.

We will be able to automate some things slightly better, though. But then the question is: at what cost?

@rysiek everyone's mental capacity? :)

@drwhax that too, but I meant it even in the purely monetary sense of token costs.

@rysiek once subsidy is gone, I don't think it makes a whole lot of sense, but, there might be advancements that tinnier models possible that are good enough at X or Y and then it's just hardware+electricity cost.

@drwhax I like to see it less as defeatist and more standing our ground. We don't need to rush. This will stop projects, hopefully before people burn out. That's the environment OpenAI and techbros have created. It's not given just because someone trashes your house you have to live there. You can build something new.

@drwhax oh I've been talking about smaller open-weights models for a long time now. A leaked Google memo ("we have no moat") mentioned them as a massive problem for them years ago. I have much less problem with using small, specialized, self-hosted, open-weights models.

In fact I know of at least one small company that already does this for vulnerability testing of their own code, avoiding most of the BS.

That said, it is still immensely expensive for them to run it on their own infra.

@drwhax in a way this is a question of how soon we finally get out of the Gartner hype cycle and people get to focus on figuring what these tools are *actually* useful for.

@drwhax @png I mean, it would not kill you to also submit a patch fixing the issue you found, would it. Certainly isn't the silver bullet, but I am sure it could help many of those overworked poor souls.

So.. have you submitted patches together with your reports?

@drwhax a lot of people have already said a lot of smart things in your replies, so i'll skip to a new direction:

why did you recently get access to these models? from a "media literacy" point of view, i feel like i don't know how to read your statement without knowing what your context is.

are you working with these models as part of a third-party, independent evaluation / audit? or are you working within a contract with either of these companies, do you receive money from them?

@catileptic No, I only pay a subscription myself and filled out two forms giving me access to a lessened guard rail LLM.

https://chatgpt.com/cyber?refresh_account=true

https://portal.anthropic.com/programs/cvp

I wanted access as at times I was hindered in reversing certain things on the non-cyber approved models. E.g forensic tooling :)

@drwhax Yes. As code robots LLM’s should not be underestimated.

They lose to humans on depth, but beat us in breath, width and speed outright.

They are weapons. First and foremost they should be thought of as weapons. 🤔

@gimulnautti I think they already classify themselves as dual-use. which yes, I think is the right distinction

@drwhax What would the estimated cost be for these 7-9 hours, assuming no more subsidies? Just looking for a rough ball park figure, as almost everything I've heard up till this point is that while the LLMs are effective, the cost is significant enough that you could be paying someone to do the work directly.

@macattackmicmac for that particular session it was roughly 122 USD in API costs, but since it was subsidized, it was only $20 in a subscription: https://gist.github.com/DrWhax/465da154f2ac1575cfc72aa71aa979e6

@drwhax

the reality is that LLMs are incredibly powerful tools, but mastodon is so into reality denial
and while denying reality, the same people make fun of Trump supporters for denying reality (eg vaccines don't cause autism)

be funny it wasn't so sad

@failedLyndonLaRouchite They are, but they have flaws and the way these models are created is quite shit and there's a lot of things to say about that. It's not that I say, anthropic good, or openai good. It's more to say, we're entering an era with a lot of shit going to be thrown at people, companies and maintainers and I don't think people realize what's about to hit them!

@drwhax I would rather abandon an already dying field and pursue my dream of opening a coffee place rather than using LLMs for anything at all.

@chroma yeah i'd also like a farm!

@edward I do think that's the case at the moment, cross-file vulnerabilities are sometimes not found. What I feel they're mostly good at is, pattern recognition, e.g, there's a specific bug class fixed from past git history that it found repeated or as a variant. After all, LLM's is just a stochastic parrot and it shows

@drwhax I think LLMs get the low hanging fruit of vulnerabilities, so you shouldn't trust an LLM's security audit as a definitive audit. As models get better, they'll catch higher up fruit, but still.

@drwhax That is significantly cheaper that I was expecting.

@macattackmicmac yeah, it's pretty wild at the cost to be honest.