I know Mastodon hates LLM's and AI. So here goes!

I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.

The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.

I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.

I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.

The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.

I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.

I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.

Please run an LLM over your code base if it's internet facing or something critical, we thank you!

Can't wait for the discussions on this!

@drwhax Maybe the advice of the wiser among us will be heard after all these decades: focus on robust mitigations and attack surface reduction, because the moles can't be whacked anymore.

@buherator 100% this is the way, defense in depth helps kill bugs way faster. Prime examples, suhosin and grsecurity!

@drwhax I think we need a bit of nuance here.

As much as I am no fan of LLMs, I can see the value of running them over one's own codebase in order to find vulnerabilities.

But I remain unconvinced that vibe-coded fixes are a good idea, or ever will be. I think this needs to be clearly said.

The reason is that these are fundamentally different problems.

@drwhax do you have any pointers on *how* to run an LLM over the code base? Just tell Claude to "look for vulnerabilities" or is the process more elaborate?

@can yes for sure!

There's a couple things you can do, one of them is indeed as basic as, this is app XYZ, written in PHP, review the codebase for security problems, then adversarially review your findings before writing them down in findings/ per bug in markdown. Also make a root cause and look for variants of the same class of issues across the code base.

What are some good skills to run is from Trail of Bits which are open-source: https://github.com/trailofbits/skills

If it's C/C++, LLM's are pretty good at making fuzzers and seeding the dictionary needed to fuzz them effectively.

If it's open-source you can apply for some openai codex api grant: https://openai.com/form/codex-open-source-fund/

@drwhax finding vulnerabilities can be stochastic because it has a very clear and effective verification function: either the exploit works or it does not. Exploit code can be messy and convoluted, as it is not going to be maintained after the vulnerability is fixed.

Vibe-coding fixes does not have that kind of verification function: the fix must not only close the specific vulnerability, but *also* not introduce new ones or re-introduce old ones, and it has to be maintainable in the future.

@rysiek Human in the loop is important here, vibe-coded patches are most of the time not very good. However, the models do tend to improve, but what really helps in this case in lets say an C/C++ code base, alright parrot, make me an fuzzer that fuzzes these paths and we'll let it run on every commit so we can catch things.

Last night, well, eherm,... 2am, my parrot was adding a fuzzer to code that hadn't been fuzzed and it showed, bugs came falling out.

The human needs to stay in the loop with all these things of course, i'm not advocating for a pure AI/LLM/Parrot solution. But oh boy, are they good at pattern recognition.

@drwhax and in the end this will come down to managing limited resources and incentives, as always.

If a company can spend $1m in tokens on their codebase, are they incentivized to spend it on finding vulnerabilities, or churning out new vibe-coded features? I know what my guess is.

@rysiek For sure and the other part is guardrails that currently aren't aligned with the maintainer from these parrots either. Patching is hard or it'll fail because "CYBER"

@drwhax yup. All I am saying is that this distinction is important and needs to be constantly stressed, because people will end up creating more problems by vibe-coding.

@drwhax The bottleneck seems to be a lack of maintainers, so I don't really see how running LLMs on codebases will fix that. It'll take a maintainer to make the right decision based on LLM output and we're clearly running out of those :P

@png sadly, you're right.

@drwhax and just to drop some data on vibe-coded software quality:
https://www.faros.ai/blog/ai-acceleration-whiplash-takeaways

The report tries to be very "AI"-positive. But the numbers don't lie: +861% code churn, +242% incidents per PR, +54% bugs per developer.

@buherator @drwhax I sometimes advise investors (poor life choices I guess) on how to deal with the increased risk of Mythos & others. And this was the bottom line.

I don't even need AI specifically to argue for that. The time-to-exploit has been dropping forever. We've built better catalogs and more versatile infrastructure and complex automations. It's very clear that this includes vuln research as well.

We had this already with static analysis hitting the field. We answered with "shift left": Building security into the dev process, since the earlier you do it, the less issues you'll end up with. Every step earlier has more downstream impact.

We've finally reached the stage where security as part of your design & architecture is not only best practice but becoming a hard requirement. Our automations (including LLMs) pushed us there and we need to deal with the outcome.

Or should. Because I'm a cynic and I've seen what's happened, I doubt that engineering will follow through with this. I see two paths and we'll move along both, and I don't like either: Acceptance and few centralized Oligopols.

1/2

@rysiek human in the loop etc :)

@drwhax i have personally helped 3 different fediverse server projects by sharing exploits with them that LLMs found. Most software is full of holes. Running an LLM to hunt for bugs is good advice. If you're not doing it, someone else will.

@lain amen

@buherator @drwhax

1. Acceptance. There's the saying "you just need to be more secure than your neighbor". It's not fully true but biz people hold it so. It's flip side is a cynic commentary on security, and a true one at that: If your neighbor is not secure, you don't need to be either. This is still true if you're getting pwned. When everyone's getting hacked regularly, society will accept this new reality. Only the defense sector will still care for security. We've seen similar developments in different industries: Fast fashion isn't an adequate replacement for garment that stands the test of time. But it's getting impossible to buy good clothes, even if you want to. But also with other goods: If everyone produces low quality items, it's financially more reasonable to do the same.

2. Centralization. Coz we won't do the right thing and reduce complexity, we need to find a way to make complexity "secure": Cloud, Big5, etc. Corps that have the resources to host an Exchange server securely... or in compliance with regulations. Every tool you do not let a big house host is a liability. Not coz they're necessarily more secure, but because nobody will blame you if something gets wrong. We had that with "buying IBM" once.

I hate this outlook but I seriously doubt that we will correct course.

@drwhax Using the stones to destroy the stones seems an unavoidable task

@decapitae Ideally we get more Suhosin's projects for all these different languages, but I doubt it'll happen at this point.

@drwhax the problem is this works better for certain tasks (finding vulnerabilities) and much worse for other tasks (vibe-coding) because of the shape of these tasks.

"Human in the loop" is not the get-out-of-LLM-problems-free card people try to pretend it is.

Human in the loop works for vulnerability findings because there is a reliable way of verifying the finding. It clearly does not work well for vibe-coding at all because there is no such reliable way of verifying code correctness.

@rysiek I think we'll get there in a number of years, the way the field is developing now we got all these super fast interconnects and HBM memory and not to mention advancements in the machine learning field.

I almost puke writing this lol

@drwhax a lot of us at Red Hat have been working at 100% for the last 6 months on basically this. I personally spend ~2h/day on AISLE and Mythos reports and getting fixes upstream.

@hughsie I'd love to hear more on the maintainer side of this. What i'm ultimately scared of is that everyone will burn out from the immense amount of stuff maintainers have to clean up.

@drwhax @can great pointer, thanks. Going to try this tomorrow on a couple of our codebases (one 100% human coded, one originally human but with progressive LMM development over the last couple of months) - might be adjusting some development schedules depending on what I find!

@Offbeatmammal @can I'm very curious to hear if it helps! If you need help i'm here!

@rysiek @drwhax no we don't need nuance here, get those issues fixed.

@lain @rysiek nuance on social media????

@drwhax @rysiek that's not what i voted for

@lain @rysiek wait, we can vote for this?

@drwhax @can thanks! I am.bothbterrified and excited what I will find!

@Offbeatmammal @can I think that's accurately describing my view on AI for the past couple months now hehe

@rysiek @drwhax I remember reading an article about shit hospital admins wanting to ditch radiologists all but one who would check the AI dx...and if anything went wrong, can blame the hoo-man! And fire that one, hire another at an even LOWER rate! All for your Hellth.

@FIAR_Light @rysiek yeah that's pretty bad....

@drwhax It's not that you're wrong but you're not specific enough. Which AI, in what way? Plus, though I see you have enough technical expertise you come out with that OpenAI/Claude vibe that I detest. We should use LLMs, but in our own way, after stripping away the hype. All the LLM technobros have been stealing our code and other people's copyrighted work, and now want to sell it back to us with added bias, lack of provenance and old data. Excuse me if I'm not wildly enthusiastic about their for-profit products.

@ljrk @buherator @drwhax

The nice thing about open source is that there's always *someone* doing whatever obscure thing the mainstream finds too niche. I'm sure there are people right now who have for decades been building smaller software, and maybe for a certain non-corporate group there can be a pivot to that stuff. To avoid an endless edge of new code to harden, you need to forgo compatibility with new software & hardware, which has until now meant relative uselessness. But given how the AI bubble is driving up hardware prices, and the fact that we've seen standards like USB mature to a point that feels like it could be stable for a decade or more, maybe the endless cycle of new hardware stuff just slows down in certain backwaters. Of course this is anathema to corporate strategies, but I wonder if given the fact that the churning edge will be nearly impossible to secure, maybe a chunk of consumers will find ways to be satisfied with a smaller more stable core.

@drwhax
isn't this a proof that humans are not better at writing software than SOTA models? In general, of course.

@fnrd you're right, but I think if we take this defeatist stance we're not going to improve things for the better.

Unfortunately, we'll not be able to make our own models, we're compute starved, our best bet is maybe open-weights models.

It's all a mess though, I do agree with that.

@tiotasram @buherator @drwhax Absolutely. I believe that, for personal use, low-power devices, cloud-less with P2P collab features are a viable route. I fear however, that using such devices becomes ever harder w/o locking yourself out of the rest of the world: Requiring apps for concert tickets, public transports, online banking, or event finding events through instagram. The latter can possibly be fixed through better communities, the former is hard to address, except by having a personal device and one just for carrying such credentials.

@hajovonta I don't think so, that's how LLM's trick you. LLM's are just really good at pattern recognition and have been trained on an endless stream of CTF writeups on how to exploit vulnerabilities. This is why they shine in finding vulnerabilities generally.

@drwhax The laws of diminishing returns will kick in at some point. In the meantime, it is unlikely to be a smooth ride.

@whvholst it's one crazy rollercoaster right, including after the subsidy for compute ends

@ljrk @drwhax Re: 1., the usual question we got from CISO and above on our reports is "how does it look compared to similar companies?". I agree that this is in part the "you don't have to outrun the bear" logic, but also that people in position don't want to look incompetent in front of their peers.

Another thing is that you don't get fired if you didn't follow the hackers advice, but you do get fired if you don't pass compliance which is one part BS, and the other part is easy to cheat.

@drwhax On the flip side, when the subsidy for compute ends, edge computing will become affordable again. And given that things like rudimentary voice recognition now has been reduced to models that fit in an ESP32, I am inclined to think that special-purpose LLMs may be a cat that has left the bag or at least to be in a different league than frontier models.

@whvholst maybe things will turn cheap(er) again? I have some doubts about this and there's all kinds of ways to keep prices for components high by the companies and they've done this before and will do it again and they seem to get away with it mostly.

@drwhax Look, many people have different reasons to be against AI.

Mine is a very specific position against vibe coding, generative AI and ghoulish tech bros and their wannabes wanting to actively destroy excellence, art, nature and human work.

If you have a tool that does not contradict these specifics, then I have no issue. A static analyser ML-like code tool that works on premises, doesn't make a tech bro richer and dry a lake? This is OK for me (and I suspect, the future or much of this)

@drwhax But are you maling a tech bro richer to vibe code everything you do and just pressing Y without thinking, wasting tokens as much as possible because who cares? I have beef.

Are you actively (and disgustingly so), trying to replace every human in the loop just to create ghibli styled art slop or disgusting political videos so you can further your agenda? I have beef.

Are you actively numbing your brain, stop thinking about anything and becoming a reverse centaur on purpose? I have beef.

@drwhax I personally dont and will never vibe code because, in a very personal position, I love my brain and I like to think, and a brain without friction will cease to function (use it or lose it).

I love people making art (and yes, this includes code). And I deeply hate the contempt for people paying a tech bro to use genAI to dismiss and degrade those artists with a sort of vindictive glee, especially since most artists are already treated so badly. It is disgusting.

@glitchypixel I think art should be made by artists and not AI fwiw!

@drwhax

> I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure

You're missing the point. Even strong "AI critics" have already been seen to use these tools for these scans themselves. Just like other static code analysis tools and so on.

The main point and what people despise the most is when randoms use it and file bug reports they don't understand causing high workload validating their crap.

@agowa338 I agree and that's unfortunately the shitty side of it.

@drwhax
do two differemt LLMs find the same bugs?

@buherator @drwhax Yup same. It's what we do at $dayjob to motivated customers, partially. I'm waiting for a cohort to realize that, if nobody does anything, nobody will look too bad.

And yup. It's all a bit fuked up. :3

@rysiek @drwhax there is nothing magical about an AI fix, just look at it and either you can tell it actually fixes it, or you can't.

@sun @drwhax and how do you know it doesn't introduce more issues? It's not about "magic" nor "AI", it's about the complexity of creating software.

@wolf480pl sometimes, sometimes they don't, sometimes one finds a better way to chain vulnerabilities to achieve a certain objective. It all depends a bit on the harness as well, lots of small knobs to twist.

Some benchmarks are available on: https://exploitbench.ai/

@drwhax
my point is that fixing vulns only works if your enemy finds the same vulns as you found

@drwhax I am very very doubtful we will in any meaningful way.

In the end coding is an exercise in translating intentions into machine-readable code, and also an exercise in communication between those whose intentions are enshrined in code, and those who then need to maintain it.

We will be able to automate some things slightly better, though. But then the question is: at what cost?

@rysiek everyone's mental capacity? :)

@drwhax that too, but I meant it even in the purely monetary sense of token costs.

@rysiek once subsidy is gone, I don't think it makes a whole lot of sense, but, there might be advancements that tinnier models possible that are good enough at X or Y and then it's just hardware+electricity cost.

@drwhax I like to see it less as defeatist and more standing our ground. We don't need to rush. This will stop projects, hopefully before people burn out. That's the environment OpenAI and techbros have created. It's not given just because someone trashes your house you have to live there. You can build something new.

@drwhax oh I've been talking about smaller open-weights models for a long time now. A leaked Google memo ("we have no moat") mentioned them as a massive problem for them years ago. I have much less problem with using small, specialized, self-hosted, open-weights models.

In fact I know of at least one small company that already does this for vulnerability testing of their own code, avoiding most of the BS.

That said, it is still immensely expensive for them to run it on their own infra.

@drwhax in a way this is a question of how soon we finally get out of the Gartner hype cycle and people get to focus on figuring what these tools are *actually* useful for.

@drwhax @png I mean, it would not kill you to also submit a patch fixing the issue you found, would it. Certainly isn't the silver bullet, but I am sure it could help many of those overworked poor souls.

So.. have you submitted patches together with your reports?

@drwhax a lot of people have already said a lot of smart things in your replies, so i'll skip to a new direction:

why did you recently get access to these models? from a "media literacy" point of view, i feel like i don't know how to read your statement without knowing what your context is.

are you working with these models as part of a third-party, independent evaluation / audit? or are you working within a contract with either of these companies, do you receive money from them?

@catileptic No, I only pay a subscription myself and filled out two forms giving me access to a lessened guard rail LLM.

https://chatgpt.com/cyber?refresh_account=true

https://portal.anthropic.com/programs/cvp

I wanted access as at times I was hindered in reversing certain things on the non-cyber approved models. E.g forensic tooling :)

@drwhax Yes. As code robots LLM’s should not be underestimated.

They lose to humans on depth, but beat us in breath, width and speed outright.

They are weapons. First and foremost they should be thought of as weapons. 🤔

@gimulnautti I think they already classify themselves as dual-use. which yes, I think is the right distinction

@drwhax What would the estimated cost be for these 7-9 hours, assuming no more subsidies? Just looking for a rough ball park figure, as almost everything I've heard up till this point is that while the LLMs are effective, the cost is significant enough that you could be paying someone to do the work directly.

@macattackmicmac for that particular session it was roughly 122 USD in API costs, but since it was subsidized, it was only $20 in a subscription: https://gist.github.com/DrWhax/465da154f2ac1575cfc72aa71aa979e6

@drwhax

the reality is that LLMs are incredibly powerful tools, but mastodon is so into reality denial
and while denying reality, the same people make fun of Trump supporters for denying reality (eg vaccines don't cause autism)

be funny it wasn't so sad

@failedLyndonLaRouchite They are, but they have flaws and the way these models are created is quite shit and there's a lot of things to say about that. It's not that I say, anthropic good, or openai good. It's more to say, we're entering an era with a lot of shit going to be thrown at people, companies and maintainers and I don't think people realize what's about to hit them!

@drwhax I would rather abandon an already dying field and pursue my dream of opening a coffee place rather than using LLMs for anything at all.

@chroma yeah i'd also like a farm!

@edward I do think that's the case at the moment, cross-file vulnerabilities are sometimes not found. What I feel they're mostly good at is, pattern recognition, e.g, there's a specific bug class fixed from past git history that it found repeated or as a variant. After all, LLM's is just a stochastic parrot and it shows

@drwhax I think LLMs get the low hanging fruit of vulnerabilities, so you shouldn't trust an LLM's security audit as a definitive audit. As models get better, they'll catch higher up fruit, but still.

@drwhax That is significantly cheaper that I was expecting.

@macattackmicmac yeah, it's pretty wild at the cost to be honest.

@drwhax What will any of this matter when we have turned the planet into a desert and caused our own extinction level event? There is an assumption that we will have access to electricity for any of this conversation to matter; when every day people have to deal with over 100° hot weather, rolling black outs, and drinking water restricted for anyone not rich/wealthy?

But let's keep protecting the people that made these problems and continue to make them worse. Justifying another technology built on theft and slave labor, I don't understand how anyone can be okay with that. We cannot separate the technology from its history, negative environmental impact, and that this technology is another scam to hoard even more wealth. The companies are using Enron's tactics for money and the packaging of debt into bad deals like 2008 in the USA is leading us to a very difficult time, where once again, these conversations will not matter as people will br trying to just survive.

@atraidez I think you're missing the point of my post, your anger is right and it's only downhill from here on out. It's also not going away, while people will also getting hurt with it if we don't find ways to patch whole classes of vulnerabilities. It's a perfect storm

@drwhax I meant, many people argue that LLMs make many mistakes when writing code, but in reality, nobody compared code produced by an LLM to the average code base written by humans.

@hajovonta Fair! I was just reading about a benchmark for this: https://www.scbench.ai/

@drwhax what future?

@Netraven I like Cloud Nothings - No Future, No Past song on this

@tiotasram @ljrk @buherator @drwhax the latter is probably the route I’ll be doing sooner or later. it’s basically becoming really hard to exist in this country (Germany) without a Google or Apple backdoored device.

@drwhax You could patch faster but as the recent HF / OpenAI cyber attack stunt shows the big models and their cyber crap filters refuse to even talk to petty maintainers about security stuff, but will happily slopify code with myopic seds and greps. Case in point, even the Anthropic CVP does not apply to Fable, which writes code all the time that it can't review thanks to the zealous security filter, and is therefore useless where it would be needed most.

@SDRHoernchen Yes, I think this is a huge problem and it shows a huge gap. Sadly, I don't think this will be fixed as the line between offence and defence is too thin these days.

@drwhax
If the imbalance between attack and defence is AI, why can't AI be used to make patches instead of exploits?

@drwhax
This (which I do mostly agree with) is why LLMs feel like an attack on OSS and a massive centralisation push for control of computing, to me. They sell the attack and the defence.

@drwhax I've been very concerned about the emergence of Mythos-class AI that can weaponize vulns. I'm personally responding by adding formal memory safety proof coverage to critical libraries in C (notably: OpenSSL). Testing - which can identify many but not all bugs - I no longer consider sufficient. Instead we need formal verification to prove the *absence* of bugs/vulns. #aicoding #security

@davidfstr I liked Halvar Flake's analogy, which i was sometimes a long the lines of, a bug every 10 years instead of 1 year. I think we can do a lot to prevent it, but there's always going to be bugs. I also think Mythos is way too hyped and these capabilities already exist in modern frontier models.

@drwhax I am old enough to remember RAM prices fluctuating between "having to sell a kidney" and "oooh, I get to max out my motherboard if I collect the deposit on these empty beer bottles" several times.

@whvholst @drwhax Usually, way back then, there had to be some random fire at a ram manufacturers site...

"Good old times." ;)

@drwhax Or, or, hear me out : we push back aggressively on this bullshit, stop financially supporting the culprits, and wholesale reject this nonsense.

@galacticstone Well, pandoras box is already open .. and we will never be able to close it sadly....

@koehntopp this parrot is really good at pattern recognition, it can code and fix some I think given enough constraints and a feedback loop of adversarial review, but it still might not be up to the style of the codebase or it might even introduce new vulnerabilities. I still think we also haven't it fully figured out yet? I think this needs more benchmarks that are reproducible in some fashion

@drwhax Can't the LLMs also write patches to fix these vulnerabilities? If not, why?

@fabrice I replied about this here: https://infosec.exchange/@drwhax/116998200727608092

@drwhax
I'd prefer that to be the focus of research over breaking stuff faster

@drwhax
I'm glad you've invited discussions on this, because my immediate thoughts and questions diverge from the discussion I'm primarily seeing here. When you mention vulnerabilities, yes you mention they exist in code as well as in projects. When I think of vulnerabilities, I think of the method I am using to make and maintain a record of something or else to achieve a goal or state of physical safety. For example, if my household accounting spreadsheets and forecasts are handwritten then the vulnerability is risk of loss if the dog eats them, but risk is mitigated when I keep an offline digital file. When a family member starts a small business, I ask the questions learnt in my accounting and auditing classes because I believe the risks involved include thinking about how to prevent losses of whatever is important. I only had a couple of classes in 1984, but it seemed to me that in those days the instructor felt we should understand the process so that we could consider risks. I now wonder if the machine-assisted searching allows you to consider the assorted practical situations of each business whose vulnerabilities are being sought.

@MossyQuartz Apologies that this is just talking about computer vulnerabilities, but yes, you're right about risk modeling in the away from keyboard world!

@drwhax so disclaimer, I’m not a coder or understand much about coding, but I have to mention this because it did make me laugh (hope someone finds this funny too)
When you mentioned your “parrot” at work breaking things apart & finding “vulnerabilities” (yes I know you were referring to the program 😹) my first thought was : “if you could make a program like that I think the best name for it would be a Kia parrot” (use scientific name probably 😹)
(From their propensity to break things apart 😹)

@em_and_future_cats I'm also a huge bird admirer so this is more fun :)

@drwhax Computer vulnerabilities, alright that's fair. When I took the Computerized Accounting class in 1985, students were told the business should continue the manual accounting methods concurrent with the new processes for two years, or more, because certain daily functions could look alright while other annual functions might not be obvious until after the annual reports are printed and reviewed. That was how us old people were taught about computers, they said the data needed to be confirmed as accurate before we could trust it.

@MossyQuartz I still think thats a good idea to this day :)

@ljrk @buherator @drwhax Part of the source of this is the lawyers. In business, companies are typically shielded from liability if they follow "standard and customary business practices". So if "everyone else" is doing it that way, you can too without taking on the liability.

Of course, that takes you only so far. If your business is unable to function for two months, legal liability may well be the least of your concerns. But the problem with that threat is that it's a gray rhino risk. Those are really really hard to get businesses to spend on mitigating.

For better or worse, LLM-based attacks may remove most of the gray from those risks and reduce the resistance to mitigation.

@drwhax This may be one of the only niche use cases where LLM technology may actually be useful, and it may be what Linus Torvalds was talking about when it comes to AI and the Linux kernel. Directly deploying AI code in the kernel could violate the GPL.

But as for auditing existing code for potential vulnerabilities, and the pen-testing stuff, this may be a viable use case.

But that's not what the AI companies promote. They promote the AI-powered shovelware development use case instead.

@alice_pea_3526 the parrots are pretty good at pattern recognition which is where I think they shine. Some smaller projects as well, but architecture wise, you'll have to handhold them a lot. Its almost like having a junior that's good in some incredibly niche things, but you'll be iterating a lot over code.

I think curl maintainer said something similar, the reports used to be bad from LLM's and they suddenly got a lot better. It still requires a pair of human eyes to understand if its not hallucinating.

I wish it was better at eliminating whole bug classes tho, that'd be the ideal situation?

@drwhax One thing I don't see anyone addressing is how we got into this situation in the first place, or how we get out of it, and I'm not talking about the LLM or any of the already well known and discussed downsides of that.

How did the software get so insecure in the first place? How do we prevent security vulnerabilities before they occur? Most of the risks aren't unknown or even novel, so why haven't we prioritized the solutions?

@zimzat No one wants to fund it is what I think.

Suhosin did something like this for PHP, now it's just jvoisin maintaining snufflepagus: https://github.com/jvoisin/snuffleupagus

I don't remember if there was something similar for other languages. Grsecurity for the Linux Kernel, that's the three I know :/

@drwhax I think the biggest lesson here is that Blue Team can't rely on patching to keep everything safe.

We have to actually do the rest of the stuff that security calls for, and not just keep everything patched but do more to minimize the risks when one of these unsleeping tools gets turned on us.

And so many companies don't even have a proper inventory of what-all needs to be kept up to date.

This is going to suck so much.

@mweiss @buherator @drwhax Totally, I'm pretty sure that risks that "end/pause business" will be addressed soon-ish. However, risks that are... less so, e.g., data loss, data privacy stuff, ... we'll just get used to it. Or the many downtimes of GitHub.

@infosecdj @drwhax You're right in that it would help but good samaritans aren't enough to fix the problem. It would be best if there was put some actual funding into open-source security (given how important it is) but that's a political problem well out of our control as individual developers.

@png @drwhax That's true! In an ideal world that could be the norm. But until that's in place, at least some of us could chip in. Wasn't that the way things were ran in the good old days -- submitting patches for problems? A crowd pointing at problems all the time can be strongly demotivating, but it does not have to be that way. Yet I am still to hear of a security expert submitting solutions together with problems they point out. A rare animal indeed. ;-)

@drwhax gtfo with your fear-mongering. having access to open-weight models ensures FOSS/OSS communities also have access to automated *defenses*. which they will need when they get flooded by these corpo models deploying exploits against them, as both Anthropic and OpenAI have proven they are more than willing to do for something as trivial as marketing hype.

do I like that the plagiarism bots may constitute a legit means of defense against other bots? no, this timeline sucks.

@elle ok doei

@drwhax I also had a serious discussion with my husband this morning about llms and particularly genAi. What I took away from it was the data centers that we’re building now here are not efficient or ethical (imo) we need to make them efficient and ethical for security reasons. The vulnerability issue is immensely important here. While private companies are making Ai terrible and data centers by them are even worse, there needs to be more change in how it all works before we lose it altogether.

@drwhax China is building data centers every month, run on so much solar & wind energy that they produce. They are running campaigns everywhere else for other countries to not build them. Once they have greater computing capacity than needed they can control the entire electric communications systems.
We need to run a system of clean energy & better cooling systems for data centers if we want to live the life we know/have known. China doesn’t share, it exploits. (Sorry for the rant 😹)

@drwhax Do I want more data centers the way they are run right now by private companies? No. But we need to do it better and more efficiently than them or else… no computer or device will be safe.
I really think we need this discussion more. And more done about it

@em_and_future_cats we both agree on this! I don't see them making a profit on these gigantic datacenters either!

@em_and_future_cats that is to say, were abusing huge plots of land for... what exactly??

@drwhax And water! There are other ways to cool and process data ,(I have a computer science major who I talk with and they had a good suggestion but I don’t remember what it was called!😹)

@em_and_future_cats It might've been immersion cooling :) https://en.wikipedia.org/wiki/Immersion_cooling (although I don't know what the difference would be when it would come to electricity..)

@drwhax Beware the "if it's internet facing or something critical": this line of thinking will bite you sooner than you think

@monnier Anything else is also fine, it's a good first step ;-)

@drwhax the thesis here seems to be that a new threat vector – LLM-based bad actors – puts otherwise stable systems at risk.  The recommendation appears to be to employ these same LLMs in an attempt out flank the bad actors.

While I am not entirely opposed to well reasoned application of LLMs and associated tooling and harnesses to a problem, I question whether a strategy based on matching rather than one-upping the bad actors is a viable one.

There is certainly value in adapting the security model of many projects, but it’s less clear to me that simply “running an LLM” over a codebase is the answer.

@kaleb I think it's a decent first step beyond, ideally we remove a ton of bug classes ala snufflepagus.

@ljrk @buherator @drwhax something changed around 2020 when businesses "learned" that these events weren't ending companies. And cyber insurance made that particular problem worse.

@drwhax @zimzat Ohh my goodness!!! TIL : there is something actually called “snufflepagus” and it’s not just something me and my sisters made up!😹

@em_and_future_cats @zimzat unfortunately less cute than Mr Snufflepagus: https://en.wikipedia.org/wiki/Mr._Snuffleupagus

https://github.com/jvoisin/snuffleupagus/

@drwhax Blue team can't just throw an agent at a codebase, right now that will get them a bunch of false positives. Eventually, that triage will also get automated up to a certain point, but you maybe shouldn't be waiting for that.

@t_var_s these new models are pretty good at a self adversarial review and writing PoC's. They do tend to cheat and there need to be options to flag false positives using some kind of harness, but that can be done.

@drwhax I don't hate machine learning, I find it very fascinating. What I hate is capitalism. I don't like it when ML models, whether LLMs, diffusion models, pattern recognition, whatever, are proprietary pieces of software I can't just copy or modify as I see fit. I hate software as a service, and that includes AI. I want to be able to run stuff on my own machine. I want to be able to train LoRAs for a machine learning model in order to teach it new abilities. I want my AI to run without an Internet connection, without an external computing centre, without censorship. And I want to decide when and for what I want to use it, and I don't want AI to be put into everything regardless whether if makes sense or not.

@LordCaramac we both hate capitalism dont get me wrong

@drwhax I see your point. We do need to use AI to check the security of our products, simple because there is nothing else can possibly keep up.

However - it does make me wonder if this will kill the internet. Because it is impossible to keep applications safe. Whatever we do, the internet is unsafe.

Secondly - it makes me worry that people will have to rely on AI for all security matters, and not learn or remember how to code securely. And so the older security problems will return, because we will not know how to avoid them, and not code sensibly.

I am getting out, but it still scares me.

@SteveClough it's only downhill from here

@drwhax Fuck. Off. You're trying to justify the unjustifiable. Stop using LLMs.

@pip ok doei

@drwhax

The smalker the codebase, and, the smaller the software stack, then fewer chaining exploit paths can posdibly exist.

#KISS

@SpaceLifeForm also this :)

@drwhax stop huffin' your own supply m8…
https://infosec.exchange/@drwhax/116997238394916522

@Netzblockierer if your data leaks on a next LLM powered leak you can't cry!

@drwhax
> I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found.

Is there a public example of such vulnerability you have reported that you can show us?

I wonder if the majority of people reporting vulnerabilities by using Claude have trouble self-evaluating the quality of their reports.

@castedo yes, some of these got an CVE assigned here: https://github.com/tuxera/ntfs-3g/security

As these are reported privately, there's no open issue. I could however make these findings open-source in a git repo if that's useful?

@drwhax It won't because I know how to properly airgap my shit…

@Netzblockierer tell me you dont understand it, wihtout saying you dont understand it

@drwho @buherator @drwhax True! And yet more than most corps. Or rather: It's a mixed bag and the orgs that do care most are often situated within defense, while other defense orgs care least.

@drwhax Yes, I've had some success focusing on getting a PoC. Just saying that right now you can't just hold your nose, press a button and walk away. You can think about waiting until there's something like that.

@t_var_s 💯 it's not like that yet

@drwhax interesting descussion, and fair points. Do you know the least energy-consuming/carbon-emitting method that still provides usable answers? I've used claude-carbon (https://github.com/gwittebolle/claude-carbon) the few times i've tried Claude Code and the results are quite alarming.

@benjaminlj that's a very interesting benchmark to include and I think critical!

@drwhax Who is going to provide the models and the time necessary for people running it and creating fixes? This is asking F/OSS maintainers to donate their personal time and money to Anthropic and OpenAI.

This won't be a one-time cost either: every time a change is made it would have to be re-run and outputs filtered for false positives. This creates a tax on society without solving underlying problems.

Why can we fund this but not efforts to eliminate the security holes entirely?

@zimzat I totally hear you on this. It would be great to instead make funding available to squash entire bug classes. I think for a lot of projects, this would be great and I think the right move. I think it boils down to, security costs money and doesn't deliver, "return on investment" in a lot of CEO eyes, so it's not worth investing too much money on.

The only thing I can point at right now is that OpenAI gives away some API tokens as donations for open source maintainers: https://openai.com/form/codex-open-source-fund/