Reposting a question for Ed Zitron, I'll forward responses. He asked on Bluesky and will get sub-Mastodon-tier answers:

"This is a serious question and I would be delighted if I only hear great things but, software engineers: both before and after LLMs, how often in your professional lives have you run into software engineers that seem completely useless or lacking in basic knowledge? I hope the answer is rarely"

@ludicity 80% (and I’m likely being generous) are absolutely useless, lacking in necessary skills and basic knowledge.

@haruki_zaemon Oh shit, I didn't know you were on Mastodon!

@ludicity online? incredible common, IRL? rarely have i met someone who doesn’t have a decent grasp of the tech they’re using

@nqd Online, I think I get exposed to way more random people. IRL, I was in a bubble of mostly incompetent people (it was huge) and now I'm in a bubble of mostly competent people (it's very small).

@ludicity I worked mostly at (pen)testing and have always been astonished how basics of basics were unclear for many people (e.g. "does this code run on the client or the server?"). My opinion in summary is that the general quality of sw engineering/ers declined since managers figured out they can bill by the hour instead of fulfillment under the guise of "agile" (see "I'm gonna write myself a new minivan this afternoon").

@ludicity For the record, I work at a software company that employs ~10k developers.

Before LLMs, I'd encounter such engineers a couple of times a month, but I interact with a lot of engineers, specifically the ones that need help or are new at the company or industry at large, so it's a selected sample. Even the most inexperienced ones are willing and able to learn with some guidance.

After LLMs, there's been a significant uptick, and these new ones are grossly incompetent, incurious, impatient, and behave like addicts if their supply of tokens is at all interrupted. If they run out of prompt credits, its an emergency because they claim they can't do any work at all. They can't even explain the architecture of what they are making anymore, and can't even file tickets or send emails without an LLM writing it for them, and they certainly lack in any kind of reading comprehension.

It's bleak and depressing, and makes me want to quit the industry altogether.

@ludicity
pre LLM: rarely in open source, often in corporate.

Now: likely in open source, mainly as security reporters who play copy&paste monkey with our project and their LLM. Cant say anything about corporate as I no longer experience that (thank the heavens).

@ludicity asking this question speaks inexperience loudly. Incompetence is widespread in all areas of life. Even before LLMs. Especially in enterprise.

@bagder I think it's the old Gel-Mann thing, where he has assumed that people in areas that aren't his own are probably real adults, because how else would the world keep working

My sweet summer Ed

@ludicity makes perfect sense. You could of course easily be mislead into believing this based on the fact that most of the world keeps working

@icing @ludicity Yes, also this! ^^^

My open-source peers are usually technically very sounds. There were some exceptions in the past, but I could count these on one hand.

Perhaps, if you do something out of the pure joy, it is hard to stay incompetent?

@ondrej @icing @ludicity lots of peeps these days do OSS as part of their job, not for fun. They found a bug or fixed something on behalf of their employer. Enterprise style. This allows the same set of incompetence, but perhaps at a lower frequency.

@buherator @ludicity I have run into security engineers a couple of times matching that description.

@bagder @ondrej @ludicity Contributing to open source means lots of people may see your code. I‘d assume that many take extra care because of this.

The worst engineer I‘ve met kept „their“ code always close to their chest. In „their“ branch/repository, etc. 💁🏻‍♂️

@ludicity i'm a bit worried about confirmation bias here, though of course incompetence has existed and will continue to exist. the difference between a competent and incompetent engineer isn't decided by the tools that they have access to but the time they choose / are afforded to develop competency and how well they have learned-to-learn.

that said, while there isn't a quantitative difference in incompetence engineers, there is a qualitative difference in incompetent engineering. expensive AI licenses move wealth from labour to capital and give management hacks a license to demand specific things from engineers at a specific rate. some of the heaviest AI users ive seen are the junior enggs and interns, and while they werent able to answer questions about what they wrote pre-LLMs either, now it's buried in an amount of noise and unaccountability that makes it hard to catch these pitfalls during code reviews.

LLMs dont make people incompetent the moment you touch them. they change the amount of code, plausibly functional code mind you, that you can create in a given amount of time. this reduces the amount of time seniors can spend in design, reviewing, and talent building, and hinders the processes that (sometimes) build competence out of incompetence. i'm not a full-time-hater of LLMs, but i do worry about the real damage they do to enterprise engineering processes moreso than the engineers themselves.

@sassdawe @ludicity 100%, I definitely not meant to piss on sw engineers in particular, median skill isn't great industry-wide.

@icing @bagder @ondrej @ludicity that ^^

i get to look at closed, "proprietary" code regularly and i am yet to see it match even the basic quality level of the average open source project.

plus, in many cases, it's more "plagiary" than proprietary.

so exactly what LLMs are good at.

@drikanis @ludicity Similar experience here. More and more people cannot function without an LLM prompt ready to answer to them, they totally lost any autonomy. If you ask anything to them, they will basically give you the output of their LLM, instead of formulating an answer by themselves, even when they know the answer. It’s pure cocaine.

@ludicity I would say for GCC, the difference is NOT pre-LLM vs post-LLM when it comes to software engineers that seems completely useless or lacking in basic knowledge.

In fact I would say the difference for GCC bug reports it would be when it became more common knowledge that there is undefined behavior in C/C++.

Since there is so much more things written about how signed integer overflow is undefined behavior and much more written about C/C++ aliasing rules; there have been much push back at their code having undefined behavior in it.

GCC seemly gets less and less bug reports that need to be closed as invalid for having undefined behavior in it. In the last 2 months, GCC has got around 3 or 4 that has had undefined behavior in it. Around 10 years ago, it would have been closer to 12 or so for a 2 month span.

These days my bug triaging is more about bug reports that have been already filed rather than invalid ones.
(been doing this for 20+ years now too so I have noticed trends like this).

@ludicity Among free software developers (a community I professionally deal with): almost never.

In corporate environments, working on enterprise software: constantly, all the time, always, everywhere. The exception was Google (~12 years ago) where everyone was pulling their weight and more; Google's problems are of another nature.

@ludicity

Uncommonly, both before and after LLMs.

I’ve generally been fortunate to work for companies that filter out people with low skill pretty well without being terrifying during the interview, and also for being on teams with mostly mid-level and higher developers/engineers.

The commonest “problem” behavior I’ve seen is people (at many levels of technical skill) having significant degrees of learned helplessness when confronted with problems outside their stronger skill sets. The developers I know mostly don’t use LLMs for coding or similar tasks, so I can’t really comment on “before vs. after” there.

@drikanis @ludicity it's a good time to become a carpenter. Being able to build a house from scratch seems to become more relevant these days.

@drikanis @ludicity

"they claim they can't do any work at all." Saying something like I can't do this terrifies me, as it says Im incompetent and should not be filling that position. Besides that this doesn't provide any information for others to give me help which I desperately need.

That's why I try to say what I want to acomplish, what I hove done, and what's the issue, and thanks to that half the time I get new ideas to check and maybe even I get to solve my problem.

@ludicity

We must welcome folks with no experience, and not deride them as being “useless”.

Lack of compassion and human engagement, and the capitalists dream of the 10x hero programmer got us into this mess.

It’s your job to develop your team. Train them. Believe in them. Support them.

It’s not a pissing contest.

@knowuh Sure, though we're talking about "Fifteen year veteran that doesn't use Git", not "Fresh grad that doesn't use Git". Like someone that is prima facie not worth their salary, and would surprise their manager if they understood how large the skill gap is.

@ludicity I'm pretty sure a bunch of my coworkers are incompetent and/or disinterested, myself included, but our engineering job is so braindead that it doesn't matter much.
Anyone can take some json event and send it off to a different event. I think many of us would count as incompetent in a less stupid environment. So the answer is "all the time, probably?"

But we also barely hired anyone since the LLM boom so maybe I'm lacking the floor comparison.

@ludicity engineer in a very large tech company here 🙋‍♂️

Like all big tech companies, we’ve got Copilot, Claude and Windsurf shoveled down our throats, with management monitoring the usage that engineers made of these AI tools - I guess to justify the huge expenses for those licenses, and lay off engineers who “didn’t adapt to the age of AI” (an “age” imposed upon us in the most top-down way imaginable).

I’m not an enemy of LLMs a priori. I have a background in AI engineering myself and these tools, when used responsibly, have actually boosted my productivity (especially when writing boilerplate code, initial scaffolding or well defined unit tests, and as long as I always review and refactor their output).

But I have huge concern when it comes to traditional knowledge transfer and human interactions in corporate environments.

There’s a lot of “soft knowledge” in companies that is best transfered through human interactions. Especially when you onboard less experienced engineers.

Nowadays those engineers are just given a Windsurf license and a couple of prompts and told “just LLM it” - like in the 2000s they’d hear “just Google it” or in the 2010 they’d hear “just StackOverflow it”. With the difference that this time there’s no room left for human exchange.

General-purpose LLMs often don’t have much domain-specific knowledge about specific quirks of a company’s tech stack. They don’t abide to good engineering practices either (they often simply provide a solution that, given a certain prompt, minimizes the number of output tokens while fulfilling the initial request). They don’t even correct the user if the initial request is nonsense, as they are trained to fullfil prompts rather than criticizing proposed approaches.

I have the impression that we’re forming a class of incompetent engineers here, and that once my generation is out or the workforce the quality of software will experience a sharp fall - unless we accept that coding in the future will be an activity solely performed, executed and corrected by machines, with no humans in the loop.

@ludicity

all. the.time. Both before and after LLMs.

Devs with minimal networking knowledge.

Web-devs with zero HTTP knowledge.

C coders writing buffer-overflows and failing proper malloc/free pairing

SQL injection vulnerabilities out the wazoo.

Interviewed folks who couldn't write a basic fizzbuzz loop (or similar Coding 101 style exercise) in languages they claimed to have expertise.

I lived through the years of "I can use Frontpage and FTP files up to a server, I must be a web-developer!" with no regard to semantic markup, accessibility, security, usability, etc.

@gumnos @ludicity I think there is certainly a negativity bias to contend with, at least I struggle with it.

It is easy to remember the glaring incompetence and forget those who range from
competent to just below excellent.

(I also try to remember when I have sucked as a wee developer too)

@jablkoziemne @drikanis @ludicity I claim that I "can't do any work at all" when a tool that I am *required* to use by my employer is broken (think source code control, the build system, whatever).

Some people might fall into this category if they work for one of those employers who *mandate* the use of LLMs to write code, and track usage, and penalise people who write code by hand.

In which case such a claim is fair enough.

(FTAOD: 🤣 😭 )

@drikanis They are not engineers.

@drikanis
Thanks for this comment. It made something click for me -- token limits for developers are probably a good thing. It's a built in signal to take a break and step back from their work for a minute. Do some planning, reading, or writing, or just go stretch your legs.
This problem is not specific to LLM-assisted programming. There's always a risk of tunnel vision in the desire to get more done, and failing to establish foundations and context for the product.

@jablkoziemne @ludicity

@adamr @drikanis @jablkoziemne @ludicity That's not the point being made.

@drikanis @ludicity It's notable that, if these useless meat sacks are in the US, someone likely either borrowed $250K or spent their own money for them to cheat their way to a degree using AI. Some days, I'm comforted by the fact that I'm old.

@Patrickoldhiker @drikanis @ludicity It remains a great fear of mine that those of us who went through education before LLMs, do not understand just how badly LLMs will affect education of young people.

I worry we will get an entire lost generation of people who will lack even the most fundamental skills in reading, writing and math. And no, sending and receiving a thousand SMS and WhatsApp messages a day, will not compensate.

@drikanis @ludicity

Tangential, I have noticed a trend with customer emails (wide spread, many multiples companies) that makes me believe more people are using LLMs to write reply emails & not reading at all.

there's a 'jje ne sais quoi' to not just them not answering questions but *how* they're not answering questions.

I can't put my finger on it, but it's tripping my spidy-sense / pattern recognition.

@wifwolf @drikanis @ludicity the students of mine that email me because they have just failed an exam in my course are *definitely* using an LLM to write their emails, because I know their English is not that flawless (I just read their exam). Credibility, meet toilet.

@ludicity Depends. Rarely professionally, but I did most of my hiring for most of my life and I’ve seen things you people wouldn’t believe during the interviews.

The worst people were exactly like LLM - stupid, loud and unable to admit they are wrong.

@ondrej @ludicity I used to ask a very open-ended interview question that I could keep making harder/more complicated basically forever, just to see if I could get a candidate to say, “I don’t know”. The ones that can’t say that, you don’t want to hire them, I’ve found.

I teach college electrical/computer engineering students programming, and they've completely lost problem-solving skills. like, actually! these people are going through my class to learn C, assembler, and how a computer's CPU works along with its peripherals, and these students are BEYOND cooked.

they, quite literally, cannot troubleshoot a simple C program without the aid of ChatGPT. they cannot read code. they cannot go through the motions of problem solving! I throw code up on the whiteboard, and two students whip out their phone to ask ChatGPT to transcribe the code (which, of course, it fails miserably at because it did not understand two-column formatting). "oh, help, my motor controller circuit doesn't work!" (the solutions are literally in the lab manual) "let me take a picture of a breadboard with a bunch of chips, wires, a microcontroller, power transistors, resistors, and capacitors and ask ChatGPT how to fix this!"

oh, the prerequisite to my class is a course on ARM assembler programming. I loved that class, because we used the oldskool ARM assembler as opposed to the GNU assembler, but also because the problems were fun to solve -- we had to write all manner of assembly programs, and there was one due every Tuesday and Thursday. unfortunately, the students in that class taking it in the post-LLM generation have done essentially none of the work themselves; the test averages are comparatively low, and cheating is RAMPANT.

in my class, I routinely get submissions for assignments that contain either total AI-generated slop, accidentally contain "as a large language model", use oscilloscope screenshot pictures I saw in spring 2024, or do not mirror the voice of the person who supposedly wrote the assignment.

so, how often do I encounter people that seem completely useless or lack basic knowledge? every day!

these people are seniors. they graduate 8 days after they give their final presentations.

@wec well that is fucking dire

@ludicity I wince imagining the responses you’ll get. But here’s mine.

Completely useless or lacking in basic knowledge? In an unexpected and undesirable way? Or in an expected and acceptable way? Every intern is this. Every junior engineer in their first role is this. Just like an apprentice in any trade. Every experienced developer who moves from a domain they know well into a domain they don’t know at all is this.

I assume there’s an implicit “when they’re not supposed to be” in the question. Or “and they’re unwilling or unable to improve enough” in the question. I’m just not sure the goal or the point of the question. I’ve run into lots of these folks over the years, but mostly in places where they’re acceptable and expected. If all is well, we bring on people who have good potential when they’re hired, but no skills, yet. And we remedy that through training and experience.

Like, someone comes in knowing language syntax and coding basics, but they have only worked in toy projects, never million-line systems that are 15 years old and the product of multiple acquisitions. They’re useless on day one in that kind of environment. But they learn (if all is well).

@paco What Ed is grappling with is the revelation that some (large) percentage of the engineering population, even before LLMs, were incapable of doing anything other than copying and pasting off StackOverflow.

He was assuming a lot of competence that isn't actually there in the general population.

(Which isn't really a software-specific thing -- I'm vaguely horrified whenever I hear about any domain from a high-level practitioner.)

@ludicity the software engineer who doesn't know anything has always been part of the industry, and will be so long as it pays well enough to attract people with no particular talent or passion for the work. Yesterday's "lmao I don't know anything, I just copy from stackoverflow" is today's LLM dipshit.

The guy who sat next to me left the company before a year was up. He couldn't do anything without the LLM. Couldn't write a document, couldn't program, couldn't search a text file for lines matching a string. One time he said he realized he wasn't learning anything about our code base by having the LLM do his work for him, so he started asking the LLM to summarize the code. Data point of one, but oh my god

@BestGirlGrace Ed asked this question because I informed him of the StackOverflow Guy. He didn't know this was the state of things pre-LLM.

@genehack
I had a supervisor do that to me in an interview... took me too long to figure out what they were doing, but I gave the right answer, eventually 🙄

Not my fault they were asking me questions in a subject I was versed in
@ondrej @ludicity

@ludicity I have run into it a lot during technical interviews both before and after LLMs.
I would not hire anyone who didn't do a white board coding exercise in front of me at this point.
I know for a fact that there's no reliable proctoring method.

@ludicity I've been in software engineering for 20 years and I haven't met any people like this yet, even by stretching the definition. Lots of completely unmotivated software engineers, lots with surprising and alarming holes in their knowledge, plenty of unreliable ones, but useless? Never

@ludicity Definitely agree. There's a huge range of skill out there. In my mind it's not how "useless" people are, but what the skill mix is. I think Silicon Valley has a distorting affect on outsiders' vision of the industry, where people associate some elite, workaholic expert with their idea of software developer. But it's like teaching: you have day-to-day teachers who teach in their communities, you have research professors, and you have fancy titled chairs of prestigious departments and labs.

I think a metaphor to education is possibly apt. If I say 'professional educator' you might think more toward the professor-type end of the spectrum, but the majority (by numbers) of the population of professional educators are day-to-day teachers seeing ordinary students.

So I don't like this label "useless" without some justification/reasoning.

@paco I was agnostic re: Willison during the first few posts, and am now convinced that he's slowly losing his mind, or at the very least very cynically cashing in on the hype for clout.