Spent time evaluating ChatGPT, Claude and Claude Code. Having spent 3 months on and off they can be useful but reliably unreliable. I found myself spending more time optimizing for their failings than if I'd done the work myself. I've also noticed major degredation in the majors over the past two months compared to self-hosted equivalents.
I see value in self-hosted models, and in using tools like CC for scaffold and occasional point use but suspect value is below cost.

Early on I put Claude and ChatGPT on my phone, and I took them off a few days ago. I was using them heavily despite their faults, which I put down to slot machine app design. The issues were clearly aggravating me, so I removed them both. Am going back to building without them. Is it slower? For some items yes, for others no.
The value I had from ChatGPT was in travel planning, not code. The value I got from Claude/Claude Code was architectural, not code - and that value was questionable.

@stevelord interesting. We have ChatGPT access at work so I have this week started tentatively* having a go with it. So far it definitely seems to be better than web search** for answering questions about coding against a framework I'm refreshing/updating my knowledge of (Django specifically).

That said, its responses are pretty slow and often in the minute+ I am waiting I find my own solution anyway, which it then confirms, or close enough... though sometimes ChatGPT responds with something that I consider a bit better design than what I came up with.

I have found it much more accurate than Perplexity (free version). I have not used any others thus far, but we have access to Claude as well I gather.

I have yet to do anything particularly hardcore, like feed it the entire codebase and ask it to fill in functionality for example.

* with quite some concern about morality in doing so

** of course this is in the context that web search seems to have got so so so much worse

@yvan @stevelord imagine if there was some sort of a search engine that was actually good... Oh, right, we used to have a couple... /s

@nina_kali_nina @yvan @stevelord Try @kagihq ;)

@buherator @stevelord @kagihq @yvan my favourite search test now is "ZX Spectrum text editor with a spell checker": first relevant hit for Kagi is result #16.

Google's first hit is 4, though most other results are incorrect.

Bing's hit is 21 on the 3rd page.

Mojeek says there were very few hits, yet returns the right result on the 5th line.

Yandex.com returns a good hit on the first page at position 9.

I'm surprised myself how poorly kagi did on this search.

@nina_kali_nina @stevelord @kagihq @yvan Fair, but - aside of this being a single data point - IMO the power of Kagi (don't want to do a sales pitch maybe other have this too) is customization. For example verbatim search by default (for my user) helps a lot with my use cases, and I also find domain ranking especially useful for tailoring results.

@nina_kali_nina @kagihq @stevelord @yvan Did a test with my settings, is it Tasword you expect to find?

@yvan There are definitely useful applications for LLMs, as there are in ML in general. ChatGPT 5 has that slow-down problem really bad. I have tried perplexity but I found it comically bad for my use cases.

@buherator @stevelord @kagihq @yvan yep, though it is not on the search results, I call this cheating. Google and Bing AI give this result too, and I'm not trusting them either:)

@buherator @kagihq @nina_kali_nina @yvan To be fair this is a fairly standard type of search for me and Tasword isn't a bad answer (but spell checking was done with the separate TasSpell product)

@nina_kali_nina @stevelord @kagihq @yvan What line do you draw for cheating? TBF I don't know how ranking to the sidebar works here, but given the complexity of the problem I'm fine with any "cheats" as long as I get the results relevant to my query - my problem with adtech engines was that they actively worked against me.

@stevelord @buherator @kagihq @nina_kali_nina took a whole 1m43s for GTP-5 "Thinking" to come up with the answer... which felt like aaaaggggeeesss... though I guess its answer is in the right ballpark. (This is not something I know anything at all about!)

Meanwhile I found the first reference for Tasword was at the top of the second page of DDG results in a few seconds. BUT if I didn't know I was looking for Tasword I don't know how helpful that would have been.

I have heard of Kagi but have never used it. I suspect for a few reasons that most (if not all) normal "search" experiences are not as good as the 2000s Google halcyon days... and one problematic fundamental reason is the "Web" is so full of old crap now, the raw content was very different 20+ years ago. Certainly some way of ranking/filtering domains would help, I find timeboxing can be sometimes helpful too.

@stevelord @buherator @kagihq @nina_kali_nina I've signed up on Kagi to see how that goes for me, whilst I'm certainly not personally likely to pay for web search... if it turns out to be actually useful/productive I can probably get work to pay ;) Amused to see it actually lists Fediverse as one of the places you can select for where you've heard about it! (And that it is Fediverse rather than Mastodon gets a nod of approval.)

@yvan @buherator @kagihq @nina_kali_nina as an aside, Protext is really really really good. I still use it on Amiga.