A friend linked to Jim Nielsenâs post about LLMs gaslighting/hallucinating Bryan Cantrill.
Jim relays what Bryanâs AI said in response, when called out on it:
What happened here is a classic AI âhallucinationâ. Because the phrase you shared used highly stylized, melodramatic language, âghoulish claimsâ, âstrike brazenly at the hearthâ, my system misidentified the tone as belonging to Sideshow Bob who is famous for speaking in exactly that kind of grandiloquent Shakespearean style.
This misidentification, i.e. conflating two separate concepts as âmeh close enough to be the same thingâ, is exactly what I see in my daily AI usage, driving agents while still reading the code.
My observation is that AI is very good at generalizing (using tokens to squint & find very useful insights / knowledge / intelligence) but generally bad at âspecializingâ that knowledge back into the artifacts itâs producing.
Itâs like a lossy translation, where they first generalize (necessary to their overall function), but then have a hard time âgoing backwardsâ, and generating output that doesnât sound awkward or esoteric.
Specifically, AI-written code frequently uses esoteric terminology, writes awkward comments, and over-engineers featuresâbut my suspicion is that, to the LLM, my âesoteric terminologyâ is actually tomato/tomato to its internal token speak.
If this naive assertion is right, then I think LLMs likely have a limit on their usefulness (I know, I know, this is ludicrous to suggest!) until they can start getting âre-specializationâ right.
Fwiw, I also think this generalization/specialization imbalance is why LLMs found programming so ripe for disruption: LLMs can âjust output over-generalized-incoherent shit while hill climbing to the desired outcomeâ (vibe coding) and it can be considered a good/acceptable outcome, because software generally has a very binary outcome: the code compiled yes/no, the tests passed yes/no, etc.
But Iâm not sure other domains (medicine? law?) will be as forgiving as programming, in that they probably donât have the same strict binary outcome, particularly one you can keep attempting over & over while hill climbing to success.
As a final disclaimer, this imbalance doesnât mean LLMsâ generalization is not useful! Itâs very useful because they are so fast & broadly trainedâit really is like cheating. But itâs why, personally, I feel like I have to keep following them around & iterating/re-specializing their output to not suck.
I asked AI itself about my ^ thesis 𤯠and it had four points, paraphrased:
-
It agrees with me that there is a âgapâ (of course it used that term đ ), which it articulated as âCan this model solve the problem?â (often yes) is different than âCan it solve the problem in a form I actually want to maintain?â
A model can preserve enough meaning to be functionally correct while discarding distinctions that matter to a human maintainer.
Sounds right.
-
It says âgeneralize vs. specializationâ is a useful metaphor but not necessarily how things workâmy observations could also be explained by training rewarding broadly acceptable answers and familiar patterns, to the detriment of potentially simpler local-specific answers.
-
It agrees that programming is particularly ripe for LLM disruption given its cheap, repeatable feedback, although pushes back somewhat, making the point: even if LLMs are good at hitting binary outcomes, it points out that âsoftware qualityâ is not actually a binary outcome.
Maintainability, security, whether the tests capture the actual requirement, and whether the abstraction fits are precisely the things those checks often miss.
Which is a good point, albeit the latest Anthropic eng blog posts assert theyâve been able to turn about everything into a metric.
-
My ânear peak usefulnessâ claim is very speculative. Agreed.
Its tldr of my post was:
LLMs are often better at producing a generally plausible solution than a locally appropriate one, and programmingâs automated feedback can conceal that gap.
Which is actually a pretty good synthesis.