How Generative and Agentic AI Shift Concern from Technical Debt to Cognitive Debt
The issue isn’t with the code itself, but with the understanding of the code.
That’s the difference between technical debt and cognitive debt.
Link tags: trust
50
David Chisnall (*Now with 50% more sarcasm!*): “I think this needs to be repeated…”
Machine learning is amazing if … the value of a correct answer is much higher than the cost of an incorrect answer.
Related to Laissez-faire Cognitive Debt:
And that’s where I start to get really annoyed by a lot of the LLM hype. It’s pushing machine-learning approaches into places where there are significant harms for sometimes giving the wrong answer. And it’s doing so while trying to outsource the liability to the customers who are using these machines in ways in which they are advertised as working. It’s great for translation! Unless a mistranslated word could kill a business deal or start a war. It’s great for summarisation! Unless missing a key point could cost you a load of money. It’s great for writing code! Unless a security vulnerability would cost you lost revenue or a copyright infringement lawsuit from having accidentally put something from the training set directly in your codebase in contravention of its license would kill your business. And so on. Lots of risks that are outsourced and liabilities that are passed directly to the user.
Laissez-faire Cognitive Debt – Smithery
I think of Cognitive Debt as ‘where we have the answers, but not the thinking that went into producing those answers’.
Lately, I have started noticing examples of not just where the debt is being accrued, but who then has the responsibility to pick it up and repay it.
Too often, an LLM doesn’t replace the need for thinking in a group setting, but simply creates more work for others.
Decentralizing quality || Matt Ström-Awn, designer-leader
I’ve personally struggled to implement a decentralized approach to quality in many of my teams. I believe in it from an academic standpoint, but in practice it works against the grain of every traditional management structure. Managers want ‘one neck to wring’ when things go wrong. Decentralized quality makes that impossible. So I’ve compromised, centralized, become the bottleneck I know slows things down. It’s easier to defend in meetings. But when I’ve managed to decentralize quality — most memorably when I was running a small agency and could write the org chart myself — I’ve been able to do some of the best work of my career.
Pluralistic: With Great Power Came No Responsibility (26 Feb 2025) – Pluralistic: Daily links from Cory Doctorow
The web is open, apps are closed. The majority of web users have installed an ad blocker (which is also a privacy blocker). But no one installs an ad blocker for an app, because it’s a felony to distribute that tool, because you have to reverse-engineer the app to make it. An app is just a website wrapped in enough IP so that the company that made it can send you to prison if you dare to modify it so that it serves your interests rather than theirs.
The future of the internet is likely smaller communities, with a focus on curated experiences | The Verge
Good news for the fediverse, the indie web, and community sites like The Session:
People are abandoning massive platforms in favor of tight-knit groups where trust and shared values flourish and content is at the core. The future of community building is in going back to the basics.
The Gist: AI, a talking dog for the 21st Century.
My main problem with AI is not that that it creates ugly, immoral, boring slop (which it does). Nor even that it disenfranchises artists and impoverishes workers, (though it does that too).
No, my main problem with AI is that its current pitch to the public is suffused with so much unsubstantiated bullshit, that I cannot banish from my thoughts the sight of a well-dressed man peddling a miraculous talking dog.
Also, trust:
They’ve also managed to muddy the waters of online information gathering to the point that that even if we scrubbed every trace of those hallucinations from the internet – a likely impossible task - the resulting lack of trust could never quite be purged. Imagine, if you will, the release of a car which was not only dangerous and unusable in and of itself, but which made people think twice before ever entering any car again, by any manufacturer, so long as they lived. How certain were you, five years ago, that an odd ingredient in an online recipe was merely an idiosyncratic choice by a quirky, or incompetent, chef, rather than a fatal addition by a robot? How certain are you now?
First Impressions of the Pixel 9 Pro | Whatever
At this point, it really does seem like “AI” is “bullshit you don’t need or is done better in other ways, but we’ve just spent literally billions on this so we really need you to use it, even though it’s nowhere as good as what we were already doing,” and everything else is just unsexy functionality that makes what you do marginally easier or better. I’m sorry we live in a world where enshittification is being marketed as The Hot And Sexy Thing, but just because we’re in that world, doesn’t mean you have to accept it.
Is Microsoft trying to commit suicide? - Charlie’s Diary
Recall undermines trust, and once an institution loses trust it’s really hard to regain it.
We Need To Rewild The Internet
Powerful metaphors in this piece by Maria Farrell and Robin Berjon on the Waldsterben of the internet:
Our online spaces are not ecosystems, though tech firms love that word. They’re plantations; highly concentrated and controlled environments, closer kin to the industrial farming of the cattle feedlot or battery chicken farms that madden the creatures trapped within.
We all know this. We see it each time we reach for our phones. But what most people have missed is how this concentration reaches deep into the internet’s infrastructure — the pipes and protocols, cables and networks, search engines and browsers. These structures determine how we build and use the internet, now and in the future.
AI Art is The New Stock Image
While some executives in Davos may get excited about its infinite possibilities this week, to a younger consumer AI Art is already ‘a bit cringe’.
Curation is the last best hope of intelligent discourse. — Joan Westenberg
The return of RSS and POSSE points to a revival of the personal website ecosystem that thrived in the early blog era. Writers, researchers, technologists and more are relaunching their independent homepages, complete with feeds, as both a public notebook and a channel for sharing insights. The personal website is the ultimate sovereign territory online, enabling creators to share content on their own terms.
I feel like Joan Westenberg has come up with the perfect tag line for personal websites (emphasis mine):
By passing high-quality, human-centric content through their own lens of discernment before syndicating it to social networks, these curators create islands of sanity amidst oceans of machine-generated content of questionable provenance.
DEF CON 31 - An Audacious Plan to Halt the Internet’s Ensh*ttification - Cory Doctorow - YouTube
A great rousing—practical—speech from Cory.
Introducing AI Help: Your Trusted Companion for Web Development | MDN Blog
As part of this pointless push, an “AI explain” button appeared on MDN articles. This terrible idea actually got pushed to production (bypassing the usual deploy steps) where it lasted less than a day.
You can read the havoc it wreaked in the short term. We’ll find out how much long-term damage it has done to trust in Mozilla and MDN.
This may be the worst use of a large language model I’ve seen since synthentic users (if you click that link, no it’s not a joke: “user research without the users” is what they’re actually proposing).
Google AMP: how Google tried to fix the web by taking it over - The Verge
AMP succeeded spectacularly. Then it failed. And to anyone looking for a reason not to trust the biggest company on the internet, AMP’s story contains all the evidence you’ll ever need.
This is a really good oral history of how AMP soured Google’s reputation.
Full disclosure: I’m briefly cited:
“When it suited them, it was open-source,” says Jeremy Keith, a web developer and a former member of AMP’s advisory council. “But whenever there were any questions about direction and control… it was Google’s.”
As an aside, this article contains a perfect description of the company cultures of Facebook, Apple, and Google:
“You meet with a Facebook person and you see in their eyes they’re psychotic,” says one media executive who’s dealt with all the major platforms. “The Apple person kind of listens but then does what it wants to do. The Google person honestly thinks what they’re doing is the best thing.”
Spot. On.
We need to tell people ChatGPT will lie to them, not debate linguistics
There’s a time for linguistics, and there’s a time for grabbing the general public by the shoulders and shouting “It lies! The computer lies to you! Don’t trust anything it says!”
s13e17: A Proposal for News Organization Mastodon Servers and More
When Dan wrote this a week ago, I thought it sounded very far-fetched. Now it sounds almost inevitable.
Trust • Robin Rendle
Robin adds a long-zoom perspective on my recent post:
I am extremely confident that pretty much any HTML I write today will render the same way in 50 years’ time. How confident am I that my CSS will work correctly? Mmmm…70%. Hand-written JavaScript? Way less, maybe 50%. A third-party service I install on a website or link to? 0% confident. Heck, I’m doubtful that any third-party service will survive until next year, let alone 50 years from now.