LLMs and the Lessons We Still Haven't Learned
Key Takeaways
- The current AI hype cycle is focused on superficial, front-end chatbot features rather than deep, integrated utility.
- Many companies are implementing 'AI' poorly, often with features less effective than simpler, established techniques like vector semantic search.
- The true value of LLMs is in empowering backend processes and solving complex, context-aware problems rapidly, as demonstrated by an accessibility project improvement.
- LLMs are significantly boosting internal productivity and enabling the creation of previously unviable internal tools, alleviating engineering backlogs.
- The author suggests that while the latest models might not offer massive leaps over predecessors, the current technology is sufficient for widespread, impactful applications.
The author critiques the current AI hype cycle, asserting that many companies are implementing superficial 'AI' features, such as simple API calls wrapped in marketing, instead of integrating deep, meaningful functionality. They point to examples like Slack and Notion leading with 'AI' in their branding when core value propositions should remain primary, noting that simple solutions like semantic search are often superior to current offerings. The piece contrasts this with Jeff Bezos's 2016 observation that AI's best application is beneath the surface, driving critical functions like search ranking and fraud detection, rather than being a visible chatbot. A powerful illustration is provided from the author's work in accessibility, where an LLM replicated a year-long research project's success (55% accuracy) to achieve 82% accuracy in a weekend. While acknowledging that coding is not the hardest part of startups, the author sees a boom in internal tooling, where LLMs like Claude allow engineering managers to rapidly build previously backlogged projects, suggesting we are approaching the peak utility of the current LLM S-curve.
