People keep asking me: “So… are we in an AI bubble?”
My real answer: I’m feeling two things at once—excited and a little nervous.
- Excited, because I think we’re about to jump to a new model paradigm beyond today’s giant transformer + RL stacks.
- Nervous, because that shift will break a lot of assumptions (and a few business models).
Here’s what I think changes next:
1) Data: quality > quantity
We’ve spent years training on “the whole internet,” and a lot of it is noise. The next wave will be licensed, curated, high-quality datasets: think trusted archives, real world movements, specialized journals, proprietary corpora—data with real signal. That means smarter ingestion (active selection, filtering, feedback loops) instead of indiscriminate scraping. Teams that build repeatable pipelines for acquiring, clearing rights to, and continuously refreshing premium data will get compounding returns in model performance and defensibility.
2) Architecture: less blob, more brain
Fully connected nets are sample-hungry and great at memorizing patterns—not always at understanding. I’m betting on dynamic, sparsely connected graphs that form/prune connections on the fly. Sparse attention, routing, and targeted updates reduce wasted compute, improve generalization, and speed training. Practically, this looks like conditional computation (only the relevant subgraph fires) and structured memory (relationships persist where they help, vanish where they don’t).
3) Specialists > one big generalist
We’ll lean harder into orchestrated specialists—adjacent to Mixture-of-Experts and closer to the brain’s layout (vision, language, reasoning, affect). Different subnets can own distinct skills (retrieval, tool use, planning, multimodal perception) and coordinate through a lightweight controller. Upside: better accuracy, lower latency/cost per request, and easier lifecycle management (you can update a single expert without retraining the world).
4) Inference: engineered “aha” moments
Today’s systems mostly follow the shortest path to the next token. There’s room for purposeful exploration at inference—sampling “distant” regions of the network or alternative experts to spark novel connections. Think of it as controlled divergence (explore) with disciplined convergence (verify and select). Done well, this yields fewer bland answers, more useful creativity, and still keeps reliability via self-checks, tools, and retrieval.
So, bubble? I don’t think so. It feels more like a phase change.
#AI #GenerativeAI #MachineLearning #LLM #DataQuality #DataCuration #AIArchitecture #SparseComputing #MixtureOfExperts #VentureCapital