Hot Takes on AI Coding

This is post 2 of a 5-post series on the AI-native development stack I built over the last 5 months.#

the 5-month journey of building the workflow that created Parley.
reflection on how to leverage AI in software product development. (this post)
1. a sneak peek of couch.
couch and pair, the coding environment I use daily. (TKTK)
ariadne, the skills and binaries that supported my workflow. (TKTK)
and lastly, where to go from here. (TKTK)

In this post, my hot takes on how we leverage AI.#

What is AI good at, and not so good at? #

1. #

AI (LLM) has all the world’s knowledge. It’s trained with basically all the written and increasingly probably all the digital assets we have ever accumulated as a species. So often, for each specific domain, especially in domains where the operator is not an expert, AI will be more knowledgeable than you.#

2. #

General purpose LLM is good at small-scale logical consistency. By small-scale, I mean: short reasoning chains before grounding. Errors compound with every step taken without grounding: if each step is right with probability $p$ , a sequence of $n$ steps is right with $p^n$ . Note that no human can write perfect code without a compiler either. So I don’t think one-shotting a very deep sequence of actions is something that will be solved; grounding is what resets the chain.#

3. #

LLM doesn’t yet seem to have a good appreciation of the relative importance of different competing facts and concerns. You may say they lack the good intuition and “common sense” human experts have, for example, regarding system architecture, user ergonomics, and performance requirements etc. I used the term “general purpose LLM” here, as I don’t think it’s the case that such intuition is human only, as for example, AlphaGo/AlphaZero seems to have pretty good intuition in playing the game Go. It seems those intuitions are deep domain understanding, needing customized training. General purpose LLM still lacks such specialization. This on the other hand, seems to indicate if the economic value is there, then likely such more specialized “LLM” will emerge. Or they will emerge when “LLM” learning efficiency increases by orders of magnitude.#

4. #

Combining 1. and 2., they are pretty good at following rules and process, over a reasonable number of steps without definitive grounding. I think that’s why coding agents are performing so well.#

What does this mean for AI coding? #

5. #

Decompose system change into shorter steps. Improve performance by a) either providing definitive grounding after those short steps; or b) at least providing some guidance on the shape of some of the intermediate state. In AI coding, a) means compile and run tests often, so languages with a fast compile/test cycle will have an advantage. And b) means to have some process, like asking it to write a plan before coding, or to write the domain entities and states down.#

6. #

From point 4., make the software and product development process legible to AI and let it drive. The AI-operated company can be real, but errors compound and we need to build one step at a time.#

7. #

I do not believe that humans can or should review AI generated code. First of all, AI generates code fast, and how much it generates is largely a function of how many tokens you are willing to pay for. Humans will never keep up in this race. During the initial introduction of AI coding to a system, humans can still review, as they have accumulated enough knowledge, but this advantage will quickly be eroded as more code is generated by AI. Or think of newcomers, who won’t have such accumulated knowledge. If we insist that humans review code, then the review will become the bottleneck, forgoing a lot of the benefit of using AI to write code in the first place. Plus, in special domains, e.g. security, AI reviewers will likely outperform humans at finding arcane issues anyway.#

Review does keep a human’s mental model of the system alive, and that matters, but reading code is a very expensive way to get it. A much more efficient way is for the human to own the system’s invariants (the state model, the data model, the interface contracts, performance behaviors) and let the machine own the code achieving them. 10., 11. and 12. are how I check them in practice, still a work in progress.#

8. #

I do not believe in spec-driven development, popularized by tools like Spec Kit, Kiro and OpenSpec. All of them use human language to describe detailed inner component behavior. Human language is notoriously imprecise and I can’t help but feel that using such an imprecise language to describe all the details of system behavior misses the whole point: we may as well write code ourselves. Maybe this is a call for a new kind of programming language at the architectural level. The precise kind of spec is code: types, schemas, property tests. That’s where the invariants from 7. belong, so the machine can check them. This is not an encouragement for humans to write those more precise invariants either; the machine writes them well. Humans just need to understand a subset of important invariants.#

9. #

I don’t review machine generated specs in detail either, for similar reasons to 8. Those specs and engineering plans tend to be bloated. I view those plans as forming the corpus that influences how the machine eventually writes code, as some internal artifact not for human consumption, i.e. point 5.b).#

10. #

I identify issues in several different ways. The first is to talk about expected surface behavior, based on my understanding of how I want the system to behave, following a logical chain. It is a lot easier for humans to understand and reason about those surface behaviors than the detailed construction of underlying components. This can be viewed as a form of 5.b) as well.#

11. #

The second is to review select intermediate artifacts, such as the state model I ask AI to generate as part of its engineering plan. It’s a small table in that big plan markdown file. Quite often, I will notice that it is both over-engineered: there’re too many states, which I find hard to reason about; and also under-engineered: the design often leaves the system in a limbo state that the end user can’t recover from. The state is right on paper, but is rather useless in practice. I view this as the 5.b) check as well. Data like this that’s worth reviewing should likely be extracted and presented to the human in a better place. I think that’s going to be the new generation of software development process.#

12. #

The third is to smoke test. That’s the ultimate grounding step where the human is still the gatekeeper. Eating your own dog food becomes a lot more important. I frequently identify issues, both ergonomic and performance issues, and uncover quite immature design choices.#

13. #

I believe we should use AI to improve those manual checks I do in 10., 11., 12. as well, a sort of recursive improvement in how it codes. I think those improvements will come from harnesses and human/machine interface design. I also believe 3. (specialized architecture design models etc.) will arrive eventually; those are going to be the core model improvements.#

14. #

I believe we don’t leverage AI enough in 1. (world knowledge) when it comes to AI coding. If machines can code, they should be able to design sensible end-user flows directly with minimal human input. The lag is probably due to the lack of clear grounding. In coding you rely on compilers and tests; in product design we still lack good grounding for good taste.#

15. #

I don’t use MCP. MCP continues the walled garden of fragmented context. Your data stays in each vendor’s system, and AI only sees the slice that vendor’s MCP server exposes, one call at a time, rather than having everything locally where it can search across all of it. And you get to pay the toll every time. The higher cost may be justified, since agents might access the data inefficiently and a lot more often than traditional human-generated traffic, but the per-call access itself is an architectural limit of MCP. If you must use a vendor system, consider fully synchronizing its data into your own repository, where AI can search it like everything else. Given how easy it is to code custom internal workflows with AI coding, I believe organizations should invest in dedicated resources to start in-sourcing more traditional SaaS, making them flexibly accessible to internal AI systems. With SaaS, you also pay for 99 features you don’t use, and for the vendor’s on-call behind them. When you are small, you don’t care about compliance; you need a business first. When you are big enough, you can hire a proper SRE team to run it, again with the help of AI.#