· 11 min read

Thoughts on AI Software Development

#development #ai #tech

This is going to be a bit of a ramble through my journey of using LLMs, how I think they have changed software development and how I think developers need to adapt to this. LLMs and agentic coding represent a massive shift in how software is being developed and will be developed - and it’s something that developers will need to accept and adapt to. The LLMs are here, they’re released, they’re not going away.

My Journey

Starting Out

I was dismissive of generative AI and LLMs for software development for a long time. I was resisting using it and felt that it was never going to be as good as code developed by a human. Then at the end of January I decided to use Copilot to do something, there were a few reasons for this:

  1. I wanted to do this project. It’s an online free version of a Continuous Performance Test of the kind used to help diagnose ADHD.
  2. I didn’t want to actually implement anything myself (because the actual CPT I did was a horrific experience that I do not want to repeat ever)
  3. My personal GitHub account was part of my employer’s organisation and I had access to Copilot.
  4. Copilot was billing based on ‘Premium Requests’ and I’d seen a Reddit post on how to get days of work from a single prompt, so I thought it would be worth a try.

I was amazed at what it could do, the quality of what was produced, but it wasn’t perfect. I had to do a lot to get it to deliver what I wanted; but within a few days it was doing what I needed. But it showed potential and I decided to push it further and try more things.

Building Sandcastles

One of the things I immediately identified was the risk of using LLMs. They don’t reason or use intelligence. They don’t think things through, they have no concept of consequences. Anyone who has read scifi will know the classic tropes and stories of AI acting to achieve its goals through whatever means it has. Terry Pratchett even referenced this in his Discworld books with the golems who would happily continue a task regardless of anything else.

There were already stories of people using an LLM which inadvertently deleted files on their computer, or wiped a production database. It was obvious to me that I needed to sandbox the agents, from both disk access and the network. Prompt fatique is a thing - with agents continually asking for permissions, the user gets complacent and just allows it or allows all actions. I didn’t want to keep clicking allow, and I didn’t want to give agents full access; it was clear I needed a sandbox.

There’s a few projects out there. I tried docker’s, I tried claude-sandbox and a few others, but nothing actually achieved what I wanted. I ended up quickly writing my own docker container sandbox with a sidecar proxy for DNS interception and SNI TLS inspection for outbound HTTP. It was good and worked fine, but I could already see the next step - running this on my laptop was just annoying. I couldn’t close the lid, I couldn’t put it to sleep. There was only one option.

Running it on Someone Else’s Computer

I had a docker container and proxy setup, I have a decent homelab and infrastructure to run everything on - but just spinning up a Linux VM and running it? That’s not what I do! I spent 14 years working at a hosting company on provisioning systems for VMs and containers, I built control panels and interfaces for this. I knew what I wanted for this:

  • Docker based
  • Web UI
  • Support for multiple workspaces (containers, all isolated)
  • Full statistics and logging, including of agent sessions into Loki, Prometheus, etc. All using OpenTelemetry.
  • Web Consoles with Tabs for accessing the terminal
  • Agent chat within a workspace
  • The ability to deploy multiple hosts and spread workspaces across them
  • SSH access to workspaces
  • Able to integrate with Jetbrains IDEs
  • Port Forwarding, TLS Forwarding, with optional authentication
  • No sudo access within containers
  • All network access via a proxy with deny/allow list, DNS logging and filtering, TLS inspection (SNI only).

I used Claude Code to build it, and already started using some spec-driven development skills like Superpowers. It took a while but I eventually built Cloud Agent. It does all of this, and a few other things. It’s currently closed source, but if you want something similar - try using Coder Community Edition, it’s basically the same but better. It doesn’t do the networking sandboxing unfortunately.

This was game-changing for me and using AI, I could just have it do work remotely, all I needed was my web browser and a laptop. Things just kept going, but I was still very hands-on. I was still having Claude Code sessions running loops to implement tasks; but I wanted to use other models. I wanted Codex to do implementation or an openweight model, or anything. I wanted automation on reviewing and quality checks - above what simple hooks could do.

I’ve probably played too much Factorio

By this point I’d moved from Superpowers to Matt Pocock’s skills for spec driven development with AI. For those unfamiliar with it, in a session with AI, you discuss your requirements, it grills you for any questions until a shared understanding is reached. It can then generate a spec, small implementation tickets and then in separate sessions, you call /implement {id} to get the agent to autonomously implement it with a TDD approach. Repeat until done.

I’d already been using a basic task execution system at this point. Something that would just invoke the agent with a basic prompt, display tickets and show the dependencies, current status, etc. But I needed something a bit better. I wanted to support multiple harnesses and agents, git worktrees, integration with the Matt Pocock skills way of doing things.

I also wanted something that can do automatic review of all work done, manage the git lifecycle and essentially deliver everything from implementation through to delivering a completed feature. So I finally got round to building Harmonic.

So now all I have to do is define the spec and get the tickets to be created. The works is delivered, reviewed, tested and quality checked. I then just take a look at it, and if I’m happy - it gets merged to main and released.

That’s where I currently am, but what about the industry? What about the future of software development?

Meanwhile…

This also seems to be where the industry is going, the ‘Agentic Software Development Lifecycle’. A way of having a Software Factory that just turns out features and fixes with minimal human involvement.

To begin with, this requires you to have really good specifications and requirements. Strictly defined use-cases, user stories, success criteria. You also need really good tests and test coverage.

To make the code maintainable, you should have static analysis, linting, code formatting and use code coverage and quality metrics. This all needs to be automated, run at every point code is pushed and have a feedback cycle of fixing any problems with it.

Notice something about that?

Everything old is new again

None of this is new. This is all what we should have been doing for years. Uncle Bob’s Clean Code came out in 2008. Static analysis has been around for decades.

The same skills we built up as software developers over our careers are still relevant - if not more relevant today than ever before. If you want to know why Senior Developers and Principals are getting on well with AI development, it’s because they have the skills for this. They know how to write good specs, they know the value of good quality tests, static analysis, etc.

For me personally, I like solving challenges, designing systems, building solutions. Coding is 95% thinking about it and 5% writing out code. Using AI to do the coding is a huge benefit to me - I can now focus on the multiple projects I’ve wanted to do. I design systems, write the specs, specify the tech choices and then let the AI do the implementation.

‘An excellent suggestion with just 2 minor drawbacks…’

This all sounds great, but there’s a few problems here.

Junior Developers

This process works great for Senior Developers, we have the skills from a career of doing this. But what about junior developers? To get seniors, you need juniors and to bring them in.

Traditionally someone would start out, you’d throw them at the codebase, maybe give them some tickets to implement and they’d do coding while slowly picking up the architecture skills, how to write specs, learning the value of tests, etc. Now there’s AI doing this coding, so how do we get new people in and teach them this? How do we teach the next generation of developers?

My answer is probably that we start teaching these skills earlier, instead of relying on gradual learning and osmosis, we teach them explicitly. Their job becomes learning these skills instead of writing lines of code. They learn what good architecture is, what a good design decision is, what makes a good test.

There is one group of developers this doesn’t work for though. You’ve probably worked with them. They don’t care about personal development, they just want to write code, go home, come back into work, write code, go home, etc. They don’t really care much about architecture, design, testing, above what’s required of them. Some of these people have been the backbone of development teams - the person you can rely on to implement something. It’s these people who are probably going to suffer from the way this is going - and I don’t have any answers to it.

AI is not perfect

AI can make mistakes, it doesn’t understand, it doesn’t think. It’s not making mistakes because it doesn’t know what’s right OR wrong. It’s achieving a task. It has its context window and it has its training data.

This is why the quality gates are needed. This is why work needs to be small units of work with success criteria that can be measured. You also need to separate our review from implementation - just as developers are the worst at QA’ing their own code, agents are terrible at reviewing their own output.

You need to have skills in-place and workflows to stop AI writing overly complex solutions or commenting code to death, but that’s easy enough with an automated review step, even with an agent doing that review of another agent’s work.

The overall code quality itself is another thing people are concerned about. By design, LLM’s are about producing the next most likely token (with some minor variation due to random-ness and fingerprinting). This means they’ll tend to produce outputs that are the median of everything. They produce mediocre output.

You can see this all the time - if you are a great PHP developer and get AI to write PHP code, it will write worse code than you will. That’s expected and normal - you are above average and the AI has to be average for it to work.

Here’s the harsh truth though - in many cases, the hand-crafted better quality code isn’t needed. As long as it’s adequate and maintainable then it probably meets the need that businesses have - and that’s before considering the LLM is probably quicker than a developer would be at writing code.

For personal projects, passion projects, open-source, or code that does need to be above average? Generative AI is not going to do a good enough job, it can get the basics but it can’t do that final bit - not without a lot of guidance. It fundamentally can’t do that.

This is also why we can easily recognise AI slop design work. Humans doing design well enough to be paid will be above average, they produce above average work. It’s why generative AI for literature, music, content, art (ignoring the ‘is code art?’ argument) is never a great idea. It just leads to average, mediocre output - and that’s by design.

Where do we go from here?

We can’t un-invent LLMs. The genie is out of the bottle and with open-weight models, anyone can run an LLM and do agentic coding. It might need hardware, but it’s doable - it’s something we have to live with.

Do I hate the impact it’s having on the environment, career development, hardware costs (I just want to buy RAM!), creative industries? Yes, it’s horrible. Especially to creative people. LLM’s can’t come close to the quality of work by skilled artists, musicians, writers and developers. It’s just that with software development, it can be good enough.

If you aren’t using LLMs for coding as part of your job/career, you are going to end up being limited as the industry is just pushing further adoption. At this point you need to treat them like any other development tool that you’ve learnt to use.

If you’re just breaking into software development and starting out - learn the fundamentals. Pick up a copy of Clean Code, learn about SOLID, DRY, TDD, BDD, static analysis and architecture.

This is a fundamental shift in commercial software development, it’s happening and there’s nothing any of us can do to avoid it if we want to remain doing commercial software development.