HanLHo. - Fractional Architect & Software Product Engineer - testing

Harness Engineering

2026-02-27T00:00:00+00:00

Today I heard the term “harness engineering” for the first time:

Harness engineering is the practice of building tooling, tests, and automation that let coding agents execute tasks safely and reliably. </blockquote>
If code is written more and more by LLMs, the focus seems to be shifting to creating guardrails so agents can validate their own work.
Heard in: The Pragmatic Engineer - Mitchell Hashimoto’s new way of writing code</a>

AI’s Opportunity: Pacing Control Loops with Development

2026-02-04T00:00:00+00:00

What caught my attention in the book Vibe Coding by Gene Kim and Steve Yegge is the idea that, as LLMs and coding agents change how we build software, control loops—tests, reviews, and other signals that tell you whether a change behaves as expected—should be faster and more integrated into development feedback loops than before. My intuition says this makes perfect sense.

For example, when there is a dedicated test stage or a QA role that tests after the fact, that role inevitably struggles to keep up with the speed of development. Over time, this makes it increasingly difficult to sustain a 'testing after the fact' organisation of quality.

So how do we solve this?

Some may think that introducing AI by implementing it at the test level after the fact, could be the solution. However, at the rate of development I see and read about, this approach will be hard to keep up with. One either has to accept not fully taking advantage of what AI can help development with, or rethink how testing is integrated into the development process.

Put bluntly: if AI lets you produce a feature in hours but the first meaningful acceptance signal only arrives days later in a separate stage, quality assurance become the bottleneck.

To me, the logical consequence is a stronger shift towards automated quality controls, including acceptance tests and code reviews at the least. I refer to acceptance tests here as writing executable specifications of expected behaviour (in domain language) before or alongside the code. This implies that testing has to move earlier in the development chain because of AI.

AI is an opportunity to start writing acceptance tests if you have not yet. It pushes us to invest time in strategic test design, testing against stable contracts, testing from a behavioural point of view, and isolating test descriptions from the actual implementation.

Put differently, the shift in development practices that LLMs are causing should inspire more adherence to testing best practices, not less. That is, if you want to keep on adding new features, fix and prevent bugs, and keep up the pace of development.

More broadly, to keep benefiting from AI over time, we should shift towards tightly coupled feedback loops embedded in everyday development. This is not limited to testing but also applies to, for example, reviews. In that sense, AI doesn’t remove quality practices; it raises the stakes if you don’t have them.

If testing 'shifts left', team structures must change as well. This evolution points towards smaller, more autonomous teams where testing, development, and feedback are inseparable rather than sequential.

AI presents us with an opportunity: not faster quality control after the fact, but to design systems, processes, and teams that make quality the fastest path forward, so control loops can keep up with the increasing pace of development loops.

On Building Reliable Software with LLMs

2026-01-27T00:00:00+00:00

This post captures my current thinking on how LLMs are impacting software development, particularly around software quality and engineering discipline.
My main observation: most of the best practices we've relied on for years are just as important—maybe even more so—in an LLM-assisted development environment. Working with LLMs requires more discipline and attention to fundamentals, not less.
When using LLMs, there is a heightened risk of losing understanding: of the problem domain, the underlying technology, and the implementation details. Code can become messy quickly without careful attention, review, and guidance. While this is certainly true, we didn't need LLMs for this to happen. Why else have so many projects failed historically? Why is technical debt a topic in most projects?
The critical difference with LLMs is the increased risk and temptation of velocity. We move too fast and skip the practices that help us maintain and change software in the future. Discipline and rigour</a> have become more important than ever.
These practices are becoming MORE crucial in an LLM-assisted workflow:

Codebase quality. This matters for LLM agents too, because they learn from existing code. A clean, well-organised codebase helps agents perform better; inconsistencies lead to poorer results. An LLM will mimic what's already there. </li>

Feedback loops and testing. If an LLM is helping you write code, you need reliable ways to verify it hasn't broken anything. A well-designed, automated test suite that's easy to extend and interpret helps maintain understanding and control of implemented functionality. </li>

Well-designed boundaries and contracts both within and outside your application. These allow you to constrain, shape, isolate, and test the work an LLM produces. </li>

Managing risk and technical debt. Be intentional and explicit about where you rely on LLMs and where you don't. Document these decisions. Maintain a technical debt log with risk assessments and timelines. </li>

Documentation of past decisions. Keep a history of architectural decisions through decision logs and ADRs, and ensure the LLM you're working with is aware of them. I've had LLMs point out inconsistencies in the codebase or flag how new change requests conflict with past decisions. </li> </ul>
Taken together, these practices are what make LLM-assisted development sustainable rather than brittle.
I don't think the skill gap in building and delivering software will ultimately be about prompt cleverness. LLM agents will be genuinely helpful tools, and working effectively with them will be an accelerator. However, as we rely on them more to write code—even when we review that code carefully—the most important work increasingly becomes the disciplined practice of boxing them in with testing, architecture, and contracts.
Avoid painting yourself into a corner. In an LLM-assisted workflow, that means being deliberate about where you let agents move fast, and where you slow them down with guardrails. LLMs make it easier to move fast, and easier to get stuck.

Using Code Coverage as a Check for Test Refactoring

2025-07-03T00:00:00+00:00

TLDR: Use code coverage reports to verify that test refactorings haven't accidentally changed what functionality you're testing. Same coverage percentage before and after refactoring gives confidence your tests still cover the same code paths.
If we refactor code, tests should cover us from breaking things. But what about refactoring tests? How do we know our tests are still testing the same functionality as before?
While test refactoring isn't widely discussed I found myself refactoring the tests of my applications whilst reviewing and researching testing approaches.
Why am I refactoring tests?</h2>
Experimentation with executable specifications</h3>
I often use Fluent Test DSLs to develop executable specifications. The DSL may change if it expresses the functionality more clearly or concisely. This might involve renaming a method, changing an assertion, or refactoring the DSL implementation. These changes are usually low risk, but it's helpful to verify we haven't accidentally missed any behaviour.
Optimize assertions</h3>
Sometimes a single test ends up testing too much and you start trimming the test to only validate the exact behaviour you intended to test, possibly aiming for a single clean assert</code>. This sometimes means creating multiple tests to cover the same functionality.
The opposite may also happen. You have too many tests that affect feedback time and you want to combine tests to improve performance. Not ideal but may be a pragmatic approach at times.
Consistent test naming</h3> Test naming can grow organically and become inconsistent over time. This happens especially when tests use sentence descriptions. Small changes pile up until your tests all follow slightly different naming styles.You might adopt Testing naming guidelines</a> for consistency. When you start applying them, you realise you haven't reviewed all tests together recently because you've been adding features one by one instead. During this naming cleanup, you often find other small improvements you want to make. Some tests may no longer seem relevant in the wider context. This leads to more small refactorings. AI-supported test refactoring</h2> You might want to use AI to help with these refactorings. Take the consistent test naming example above. I use AI agents for this task, asking them to review my current tests and align with testing guidelines</a>. I also ask the AI to check for small cleanup opportunities. An AI assistant can make a plan and list all the changes I need to implement. I then work through them one by one, often by copy-pasting. This isn't the most exciting work, so it would be easier if the AI could make these changes directly after I approve them. This raises a question: how do I know I'm still testing the same functionality without checking every line of code? Test renaming shouldn't change the test logic, but splitting or trimming tests might. You might even let an AI agent change code directly, which could lead to unintended changes. Verifying test refactorings with code coverage</h2> The first check after any refactoring is obvious: do the tests still compile and run? But how do we know the tests still cover the same functionality after refactoring? I use code coverage as a basic check for this. Code coverage shows which code lines your tests execute, and it can catch obvious mistakes during test refactoring. If I have exactly the same coverage percentage before and after refactoring (like 94.614%</code>), it suggests I haven't accidentally deleted entire tests or changed which code paths are exercised. This isn't code coverage's intended purpose, and it has clear limitations - it won't catch changes in test logic or assertions that still execute the same code lines. But it's a useful test for catching the most obvious refactoring mistakes. When to refactor tests</h2> I wanted to include the types of test refactorings that code coverage can help verify. These fall into two categories: individual test refactorings are usually easy to manage, but sometimes we need to, or should, refactor tests at the level of entire test suites. Refactor individual tests when: The test is too complex or difficult to understand </li> The test asserts too many behaviours at once </li> The test is too slow or resource-intensive </li> The test doesn't match the rest of the test suite style </li> The test is outdated or no longer relevant </li> </ul> Refactor entire test suites when: The test suite fails with almost every production code change </li> The test suite lacks consistency across tests </li> The test suite doesn't align with current testing guidelines </li> The test suite has become too large or complex to manage </li> The test framework or design needs upgrading </li> </ul> Conclusion</h2> Using code coverage as a basic check for test refactorings isn't comprehensive, but it's a simple technique that catches obvious mistakes. I'm curious how others approach this challenge. Do you have other methods for ensuring your tests haven't been unintentionally modified during refactoring? Thank you for reading, Hans
Test Naming Guidelines 2025-06-26T00:00:00+00:00 TLDR: Below are test naming guidelines that help me write consistent, clear test names. This is quick post with the Test Naming Guidelines I have been using to make my test names consistent across multiple projects. I use also these with LLMs when writing tests or refactoring multiple tests to this style. I used one of my personal projects as a basis to develop, refine and structure these guidelines with help from an LLM. Below are the guidelines. Format</h2> [subject]_should_[expected_behavior]_[optional_when_condition] </code></pre> Components</h2> Subject: The component, feature, or system under test Examples: user</code>, api</code>, filter</code>,time_report</code></li> </ul> Expected Behaviour: What should happen, described as an action or outcome Examples: return_success</code>, validate_input</code>, fail_with_error</code></li> </ul> Optional When Condition: Include only when necessary for clarity or disambiguation Format: when_[condition]</code> </li> Examples: when_input_valid</code>, when_user_authenticated</code> </li> </ul> Guidelines for When Conditions</h2> Include when_[condition]</code> if:</h3> Essential for understanding: The condition is crucial to know what's being tested </li> Multiple variants exist: Similar tests with different conditions need distinction </li> Specific circumstances: The behavior only occurs under particular conditions </li> </ul> Omit when_[condition]</code> for:</h3> Basic/default behavior: Standard functionality that doesn't require special conditions </li> Self-evident scenarios: Cases where the expected behavior already implies the context </li> Overly obvious conditions: When the condition adds no meaningful information </li> </ul> Examples</h2> Good (concise when appropriate): health_check_should_return_200 user_should_be_created </code></pre> Avoid (unnecessarily verbose): health_check_should_return_200_when_request_valid // "valid" is implied by 200 response user_should_be_created_when_data_provided // data is obviously needed </code></pre> Good (meaningful distinctions): user_should_login_successfully_when_credentials_valid user_should_be_rejected_when_credentials_invalid </code></pre> Principle</h2> Keep test names as short as possible while maintaining clarity. The when_</code> clause is a tool for disambiguation, not a mandatory requirement. Key Takeaways</h2> Architecture benefits from being grounded in implementation. Adding a Verification section to your ADRs creates feedback loops and improves their quality. The verification section serves as more than a checklist. It: Connects architecture with implementation </li> Provides a mechanism for feedback in both directions </li> Reveals when decisions aren't being acted upon </li> Makes ADRs more actionable and effective </li> </ul> This grounding helps prevent architectural decisions from remaining theoretical or becoming shelfware. Notes on TDD & DDD From the Ground Up Live Coding 2024-11-14T00:00:00+00:00 Some short notes from Chris Simon's Talk TDD & DDD From the Ground Up Live Coding</a> When choosing the right testing level, developers face an important trade-off. Higher-level tests provide better coverage for refactoring, but make it harder to pinpoint the exact location of failures. Finding the right balance is crucial for maintainable tests. Domain modelling through traditional entity diagrams can lead to hidden assumptions. Event storming, by contrast, helps create more explicit models that better reflect how systems change over time. This approach brings us closer to the actual domain and helps reveal important details we might otherwise miss. Notes on 🚀 TDD, Where Did It All Go Wrong 2024-11-14T00:00:00+00:00 My short notes on 🚀 TDD, Where Did It All Go Wrong</a>. Still a lot of testing wisdom in this talk (re-watched 6 years after publication). Focus on behaviour</h2> Behaviour should be your primary focus when writing tests. The need for a new test should arise from new behaviour or requirements. While TDD is often contrasted with BDD, Kent Beck's early work already emphasised the importance of focusing on behaviour. When writing tests, be mindful of coupling. Your tests should not all fail when you change your code. Write tests against public contracts where possible. In my experience, you can deviate from this rule if your unit under test has a stable internal contract. This approach leads to better 'refactorability'. For a practical demonstration, I recommend watching TDD & DDD From the Ground Up Live Coding by Chris Simon</a>. ATDD tools are not worth it if business is not actively participating</h2> While I have promoted the use of ATDD (Acceptance Test Driven Development) tools like Gherkin, I must agree, the burden of translating between natural language and code can be 'horrible', to the point where internal frameworks are implemented to manage this complexity. More importantly, the effort may not be worthwhile without an engaged business analyst or product owner. In my experience, business stakeholders rarely show interest in participating. While I've questioned whether I could have done more to encourage engagement, this talk confirms this is a common challenge. Though disappointing to acknowledge, as the practice appears promising on paper, this seems to be the reality we face. That said, the consistent style these tools promote can still be valuable in test writing. If you had success using Gherkin or similar tool I would be interested in learn how. Do not forget about Refactor</h2> The TDD-cycle is Red-Green-Refactor. First fix the problem, then improve the design. The central idea is to decouple thinking about the problem from thinking about the design. The Refactor step is where the design is improved and is an integral part of the cycle. This methodical approach leads to more maintainable code, contrasting with the approach of the 'Duct-tape Programmer' (this talk) or 'Tactical Tornado' (Ousterhout</a>) approach. When you can improve the design without changing tests, you have achieved good decoupling.

HanLHo. - Fractional Architect & Software Product Engineer - testing

Harness Engineering

AI’s Opportunity: Pacing Control Loops with Development

On Building Reliable Software with LLMs

Using Code Coverage as a Check for Test Refactoring

Test Naming Guidelines

Principle</h2> Keep test names as short as possible while maintaining clarity</strong>. The when_</code> clause is a tool for disambiguation, not a mandatory requirement.</p>

Notes on TDD & DDD From the Ground Up Live Coding

Notes on 🚀 TDD, Where Did It All Go Wrong

Principle</h2>
Keep test names as short as possible while maintaining clarity</strong>. The `when_</code> clause is a tool for disambiguation, not a mandatory requirement.</p>`