Show all articles

Why agent testing won't fix your visual UI

Manual testing, automated testing, and now teams of testing agents. Each step made QA faster. None of them can tell you whether your product still looks like your product.

By Grum

Testing has moved in clear steps. People clicked through the app by hand. Then scripts ran the same checks on every build. Now vendors pitch multi-agent testing: a planner agent, a test-writing agent, a runner, a bug sorter, and a release gate, all sharing context so no one has to coordinate them in Slack.

Adoption is moving fast. The ThinkSys QA Trends Report 2026 expects 80% of software teams to adopt AI-driven testing. Results are not keeping pace: S&P Global found that 42% of companies abandoned most of their AI initiatives in 2025, up from 17% the year before.

That is real progress for behavior. Does the form submit? Does checkout total correctly? Does the API return the right error? Agents are good at those questions because each one has a right answer.

Visual quality is different. A button with the wrong radius, a heading in the wrong weight, or a card with a hand-picked gray passes every functional test. Adding more agents to the test suite does not change that.

Manual, automated, agent: what each one fixed

Each step in testing solved the bottleneck the last one left behind.

Manual testing: a person checks the product
A tester can notice anything, including "this looks off." But it is slow, it depends on who is looking, and it does not keep up with daily releases.
Automated testing: scripts check the same thing every time
Scripts are fast and repeatable. They only check what someone wrote down, and someone still has to write, run, and maintain them.
Agent testing: agents plan, write, run, and sort tests
Agents take over the coordination. They read requirements, generate cases, retry flaky runs, and score a release. The human work shrinks to reviewing what they report.

Notice what each step optimizes: speed and coverage of checks that have a clear pass or fail. That is exactly the part of quality that visual issues sit outside of. Even the ThinkSys report, which is bullish on AI testing, notes that complex UX testing still needs human review.

Where agent testing stops

"Looks right" is not a test assertion

A test needs an expected result. For behavior, the requirement gives you one. For visuals, the expected result is a set of decisions your team made: this blue, this radius, this spacing, this button for the main action. If those decisions are not written down somewhere a tool can read, no agent can test against them. It can only guess.

Two notification settings pages side by side, each marked 14 of 14 tests passed. The left has the approved Save button. The right has a pill-shaped pink Save button with notes: pill radius instead of 8px, color not in the contract, extra shadow and weight.
Functional tests can't tell these two apart. Your design team can, at a glance.

Agents checking agents share the same blind spot

More and more UI is now written by coding agents. When another agent reviews that work, it starts from the same missing information. If the coding agent did not know your approved button, the testing agent does not either. Both can agree that a one-off button "looks fine."

Screenshot diffs flag too much and explain too little

Visual regression tools compare screenshots pixel by pixel. That catches change, not intent. Fonts, browsers, and hardware render slightly differently, which Playwright's own docs call out for its screenshot comparisons. A diff can tell you something moved. It cannot tell you whether the move was an approved redesign or a hardcoded color an agent invented.

Left: a screenshot diff with five red boxes labeled 1px shift, font smoothing, font smoothing, anti-aliasing, and button, with no way to tell which matters. Right: the same page with only the Save button outlined and a note reading Primary button changed, radius and color differ from the approved design, labeled 1 departure to review.
A pixel diff reports that something changed. A contract check names what changed and what it should have been.

The UI is not built the same way twice

Ask an agent for a settings page twice and you may get two different buttons. Both work. Both pass. Over a few weeks, the product collects small differences that no single test was ever going to fail.

What visual quality needs instead

Testing still matters. It just needs to sit next to four things that are about agreement, not only checking.

A loop around a central Design contract: 1 Determinism, build from approved pieces; 2 Visualization, see it on the page; 3 Review management, route it to a person; 4 Collaboration, decide together, which feeds back into the contract.
The four parts form a loop. Each decision the team makes goes back into the contract, so the next change starts from it.

1. Collaboration: agree on what "right" means

Designers, engineers, and product people need one shared answer to "which button is the main action?" and "which gray is the border?" That answer has to live where the code is written, so a person or an agent can read it before they build, not after someone complains.

2. Review management: route each departure to a person

Some differences are mistakes. Some are new ideas worth keeping. A team needs a clear path for both: fix it, approve it as an exception, or adopt it as a new pattern. And it needs to know which reviews are open, so a departure does not get lost in a chat thread.

3. Visualization: show the change, not just report it

"Line 42 uses a raw hex color" is hard to judge. Seeing the page with the change outlined is easy. Reviewers make faster, better calls when they look at the actual screen, next to the approved version.

4. Determinism: build the UI from fixed, named decisions

If every color, size, and button comes from a named token or an approved component, the same request produces the same result. That turns a fuzzy question ("does this look right?") into a clear one ("does this use the approved pieces?"), which a tool can actually check.

One button, four parts

Say an agent adds a "Save" button to a new settings page. Every functional test passes: the button saves. Here is how the four parts handle what the tests miss.

  1. Determinism. The agent reads the approved Primary button and its tokens before writing code, so the default path produces the right button.
  2. Check. If the agent overrides the radius anyway, a source check flags that line as a departure from the agreement.
  3. Visualization. The reviewer sees the page with the new button marked, next to the approved one.
  4. Review and collaboration. The team decides: change it back, approve it for this page, or adopt it as a new shared pattern. The decision is recorded, so the next agent knows the answer.

None of these steps replaces testing. They cover the part of quality that a test suite, with or without agents, was never designed to own.

How we handle it in Grum

Grum keeps a design contract for each repository: the approved tokens, buttons, and inputs, plus the rules for using them. Coding agents read it before they change the UI, and check their work for drift before they finish.

When something falls outside the contract, it goes to review with the page it appears on, so the team can see it and decide. Approved patterns become part of the contract, and the next change starts from them.

Four cards under the title Where UI review time goes. Visualization: see the change on the page. Determinism: only real departures. Review management: every review has an owner. Collaboration: decide once. Each card contrasts the old way with the new one.
The goal is not faster testing. It is less time hunting for what changed and more time deciding what to keep.

A source check has limits. It can show where code leaves the agreement. It does not prove a live page looks right, and it does not replace accessibility testing or product judgment. That is the point: visual quality is a team decision, and the tools should help the team make it, not pretend to make it for them.

Want to see Grum on your own code? Book a demo and we'll walk you through it.

Request early access
X Discord LinkedIn