End-to-end tests are brittle.
Any team that has maintained a large Playwright suite knows the drill. A UI redesign ships, three classes get renamed, and suddenly a dozen tests fail on selectors that no longer exist. The tests were right. The feature still works. But the suite is red, and someone has to go fix it.
AI test healing addresses that problem directly. But where AI belongs in the execution path matters enormously.
A UI redesign ships. Three classes get renamed. A dozen tests fail on selectors that no longer exist. The tests were right. The feature still works. But the suite is red, and someone has to go fix it.
This is the daily reality of maintaining large Playwright suites — and it erodes confidence in the entire test infrastructure over time.
When a selector fails, a healing layer examines the current DOM, infers the element you were targeting, and returns a new selector. It's a genuine improvement over maintaining selectors manually after every significant UI change.
Using AI healing is not really up for debate anymore. Where it belongs in the execution path is. Teams that get that wrong end up with a test suite that is harder to trust than the one they started with.
The appeal of AI-first test healing is obvious: let the model find every element, every time. No fragile selectors to maintain. But that framing trades one class of problem for a worse one.
LLMs are probabilistic. The same DOM, the same prompt, and the same model can return different selectors across runs. That's acceptable for a one-time fix. It's disqualifying as a primary test execution strategy. It turns green tests into coin flips.
If the element you're looking for genuinely doesn't exist because a feature was broken in a recent deploy, AI healing will still try to find something close. The test passes. The bug ships. That's the most dangerous failure mode: a green result that means nothing.
A test suite with 500 tests that each make an API call to an AI model will take dramatically longer to run. CI pipelines that used to finish in four minutes now take forty. Engineers stop running the full suite locally. Feedback loops collapse.
Trust is a test suite's most valuable property. When engineers know that tests sometimes pass because an AI found something close enough, they start discounting the green signal. Once that skepticism sets in, it is very hard to recover.
AI healing belongs in one place: the fallback layer. The structure is simple, and it should be enforced at the infrastructure level, not left to individual test authors.
This structure means that AI is engaged only when a human-authored selector has already broken. It is a signal that something needs attention, not a mechanism for hiding that signal. AI confirms what deterministic code cannot find — it doesn't decide what the test is looking for.
Key Principle: If the selector succeeds, the AI never runs. If it fails, pass the current DOM to the AI model. If the AI also fails, throw an error that tells an engineer exactly which test failed and why. Do not swallow the failure. Do not retry silently.
Take a standard Playwright test that breaks the moment a developer renames a class or restructures the checkout form. This test fails every time the UI team ships a styling refactor, even when the feature itself is fine. It creates noise, burns time, and erodes confidence in the suite.
// test/checkout.spec.ts (brittle version)
import { test, expect } from '@playwright/test';
test('user can complete checkout', async ({ page }) => {
await page.goto('/checkout');
// Breaks if the class changes, the element moves, or the designer
// wraps it in a new container
await page.click('.checkout-form__submit-btn--primary');
await page.fill('.checkout-form__email-input', 'user@example.com');
await page.fill('.checkout-form__card-number', '4111111111111111');
await expect(page.locator('.order-confirmation__heading')).toBeVisible();
});
Every class name in this test is a ticking time bomb. A single CSS refactor — with zero functional changes — will turn this entire suite red.
The healed version separates concerns cleanly. Stable selectors are tried first. AI is a fallback. Failures are explicit. The findElement helper in lib/healing-locator.ts encapsulates the entire fallback chain:
// lib/healing-locator.ts
import { Page, Locator } from '@playwright/test';
import { healSelector } from './ai-fallback';
import { readSelectorCache, writeSelectorCache } from './selector-cache';
export async function findElement(
page: Page,
intent: string,
primarySelectors: string[]
): Promise<Locator> {
// Step 1: Try each deterministic selector in priority order
for (const selector of primarySelectors) {
const locator = page.locator(selector);
if (await locator.count() > 0) {
return locator;
}
}
// Step 2: Check the cache for a previously healed selector
const cached = await readSelectorCache(intent);
if (cached) {
const cachedLocator = page.locator(cached);
if (await cachedLocator.count() > 0) {
return cachedLocator;
}
}
// Step 3: Ask the AI (pass the current DOM and the element description)
const dom = await page.content();
const healed = await healSelector(intent, dom);
if (!healed) {
throw new Error(
`[AI Healing] Could not locate element for intent: "${intent}"\n` +
`Tried selectors: ${primarySelectors.join(', ')}\n` +
`AI fallback returned no result. Update the selector in your test.`
);
}
// Step 4: Validate the healed selector actually finds something
const healedLocator = page.locator(healed);
if (await healedLocator.count() === 0) {
throw new Error(
`[AI Healing] AI returned selector "${healed}" for intent "${intent}" but it matched nothing.`
);
}
// Step 5: Cache the result so we don't call the AI again
await writeSelectorCache(intent, healed);
return healedLocator;
}
// test/checkout.spec.ts (healed version)
import { test, expect } from '@playwright/test';
import { findElement } from '../lib/healing-locator';
test('user can complete checkout', async ({ page }) => {
await page.goto('/checkout');
const submitBtn = await findElement(page, 'checkout-submit-button', [
'[data-testid="checkout-submit"]',
'[aria-label="Complete purchase"]',
'button[type="submit"]',
]);
const emailInput = await findElement(page, 'checkout-email-input', [
'[data-testid="checkout-email"]',
'input[name="email"]',
'input[type="email"]',
]);
await emailInput.fill('user@example.com');
await submitBtn.click();
// Assertions are NOT healed. If the confirmation heading is gone, that's a real bug.
await expect(page.locator('[data-testid="order-confirmation-heading"]')).toBeVisible();
});
The AI fallback layer sends a stripped-down DOM representation to the model and asks it to return a CSS selector. Keep the prompt tight and the DOM trimmed. Full-page HTML is noisy and expensive.
// lib/ai-fallback.ts
import Anthropic from '@anthropic-ai/sdk';
const client = new Anthropic();
export async function healSelector(
intent: string,
fullDom: string
): Promise<string | null> {
// Trim the DOM to reduce token usage (keep only interactive elements)
const trimmedDom = trimDomForHealing(fullDom);
const message = await client.messages.create({
model: 'claude-sonnet-4-6',
max_tokens: 256,
messages: [
{
role: 'user',
content: `You are a test automation assistant. Given the following HTML, return a single CSS selector that uniquely identifies the element described below.
Element description: ${intent}
Rules:
- Prefer data-testid attributes over class names
- Prefer ARIA attributes over positional selectors
- Return ONLY the CSS selector string, nothing else
- If you cannot identify a unique match, return the word NULL
HTML:
${trimmedDom}`,
},
],
});
const result =
message.content[0].type === 'text'
? message.content[0].text.trim()
: null;
return result === 'NULL' || !result ? null : result;
}
function trimDomForHealing(html: string): string {
// Strip scripts, styles, and deeply nested non-interactive content
// A real implementation would use a proper DOM parser; this is illustrative
return html
.replace(/<script[\s\S]*?<\/script>/gi, '')
.replace(/<style[\s\S]*?<\/style>/gi, '')
.replace(/\s{2,}/g, ' ')
.slice(0, 12000); // Hard cap to avoid runaway token costs
}
Model choice: claude-sonnet-4-6 is used here for its balance of speed and accuracy. The max_tokens: 256 cap keeps costs low — a CSS selector never needs more than a few dozen tokens.
Each AI call is slow and costs money. More importantly, if the same selector breaks across multiple test runs, you want one AI call to fix it, not one per run. A simple JSON cache handles this.
// lib/selector-cache.ts
import fs from 'fs/promises';
import path from 'path';
const CACHE_PATH = path.resolve(__dirname, '../.selector-cache.json');
type SelectorCache = Record<string, string>;
async function loadCache(): Promise<SelectorCache> {
try {
const raw = await fs.readFile(CACHE_PATH, 'utf-8');
return JSON.parse(raw);
} catch {
return {};
}
}
export async function readSelectorCache(intent: string): Promise<string | null> {
const cache = await loadCache();
return cache[intent] ?? null;
}
export async function writeSelectorCache(
intent: string,
selector: string
): Promise<void> {
const cache = await loadCache();
cache[intent] = selector;
await fs.writeFile(CACHE_PATH, JSON.stringify(cache, null, 2), 'utf-8');
}
Commit .selector-cache.json to version control. When the AI heals a selector, that fix is permanent, reviewable in the next PR, and won't trigger another LLM call on the next run.
If a cached selector stops matching (because the UI changed again), the cache miss falls through to the AI layer automatically. The old entry is overwritten with the new healed selector after validation.
AI healing is appropriate for navigation and interaction selectors. A few contexts call for disabling it entirely.
Never apply AI healing to assertion targets. If expect(locator).toBeVisible() fails because the element is absent, that is the failure you were testing for. Healing over it defeats the purpose of the test.
Consider a strict mode flag that disables healing entirely on main branch CI runs. Healing should catch regressions in feature branches, not silently paper over broken selectors that reach production. Gate the flag with an environment variable.
These tests validate structural properties of the DOM itself. An AI that rewrites selectors to find "something close" will produce meaningless results for axe-core assertions or Lighthouse audits.
If you run tests against Chromium, Firefox, and WebKit, AI healing adds latency and non-determinism to every browser variant. Heal on Chromium in development. Run strict selectors only in your full cross-browser suite.
Wrap the healing logic in a Playwright fixture and test authors get the behavior automatically, without touching how they write locators.
// lib/fixtures.ts
import { test as base, Page } from '@playwright/test';
import { findElement } from './healing-locator';
type HealingFixtures = {
healingPage: HealingPage;
};
class HealingPage {
constructor(private page: Page) {}
async find(intent: string, selectors: string[]) {
return findElement(this.page, intent, selectors);
}
// Delegate everything else to the underlying page
get locator() { return this.page.locator.bind(this.page); }
get goto() { return this.page.goto.bind(this.page); }
get waitForSelector() { return this.page.waitForSelector.bind(this.page); }
get content() { return this.page.content.bind(this.page); }
}
export const test = base.extend<HealingFixtures>({
healingPage: async ({ page }, use) => {
await use(new HealingPage(page));
},
});
export { expect } from '@playwright/test';
Import from lib/fixtures instead of @playwright/test. Standard locators work exactly as before:
// test/checkout.spec.ts (with fixture)
import { test, expect } from '../lib/fixtures';
test('user can complete checkout', async ({ healingPage }) => {
await healingPage.goto('/checkout');
const emailInput = await healingPage.find('checkout-email-input', [
'[data-testid="checkout-email"]',
'input[name="email"]',
]);
await emailInput.fill('user@example.com');
const submitBtn = await healingPage.find('checkout-submit-button', [
'[data-testid="checkout-submit"]',
'button[type="submit"]',
]);
await submitBtn.click();
// Standard assertion. No healing applied here.
await expect(healingPage.locator('[data-testid="order-confirmation-heading"]')).toBeVisible();
});
The fixture is also a natural enforcement point. Add a STRICT_SELECTORS environment check inside HealingPage.find() and the entire fallback layer is disabled in one line for situations where you need a fully deterministic run.
Scoped correctly, AI test healing changes the economics of E2E maintenance.
The cache ensures a single AI call fixes a broken selector permanently across all future runs.
Every failure path throws an explicit, descriptive error. No swallowed exceptions, no silent retries.
Deterministic selectors → cache → AI → validation → explicit error. Each step has a clear purpose.
A test suite that heals silently over real bugs provides false confidence. One that heals structural changes and still fails loudly on logic failures provides genuine confidence. The approach described here draws that line deliberately and enforces it in code.
Selectors break less often. When they do break, the fix is automatic and logged. Engineers spend less time playing whack-a-mole with CSS classes and more time writing coverage for new features. But the payoff only lands if the failure mode stays intact — and this architecture ensures it does.
As systems become more complex, maintaining trust in your test infrastructure becomes increasingly important. If your organization is looking to modernize quality engineering, reduce automation maintenance, or improve software delivery confidence, Vericence can help.
Contact us to start the conversation.