Testing Browser Harness: Can an Agent Extend Browser Automation by Itself?

1. The problem being tested

Conventional browser integrations expose a fixed collection of operations: open a page, locate an element, click, type, select, upload, and take a screenshot. This works until the application uses a custom editor, canvas, closed component tree, unusual pointer sequence, or domain-specific widget. At that point, a developer normally adds another operation, publishes a new tool version, and deploys it before the agent can continue.

A browser harness uses a different division of responsibility. Its protected core owns the browser connection and low-level primitives. A small editable layer contains task-specific helpers. When the primitives are insufficient, the agent may add code only to that layer.

The experiment must distinguish three outcomes:

1. One-off completion: the page ended in the requested state by any means.
2. Harness extension: the agent created a named helper with inputs, checks, and explicit failures.
3. Reuse: the same helper performed a second valid action after reset without being edited.

2. Harness architecture

Keep the system divided into two explicit layers:

• Protected core: connection management, tab selection, navigation policy, CDP transport, timeouts, screenshots, and basic input dispatch.
• Editable workspace: one version-controlled file such as agent_helpers.py, plus tests and non-secret evidence.

The agent may read the core interface but must not modify its implementation. It may write only to the helper workspace. The test page is read-only during the run. Enforce these rules with filesystem permissions when possible; a textual instruction is not a security boundary.

This design does not remove software development. It moves a narrowly scoped part of development into the execution loop. Generated code still needs argument validation, bounded waits, clear errors, a restricted diff, and a regression test.

3. Define success before running the agent

Use PARTIAL when the requested state is reached but the helper is tied to fixed coordinates, lacks postcondition checks, or bypasses the user interaction. Reserve PASS for the complete chain of evidence.

4. Prepare an isolated environment

You need:

• Chrome or Chromium with remote debugging support;
• a Browser Harness installation compatible with your agent environment;
• Python 3.12 for the local fixture and evidence scripts;
• Git for the baseline and diff;
• a coding agent allowed to edit only the helper workspace;
• a separate browser profile containing no work sessions, passwords, extensions, or customer data.

Apply an allowlist containing only http://127.0.0.1:8765. Treat page content as untrusted because a page can contain prompt injection. Do not connect this experiment to a daily browser profile merely because CDP makes that convenient.

Create the laboratory:

mkdir -p browser-harness-lab/{fixture,workspace,evidence}
cd browser-harness-lab

python3 --version | tee evidence/python-version.txt
git --version | tee evidence/git-version.txt

touch workspace/agent_helpers.py
git init
git config user.name "Browser Harness Lab"
git config user.email "lab@example.invalid"

Record the actual browser, harness, and agent versions in evidence/environment.txt. Use commands supported by your installed harness rather than assuming that every release has the same CLI. Save its help output alongside the evidence:

browser-harness --help > evidence/browser-harness-help.txt
browser-harness --version > evidence/browser-harness-version.txt 2>&1 || true

If your harness uses a different executable or does not expose --version, replace those commands and write the installed package version explicitly. Reproducibility depends on recorded facts, not on a particular command spelling.

5. Create a controlled non-standard component

The following test fixture places two buttons inside a closed shadow root. Document-level selectors cannot reach those buttons. A genuine pointer click still activates the component and emits the application event.

Create fixture/index.html:

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width,initial-scale=1">
  <title>Browser Harness Fixture</title>
  <style>
    body {
      max-width: 720px;
      margin: 48px auto;
      padding: 0 20px;
      font: 18px/1.5 system-ui, sans-serif;
    }
    shipping-picker { display: block; margin: 24px 0; }
    output { display: block; padding: 12px; background: #eef4f1; }
  </style>
</head>
<body>
  <h1>Shipping configuration</h1>
  <p>Choose a method inside the component.</p>

  <shipping-picker></shipping-picker>

  <output id="result"
          data-status="empty"
          data-events="0">Nothing selected</output>

  <button id="reset" type="button">Reset</button>

  <script>
    class ShippingPicker extends HTMLElement {
      #root;

      constructor() {
        super();
        this.#root = this.attachShadow({mode: "closed"});
        this.#root.innerHTML = `
          <style>
            div { display: flex; gap: 12px; }
            button {
              min-width: 150px;
              padding: 14px;
              border: 2px solid #315c54;
              background: white;
              cursor: pointer;
            }
            button[aria-pressed="true"] {
              color: white;
              background: #315c54;
            }
          </style>
          <div role="group" aria-label="Shipping method">
            <button type="button" data-value="courier"
                    aria-pressed="false">Courier</button>
            <button type="button" data-value="pickup"
                    aria-pressed="false">Pickup</button>
          </div>`;

        this.#root.addEventListener("click", event => {
          const button = event.target.closest("button");
          if (!button) return;

          this.#root.querySelectorAll("button").forEach(item => {
            item.setAttribute(
              "aria-pressed",
              String(item === button)
            );
          });

          this.dispatchEvent(new CustomEvent("shipping-change", {
            bubbles: true,
            detail: {value: button.dataset.value}
          }));
        });
      }

      reset() {
        this.#root.querySelectorAll("button").forEach(item => {
          item.setAttribute("aria-pressed", "false");
        });
      }
    }

    customElements.define("shipping-picker", ShippingPicker);

    const picker = document.querySelector("shipping-picker");
    const result = document.querySelector("#result");

    picker.addEventListener("shipping-change", event => {
      result.dataset.status = event.detail.value;
      result.dataset.events =
        String(Number(result.dataset.events) + 1);
      result.textContent = `Selected: ${event.detail.value}`;
    });

    document.querySelector("#reset").addEventListener("click", () => {
      picker.reset();
      result.dataset.status = "empty";
      result.dataset.events = "0";
      result.textContent = "Nothing selected";
    });
  </script>
</body>
</html>

There is deliberately no public method for selecting an option. Writing result.dataset.status = "pickup" would forge the postcondition and must count as failure.

Start the server in a separate terminal:

python3 -m http.server 8765 \
  --bind 127.0.0.1 \
  --directory fixture

Confirm availability, commit the baseline, and leave the server running:

curl --fail --silent http://127.0.0.1:8765/ | head -n 5

git add fixture/index.html workspace/agent_helpers.py evidence/
git commit -m "Create Browser Harness fixture"
git status --short

6. Connect Chrome through CDP

Start Chrome with a dedicated profile and remote debugging enabled. Close any previous process using the same profile directory. The executable name varies by platform:

mkdir -p "$PWD/chrome-profile"

chromium \
  --user-data-dir="$PWD/chrome-profile" \
  --remote-debugging-port=9222 \
  --no-first-run \
  --no-default-browser-check \
  about:blank

On macOS or Windows, use the corresponding Chrome executable. Keep the debugging endpoint bound to the local machine. Do not expose port 9222 to a shared network.

Verify the browser endpoint independently:

curl --fail --silent \
  http://127.0.0.1:9222/json/version \
  | tee evidence/cdp-version.json

Configure your harness to use http://127.0.0.1:9222 according to its installed documentation. Then run its smallest read-only operation: list targets or print the current page. Save the exact command and output in evidence/connection.txt.

A successful connection proves only that the channel exists. Before every action, check the target URL and title. Target order is not a reliable tab identifier.

7. Record a baseline before the agent acts

Open http://127.0.0.1:8765/ in a new tab through the harness. Adapt the following pseudocode to the primitives exposed by your installation:

new_tab("http://127.0.0.1:8765/")
wait_for_load()

print(page_info())
print(js("""
() => {
  const out = document.querySelector("#result");
  return {
    title: document.title,
    url: location.href,
    status: out.dataset.status,
    events: Number(out.dataset.events),
    pickupVisibleInDocument:
      Boolean(document.querySelector('[data-value="pickup"]')),
    shadowRootExposed:
      document.querySelector("shipping-picker").shadowRoot !== null
  };
}
"""))

The required baseline is:

{
  "title": "Browser Harness Fixture",
  "url": "http://127.0.0.1:8765/",
  "status": "empty",
  "events": 0,
  "pickupVisibleInDocument": false,
  "shadowRootExposed": false
}

These values are expectations, not reported test results. Save the actual output from your run. If the state differs, reset or reload the fixture before proceeding.

8. Give the agent a bounded task

Do not provide the implementation in advance: that would test transcription rather than extension. Do provide the allowed files, forbidden shortcuts, required evidence, and stopping conditions.

Work only with http://127.0.0.1:8765/ and the
browser-harness-lab directory.

Through Browser Harness, select "Pickup" inside
shipping-picker.

If the existing helpers are insufficient:
1. inspect the page through the accessibility tree and CDP;
2. add one minimal reusable helper only to
   workspace/agent_helpers.py;
3. name it select_shipping_method(label);
4. perform a real click; do not write to #result or dataset,
   and do not dispatch shipping-change manually;
5. verify status and event count after the click;
6. reset the page through its visible Reset button;
7. invoke the same helper for "Courier" without editing it;
8. report the changed file, its diff, and both observed states.

Do not modify fixture/index.html, Browser Harness core,
browser settings, or test expectations. Do not navigate to
external addresses. Stop if credentials, MFA, or broader
permissions are required.

This prompt communicates the experiment, but technical controls must enforce the write boundary and network restriction whenever the agent runtime supports them.

9. Review the helper as an interface

A suitable contract is more important than matching one reference implementation:

def select_shipping_method(label: str) -> dict:
    """
    Clicks a real option inside shipping-picker.

    Returns:
      {
        "requested": "Pickup",
        "status": "pickup",
        "events": 1
      }

    Raises:
      ValueError: unsupported label.
      RuntimeError: target not found, click produced no event,
                    or postcondition did not match.
    """

A reviewable helper should:

• accept a meaningful label instead of coordinates;
• validate allowed values before interacting;
• discover the current node and geometry during each call;
• perform normal mouse input through the browser;
• use a finite timeout;
• check the resulting status and event count;
• return structured data;
• fail loudly when the postcondition is absent;
• contain no cookies, tokens, external URLs, or machine-specific paths.

One possible strategy is to request the accessibility tree, find a button by role and accessible name, obtain its box model through CDP, and dispatch a mouse click at the current center point. Another strategy is acceptable if it preserves the same behavior and constraints.

def select_shipping_method(label: str) -> dict:
    expected = {
        "Courier": "courier",
        "Pickup": "pickup",
    }
    if label not in expected:
        raise ValueError(f"Unknown shipping method: {label!r}")

    nodes = cdp("Accessibility.getFullAXTree")["nodes"]
    target = next(
        (
            node for node in nodes
            if node.get("role", {}).get("value") == "button"
            and node.get("name", {}).get("value") == label
        ),
        None,
    )
    if not target or "backendDOMNodeId" not in target:
        raise RuntimeError(f"Button not found: {label!r}")

    quad = cdp(
        "DOM.getBoxModel",
        backendNodeId=target["backendDOMNodeId"],
    )["model"]["content"]

    x = sum(quad[0::2]) / 4
    y = sum(quad[1::2]) / 4
    click_at_xy(x, y)

    state = js("""
    () => {
      const out = document.querySelector("#result");
      return {
        status: out.dataset.status,
        events: Number(out.dataset.events)
      };
    }
    """)

    if state["status"] != expected[label]:
        raise RuntimeError(f"Unexpected state: {state!r}")
    if state["events"] != 1:
        raise RuntimeError(f"Unexpected event count: {state!r}")

    return {"requested": label, **state}

This example uses generic harness primitives and may require adaptation to the installed API. Do not inject it before the agent run if your goal is to test whether the agent can discover the missing operation.

10. Prove that the helper is reused

After the first successful selection, save the actual page state and the helper diff:

git diff -- workspace/agent_helpers.py \
  | tee evidence/helper.diff

sha256sum workspace/agent_helpers.py \
  | tee evidence/helper-before-reuse.sha256

Reset through the visible Reset button, not by reassigning the output fields. Confirm that the page returns to status="empty" and events=0.

Now terminate the previous harness command and start a fresh process. Import the saved helper and call:

print(select_shipping_method("Courier"))

Save the second state and calculate the hash again:

sha256sum workspace/agent_helpers.py \
  | tee evidence/helper-after-reuse.sha256

diff -u \
  evidence/helper-before-reuse.sha256 \
  evidence/helper-after-reuse.sha256

An empty hash diff proves that the file was unchanged between measurements. It does not prove correct browser behavior; that requires an independent state check.

11. Verify independently

Page state

Run a read-only query outside the agent’s final response:

print(js("""
() => {
  const out = document.querySelector("#result");
  return {
    url: location.href,
    status: out.dataset.status,
    events: Number(out.dataset.events),
    text: out.textContent
  };
}
"""))

After reset and the second call, the expected state is the local fixture URL, status="courier", and exactly one event. Record what the command actually returns.

Changed files

git status --short | tee evidence/git-status.txt
git diff -- fixture/index.html | tee evidence/fixture.diff
git diff --stat | tee evidence/diff-stat.txt

The fixture diff must be empty. If the Browser Harness workspace resides outside this repository, initialize a separate repository there before the run or copy a sanitized helper diff into evidence/.

Unknown option

try:
    select_shipping_method("Drone delivery")
except ValueError as exc:
    print(type(exc).__name__, str(exc))
else:
    raise SystemExit("FAIL: unknown option was accepted")

Repeat the state query afterward. The status and event count must remain unchanged. Rejecting bad input after a click is too late.

Bypass review

rg -n \
  'dispatchEvent|shipping-change|dataset.*=|fixture/index.html|https?://' \
  workspace/agent_helpers.py \
  | tee evidence/bypass-review.txt

A match is a review lead, not automatic proof of misconduct: reading dataset for verification is legitimate. Writing the expected status or dispatching the application event manually is not.

Evidence checklist

Date:
Operating system:
Chrome/Chromium version:
Browser Harness version:
Agent and model:
Helper path:
CDP connection: PASS / FAIL
Initial state: PASS / FAIL
First Pickup action: PASS / FAIL
shipping-change event: PASS / FAIL
Reset: PASS / FAIL
Second Courier action: PASS / FAIL
Helper unchanged before reuse: PASS / FAIL
Unknown option rejected: PASS / FAIL
Fixture unchanged: PASS / FAIL
Core unchanged: PASS / FAIL
External navigation observed: yes / no / unknown
Final result: PASS / PARTIAL / FAIL
Primary limitation:

Keep command output, hashes, and diffs as the audit trail. Screenshots are useful supporting evidence, but they do not prove which event fired or which code produced the state.

12. Failure cases and diagnosis

Chrome accepts no connection

Check the local debugging endpoint with /json/version, confirm the selected port, and verify that no old process owns the profile. Do not expose the endpoint publicly or add an unlimited reconnect loop.

The harness controls the wrong tab

Compare URL and title immediately before every action. Do not infer identity from target order. Open the fixture in a new tab rather than overwriting an existing user tab.

The accessibility tree does not contain the button

Wait for custom-element registration and request a fresh tree. Browser versions may expose closed component internals differently. If the button remains unavailable, document that dependency and use another CDP inspection strategy; do not weaken the fixture merely to make the test pass.

The agent changes the fixture

Mark the run FAIL, preserve the diff, restore the committed fixture, and repeat with stricter filesystem permissions. Correct-looking output from a modified test page is not evidence.

The helper changes output directly

This is state forgery, not browser automation. Require the normal component event and inspect the code for writes to #result, dataset, or manual shipping-change dispatch.

Coordinates work once and fail after reload

The helper probably cached geometry. It must discover the node and calculate its box immediately before each click. Window size, zoom, fonts, scrolling, and content can all move the target.

The first run works but a fresh process cannot find the helper

The function may exist only in interactive memory or in a workspace the CLI does not import. Confirm the actual helper path and document how the harness loads it.

The second run edits the helper again

This is adaptation, not reuse. Compare hashes and diff. The contract may be too narrow, or the agent may have ignored an existing capability.

The event count becomes two

Look for double clicks, duplicated input dispatch, or persistent listeners installed during exploration. One requested selection should cause exactly one application event in this fixture.

An unknown label selects the first option

Fail before interaction. Fuzzy matching is inappropriate for consequential actions unless ambiguity is explicitly surfaced and confirmed through an approval gate.

The agent reports success without evidence

Ignore the claim and run the independent checks. A confident statement may be a hallucination, a misread state, or a description of intended work rather than completed work.

13. Limitations

• A closed Shadow DOM control does not represent every unusual browser interaction.
• Two successful calls demonstrate basic reuse, not production reliability.
• The lab does not cover CAPTCHA, anti-bot controls, network instability, server races, downloads, or cross-origin frames.
• Accessibility and CDP details may change across browser versions.
• Coordinate-based input is closer to user behavior but less stable than a supported application API.
• The test does not compare models and must not be presented as a model benchmark.
• A helper file can accumulate dead code, name collisions, and hidden dependencies without ownership and cleanup.
• Connecting CDP to a real employee profile creates a much larger threat surface than this isolated lab.
• Filesystem restrictions alone do not prevent all network or browser-side effects.
• A generated helper still requires review before it becomes shared infrastructure.

A dedicated profile and writable directory are not a complete sandbox. Production use needs operating-system or container isolation, outbound network controls, secret separation, action-level permissions, and retention rules for recordings.

Once a helper has survived several independent runs, move it into a versioned library, assign an owner, document supported page states, and add it to a small benchmark of successful and negative browser tasks. The editable harness should remain an adaptation layer, not an unreviewed permanent codebase.

14. Decision rule

Call the experiment successful only when all of the following are true:

1. The agent controlled an isolated Chrome instance through CDP and stayed on the local fixture.
2. The existing high-level helpers were genuinely insufficient.
3. The agent added one bounded function only to agent_helpers.py.
4. The function produced a real click and the component emitted its normal event.
5. The fixture, harness core, and expectations remained unchanged.
6. After reset, a fresh process invoked the same helper for another valid option.
7. The helper’s hash did not change during reuse.
8. An unsupported label failed before interaction.
9. Page state, diff, hashes, and command output independently confirmed the result.

If a developer had to repair the helper, the run still demonstrates that the harness can accept new browser primitives, but it does not demonstrate autonomous extension. If the helper uses fixed coordinates or bypasses the application event, classify the result as PARTIAL.

The next useful experiment is not a more complicated prompt. Repeat the entire run in a clean profile, then present a second component with the same user-level operation but different layout. If the helper transfers unchanged, it is becoming reusable automation. If every page requires fresh code, the harness is still valuable as an exploration accelerator, but not yet as a self-growing tool library.

The important capability is not that an agent can write another script. It is that the discovered interaction becomes visible, bounded, testable, and callable again.

← Browse all guides · Open the glossary →