Windows automation · Advanced

An AI Agent Without Mouse Hijacking: Background Window Control on Windows

AI-агент без перехвата мыши: фоновое управление окнами в Windows
Temporary fallback cover; replace in editorial pass.
40 min read Updated November 19, 2026 By Agent Lab Journal

A computer-use agent that moves the physical mouse and activates windows is easy to demonstrate and difficult to share with a human operator. The agent can steal focus while someone is typing, move the pointer under a person’s hand, or click an unrelated application after a layout change.

This article develops a safer prototype for Windows: the agent sends narrowly scoped messages to an approved background window, refuses ambiguous targets, supports an explicit Arm mode, verifies the result after every action, and writes an audit log. The design is intentionally conservative. It is not a universal remote-control layer and it does not prove that an application is safe to automate.

The concrete problem

Consider a person working in a browser, terminal, or document editor while an agent performs a repetitive task in a separate desktop application. A conventional computer-use loop usually looks like this:

  1. capture the screen;
  2. move the system cursor to a coordinate;
  3. click or type;
  4. inspect the new screen;
  5. repeat.

That loop changes global state. It changes the physical cursor, the active window, and the destination of keyboard input. Even if each individual action is correct, the person and the agent are competing for the same interaction channel.

Background control changes the channel. Instead of asking Windows to activate a window and inject input into the foreground desktop, the prototype addresses a specific window handle and sends only the messages that the target control is documented or observed to accept. The human’s window focus, pointer position, and active application remain outside the agent’s control path.

What the prototype will and will not do

The prototype has five boundaries:

The prototype will not promise compatibility with every Windows application. It will not bypass permission boundaries, defeat secure desktop prompts, automate protected controls, or guarantee that a message was interpreted by the target application. Those limitations are central to the safety model.

Why background messages are different

Windows applications expose several possible automation surfaces. They are not interchangeable.

Win32 messages

Win32 applications communicate with controls through window messages. A text box may accept WM_SETTEXT; a button may respond to BM_CLICK; a list or custom control may implement its own protocol. The message is addressed to a window handle, or HWND, rather than to the screen coordinate under the mouse.

This is useful because it avoids foreground activation. It is also fragile because message behavior is control-specific. Sending WM_SETTEXT to a custom-rendered editor may do nothing, while sending BM_CLICK to the wrong child window may produce an unexpected result.

UI Automation

UI Automation exposes semantic elements and patterns such as Value, Invoke, Toggle, and Selection. It is usually a better fit when the application exposes a complete accessibility tree. It can still fail when elements are virtualized, custom-drawn, remote, elevated, or implemented with an incomplete provider.

Physical input

Physical mouse and keyboard injection is the broadest fallback and the worst default for shared use. It necessarily interacts with the foreground desktop. Treat it as a separate, visibly announced mode with stronger confirmation requirements, not as an invisible fallback when background control fails.

Threat model for a shared workstation

Assume the human and agent use the same interactive Windows session. The agent may receive malformed model output, stale screenshots, a wrong window title, or an instruction that is valid in one application state but unsafe in another.

The relevant failures are not limited to malicious behavior:

The design therefore treats a window handle as an ephemeral reference, not an identity. It resolves the target immediately before each action, checks its process and title, performs one action, then verifies an observable state change.

Project layout

Create a small .NET console project on a Windows test machine:

dotnet new console -n BackgroundControlPrototype
cd BackgroundControlPrototype

The prototype can begin with one source file and a configuration file:

BackgroundControlPrototype/
  Program.cs
  policy.json

Use a real test application that you control. A simple Win32 sample with a text field and button is preferable because you know which messages its controls accept. Do not begin with a password manager, financial application, administrator console, or any application containing personal data.

Policy before implementation

Keep policy outside the model prompt. The model may propose an action, but it must not define the set of applications or message types that are permitted.

{
  "allowedWindows": [
    {
      "processName": "SampleTarget",
      "titleContains": "Background Test",
      "allowedActions": [
        "setText",
        "invokeButton"
      ]
    }
  ],
  "armRequiredFor": [
    "invokeButton"
  ],
  "verificationTimeoutMs": 1500,
  "auditPath": "audit.jsonl"
}

The process name and title are only an initial filter. In a stronger implementation, add a signed executable identity, a known installation path, a parent-process constraint, or an operator-approved application identifier. A title substring alone is not a sufficient security boundary.

Resolve a window without activating it

The first implementation step is enumerating visible top-level windows, reading their process names and titles, and returning a candidate only when exactly one window matches.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Text;

internal static class WindowResolver
{
    private delegate bool EnumWindowsProc nint hwnd, nint lParam);

    [DllImport("user32.dll")]
    private static extern bool EnumWindows(EnumWindowsProc callback, nint lParam);

    [DllImport("user32.dll", CharSet = CharSet.Unicode)]
    private static extern int GetWindowText(nint hWnd, StringBuilder text, int count);

    [DllImport("user32.dll")]
    private static extern uint GetWindowThreadProcessId(nint hWnd, out uint processId);

    [DllImport("user32.dll")]
    private static extern bool IsWindowVisible(nint hWnd);

    public sealed record Candidate(nint Hwnd, string Title, string ProcessName, uint ProcessId);

    public static Candidate Resolve(string processName, string titleContains)
    {
        var matches = new List<Candidate>();

        EnumWindows((hwnd, _) =>
        {
            if (!IsWindowVisible(hwnd))
                return true;

            var titleBuffer = new StringBuilder(512);
            GetWindowText(hwnd, titleBuffer, titleBuffer.Capacity);

            if (!GetWindowThreadProcessId(hwnd, out var pid))
                return true;

            try
            {
                using var process = Process.GetProcessById((int)pid);
                var actualName = process.ProcessName;

                if (actualName.Equals(processName, StringComparison.OrdinalIgnoreCase) &&
                    titleBuffer.ToString().Contains(titleContains, StringComparison.OrdinalIgnoreCase))
                {
                    matches.Add(new Candidate(
                        hwnd,
                        titleBuffer.ToString(),
                        actualName,
                        pid));
                }
            }
            catch (ArgumentException)
            {
                // The process exited while the window list was being inspected.
            }

            return true;
        }, nint.Zero);

        return matches.Count switch
        {
            1 => matches[0],
            0 => throw new InvalidOperationException("No allowed window matched."),
            _ => throw new InvalidOperationException("More than one allowed window matched.")
        };
    }
}

The exact delegate declaration in a compilable project is:

private delegate bool EnumWindowsProc(nint hwnd, nint lParam);

The split-out declaration is intentional here because it makes the safety rule visible: zero matches and multiple matches are both failures. Do not silently choose the first result.

Find child controls by stable properties

A top-level window is not enough. An action must also identify the intended child control. For a controlled Win32 test application, a child control ID is preferable to a screen coordinate. If the application does not expose stable IDs, use UI Automation with an explicit control type, automation ID, and accessible name, then fail when the match is not unique.

For a small Win32 sample, a child enumeration helper can inspect class names and control IDs:

internal static class ChildFinder
{
    private delegate bool EnumChildProc(nint hwnd, nint lParam);

    [DllImport("user32.dll")]
    private static extern bool EnumChildWindows(nint parent, EnumChildProc callback, nint lParam);

    [DllImport("user32.dll")]
    private static extern int GetDlgCtrlID(nint hwnd);

    [DllImport("user32.dll", CharSet = CharSet.Unicode)]
    private static extern int GetClassName(nint hwnd, StringBuilder className, int maxCount);

    public static nint FindById(nint parent, int expectedId)
    {
        nint found = nint.Zero;
        var count = 0;

        EnumChildWindows(parent, (hwnd, _) =>
        {
            if (GetDlgCtrlID(hwnd) == expectedId)
            {
                found = hwnd;
                count++;
            }
            return true;
        }, nint.Zero);

        if (count != 1)
            throw new InvalidOperationException("Child control ID was not unique.");

        return found;
    }
}

Do not use a child ID from an untrusted model response. Put the ID in the application policy and associate it with a named action such as setText.

Implement a narrow message adapter

The adapter should expose intent-level operations and keep raw Windows message constants out of the model-facing interface. In this example, the model can request setText, but it cannot choose an arbitrary message number or target handle.

internal static class BackgroundActions
{
    private const uint WM_SETTEXT = 0x000C;
    private const uint BM_CLICK = 0x00F5;

    [DllImport("user32.dll", CharSet = CharSet.Unicode)]
    private static extern IntPtr SendMessage(
        nint hWnd,
        uint message,
        nint wParam,
        string? lParam);

    [DllImport("user32.dll")]
    private static extern bool IsWindow(nint hWnd);

    public static void SetText(nint control, string value)
    {
        if (!IsWindow(control))
            throw new InvalidOperationException("The target control no longer exists.");

        if (value.Length > 1000)
            throw new ArgumentOutOfRangeException(nameof(value));

        var result = SendMessage(control, WM_SETTEXT, nint.Zero, value);

        // For WM_SETTEXT, a nonzero result indicates success for the target control.
        if (result == nint.Zero)
            throw new InvalidOperationException("The control rejected WM_SETTEXT.");
    }

    public static void InvokeButton(nint button)
    {
        if (!IsWindow(button))
            throw new InvalidOperationException("The target button no longer exists.");

        SendMessage(button, BM_CLICK, nint.Zero, null);
    }
}

This is an intentionally small adapter. It does not call SetForegroundWindow, SetCursorPos, mouse_event, keybd_event, or a global keyboard hook. Those omissions are the point of the prototype.

For a production implementation, prefer SendMessageTimeout with a bounded timeout when the target application may block. A hung target should produce a failed action and an audit record, not an indefinitely waiting agent process.

Arm mode as a state machine

Arm mode should not be a Boolean hidden in a prompt. Treat it as an explicit state with an expiry and a scope. A useful minimum is:

internal sealed class ArmState
{
    private DateTimeOffset? expiresAt;
    private string? scope;

    public void Arm(string actionScope, TimeSpan duration)
    {
        if (duration <= TimeSpan.Zero || duration > TimeSpan.FromMinutes(10))
            throw new ArgumentOutOfRangeException(nameof(duration));

        scope = actionScope;
        expiresAt = DateTimeOffset.UtcNow.Add(duration);
    }

    public bool Allows(string actionName)
    {
        if (expiresAt is null || DateTimeOffset.UtcNow >= expiresAt)
            return false;

        return string.Equals(scope, actionName, StringComparison.Ordinal);
    }

    public void Disarm()
    {
        scope = null;
        expiresAt = null;
    }
}

The human approval event should be outside the model’s control. Examples include a local console command, a tray UI button, or a hardware confirmation step. The audit record should identify who or what initiated arming without storing credentials or personal data.

Use separate scopes for low-risk and consequential actions. Setting draft text may be allowed while disarmed; clicking “Send”, “Delete”, “Publish”, or “Apply” should require an explicit scope such as invokeButton:publish.

Post-action verification

A successful API return is not proof that the intended application state changed. Verification must observe the target state through an independent read path.

For text entry, read the control’s current value and compare it with an expected value. For a button, verify a state transition such as a status label changing, a row appearing, or a dialog opening with an expected title. Avoid verifying only that the target window still exists.

internal static class ControlReadback
{
    [DllImport("user32.dll", CharSet = CharSet.Unicode)]
    private static extern int GetWindowText(
        nint hWnd,
        StringBuilder text,
        int count);

    public static string ReadText(nint control)
    {
        var buffer = new StringBuilder(2048);
        GetWindowText(control, buffer, buffer.Capacity);
        return buffer.ToString();
    }

    public static bool WaitForText(
        nint control,
        string expected,
        TimeSpan timeout)
    {
        var deadline = DateTimeOffset.UtcNow + timeout;

        while (DateTimeOffset.UtcNow < deadline)
        {
            if (ReadText(control) == expected)
                return true;

            Thread.Sleep(50);
        }

        return false;
    }
}

For asynchronous applications, use bounded polling with a short interval and a clear timeout. If the verifier fails, do not automatically repeat a consequential action. The correct recovery may be inspection, operator review, or a separate idempotent retry policy.

Audit logging without secrets

Use newline-delimited JSON so each action is independently parseable. Log identifiers and hashes where possible, not document contents, passwords, tokens, or full screenshots.

using System.Text.Json;

internal sealed record AuditEvent(
    DateTimeOffset Timestamp,
    string Event,
    string Action,
    string ProcessName,
    uint ProcessId,
    string WindowTitle,
    string Result,
    string? Verification,
    string? ErrorCode);

internal static class AuditLog
{
    public static void Append(string path, AuditEvent item)
    {
        var line = JsonSerializer.Serialize(item);
        File.AppendAllText(path, line + Environment.NewLine);
    }
}

Useful events include window_resolved, action_requested, action_rejected, action_sent, verification_passed, verification_failed, arm_enabled, and arm_expired.

Keep the log append-only for the prototype. Restrict its file permissions, rotate it, and define a retention period before using it on a shared machine. If the log itself can contain sensitive titles or values, hash or redact those fields before writing them.

Put the pieces behind one guarded operation

The model-facing function should accept a small typed request. It should not accept arbitrary DLL names, message IDs, window handles, coordinates, or shell commands.

internal sealed record AgentAction(
    string Name,
    string? Text);

internal sealed class GuardedExecutor
{
    private readonly ArmState arm = new();

    public void Execute(AgentAction request)
    {
        var started = DateTimeOffset.UtcNow;

        try
        {
            var policy = LoadPolicyFor(request.Name);

            if (policy.RequiresArm && !arm.Allows(request.Name))
                throw new InvalidOperationException("Action requires Arm mode.");

            var window = WindowResolver.Resolve(
                policy.ProcessName,
                policy.TitleContains);

            var control = ChildFinder.FindById(
                window.Hwnd,
                policy.ControlId);

            if (request.Name == "setText")
            {
                BackgroundActions.SetText(control, request.Text ?? "");
                var verified = ControlReadback.WaitForText(
                    control,
                    request.Text ?? "",
                    TimeSpan.FromMilliseconds(policy.TimeoutMs));

                if (!verified)
                    throw new InvalidOperationException("Text verification failed.");
            }
            else if (request.Name == "invokeButton")
            {
                BackgroundActions.InvokeButton(control);

                // Replace this with an application-specific state predicate.
                if (!WaitForApplicationState(policy))
                    throw new InvalidOperationException("Button verification failed.");
            }
            else
            {
                throw new InvalidOperationException("Action is not in the allowlist.");
            }

            AuditLog.Append(policy.AuditPath, new AuditEvent(
                started,
                "action_completed",
                request.Name,
                window.ProcessName,
                window.ProcessId,
                window.Title,
                "success",
                "passed",
                null));
        }
        catch (Exception ex)
        {
            AuditLog.Append("audit.jsonl", new AuditEvent(
                started,
                "action_failed",
                request.Name,
                "unknown",
                0,
                "redacted",
                "failure",
                "not_verified",
                ex.GetType().Name));

            throw;
        }
    }

    private static dynamic LoadPolicyFor(string actionName) =>
        throw new NotImplementedException();

    private static bool WaitForApplicationState(dynamic policy) =>
        throw new NotImplementedException();
}

The example leaves policy loading and the application-specific predicate explicit because those are not generic implementation details. A useful verifier depends on the target application’s contract. Do not fill the gap with “sleep for one second and assume success.”

Reproducible verification plan

The following checks are a test plan, not reported results. Run them on a Windows test machine after completing the adapter and policy wiring.

  1. Open the approved test application and a separate text editor.
  2. Place the human cursor in the text editor and begin typing.
  3. Run a disarmed setText action against the test application.
  4. Confirm that the text editor remains focused and the physical cursor does not move.
  5. Inspect the target control and verify the expected text through readback.
  6. Attempt an unapproved action name and confirm that it is rejected before a Windows message is sent.
  7. Open a second window with the same title pattern and confirm that resolution fails as ambiguous.
  8. Close the target between resolution and execution and confirm that the action fails closed.
  9. Request invokeButton while disarmed and confirm that Arm mode is required.
  10. Arm only the named action for a short duration, execute it, and confirm that the arm state expires.
  11. Make the application ignore or delay the expected state change and confirm that verification fails without an automatic consequential retry.
  12. Review the JSONL records and confirm that they contain action metadata but no passwords, tokens, or full document contents.

For stronger evidence, add automated tests around policy evaluation, duplicate-window rejection, arm expiry, action-schema validation, redaction, and verifier timeouts. Add an integration test against a purpose-built sample application whose controls and state transitions are known.

Failure cases to design explicitly

Window title changes

Many applications put document names, connection states, or unsaved-change markers in the title. A title substring can remain useful as a first filter, but it should not be the only identity. Prefer a stable application identifier and inspect the process path when possible.

Elevated and unelevated processes

Windows integrity levels can prevent one process from interacting with another in expected ways. Do not “solve” a failed interaction by silently running the agent as administrator. Treat elevation as a separate deployment decision with a narrower policy and an explicit operator review.

Custom-rendered controls

Games, canvas-based interfaces, remote desktop clients, and many modern toolkits may not implement the standard Win32 message contract. A control that looks like a button may not respond to BM_CLICK. Use the application’s supported automation interface where available; otherwise record that the target is unsupported.

Modal dialogs

A save prompt or permission dialog can appear between resolution and action. The prototype should not search every visible window and guess what to click. Re-resolve only against an explicit dialog policy, and require a distinct action schema for that dialog.

Stale handles

An HWND can become invalid after a window closes. Check it immediately before use, but still assume a race can occur. The target application may destroy the control after the check and before the message is processed.

Messages that have side effects

Some messages trigger validation, navigation, network requests, or irreversible actions. Classify operations by consequence. “Background” describes how input is delivered; it does not make the resulting action harmless.

Security and privacy checklist

Where this approach fits

Background window control is a good fit for bounded, repetitive tasks in applications with stable control contracts: filling a test form, updating a local draft, triggering a non-destructive refresh, or exercising a legacy Win32 program in a lab.

It is a poor fit when the application offers a reliable API, when the interface is custom-rendered, when the task is highly consequential, or when the only available signal is a screenshot. In those cases, use the supported API or a human-confirmed foreground workflow with an explicit safety envelope.

The most important architectural decision is to keep the language model away from low-level authority. The model may choose among approved actions and provide bounded data. A deterministic local executor resolves the window, validates policy, sends one message, verifies the result, and records what happened.

Conclusion

An AI agent does not need to own the system cursor to operate a Windows application. A narrow background-control layer can preserve the person’s focus while still supporting useful automation, provided that window selection, control selection, permissions, verification, and logging are treated as first-class boundaries.

The prototype in this article is deliberately incomplete at the application-specific edges. That is where a safe implementation should remain explicit: the target application must define which controls are stable, which messages are supported, which state change proves success, and which actions require a human’s Arm-mode approval.

For related implementation patterns, see the guides and the glossary.

We publish what works for us—and implement the same solutions for your business. We design AI automation, Telegram bots, chats, and AI agents for real-world processes. Discuss your project →