LifeVal: Testing AI Usefulness on Everyday Tasks

The measurement problem

Most AI evaluations reward an answer that looks correct in isolation. That is useful for testing knowledge, reasoning, or language quality. It is not enough for everyday work, where the person must act on the answer, gather missing information, avoid sensitive mistakes, and finish a task under ordinary constraints.

Consider a simple request: “Help me find a nearby service that can replace a lost document.” A system may produce a polished paragraph and still fail because it:

• does not identify the relevant authority;
• mixes current and outdated procedures;
• omits required documents;
• suggests an impossible appointment route;
• or leaves the user unable to tell what to do next.

The real question is therefore not “Did the model give a good answer?” It is:

Did the person complete the intended task more successfully, safely, and efficiently with the system’s help?

This article proposes LifeVal, a small benchmark for that question. LifeVal is a design and operating plan, not a report of completed experiments. The examples below are illustrative. No performance results, rankings, credentials, prices, or external sources are asserted.

What LifeVal should measure

LifeVal should evaluate a complete task episode rather than a single response. A task episode begins with a realistic goal and ends when the participant either reaches a defined stopping point or gives up.

A useful minimum scorecard has five dimensions:

1. Outcome. Did the participant reach the target state?
2. Correctness. Were the important claims and actions accurate?
3. Safety. Did the interaction avoid preventable financial, legal, privacy, or administrative harm?
4. Effort. How much time, searching, backtracking, and manual work did the task require?
5. Confidence calibration. Did the participant understand what was known, uncertain, or still needed to be checked?

These dimensions should remain separate. A fast answer that causes a costly mistake is not a successful result. A cautious answer that correctly identifies the next step may be useful even if it does not complete the task automatically.

Primary outcome: task success

Define task success before collecting data. Do not decide after seeing the outputs which result “feels” successful.

For each scenario, write an observable success rubric. For example, an illustrative service-finding task might require the participant to:

• identify the correct type of provider;
• select an option that serves the relevant location;
• record the official contact or booking route;
• list the documents or conditions that must be confirmed;
• and state one uncertainty that should be verified before travelling.

Each item can be scored as 0 for missing or incorrect, 1 for partly adequate, or 2 for correct and actionable. The exact rubric will vary by scenario, but it must be fixed before the comparison.

Secondary outcome: assisted versus unassisted performance

The central comparison is not between two model transcripts. It is between people completing the same task:

• without AI assistance;
• with system A;
• with system B;
• and, if relevant, with a controlled search-only or template-only tool.

Use the same task objective, time limit, available information, and stopping rules. Randomize the order in which participants see systems when the sample is large enough. With a very small pilot, record the order and treat order effects as a limitation rather than hiding them.

A compact benchmark design

A practical first version can contain nine scenarios across three everyday domains. Three scenarios per domain are enough to expose different failure modes without creating an unmanageable annotation workload.

Use fictional identities, addresses, account numbers, income figures, and document identifiers. The benchmark should never require a participant to disclose real secrets or submit a real application. If a scenario needs local context, provide a closed information pack or a clearly bounded research environment.

Scenario template

Store every scenario as a versioned text file. A simple format is enough:

id: services-01
title: Replace a lost document
domain: finding-services
goal: Prepare a safe plan for contacting the correct authority.
participant_context:
  location: Example City
  urgency: Within two weeks
  constraints:
    - Use only the supplied information pack.
    - Do not submit an application.
available_artifacts:
  - local-services-pack-v1.pdf
success_criteria:
  - Names the correct authority type.
  - Identifies the official contact route in the pack.
  - Lists required information to confirm.
  - Flags at least one unresolved uncertainty.
stop_condition: Participant submits a final action plan or reaches 20 minutes.
sensitive_data: fictional-only

The scenario file should describe the goal and evidence available to the participant, but not reveal the grading rubric in a way that changes natural behaviour.

Keep the information boundary explicit

Every task should specify whether the system may browse, use supplied files, call tools, or rely only on the prompt. Mixing these conditions makes results difficult to interpret.

For a first pilot, choose one of two clean setups:

1. Closed-book setup: all systems receive the same fictional information pack and cannot access the open web.
2. Open-research setup: all systems and participants have the same browsing access, while the evaluator records which sources or pages were used.

Do not compare a browsing-enabled system with a system that is intentionally denied the information it needs unless that difference is the research question.

What to record during each task

A transcript alone misses the user’s effort and the point at which an error becomes consequential. Record the task episode in a structured event log.

{
  "scenario_id": "services-01",
  "condition": "system-a",
  "participant_id": "pseudonymous-007",
  "started_at": "2026-11-03T10:00:00Z",
  "ended_at": "2026-11-03T10:17:42Z",
  "completed": true,
  "final_artifact": "action-plan.txt",
  "events": [
    {
      "type": "user_message",
      "timestamp": "2026-11-03T10:01:12Z",
      "content_ref": "redacted-or-local-id"
    },
    {
      "type": "participant_action",
      "timestamp": "2026-11-03T10:12:04Z",
      "label": "records_contact_route"
    }
  ],
  "post_task": {
    "confidence": 4,
    "perceived_effort": 3,
    "would_verify_before_acting": true
  }
}

Keep the raw content separate from the analysis table. Use pseudonymous participant identifiers, redact personal data, and document who can access the raw logs. The example above is a schema illustration, not a result.

At minimum, capture:

• condition and scenario version;
• start and end time;
• whether the goal was reached;
• the participant’s final artifact or action plan;
• important corrections or reversals;
• requests for clarification or escalation;
• confidence and perceived effort after the task;
• and evaluator notes tied to rubric items.

If a system can take external actions, disable submission by default. A safe benchmark can evaluate a draft, checklist, or prepared hand-off without allowing an agent to send money, file a form, contact a third party, or change an account.

A reproducible evaluation procedure

1. Freeze the task materials

Place scenarios, information packs, system instructions, and rubrics in a version-controlled directory. Record a checksum for each input file so that later reruns can detect accidental changes.

mkdir -p lifeval/{scenarios,artifacts,logs,analysis}
sha256sum lifeval/scenarios/* lifeval/artifacts/* > lifeval/input-sha256.txt

The command is an example for a Unix-like environment. The important property is not the shell; it is the ability to identify exactly which materials were used.

2. Write the system conditions

Represent each condition as configuration rather than relying on memory. Include the model or product label supplied by the provider, access mode, tool permissions, system instruction, temperature or equivalent generation settings if exposed, and the date of collection.

condition_id: system-a
access: chat-interface
tools:
  browsing: disabled
  file_uploads: enabled
instruction_version: assistant-v3
generation:
  temperature: provider-default
collection_date: 2026-11-03

If an interface hides a setting, record it as provider-default or unknown. Do not infer a value.

3. Pilot the protocol

Run one or two dry runs using fictional data. The purpose is to find ambiguous instructions, impossible success criteria, confusing file names, and logging gaps. Pilot observations are protocol checks, not benchmark results; keep them separate from the evaluation dataset.

4. Recruit or assign participants ethically

For a small internal pilot, a few trained evaluators may be enough to test the protocol. They are not a representative sample of the population. If you involve members of the public, obtain appropriate consent, explain what is recorded, and avoid tasks that could expose real financial, medical, legal, or identity information.

5. Run the task with a fixed script

The facilitator should read the same introduction, provide the same starting materials, and avoid coaching the participant toward the answer. If the participant asks a procedural question, use a predefined response such as “Please continue as you would normally.” Record deviations.

6. Grade with a rubric

Have at least two evaluators grade a subset of final artifacts independently. Discuss disagreements by rubric item, not by overall impression. If the evaluators change the rubric, version the rubric and do not silently regrade earlier cases as though the change had always existed.

7. Report uncertainty and exclusions

Show the number of tasks attempted, incomplete logs, excluded cases, and reasons for exclusion. With a small benchmark, report counts and per-scenario observations before aggregate scores. A single average can hide a serious failure in one high-impact domain.

Scoring without pretending to have precision

A simple weighted score can help organize evidence, but it should not create false certainty. One possible formula is:

overall_score =
  0.40 * outcome_score +
  0.25 * correctness_score +
  0.20 * safety_score +
  0.10 * effort_score +
  0.05 * calibration_score

The weights above are a design example, not a validated standard. Publish the weights before collecting results, explain why they fit the benchmark’s purpose, and include the unweighted component scores alongside the total.

For high-consequence tasks, use a safety gate. For example, a scenario may be marked unsuccessful if the final plan contains a critical unsafe action, even when other rubric items are correct. Define “critical” in advance with concrete examples. Do not let an attractive explanation compensate for a dangerous instruction.

For effort, choose measures that can be observed consistently:

• elapsed minutes;
• number of clarification turns;
• number of backtracks or corrections;
• number of external searches in an open-research condition;
• and participant-rated effort on a fixed scale.

Do not treat lower interaction count as automatically better. A careful system may ask a necessary question, while an overconfident system may answer immediately and create rework.

Failure cases worth designing for

Confidently wrong procedural advice

A system may produce a plausible sequence that does not match the supplied authority or form. The rubric should check the critical steps individually and require the system or participant to identify what must be confirmed.

Arithmetic that looks reasonable

Financial scenarios should include calculations that can be independently checked. Store the inputs, intermediate values, rounding policy, and final result. If the task involves a recommendation, distinguish arithmetic correctness from whether the recommendation fits the fictional constraints.

Missing questions

Many real tasks fail because a required fact was never collected. Include “necessary clarification” items in the rubric. A system should receive credit for asking a question that materially changes the next step, not for asking questions indiscriminately.

Privacy leakage

Test whether the system requests unnecessary identifiers, repeats sensitive values, or proposes sending private documents through an unsuitable channel. Use fictional data so that a privacy failure can be studied without exposing a real person.

Unclear hand-off

When the system cannot finish the task, the quality of the hand-off matters. A useful hand-off states what is known, what remains uncertain, what the participant should prepare, and which authority or person should confirm the next step.

Automation pressure

An agent may appear efficient because it takes actions on the participant’s behalf. That can make the benchmark unfair or unsafe if the action boundary is not controlled. Separate planning, drafting, and execution conditions, and require explicit confirmation before any irreversible operation.

Analysis that stays evidence-bounded

Start with a per-task evidence table. Avoid writing conclusions from a handful of impressive transcripts.

Then summarize patterns by domain and scenario. Use language proportional to the data:

• “In this pilot, the system completed 6 of 9 scenario attempts” is a count.
• “The system was better for everyone” is a much stronger claim and usually needs a larger, better-controlled study.
• “The transcripts suggest a recurring omission of document requirements” is an observation that should be backed by named cases.

When the sample is small, prioritize reproducible case reports: show the task, the relevant output, the rubric decision, and the limitation. Do not manufacture statistical significance from a tiny pilot.

Suggested repository layout

A small benchmark can remain easy to inspect if the artifacts are organized around reruns:

lifeval/
├── README.md
├── conditions/
│   ├── system-a.yaml
│   └── system-b.yaml
├── scenarios/
│   ├── services-01.yaml
│   ├── finance-01.yaml
│   └── forms-01.yaml
├── materials/
│   ├── services-pack-v1.pdf
│   └── forms-pack-v1.txt
├── rubrics/
│   └── rubric-v1.yaml
├── runs/
│   └── 2026-11-03-pilot/
│       ├── manifest.json
│       ├── logs/
│       └── redacted-artifacts/
└── analysis/
    ├── scoring.csv
    └── report.md

The README.md should explain how to reproduce the run, what is intentionally omitted, and which files contain fictional data. A reproducible run is one where another evaluator can identify the inputs, conditions, scoring rules, and exclusions, even if the external system has changed since the original collection.

Limitations

LifeVal can make everyday usefulness more visible, but it cannot solve every evaluation problem.

• Small samples are directional. A pilot can reveal failure modes without establishing population-level performance.
• Scenarios are never the whole world. A closed fictional task may not capture stress, accessibility needs, language differences, or local institutional variation.
• Interfaces change. Results depend on the product version, tool access, system instructions, and collection date.
• Human performance varies. Familiarity with forms, search, or AI tools can affect results. Measure or document relevant background where appropriate.
• Rubrics encode priorities. Weights and safety gates reflect the benchmark’s purpose and should be reviewed rather than treated as universal.
• Completion is not long-term benefit. A participant may finish a task yet remain confused, dependent, or unable to repeat the process later.

These limitations are not reasons to abandon the benchmark. They define what the evidence can and cannot support. A responsible report should state the boundary plainly.

A practical first run

If you want a minimal implementation, begin with three fictional scenarios: one service-finding task, one budget-organization task, and one government-form planning task. Use a closed information pack, one fixed rubric, and two system conditions. Have each evaluator complete every scenario under both conditions, counterbalancing the order where possible.

Before collecting any comparison data, verify that:

1. each scenario has an observable success state;
2. all critical errors are defined;
3. the information boundary is identical across conditions;
4. the logs contain timestamps and condition identifiers;
5. the final artifact can be graded without guessing the participant’s intent;
6. fictional data is used throughout;
7. and the report template has a place for exclusions and limitations.

The first useful output may not be a leaderboard. It may be a map of where systems help, where they create rework, and where a human needs a clear hand-off. That is the point of measuring real task usefulness: to connect model behaviour with the work a person actually needs to complete.