Arrow 
Measure an AI pilot from the incoming task to an accepted result. Include review, corrections, rejected outputs, manual fallback, and operating overhead; a faster draft can still create more work for the team.
Compare the same task in the baseline and the pilot
A useful comparison starts with the same definition of a completed task. Select eligible requests from the workflow's approved scope and describe their mix. A pilot that handles only easy questions should not be compared with the entire support queue.
For the baseline, record human handling time through acceptance. For the pilot, record human review and correction time, fallback work, and recurring batch-level effort. Track model or integration cost separately if it matters to the decision. Keep quality and access failures visible rather than trading them away inside one average.
Agree on the measurement plan in the scoping brief. Use the acceptance tests to establish whether the results satisfy the task before discussing efficiency.
A worked example with rejected drafts included
Synthetic arithmetic, not an Arrow customer result: assume 20 comparable support requests take six minutes each in the manual process. That gives a baseline of 120 human minutes. In a proposed pilot, 14 drafts are usable after review and correction, while six require manual completion.
| Work | Illustrative calculation | Human minutes |
|---|---|---|
| Manual baseline | 20 tasks × 6 minutes | 120 |
| Usable pilot drafts | 14 × (2 review + 1 correction) | 42 |
| Rejected pilot drafts | 6 × (2 review + 6 manual completion) | 48 |
| Pilot operating overhead | 10 minutes for this batch | 10 |
| Total pilot effort | 42 + 48 + 10 | 100 |
| Observed difference in this example | 120 − 100 | 20 |
Counting only the 42 minutes spent on usable drafts would hide most of the remaining work. The 20-minute difference is specific to the assumptions in this constructed batch. It is not a forecast, proof of financial return, or evidence that staffing can be reduced.
Separate human effort, elapsed time, and quality
Human effort and elapsed time answer different questions. A draft can arrive immediately but wait in a review queue. An asynchronous lookup may consume time without occupying a person. Record both when turnaround matters, and identify where the measurement begins and ends.
Report how many outputs were reviewed, accepted with corrections, rejected, or routed elsewhere. Explain whether corrections were cosmetic or changed the substance. Include work that shifts to another team, such as a source owner resolving missing documentation.
Keep setup work separate from recurring operation, then show both when making a broader investment decision. Do not spread initial effort over an assumed future request volume without making that assumption explicit.
Use the evidence to change one thing at a time
When the pilot adds work, inspect the cause before changing models. Missing sources may produce repeated escalations. An unclear review rule may make every draft slow to verify. A narrow request filter may improve the first release while the team fixes those issues.
Document the sample, measurement period, source and configuration versions, and any differences from the baseline. Small samples and changing request mixes limit what can be concluded. Do not convert unmeasured effort into zero.
The NIST AI RMF Playbook offers voluntary suggestions to adapt to an organization's context. This worked accounting method is Arrow AI's own illustration, not a NIST metric. Use it with the pilot decision record and the operating handoff.
Sources and editorial scope
Original operational guidance by Arrow AI, published and reviewed October 19, 2026. Worked examples and calculations are synthetic, not customer results or forecasts. Adapt the method to the workflow and its operating context.
The Arrow read
Build the whole pilot decision
Follow the AI pilots and operations collection, from scope through acceptance and measurement to handoff. Explore the pilot planning approach for the wider sequence.