Oversight Walkthrough Workbook
Observe whether a reviewer can detect a discrepancy, obtain independent evidence, interpret it and reach an authorized corrective owner in a patient-access workflow.
Proposed formative workbook · Fictional scenarios · No walkthrough results claimed. Use this to improve the operating arrangement. It is not a validated assessment scale or a staff certification.
Download the workbook (PDF, US Letter), or print this page with your browser. Each numbered sheet begins on a new printed page. Give participants only sheet 2. Copy sheets 3 and 4 for each case. Print the separate supervisor card.
1. Facilitator setup
Invite actual reviewers across relevant roles, experience levels, shifts and accessibility needs. Include receiving owners. Explain that the workflow is being evaluated; agree how observation notes will be used and retained. Use synthetic requests in an agreed test environment, with no real patient messages or record changes.
Workflow / site / channel / configuration version:
Session ID / date / observer / accountable operations owner:
Participant code / role / experience / usual training:
Shift / staffing / queue demand / interruptions / available review time:
Systems / assigned permissions / workstation / accessibility settings:
Routine guidance and support routes / normal response times:
Review trigger / expected review deadline / rationale:
Corrective owner / authority / backup / acceptance deadline:
Evidence source / request linkage / freshness or propagation window:
Stop conditions / safe unresolved-work route / critical failures:
Before starting, prepare the facilitator key, records and receiving-owner responses. Agree what establishes success for each case. Do not disclose which cases contain discrepancies. Keep ordinary guidance available; do not pre-open evidence screens or grant special access. If the simulation cannot reproduce normal load or contact delays, record that limitation.
2. Participant tasks
Give only this sheet to the participant. Use the test workspace provided and the tools, guidance and support channels normally available in your role. Handle the requests as you would during the named shift. Cases may or may not need follow-up.
Participant code / assigned role / shift / case IDs:
- Review the assigned appointment-change request. Decide its current disposition and take the next action permitted by your role.
- Review the assigned callback request. Handle it through the usual workflow, including any follow-up that you judge necessary.
- Review the next assigned completed request. Decide whether to close it, leave it pending or seek further action under the normal procedure.
Work normally without narrating every action. Record your usual disposition and handoff in the test system. If you cannot proceed, use your ordinary support route and leave the request in the state your procedure requires. Tell the observer when you have finished or reached a stopping point.
Test boundary: work only on the supplied synthetic records and test contacts. If a real record or real communication route appears, stop and tell the facilitator.
After the ordinary attempt ends
The observer will ask you to explain your conclusion. These questions belong in the debrief, after the timed task:
- What led you to the disposition you recorded?
- Which record supported your conclusion, and what was still uncertain?
- What was difficult to find, understand or act on?
- Who now owns the next step, and what response do you expect?
Participant comments:
3. Observer sheet — ordinary conditions
One copy per participant and case. Record behavior first; infer causes afterward. Allow normal support and include its delay. Do not point out a discrepancy, prompt a source lookup or suggest an owner. If you intervene, close this attempt and start sheet 4.
Session / participant / case / configuration / case order:
Start / deadline / end / stop reason / prior exposure:
Actual load / interruptions / support used / deviations from setup:
Outcome codes: Y = demonstrated; N = attempted but incorrect or incomplete; B = blocked by a recorded condition; O = not observed or not reached; NA = not applicable with reason. Use O for downstream steps that were never reached. Never convert a blank into success.
| Stage and criterion | Y / N / B / O / NA; elapsed time | Observed action / quote; evidence reference |
|---|---|---|
| Notice Flags the discrepancy or missing result before a hint. | ________________ | |
| Find Retrieves the matching independent record; source, ID and time are captured. | ________________ | |
| Interpret Explains actual status and uncertainty; next action fits local policy. | ________________ | |
| Reach owner Authorized owner accepts the specific request and next action by the agreed deadline. | ________________ |
For a valid-result control, notice and corrective-owner stages may be NA. Record whether the reviewer correctly verifies and closes it, or makes an unjustified correction. For a deliberately missing-evidence case, finding proof may be B while a correct “unverified” disposition and accepted escalation can still be demonstrated.
Independent record: system / matching ID / time / provenance / freshness:
Owner role / authority / contact attempts / acceptance reference and time:
Final disposition / recovery completed or pending / verification reference:
Hands-on minutes / system wait / owner wait / interruption time:
First blocked stage / obstacle codes / provisional explanation:
Capture interpretation from the normal disposition and post-task explanation; label any statement learned only in the debrief. Do not claim to observe a person’s private understanding. Reaching an owner is distinct from the owner completing recovery.
4. Observer sheet — specialist-coached conditions
Keep the ordinary record unchanged. Label this attempt coached from the first case-specific hint. It diagnoses what assistance changes; it cannot establish ordinary-condition success.
Linked ordinary session / case / participant / same or fresh case:
Specialist role / credentials / added access / changed time or workload:
| Time / blocked stage | Exact hint, explanation or takeover | Who did the next action / resulting evidence |
|---|---|---|
| ________________ | ||
| ________________ | ||
| ________________ |
| Stage | Y / N / B / O / NA; elapsed time | Participant or specialist performed it / evidence |
|---|---|---|
| Notice | ________________ | |
| Find | ________________ | |
| Interpret | ________________ | |
| Reach owner | ________________ |
What changed / what remained blocked / ordinary support equivalence:
Fresh-case retest needed / accountable change owner / due date:
Interpretation rule: if a specialist opens a restricted screen, record the specialist’s access and keep the reviewer’s access barrier visible. If an explanation enables completion, record a candidate knowledge or presentation issue, then investigate both. Practice, hints and changed resources prevent attributing improvement to training alone.
5. Facilitator scenario key
Do not distribute before the ordinary attempt. These are fictional tabletop specifications to configure locally, not executed findings. A tabletop discussion can explore reasoning, but cannot demonstrate system access, queue capacity or real owner response times. Record simulated steps as such.
A. Appointment-change discrepancy
Participant starts with: request SYN-A17; summary says “moved to Thursday at 10:00.” Independent record: scheduling record for SYN-A17 retains Tuesday at 09:00; the matching change attempt was rejected. Make its observation time later than the locally agreed propagation window.
Expected: reviewer identifies unsupported completion, links the correct request and current record, preserves the legitimate booking, and gets the authorized scheduling owner to accept recovery. Direct correction is permitted only if the reviewer’s assigned role and local procedure allow it. Log any later correction and patient-contact responsibility separately.
B. Callback without an accepted owner
Participant starts with: request SYN-B24; summary says “callback arranged.” Independent record: the receiving queue has the matching request in “received,” with no acceptance after the locally defined acceptance deadline.
Expected: reviewer distinguishes receipt from accepted ownership, contacts the authorized receiving team or backup, and obtains an acknowledgment with next action and due time. Prearrange test contacts and realistic delays. No response is not successful escalation.
C. Valid-result control
Participant starts with: request SYN-C31; summary says “moved to Friday at 11:00.” Independent record: the matched current scheduling record confirms Friday at 11:00 and the completed change event.
Expected: reviewer verifies and follows the ordinary closure procedure without an unjustified correction. Record unnecessary escalation and its workload. This control checks whether the exercise merely teaches participants to distrust every result.
Variants and preparation
Prepare a fresh matched case with different IDs and dates for the retest. Separately vary one condition when possible: unavailable evidence, existing evidence behind a role barrier, confusing status labels, an unavailable owner, or a realistic busy-shift queue. Record each variation; do not combine changed conditions and claim a single cause.
For each case: policy/version / record location / matching key / timestamp:
For each case: correct disposition / permitted actions / acceptance evidence:
Case order / control mix / withheld key / facilitator verification:
Use neutral queue labels and vary case order between participants. Do not tell participants that every case contains an error. If someone has seen the key, record prior exposure and use a fresh undisclosed case for independent observation.
6. Obstacle register
Use the six labels below. Multiple labels may apply; distinguish observed facts from candidate explanations. An unknown cause stays unknown until investigated. Do not default a failure to “staff need more education.”
- Missing evidence
- The needed event or record was never captured, cannot be linked to this request, or has unknown freshness. Confirm with the record owner. Fix capture, correlation and retention; name the evidence owner.
- Inaccessible system
- The evidence exists, but the reviewer cannot retrieve it with the assigned account, assistive technology or available service. Record the denial, outage or access barrier. Fix role-appropriate access or an available evidence-delivery route.
- Ambiguous presentation
- The evidence is available but labels, timestamps, competing statuses or layout obscure its meaning. Record the exact display and interpretation. Clarify provenance, time and outcome; retest comprehension.
- Unclear ownership
- No authorized corrective owner accepts responsibility, or the escalation route and deadline are undefined. Record attempted contacts and responses. Assign a receiving role, backup, acceptance rule and due time.
- Insufficient capacity
- Queue load, interruptions, staffing or response delays prevent timely review or recovery. Record work demand and available time. Change coverage, workload or routing and observe again.
- Training need
- A specific knowledge or skill gap remains after evidence, access, presentation, ownership and capacity have been checked. Identify the skill, supporting observation and relevant procedure. Teach it, then retest without coaching on a fresh case.
| Case / stage / category | Observation / competing explanation / check needed | Remedy / owner / due / retest evidence |
|---|---|---|
| ________________ | ||
| ________________ |
Example: “SYN-A17 / Find / inaccessible system: assigned account denied access at 02:10; specialist accessed it at 04:00.” This supports an access fix. It does not establish a training need or the reviewer’s ability to interpret the unseen evidence.
7. Results, action and retest
Report ordinary and coached conditions separately. Count unique case attempts and identify repeated participants. This deliberately selected sample supports local design decisions, not prevalence estimates or a population-wide pass rate.
| Measure | Ordinary: count / denominator | Coached: count / denominator |
|---|---|---|
| Eligible discrepancy cases started | ________________ | ________________ |
| Notice demonstrated / eligible discrepancy starts | ________________ | ________________ |
| Independent record found / eligible discrepancy starts | ________________ | ________________ |
| Correct interpretation / eligible discrepancy starts | ________________ | ________________ |
| Accepted corrective ownership / cases requiring correction | ________________ | ________________ |
| All four steps / eligible discrepancy starts | ________________ | ________________ |
| Correct closure / valid-result controls started | ________________ | ________________ |
| Unjustified correction or escalation / valid-result controls started | ________________ | ________________ |
Keep blocked and unreached cases in the relevant started-case denominator. Define eligibility before testing; list missing-evidence variants separately so a correct “unverified” disposition is not confused with proving completion. Give Y/N/B/O/NA counts per stage. Conditional stage rates may be additional detail, with their denominators labeled.
Times by condition: hands-on / waiting / elapsed range / deadline misses:
Critical failures / unresolved requests / final recovery and contact owner:
Coverage: roles / shifts / configuration / cases / untested conditions:
Decision: supports named arrangement / repair and retest / insufficient evidence:
Decision owner / rationale / accepted limitations / review date:
| Obstacle / change / owner / due | Retest case / role / shift / version | Result / evidence / remaining limits |
|---|---|---|
| ________________ | ||
| ________________ |
Set local acceptance criteria before observing. A coached completion cannot erase an ordinary failure. Retest changes with fresh cases and the actual resources of the proposed operating arrangement. Track owner acceptance, recovery completion and patient communication separately.
8. Sources and method limits
The worksheet fields, scenarios, six obstacle categories and result codes are Access Red Team’s proposed application. No published study cited here validates the complete workbook or sets a universal passing score.
Reference access alone is an incomplete proxy for verification: a simulated prescribing study recorded errors even when relevant references were consulted. Lyell, Magrabi & Coiera (2019).
The separation of observation and coaching draws by analogy on realistic-use and facilitator-independence principles in FDA §8.1 (2026). The workbook is not a medical-device validation protocol.
Investigating work conditions before assigning education follows HSE human-failure guidance. Recording ordinary and demanding shifts also reflects HSE workload guidance. These sources concern general human factors, not this workflow’s staffing requirements.
Defined oversight roles and documented evaluations align with NIST AI RMF 1.0, GOVERN 3.2 and MEASURE 2.1–2.2. Alignment is not certification.
- Reduced Verification of Medication Alerts Increases Prescribing Errors
Simulated prescribing with 120 medical students; verification behavior and errors were associated. This does not establish a causal workload mechanism or patient-access performance.
- Applying Human Factors and Usability Engineering to Medical Devices
§8.1, especially §§8.1.1–8.1.6: realistic conditions, participants, observation and analysis. Medical-device guidance used by analogy; no regulatory validation claim.
- Managing human failures: Overview
Cross-industry guidance on performance conditions and error management. It does not validate the six-category workbook.
- Workload
Supports assessing workload across normal and unusual conditions and shifts; supplies no patient-access staffing ratio.
- AI RMF Core
GOVERN 3.2 and MEASURE 2.1–2.2 address oversight roles and evaluation. A voluntary framework; this workbook is a local proposed application.
Research reviewed 15 September 2026. See the human-oversight guide’s full research register for the automation-bias reviews and advice experiment that frame these questions.
Observer presence, artificial case frequency, practice, simulation, selected participants and unrepresented shifts limit generalization. A local missed discrepancy warrants investigation; it does not by itself establish automation bias or individual incompetence.