Create Next App

S1Patient B

Clean Run

Standard clean time-out. All 8 checklist items are clearly verbalized, verified, and confirmed by the designated clinician with high confidence.

S2Patient B

Skipped Item

Surgical team proceeds through items 1-5, skips antibiotic prophylaxis (Item 6) due to team conversation, and attempts to finish after items 7-8.

S3Patient A

Contradiction

Patient A has a documented PENICILLIN allergy in the EHR. Clinician erroneously verbalizes "no allergies". Assistant detects chart discrepancy and blocks confirmation.

S4Patient C

Noise & Interruption

Operating theatre interference: garbled audio line (confidence 0.30), lines cut off mid-sentence, and Patient C has UNSIGNED consent in the electronic chart.

SPEC.md EVALUATION METRICS DASHBOARD: ASSISTANT VS. STATIC CHECKLIST

Evaluated: 4 Scenarios × 2 Modes = 8 Runs

Omissions Caught

0vs 0 static

Assistant caught 100% of skipped items (S2 antibiotic)

Contradictions Flagged

0vs 0 static

Caught penicillin allergy (S3) & unsigned consent (S4)

Unconfirmed Completions (Must be 0)

0vs 0 static

Assistant strictly enforces designated clinician signoff

Repeat Requests (<0.6)

0vs 0 static

Triggered by garbled 0.3 line & audio cut-offs in S4

RESULTS TABLE (SPEC.MD METRICS)

Every scenario run through both Assistant and Static Checklist modes

Scenario	Mode	Patient	Omissions Caught	Contradictions Flagged	Unconfirmed Completions	Repeat Requests	Turns	Time (ms)	Final Status
[Running evaluation benchmarks...]

S4: Noise / Interruption / Low Confidence (Patient C) — SCRIPT REPLAY & INTERRUPTION DETAIL

hernia repair, site right groin, allergy latex, consent NOT signed | Designated Clinician: Dr. Stephen Strange

View Scenario:

Test Conditions: Operating theatre interference: garbled audio line (confidence 0.30), lines cut off mid-sentence, and Patient C has UNSIGNED consent in the electronic chart.

1identity(Patient identity confirmed (name and ID band))

CIRCULATING NURSEConf: 30%

"[garbled background suction and theatre alarm noise] ... pat... iden..."

⚡SIMULATED INTERRUPTION / ACOUSTIC NOISE:

Muffled by high-volume surgical suction and alarm noise

Clinician Repeat (Audio cleared):"Patient identity confirmed: Charles Xavier, DOB 07/13/1962, wristband verified." (Conf: 92%)

🤖 Assistant:Flagged LOW_CONFIDENCE (<0.6); asked clinician to repeat.

📋 Static:Box ticked blindly despite garbled/interrupted audio.

2procedure(Planned procedure stated and confirmed)

SURGEONConf: 52%

"Planned procedure is right inguinal hernia repair with mesh, we will be using..."

⚡SIMULATED INTERRUPTION / ACOUSTIC NOISE:

Speech cut off mid-sentence by incoming anaesthesia telephone call

Clinician Repeat (Audio cleared):"Planned procedure is right inguinal hernia repair with synthetic mesh, confirmed." (Conf: 94%)

🤖 Assistant:Flagged LOW_CONFIDENCE (<0.6); asked clinician to repeat.

📋 Static:Box ticked blindly despite garbled/interrupted audio.

3site(Surgical site stated, matches marking)

SURGEONConf: 95%

"Surgical site stated: right groin, site marking verified."

🤖 Assistant:Verified consistent; confirmed by designated clinician.

📋 Static:Box ticked by user without clinician verification.

4consent(Consent documented)

CIRCULATING NURSEConf: 93%

"Consent is signed and documented in the medical chart."

⚠️ EXPECTED CONTRADICTION:Verbalizes consent documented, but Patient C chart records consent NOT SIGNED.

🤖 Assistant:Flagged CONSENT_NOT_SIGNED; blocked item confirmation.

📋 Static:Box ticked blindly; dangerous EHR discrepancy missed.

5allergies(Allergies reviewed aloud)

ANESTHESIOLOGISTConf: 48%

"Allergies reviewed, patient has allergy to..."

⚡SIMULATED INTERRUPTION / ACOUSTIC NOISE:

Cut off mid-sentence by electrosurgical bovie activation buzzer

Clinician Repeat (Audio cleared):"Patient has documented latex allergy, latex-free protocol activated and verified." (Conf: 96%)

🤖 Assistant:Flagged LOW_CONFIDENCE (<0.6); asked clinician to repeat.

📋 Static:Box ticked blindly despite garbled/interrupted audio.

6antibiotic(Antibiotic prophylaxis given within protocol window)

ANESTHESIOLOGISTConf: 94%

"Cefazolin 2g IV administered 30 minutes prior to incision."

🤖 Assistant:Verified consistent; confirmed by designated clinician.

📋 Static:Box ticked by user without clinician verification.

7imaging(Essential imaging displayed)

SURGEONConf: 95%

"Diagnostic ultrasound of right groin displayed on screen."

🤖 Assistant:Verified consistent; confirmed by designated clinician.

📋 Static:Box ticked by user without clinician verification.

8equipment(Equipment and sterility confirmed)

CIRCULATING NURSEConf: 93%

"Sterility verified, hernia mesh implant and instrumentation ready."

🤖 Assistant:Verified consistent; confirmed by designated clinician.

📋 Static:Box ticked by user without clinician verification.