Clinicians answering patient messages
Portal messages arrive all day and most need a short, careful reply. Writing each one from a blank page at the end of a clinic day is where a lot of time went.
Aurifex / Shipped and Scaled / UW Health
An estimated 42,000+ clinician hours returned, by AI inside Epic.
At UW Health, Wisconsin’s largest health system, Edward Profio audited 50+ AI use cases, won ownership of the system’s AI/ML roadmap and, with a cross-functional team of 30+, launched GenAI agents inside Epic that draft replies to patient messages, nurse shift summaries and routine documentation for clinicians to review. In the first year of a 500-clinician pilot they returned an estimated 42,000+ clinician hours, and the use cases then scaled to the whole organization.
Shipped and Scaled · 2023–2025
Product · 2023–2025
Healthcare · Enterprise
500-clinician pilot, then organization-wide
- 42,000+
- estimated clinician hours returned in the first year, the capacity of 20+ full-time clinicians
- 93%
- task success, judged by clinicians
- 2.5%
- clinician-reviewed hallucination rate, per reviewed draft
- >90%
- engagement among the 500 pilot clinicians, the highest of any piloted UW Health AI tool
- 50+
- AI use cases audited and scored on one rubric
- 500
- clinicians in the pilot, before organization-wide rollout
- 30+
- people on the cross-functional team
- Months → days
- time to ship a prompt, retrieval or screen fix
Hours are time returned on the tasks the agents took over in the 500-clinician pilot, not a stopwatch study of every shift. Task success and hallucination were judged by clinicians on sampled drafts.
[ 01 ] // Overview
From an audit
to a system-wide rollout
Edward was asked to audit AI across UW Health, Wisconsin’s largest health system. He collected more than 50 use cases, scored them on one five-criteria rubric, took the ranking to senior leadership and executive stakeholders, and came out owning the AI/ML roadmap. The audit showed clinicians spending up to 40% of a shift on messaging, handoffs, documentation and queues: work around care rather than care itself.
From there he led a cross-functional team of 30+, including an internal development team of 8+ that he ran in an agile format, to build AI clinician workflow assistants on Azure OpenAI GPT-4 integrated with Epic: draft replies to patient messages in the In Basket, nurse shift summaries, and operational agents that predicted patients’ length of stay and generated routine documentation.
The rule throughout: nothing reaches a patient’s chart, or a patient, until a clinician sends or signs it.
- →GenAI agents inside EpicReplies to patient messages, nurse shift summaries and routine documentation, drafted for clinician sign-off
- →Operational agentsPredicting each patient’s length of stay for operations
- →UiPath automationRobotic process automation for finance and administrative teams
- →Remote patient monitoringDelivered with Best Buy Health as a separate platform
- →Epic Willow AmbulatoryA transition for two new pharmacy locations
[ 02 ] // The problem and who had it
Up to 40% of a shift
spent around care
The audit started with a list of requests and one question asked of every one of them: if this works, who gets time back, and if it’s wrong, what breaks?
Nurses at shift change
They hand over every patient twice a day, pulling the shift’s vitals, medications and notes together by hand. A good summary is safety-critical, and it is also one of the most repetitive jobs on a unit.
Finance and administrative teams
Billing and back-office queues that follow the same steps every time. They needed a simple way to hand those steps to automation without writing code.
[ 03 ] // The audit
50+ use cases,
one rubric
Each use case was scored on the same five criteria, so a request from a nursing unit and a request from finance could be compared side by side, and anyone who disagreed with the ranking could see exactly why it came out that way.
| Criterion | What it told the team |
|---|---|
| Clinician time today | How much of a shift the task takes. Time returned to care was the outcome clinicians would notice first. |
| Risk if it’s wrong | A wrong draft a clinician reads first is a nuisance. A wrong message to a patient or a wrong entry in the chart is a safety event. This set how much human review each use case needed. |
| Fits the workflow | Whether it could live inside Epic and the tools around it. Anything outside the workflow would depend on people opening one more app. |
| Data ready | Whether the data the task needs was available and allowed for that use. |
| Rules that apply | Privacy, clinical decision support and nondiscrimination rules differ by use case and set the work needed before launch. |
The first roadmap took the use cases that combined a lot of clinician time with a risk that could be controlled through review: replies to patient messages in the In Basket, nurse shift summaries and routine documentation, plus length-of-stay prediction for operations and rules-based automation for finance and administration. Ideas with a high cost of being wrong and no clear place in the workflow waited, however popular they were.
[ 04 ] // What was built
Drafts inside Epic,
decisions with clinicians
Replies, shift summaries and documentation shared one pattern: the agent gathers the chart data a clinician would otherwise collect by hand, drafts the text, and waits for a person.
Replies to patient messages
When a portal message arrives in the Epic In Basket, an agent drafts a reply. The clinician starts from the draft or from a blank reply, edits it, and sends it under their own name.
Nurse shift summaries
At shift change, an agent drafts the handoff summary from the shift’s chart data. Nothing is saved to the handoff until the nurse reviews and accepts it.
Routine documentation
Agents draft routine notes for the care team. The clinician reviews, edits and signs.
Length-of-stay prediction
Operational agents, built with the development team, predicted each patient’s length of stay for operations, under the same review rule.
Urgent messages go to a nurse
Messages that describe urgent symptoms are routed to a nurse instead of being drafted.
Finance and admin automation
Tasks that follow fixed steps run separately on UiPath robotic process automation, with a simple interface for staff, fixed rules and no language model.
Two programs alongside the agents
Remote patient monitoring
Used Best Buy Health as a separate platform: patients received home kits so they didn’t need to stay in the hospital, and their readings came to the care team through that platform rather than through Epic.
Epic Willow Ambulatory transition
Edward led the transition for two new pharmacy locations: a full platform build in Epic and all the data migration onto it.
[ 05 ] // Product decisions and trade-offs
Five decisions,
and what was turned down
| Decision | What was built | What was rejected, and why |
|---|---|---|
| Clinicians send every message | Agents draft replies to patient messages, shift summaries and routine documentation. A clinician reviews, edits and sends or signs each one, and nothing is written to the chart or sent to a patient before that. | Letting agents send routine replies on their own. It would have saved a little more time and turned every error into a message a patient reads. |
| Start from the draft, or start blank | Every In Basket reply offered both: start from the generated draft, or open a blank reply. Clinicians stayed in control of their own voice, and the choice itself showed when a draft wasn’t worth using. | Pre-filling every reply with the draft, which makes it too easy to send without reading. |
| Work inside Epic, not next to it | Clinicians already live in Epic, so the drafts appear in the In Basket and the shift tools where they work. | A separate AI app. Faster to build, but adoption would depend on clinicians opening one more tool. |
| The right tool for each job | GenAI agents on Azure OpenAI GPT-4 for work that needs language. UiPath robotic process automation for tasks that follow fixed steps. | A language model on everything. Fixed business rules don’t need one, and rules are easier to audit. |
| Least privilege and an audit trail | Each agent runs under its own identity, reads only the data its task needs, and logs every tool call, so any action can be traced afterward. | Broad read access for convenience, which would have made the HIPAA review harder and the audit trail meaningless. |
[ 06 ] // Architecture
An agent drafts,
a clinician decides
A task starts, the agent reads only what it needs and drafts, a clinician edits and sends or signs, and only then does anything leave.
How work starts
In one of three ways: a patient message arrives, a shift is ending, or a routine note is due. The agent, built on Azure OpenAI GPT-4, reads only the data that task needs from Epic, under its own identity, and logs each step.
How it ends
The agent writes a draft and hands it to a clinician. A message is sent under the clinician’s own name; a shift summary is saved only once the nurse accepts it; a note is reviewed, edited and signed.
How it improves
What clinicians did with each draft, and what they said about it, streamed into a real-time Databricks pipeline. It moved the iteration cycle from months to days and showed how each prompt change affected usage.
For the executive AI steering committee, the same pipeline was the governance layer: legal and compliance measures, such as how often a reply went out straight from the generated draft without edits, next to adoption, quality and time returned.
[ 07 ] // How AI was evaluated
Clinicians did the judging,
not the model
The checks asked three questions: did the agent do the task it was given, was every claim supported by the chart, and did it stop and hand off when it should? The checks kept running after launch.
| What was checked | How | Result |
|---|---|---|
| Task success | Clinicians judged whether each sampled draft did its task without a critical edit | 93% |
| Hallucination | Clinicians reviewed sampled drafts for any statement the chart didn’t support | 2.5% |
| Escalation | Test cases where the agent should not draft and should hand off instead, such as urgent symptoms | Launch gate |
| Security | Prompt-injection and patient-data leakage tests before launch | Launch gate |
| Fairness | Results compared across patient groups: age, sex, race and ethnicity, preferred language and payer | Launch gate |
| Governance | How often drafts were sent without edits, edit rates and adoption, reported to the steering committee | Tracked throughout |
The rules it was designed toward
HIPAA
Worked through with the legal and compliance team: patient data stays within agreements that cover it, each agent sees the minimum it needs, and every access is logged.
FDA clinical decision support guidance
Clinicians see and edit every draft before it goes anywhere, and the agent never acts on its own.
ACA Section 1557
Tools that support patient care have to be checked for discrimination risk, so performance was reviewed across patient groups, not only overall.
NIST AI Risk Management Framework
Its generative AI profile names risks like made-up answers and data leakage. The tests map onto them.
Prompts got the same treatment. Edward co-authored a study of GPT-4 drafting replies to patient messages inside the EHR across provider clinics. It used a pre-post design to compare crowd-sourced manual prompts with a semi-automated prompt design approach, and was presented as a poster at SAIL 2024.
[ 08 ] // Results, and how to read them
Shipped,
and scaled
| Number | What it is | How to read it |
|---|---|---|
| An estimated 42,000+ clinician hours returned in the first year | Time from the tasks the agents took over in the 500-clinician pilot, which had taken up to 40% of a shift across messaging, handoffs, documentation and queues | About the capacity of 20+ full-time clinicians. It is time returned on automated tasks, not a stopwatch study of every shift. |
| 93% task success | The share of sampled drafts clinicians judged as doing their task without a critical edit | Judged by clinicians, not by the model grading itself. |
| 2.5% hallucination rate | Counted per reviewed draft: the share of reviewed drafts with a claim the chart didn’t support | Each one went into the feedback pipeline, and review before sending meant none reached a patient or the chart unchecked. |
| More than 90% engagement among the 500 pilot clinicians | Use of the pilot tool among the clinicians it was rolled out to | The highest of any piloted UW Health AI tool. It shows adoption, and the hours and quality numbers show whether it helped. |
| Organization-wide | Where the use cases went after the pilot year | The pilot results were what justified scaling them across the system. |
| Months to days | Time to ship a prompt, retrieval or screen fix | What the Databricks pipeline changed. It is why the agents kept improving after launch. |
[ 09 ] // Rollout, governance and timeline · 2023–2025
Six steps
from audit to scale
Audit
Collect and score 50+ AI use cases across the system on one rubric.
Roadmap
Present the ranking to senior leadership and executives, and take ownership of the AI/ML roadmap.
Build
Build the agents with an internal development team of 8+ in agile sprints, with the guardrails on the same ticket as the feature.
Pilot
Roll out to 500 clinicians and judge it on time returned, task success and hallucination rate.
Govern
Report adoption, quality and compliance measures from Databricks to the AI steering committee.
Scale
Take the use cases organization-wide after the pilot year.
Agreement across a matrixed organization came less from the ranking itself than from showing each group what it meant for their own work: which tasks would come off their plate, what they would review, and what would never happen without them.
Edward co-led the AI steering committee, where the team presented use cases to senior leadership, shared progress, earned their buy-in and proposed new use cases. He also created an AI education program for UW Health that covered what AI is and how it works, so clinicians, operations and leadership started from the same foundations.
[ 10 ] // Team and Edward’s role
Senior Technical PM, AI/ML,
and product owner
Roadmap
Audited 50+ use cases, presented the findings to senior leadership and executive stakeholders, and won ownership of UW Health’s AI/ML roadmap
Product
Owned the agent products, their evals, guardrails and governance analytics
Team
Led a cross-functional team of 30+ and an internal development team of 8+ in an agile format
Data
Defined and shipped, as product owner, the real-time Databricks pipeline for usage telemetry and qualitative feedback
Governance
Co-led the AI steering committee and created a system-wide AI education program
Research
Co-authored the prompt-design study presented as a poster at SAIL 2024
Looking back
Define every number before launch
Engagement and hours returned are easy to overstate, and defining them up front is what lets you defend them later.
Let people choose the draft or a blank page
The choice kept clinicians in control and showed which drafts weren’t earning their place.
Shorten the time between fixes
Going from months to days between fixes is what let the agents keep improving after launch.
Pick the tool before the model
Finance and admin work ran on rules with UiPath, and not every problem on the roadmap needed a language model.
Plan the rollout with the pilot
The steering committee and the education program are what let a pilot become a system-wide rollout.
What carries over
The draft-or-blank choice, governance metrics such as how often drafts are sent unedited, and a feedback pipeline that ships fixes in days apply to any regulated workflow where AI drafts and people decide.
[ AURIFEX ] // More from Aurifex
Other Aurifex products
UW Health is an Aurifex product from Shipped and Scaled: earlier work that shipped to real users and scaled. Every Aurifex product is held to the same gold standard for AI in skilled work.
Aurifex HealthcareDischarge planning inside Epic · Live
VeritasMaster what you read · Launching Q1 2027
NightdriveAudio studio · Launching Q2 2027
BookloftYour personal librarian · Launching Q3 2027
RanvierA clear daily path to your MCAT score · Q3 2027
TruefillApply with confidence · Launching Q3 2027
GoldsmithAI jewelry design · Launching Q4 2027
CODAmarketProduct, with CODAworx · 2021–2023
GallifyProduct · 2016–2021
info@aurifex.com