2024 Buyer’s Guide to AI Scribes
AI scribes are having a moment.
It feels like most provider organizations today, large and small, are currently investigating AI scribe vendors given the promise these solutions show in addressing their biggest pain points.
Evaluating them can be difficult though, given the novelty of the technology and the rapid pace at which they’re evolving.
To help, Elion put together this guide to support prospective buyers as they navigate their vendor selection journey. Though it’s geared slightly toward larger provider organizations, such as hospitals and health systems, the principles we’ve outlined are relevant to provider groups across the spectrum.
What is an AI scribe?
Before we dive into the nuances of evaluating and acquiring an AI scribe, it’s important to clarify what we’re talking about here. For the purposes of this article, “AI scribes” refer to systems that:
Ambiently capture both patient consultations and clinicians’ dictations
Convert these audio recordings into written transcripts through speech-to-text technology
Synthesize a clinical note from the transcript by employing artificial intelligence, notably a large language model (LLM) specifically tuned for this application
It’s important to note that we’re not addressing simple dictation tools or considering virtual/remote scribes here. We do, however, include platforms that integrate AI scribing with human oversight, where the human element typically acts as a verification layer before the clinical note reaches the physician-user for approval.
The benefits of AI scribes
AI scribes are the first generative AI product to take off in the context of care delivery.
Their success can be attributed in part to the impressive capabilities of standalone large language models (LLMs). Given their ability to summarize and condense information, LLMs are particularly well-suited for this use case.
However, the bigger driver behind the emergence of AI scribes is the significant burden clinical documentation places on healthcare providers, organizations, and patients. AI scribes show promise in addressing many of the problems that plague the current process of documenting clinical visits. To name a few of the benefits:
Mitigation of provider burnout by cutting down on the time clinicians dedicate to documenting, especially during off-hours or the infamous “pajama time,” leading to improved retention and lower rates of provider churn.
Enhanced patient engagement during visits, since clinicians can focus on patients instead of screens.
[Our AI scribe solution] also has the unexpected benefit of improving patient interaction, as I provide a summary and explanation of their background and history during the visit that the patient can hear and contribute to.
Medical Director, Oncology Practice
View full buyer transcript here.
Increased clinician productivity, either by affording time to see additional patients or by spending more quality time with existing ones.
Decreased documentation expenses by reducing the reliance on human scribes.
Improved note quality as AI scribes can faithfully capture details from conversations and tailor the notes to the clinician’s style and specialty nuances, as well as provide greater depth in support of revenue cycle and care coordination needs.
Quicker note finalization, with AI-powered notes often ready within minutes post-consultation.
Better patient adherence post-visit, facilitated by AI solutions that can craft summaries and instructions tailored to an individual patient’s language, literacy, and medical context.
Enhanced medical coding accuracy and comprehensiveness, as some of these technologies can accurately capture and propose relevant codes.
Reduced prior authorization denials as AI scribing solutions can be calibrated to capture essential information for authorization requests.
Given the many potential benefits, it’s easy to understand the tremendous interest in this category. AI scribes are uniquely well positioned to address healthcare providers’ key priorities, including reducing burnout, boosting revenue, and cutting administrative costs.
This impact is not speculation either. We spoke recently with a clinical leader who stated that AI scribes have done more to improve the clinician experience than any other technological or care model innovation she’s witnessed in her career. We’re hearing other clinical organizations sharing similar sentiments as well. So it’s no surprise that many healthcare organizations are looking into this space.
Interestingly, I had one clinician who was exhausted and didn’t particularly enjoy the care model, but she said she was staying with our company because she knew she wouldn’t have access to [our AI Scribe Vendor] if she went anywhere else. I cannot stress enough how much the clinicians appreciate it.
Chief Medical Officer, Women’s Health Practice
View full buyer transcript here.
What to look for in an AI scribe solution
Although AI scribes are conceptually straightforward — record an encounter, transcribe the audio, and craft a clinical note from that transcription — the reality of selecting the right solution involves a nuanced evaluation.
Here’s a breakdown of the main aspects to consider when assessing AI scribes. For those looking for a deeper dive, we’ve created a detailed requirements grid that can be adapted to your organization’s specific criteria.
1) Note Quality
The first aspect to consider is the ability of the AI scribe to produce notes that meet your quality standards. Since “quality” can be a somewhat nebulous term, it’s helpful to break it into more actionable components:
Note Structure: Check if the AI scribe has templates that align with your clinical specialty and documentation requirements. How flexible is the system in allowing you to modify these templates to suit your organization’s unique preferences?
Note Content: Assess the caliber of the writing. Does it match or surpass the quality typically achieved by your clinicians? Ensure that the scribe captures essential details relevant to your specialty or subspecialty. Additionally, evaluate whether the vendor can customize the model based on specific content provided by your organization. Finally, consider whether the tool can present the note in your preferred format, whether it’s bullet points for easy scanning or comprehensive text blocks for detail.
Accuracy: Understand how the vendor evaluates the accuracy of their solution. Ask about what measures they use and whether they have any studies or data to verify their claims.
Revenue Cycle Support: Evaluate the AI scribe’s capabilities in facilitating billing and coding processes. Does it generate accurate diagnosis and procedure codes from spoken information during the visit? Is the system able to suggest codes based on the transcript content? Additionally, check if the documentation includes all necessary elements to back up prior authorization requests, which can be critical for streamlining reimbursement and reducing denials.
2) Ability to Manage Complicating Factors
Clinical interactions take place in environments that are often noisy, chaotic, and unpredictable. It’s essential for AI scribes to navigate these complexities seamlessly to ensure clinicians will actually utilize them. Be sure to evaluate how different solutions perform in scenarios like:
Loud Background Noise: The AI scribe should be able to differentiate relevant dialogue from ambient disruptions, such as other conversations or equipment sounds.
Multiple Speakers: Often, consultations don’t involve just the clinician and the patient. The AI needs to discern and accurately document when multiple voices, such as those of family members or other caregivers, contribute to the conversation.
Multilingual Interactions: In encounters involving patients who speak other languages and their interpreters, the AI scribe must accurately capture the translation without getting tripped up by the presence of multiple languages.
Interruptions and Side Conversations: It’s not uncommon for an appointment to be momentarily sidetracked by an unrelated matter, like an urgent query about a different patient. The AI should maintain focus on the primary discussion without erroneously incorporating the content of the interruption into the clinical note.
Speakers with Varied Accents: Different accents shouldn’t throw the AI scribe off—it needs to be robust enough to understand and transcribe speech regardless of accent or dialect.
Technical Glitches: Finally, consider the AI scribe’s reliability in face of imperfect technology. How does it perform when confronted with unstable connectivity, or worse, a complete internet outage?
3) Workflow Details
The reality is that clinical workflows can differ substantially, not just from one organization to another but within the same institution. It’s critical to ensure that your AI scribe can support the particular workflows of your practice, department, or system. Here are some workflow-related factors that should be on your radar:
Support for Different Visit Modalities: Confirm that the AI scribe is versatile enough to handle various types of visits—be it in-person, video calls, or telephone appointments. Ensure it’s capable of smoothly handling transitions, such as switching from a video to a phone visit mid-consultation.
Support for Different Visit Types: Ensure the solution can support the documentation and workflow requirements of various types of visits—for example new patient visits, follow-ups, and annual wellness exams.
Support for Different Care Settings: Confirm that the AI scribe can handle the nuances of the various care settings in which you plan to use it, including both ambulatory and acute care settings as appropriate.
Degree of EHR Integration: Verify that the AI scribe can integrate with your organization’s EHR and that the depth of the integration is sufficient to support your desired workflows.
Pulling Data: Can the AI scribe extract pertinent information such as physician schedules and specifics about upcoming visits from the EHR? Can it pull forward and edit previous notes, for example to document care across multiple, longitudinal visits?
Pushing Data: Look into its ability to not only push notes into the EHR but also place them correctly within it, including into specific sections of the note and structured data fields, such as flowsheets.
Human-in-the-Loop Involvement: Determine whether the solution necessitates or offers the option of human verification before the clinical note is finalized. This might be a critical feature for some users who prefer an additional layer of review.
Handling Multiple Recordings: In cases where a visit may involve multiple audio inputs, such as the main visit dialogue supplemented by pre- and post-visit dictation, confirm that the AI scribe can integrate these to create one comprehensive note (including recordings from multiple users, such as a physician and a nurse practitioner or medical assistant, each of whom handle different aspects of the visit).
4) Enhanced Capabilities
Beyond the core functions of transcribing consultations and generating billing codes, some AI scribe vendors are offering additional capabilities. Several areas we’re beginning to see advancements include:
Diverse Note Types: Examples include patient-oriented visit summaries and instructions, prior authorization letters, and referral letters to other specialists.
Verbal Prompts: Some solutions can provide real-time, adaptive prompts to clinicians during patient encounters. Given that clinicians might not be interacting with the EHR during a visit any longer, these prompts can guide them on visit-specific necessary actions, like ordering particular tests or inquiring about prior diagnoses, without involving disruptive screen time.
Clinical Documentation Improvement (CDI) Capabilities: In addition to code generation, some solutions are starting to cover CDI, ensuring that the clinical documentation, CPT and ICD-10 codes all appropriately support one another. This aids in reimbursement, as well as compliance and quality reporting.
Platform Approach: Some vendors are taking a platform approach, offering integrations with third-party applications to extend their capabilities (e.g., incorporating clinical decision support tools).
5) Technical Considerations
Given the highly sensitive nature of the information being shared with AI scribe solutions, it’s critical to understand the underlying technology being used, how the data will be handled, and whether the vendor meets your security requirements. Here are some key technical considerations to evaluate:
Foundational Technology: While some details are likely to be proprietary, you should get a solid overview of how the solution works. Key questions might include the specifics of the large language model used (like GPT-4), the datasets it was trained on, and approaches to handle potential ‘hallucinations’ or inaccuracies in the generated content.
Data Privacy and Security:
Data Storage: Understand the specifics of data storage practices – find out what information is stored, where it is stored, and how long it is retained.
Data Processing: Likewise, understand the particulars regarding data processing – clarify what data will be processed, who will process it, and the steps taken to ensure continued privacy.
Data Security: Confirm that the data is secured while it’s moving (in transit) and when it’s stored (at rest).
Data Utilized for Model Training: Ensure you have the ability to specify how your data may (or may not) be used for improving the AI model.
Certifications: Finally, evaluate whether the solution meets your organization’s specific compliance needs. This includes HIPAA for patient privacy, along with certifications that might be necessary for your organization, like SOC 2 (types 1 and 2), HITRUST, GDPR for data protection within the EU, and ISO 27001 for information security.
6) Implementation and Support
When it comes to rolling out AI scribe technology, the scale and complexity of implementation and support can vary greatly based on the size and nature of your organization. Smaller practices might just need some basic training and a communication channel for troubleshooting, while larger institutions introducing the tech to numerous users across various departments will require a more comprehensive approach.
Here we highlight a few of the key elements to assess, but recommend prospective buyers spend significant time evaluating a vendor’s ability to support their needs in these areas.
Implementation Needs:
Custom Tuning: Determine if the AI scribe can be fine-tuned to align with the specific demands and preferences of your organization, department, subspecialty, or individual clinicians. Understand which aspects of the solution can be adjusted – for example, the note content, note structure, visit type, or how the notes are ingested into your EHR. Assess how prescriptive the fine-tuning process is – does it seek to mimic your style or does it allow you to specify exact preferences (e.g., “always capture hours of sleep in this format Hours of Sleep: X“)? Additionally, inquire about what this customization process entails, and whether it can be done directly by your team or if vendor involvement is required.
Initial Onboarding and Training: Assess the onboarding process for clinicians. How does the vendor ensure adoption and respond to queries or technical issues during the early stages of use? Can they support both 1-to-1 and group training sessions?
Support Considerations:
Ongoing User Support: Evaluate the ongoing support infrastructure for clinical users. How does the vendor offer help? Through direct communication channels like email and phone support, or are there messaging options like Slack or Microsoft Teams? Additionally, look into their service level agreements (SLAs) for response and resolution times.
Account Management: Assess the nature and quality of account management provided. Will you receive dedicated support and what does the cadence of regular check-ins look like?
ROI Demonstration: Examine how the vendor plans to showcase the return on investment (ROI) for their solution. Do they offer detailed reports or dashboards highlighting key performance indicators (KPIs) such as usage levels, time savings, and productivity improvements?
7) Pricing
Cost is always a crucial factor in any technology evaluation, AI scribes included. As you assess various vendors, make sure to clarify the following details regarding their pricing models and terms.
Price Structure: Is it subscription-based, such as per user per month? Or is it dependent on usage, such as per recorded session? Will there be a separate rate for different user types (e.g., physicians vs. advanced practice providers)?
Price Amount: What’s the actual cost of using the service? Does the price fluctuate based on the number of users or the volume of transcription sessions?
Contract Terms: Get clarity on the contract’s duration—what is the minimum commitment required by the vendor? Is there room to scale up or down based on your evolving needs without penalty?
This sector is evolving rapidly, and I’m glad that I’m not tied to an annual contract that would limit my ability to use new tools.
Product Manager, Digital Health Primary Care Provider
View full buyer transcript here.
How to evaluate AI scribes
Now that we’ve covered what to consider in an AI scribe, we’ll go over some best practices for evaluating these solutions.
1) Build a Comprehensive Evaluation Team
Though it may seem straightforward, assembling the right evaluation team is a step where many organizations stumble, often leading to complications down the line. The most common roles we see organizations overlook are:
Clinician Representatives from Key Departments: Be sure to include clinicians from the departments where the AI scribe solution will first be deployed. Ideally, aim to represent a range of specialties and subspecialties to ensure the technology can cater to the diverse requirements of different clinical teams.
Revenue Cycle Representatives: Consider including a revenue cycle director or a clinical documentation improvement (CDI) leader. A revenue cycle team member should be involved in evaluating the output of the AI scribe to determine how well it supports the organization’s reimbursement-related needs.
2) Align on Your Requirements
With your evaluation team ready, the next step is achieving consensus on the criteria for assessing AI scribe solutions. For help pulling together your initial list, refer to the breakdown above and our in-depth requirements grid.
3) Develop Your Vendor Consideration Set
At the time we’re writing this (early 2024) the AI scribe space is evolving rapidly, with new entrants coming into the market seemingly every day and incumbent software organizations rapidly adding scribing capabilities.
The market map below offers a snapshot of the vendors we’ve come across in the course of our research for this article.
4) Narrow Down to a Shortlist
Narrowing the broad set of vendors above into a manageable shortlist can be difficult and time consuming. Here are some tips to streamline this stage:
First, utilize an RFI or requirements grid to gather detailed information from each vendor. If you’re building off our template, many vendors will already be familiar with the format, which should make for a smoother process.
Second, take advantage of in-depth reviews found in our reviews library. These real-world insights can provide a clearer picture of what actual users think about their experience with the vendors in question.
Third, whenever possible, ask for live demonstrations of the AI scribe offerings. Some vendors might offer a hands-on trial of their solution that you can explore without sales intervention, while others may provide a live demo in a call, showcasing their solutions in action through realistic patient-clinician scenarios.
Key Elements of an Effective AI Scribe Testing Program
Program Set Up
Identify the clinical specialties where you expect to implement the AI scribe
Include clinical representatives from each of these specialties, as well as relevant revenue cycle representatives
Run multiple tests for each specialty, with different clinicians taking turns acting as the patient and the clinician
Have a separate clinician document the test appointment alongside the AI scribe as a comparator
Test Appointment Best Practices
Conduct visits in an unscripted manner and ensure they involve significant complexity, including lengthy medical histories and complicated symptoms
Run visits in busy environments, with lots of background noise
Interrupt visits with discussions of unrelated topics and patient situations
Incorporate multiple speakers and languages
Test technical difficulties, including wifi drops and switching from one device to another
Record pre-visit and post-visit dictations that need to be incorporated into the visit documentation
Evaluation Best Practices
Use the clinician-generated comparator output as the standard the clinical and revenue cycle representatives use to evaluate the AI scribe-generated note and billing codes against
Assess the end-to-end workflow, including the quality of the EHR integration and the ease of editing the output
5) Run a Pilot Program with Your Finalists
Given the pivotal role an AI scribe solution will play in your clinical processes and revenue cycle management, we strongly recommend conducting a pilot program with your top candidates.
Pitting two or three vendors against each other in a real-world setting can be incredibly informative, revealing the tangible differences in quality and performance under actual working conditions. Not only does this approach enable a thorough comparison, but it also positions you with an alternative option should something prevent you from moving ahead with your preferred choice.
To assess note quality, we plan to employ strategies including a note comparator tool. This tool will help us determine discrepancies between the notes that have been processed by [scribes] and the AI-generated final notes. That will help us distinguish between minor grammatical issues related to human processing and significant clinical inaccuracies, such as medication or diagnosis errors, made by [the AI scribe vendor], which could lead to adverse outcomes.
Director of Clinical Innovation, Large Health System
View full buyer transcript here.
6) Make Your Decision
If you’ve followed the steps to this point, you should be in a good position to make your decision:
You’ve engaged key stakeholders from various departments and included their feedback throughout the evaluation process.
You’ve conducted an exhaustive review of the available vendors in the market and systematically filtered them to identify the most promising fits for your needs.
You’ve rigorously put your finalists to the test within the context of your organization’s operations and have a clear understanding of which vendor stands out as the best match.
All that remains are the final steps of your procurement process: finalizing negotiations on pricing, working through the specifics of the contract, and performing a detailed security review to ensure compliance and data protection.
Where to go from here
At this point you should have everything you need to get started evaluating AI scribes. Our hope is that this guide, along with our template requirements grid and in-depth vendor reviews, can help get you much of the way there.
However, should you have more questions or need further assistance as you navigate this path, we encourage you to reach out. We want to support you in finding the right AI scribe solution for your organization, so please don’t hesitate to contact us if we can help in any way!