Northwestern Medicine’s RMS: Improving Patient Care with AI Imaging Innovation

Posted May 11, 2022 | Technology |
reminder alert

Northwestern Medicine has built a natural language processing (NLP) system to analyze reports associated with medical imaging studies in order to identify incidental findings unrelated to the original indication for a study.


This artificial intelligence (AI) application addresses an important problem in healthcare where incidental findings in imaging studies are sometimes accidentally overlooked, resulting in treatment for a patient being delayed, and which can lead to poorer outcomes.

This is important because medical imaging is one of the most widely used forms of diagnostics by healthcare providers. Consequently, according to Northwestern Medicine officials, delayed and missed follow-ups on incidental findings in diagnostic imaging studies is a significant problem in healthcare today because it has the potential to harm millions of patients as well as increase costs for providers.

The AI application — named the Results Management System (RMS) — is innovative for a number of other reasons. First, it is fully integrated within the hospital's electronic health record (EHR) system, so that it examines (in real time) nearly every imaging study ordered in Northwestern Medicine’s health network comprised of 11 hospitals.

This embedded functionality enables RMS to automatically alert the patient’s care team and the patient in order to prevent delayed and missed care. Once the application triggers an alert, it will track the completion of recommended follow-ups for both the incidental and expected findings. If action is not taken to schedule imaging or other follow-up actions, RMS will generate another alert that is sent to the patient and their care team. (Patients receive an alert in their online portal — including the results of the imaging study. If they don’t have an online portal, they receive a letter with the results and/or are managed by a team of follow-up nurses.) Importantly, these processes all take place within the EHR system workflow, meaning that physicians and other care team members do not have to access another separate system.

Application Design & Development

RMS was completely developed in-house at Northwestern Medicine by a multidisciplinary team consisting of members from radiology, patient safety, quality, process improvement, primary care, nursing, and healthcare informatics.

Initial application design started in August 2018, with actual implementation of RMS within the EHR system taking place in December 2020. Continuous monitoring of the use of RMS and model performance is ongoing to this day. The current version of RMS focuses on searching for lung and adrenal findings in radiographic imaging notes. Lung findings were selected because, according to Northwestern Medicine designers and developers, they are among the most commonly encountered radiographic findings that require additional follow-up. Moreover, given the anticipated high volume of lung findings and well-established follow-up guidelines, application designers and developers deemed that lung findings detection was both a realistic and impactful use case for clinical implementation of the RMS application.

In comparison, adrenal findings have a much lower incidence. Consequently, they were selected to test the limits of the application under development and to identify challenges that would be necessary to overcome when it came time to expand RMS to analyze other findings in the future.

Model Development

Developers built four separate neural net models — including models based on deep learning architectures — to extract and classify target information from radiology reports. Developers continue to refine and update the models; the latest focus involves using RoBERTa — the most recent improvement on the BERT transformer architecture — to update the RMS application's classification models.

One particularly interesting aspect of application development was Northwestern Medicine’s use of its own clinical experts — in this case nurses — for labeling the training and testing data used for building RMS’s neural net models. Data acquisition and labeling — which requires experts to pour through and annotate large data sets — is one of the most time-consuming tasks when it comes to AI model development. Organizations frequently turn to outside, third-party companies with experts specializing in such efforts for particular industries and applications to assist them with their AI projects.

Northwestern Medicine developers decided to use the organization’s own nurses and other front-line staff to handle the data labeling part of the project due to data safety and privacy concerns. They also felt it would give them finer control over the quality of annotations — based on the fact that identifying findings and respective recommended follow-ups in radiology reports requires a certain degree of medical knowledge.

The staff, who were on light-duty restrictions and therefore unable to perform their regular clinical duties, were trained as annotators and were used to select relevant text from curated radiology reports (with many of them working remotely). According to Northwestern Medicine officials, use of in-house staff for data labeling resulted in higher quality and expertly reviewed data that was cost effective.

The Catalyst Study

Northwestern Medicine tested RMS for more than a year. In March 2022, it published the results of the study in the NEJM Catalyst Innovations in Care Delivery, which examines a year of data following live implementation of RMS in December 2020.

One of the key points of the Catalyst study indicates just how common incidental findings in diagnostic imaging actually are: out of the more than 460,000 imaging studies that were screened by RMS, 23,000 were flagged as containing lung follow-up recommendations. This represents a 5% rate of lung-finding occurrence on relevant imaging studies. In other words, 68 findings per day required follow-up!


Northwestern Medicine has developed a very innovative NLP application — integrated within the hospital EHR system — designed to solve a practical and pressing healthcare problem: missed follow-ups associated with incidental findings from diagnostic imaging.

In its current form, RMS is tailored to search for lung and adrenal findings. However, Northwestern Medicine is now working on expanding the system to cover hepatic, thyroid, and ovarian findings that require follow-up. It will be interesting to see how this upgrading works out, and we look forward to examining subsequent studies on the healthcare system’s latest effort in this area.

If interested, the full study can be found here. Northwestern Medicine also has a companion website covering the process of building and deploying the RMS system that other hospitals and healthcare providers can refer to when implementing similar results management applications.

Finally, I’d like to get your opinion on AI alerting and advisory systems for healthcare and other industries. As always, your comments will be held in strict confidence. You can email me at or call +1 510 356 7299 with your comments.

Photo by Kelly Sikkema on Unsplash

About The Author
Curt Hall
Curt Hall is a Cutter Expert and a member of Arthur D. Little’s AMP open consulting network. He has extensive experience as an IT analyst covering technology and application development trends, markets, software, and services. Mr. Hall's expertise includes artificial intelligence (AI), machine learning (ML), intelligent process automation (IPA), natural language processing (NLP) and conversational computing, blockchain for business, and customer… Read More