A 2019 study published in Science revealed significant racial bias in a widely used healthcare algorithm. The algorithm, designed to identify high-risk patients for additional medical care, prioritized white patients over Black patients with similar medical conditions. This bias arose because the algorithm used predicted future healthcare costs as a proxy for health risk, unintentionally reinforcing racial disparities in medical care access.
By relying on financial data rather than direct health indicators, the algorithm systematically underestimated the medical needs of Black patients, leading to inequitable access to necessary healthcare interventions. This case study highlights the risks of integrating machine learning into critical decision-making without carefully considering systemic biases.
The Role of Predictive Algorithms in Healthcare
Hospitals and insurance companies employ predictive algorithms to determine which patients should be enrolled in special care programs. These programs are designed to:
- Provide more frequent check-ups and medical follow-ups
- Offer additional nursing and support services
- Improve chronic disease management
- Reduce emergency room visits and hospitalizations
The algorithm in question was intended to optimize healthcare resources by identifying patients who would benefit the most from these programs. However, its design introduced significant bias, leading to unintended but serious consequences.
How the Algorithm Functioned
The healthcare algorithm assigned patients a risk score based on their predicted future healthcare costs. The process involved three key steps:
1. Data Collection
The algorithm analyzed historical patient records, including:
- Diagnosed medical conditions
- Previous hospital visits and prescribed medications
- Total healthcare costs incurred in previous years
2. Risk Prediction
Using this data, the algorithm predicted how much each patient would cost the healthcare system in the future.
3. Decision-Making
Patients with high predicted costs were classified as high-risk and prioritized for extra care programs, while those with lower predicted costs were not given access to additional services.
The Fundamental Flaw: Cost as a Proxy for Medical Need
The primary issue with the algorithm was its reliance on future healthcare costs as a substitute for actual medical risk. The system operated under the assumption that patients who incurred lower medical costs were healthier and required less intervention.
However, this assumption failed to account for systemic disparities in healthcare access. Black patients, on average, incur lower healthcare costs not because they are healthier, but because they receive less medical care due to economic, geographic, and systemic barriers. This led to a significant underestimation of their actual medical risk.
Impact on Patient Enrollment
The result of this flawed methodology was that Black patients were significantly under-enrolled in the extra care programs. The study found that at the same algorithm-assigned risk score, Black patients were sicker than white patients, yet were still less likely to be selected for additional care.
A direct comparison illustrates the bias:
- White Patient (Alice): Diagnosed with hypertension and diabetes, visits the doctor regularly, and incurs $10,000 in healthcare costs per year. The algorithm classifies Alice as high-risk, granting her access to additional care.
- Black Patient (Bob): Diagnosed with the same conditions but has fewer doctor visits and incurs only $5,000 in healthcare costs. The algorithm classifies Bob as low-risk, denying him access to the same extra care.
Since the model did not measure actual health indicators, it perpetuated racial disparities by failing to provide necessary medical support to those who needed it most.
Key Findings from the Study
1. Racial Disparities in Enrollment
- Black patients were enrolled in extra care programs at half the rate of white patients with similar medical conditions.
- Without this bias, the number of Black patients receiving additional medical support should have been nearly twice as high.
2. The Root Cause of the Bias
- The algorithm treated lower past healthcare spending as an indicator of better health, which was incorrect.
- Black patients often receive less medical care due to systemic inequities, leading to lower recorded healthcare costs despite equal or greater medical need.
3. Potential Solutions
- The study suggested replacing cost-based risk scoring with direct health indicators, such as:
- Number of chronic conditions
- Frequency of hospital admissions
- Blood pressure and blood sugar levels for conditions like hypertension and diabetes
- According to the study’s simulations, these adjustments would have reduced racial bias in the algorithm by 84%.
Why Was Cost Chosen as the Risk Measure?
The company that built the algorithm likely chose cost as the primary risk factor for several reasons:
- Cost is easy to measure. Unlike medical conditions or disease severity, cost data is well-documented in healthcare systems.
- Hospitals and insurance companies prioritize financial metrics.
Since these organizations manage budgets and resources, they often use spending patterns to identify high-risk patients. - Cost correlates with health risk—but not perfectly. While high-cost patients often have greater health needs, this correlation breaks down in the presence of systemic inequities (e.g., racial disparities in healthcare access).
Challenges in Fair AI Implementation
- Bias in Historical Data: Algorithms that rely on past data without accounting for systemic discrimination will continue to reflect historical inequalities.
- Ethical Responsibility in AI Design: Organizations deploying machine learning in healthcare must ensure that the chosen metrics accurately reflect medical risk, not just financial considerations.
- The Need for Regular Audits: AI models should be subject to continuous evaluation and refinement to identify and mitigate hidden biases before they impact real-world patient care.
The Aftermath: What Changed?
Following the publication of the study, the company responsible for the algorithm, Optum (a subsidiary of UnitedHealth Group), agreed to modify the model to reduce racial bias. Additionally:
- Healthcare providers began reassessing their use of AI in decision-making.
- Policymakers and researchers increased scrutiny on how machine learning models are designed and deployed in clinical settings.
To ensure fairness in AI-driven healthcare, stakeholders—including hospitals, insurers, policymakers, and AI developers—must adopt more rigorous bias detection, transparent model design, and ethical oversight. AI is shaping the future of medicine, but without deliberate intervention, it risks reinforcing the very inequities it seeks to address.
For those interested in reading the original study, you can find it here:
Link to the case study: Dissecting racial bias in an algorithm used to manage the health of populations (Published in Science, 2019).
Leave a Reply