In late 2023, a disability insurer's internal review uncovered a troubling pattern. Two teachers, both in their early 50s, both diagnosed with the same chronic condition, and both holding identical long-term disability policies, filed claims within weeks of each other. One claim was approved. The other was denied. The difference, it turned out, was not in their medical histories or policy terms, but in the artificial intelligence system that had underwritten their applications years earlier.

When Identical Claims Diverged

The case, described in a 2023 internal review that was later shared with regulators, involved two elementary school teachers from the same district. Both had purchased the same disability policy through their employer. Both developed a degenerative spinal condition that made full-time work impossible. Their claims were submitted almost simultaneously, and both were processed by the same claims system, which used the underwriting data from the original applications.

The AI underwriter that had evaluated their initial applications had assigned different risk scores to each teacher. The first teacher, whose claim was approved, had been scored as a lower risk. The second teacher, whose claim was denied, had been scored as higher risk, despite having nearly identical health profiles. The insurer's automated claims system used these scores to flag the second claim for denial, while the first sailed through.

When the review team dug into the model's logic, they found that the difference in scores traced back to a single variable: the second teacher's zip code. The model had learned, from historical data, that claims from that particular neighborhood had a slightly higher incidence of fraud. It applied that proxy to the individual, overriding her actual medical records. The first teacher lived in a different zip code, so she escaped the penalty.

This case, while anonymized in the review, became a touchstone for discussions about the fairness of AI-driven underwriting. It demonstrated, in stark terms, how a model trained on aggregate data can produce disparate outcomes for individuals who are, in every relevant way, identical. The insurer's own actuaries acknowledged that the zip code variable had no actuarial justification for this specific condition.

The review recommended changes, including the removal of geographic proxies and a mandatory human review for any denial where the claimant's physician disagreed with the insurer's assessment. But the damage was done. The denied teacher appealed, and after a lengthy process, her claim was ultimately approved. Yet the case raised questions that the industry is still grappling with.

The Black Box Underwriting Shift

Over the past decade, insurers have moved away from traditional rules-based underwriting, where human underwriters applied a fixed set of criteria, toward machine learning models that can analyze hundreds of variables and identify complex patterns. This shift promised greater efficiency and more accurate risk assessment. In practice, it has created a new set of challenges.

In many carriers, the human underwriter's role has changed from making decisions to auditing the AI's suggestions. A senior underwriter might review a sample of cases, or only those that fall outside certain confidence thresholds. For the vast majority of applications, the AI's decision is final. This is especially true in the group market, where policies are sold to employers and individual underwriting is often streamlined.

Claims adjudication has seen a similar automation. Some carriers now use algorithms to flag claims for review or even to approve straightforward cases automatically. The teacher's case illustrates what can happen when those algorithms are trained on biased data.

However, adoption is uneven. Large national carriers have invested heavily in proprietary models, while regional mutuals often rely on older systems or third-party software. A 2024 survey of U.S. insurers found that roughly 60% of large carriers used AI in some part of the underwriting or claims process, but only about a third of small carriers did. The disparity means that the regulatory and ethical questions are not yet universal.

Regulators have taken notice. The National Association of Insurance Commissioners has been developing model rules on the use of AI and algorithmic decision-making, focusing on transparency and fairness. Some state insurance departments have already issued guidance requiring insurers to be able to explain the factors that led to a decision, especially when it results in an adverse action.

Data Bias in Disability Models

The root cause of the divergent outcomes was data bias. The AI model had been trained on a large dataset of past disability claims, but that dataset was not representative of all occupations. Teachers, for example, were under-represented. When the model encountered teacher claims, it had less data to draw on, so it leaned on related variables, like geography or age, that were not truly predictive.

The colleague's file was flagged by proxy variables. The model inferred risk from non-medical factors, such as the applicant's credit history, neighborhood, and even the type of vehicle they drove. These variables, while legal in many states, can correlate with socioeconomic status and race, raising concerns about disparate impact.

Actuarial fairness is a long-standing principle in insurance. Premiums should reflect risk, not arbitrary characteristics. But when models are trained on historical data that itself contains biases, those biases become encoded in the algorithm. The teacher's case is a textbook example of what researchers call algorithmic bias.

One study, published in 2022, examined disability insurance claims across several carriers and found that Black applicants were denied at higher rates than white applicants, even when controlling for income and health status. The study's authors attributed part of the gap to the use of proxy variables in underwriting models.

The insurance industry has pushed back, arguing that the use of such variables is actuarially justified and that regulators have long permitted them. But the teacher's case, and others like it, are eroding that defense. When a model makes an error that a human would not have made, the cost of that error is borne by the most vulnerable applicants.

The Human Override That Saved a Claim

In the teacher's case, the initial denial was not the final word. Her attorney filed an appeal, citing the insurer's own policy language, which guaranteed a review by a senior underwriter if the claimant provided additional medical evidence. The senior underwriter, a veteran with 20 years of experience, reviewed the file and immediately spotted the anomaly.

The override protocol was invoked. The senior underwriter overruled the AI's decision, approved the claim, and ordered a review of the model's performance. This human intervention was critical, but it was also a matter of chance. The teacher's attorney knew the right language to trigger the appeal. Her colleague, who had been denied, did not have the same resources.

The colleague's denial was upheld without review. She had filed her claim pro se, without legal representation, and did not know that she could request a human review. The insurer's automated system marked her file as closed, and she did not pursue further action. Her claim was never reconsidered.

This disparity in outcomes, driven by access to information and advocacy, is a governance gap. Insurers may have robust human-review protocols in place, but they are not always communicated clearly to claimants. And when they are, the process can be opaque.

The teacher's case led the insurer to change its appeal process, making it easier for claimants to request a human review. But the broader industry has been slower to adopt such changes. Many carriers still treat AI decisions as final, with human review reserved for cases that meet certain criteria.

Regulatory and Legal Repercussions

The case prompted an inquiry by the state insurance department, which has the authority to examine underwriting practices. The inquiry focused on whether the insurer had violated state laws prohibiting unfair discrimination. The department's investigators found that the insurer had failed to document the rationale for the AI's decisions, making it impossible to determine whether discrimination had occurred.

The inquiry also raised the question of whether fair lending laws, which prohibit discrimination in credit and insurance, apply to AI underwriting. Some legal scholars argue that they do, and that insurers must be able to demonstrate that their models do not have a disparate impact on protected classes.

So far, fines have been modest. The insurer in question paid a penalty of a few hundred thousand dollars, a fraction of its annual revenue. But the precedent is significant. Regulators are signaling that they will not tolerate unexplained algorithmic decisions.

Proposed rules, still pending in some states, would require insurers to conduct bias testing before deploying models and to maintain model cards, which document the model's purpose, data, and limitations. These rules would apply to both underwriting and claims, and they would give regulators the tools to audit AI systems.

The industry has pushed back, arguing that such rules would be costly and would stifle innovation. But the teacher's case suggests that the cost of inaction may be higher. A single lawsuit, if it goes to class action, could dwarf any fine.

What Carriers Can Do Differently

Carriers can take several steps to prevent similar incidents. First, they should conduct mandatory bias testing before deploying any AI model. This testing should include not only statistical measures of discrimination but also scenario-based tests that simulate real-world claims.

Second, they should document an override trail for every denial. If a human reviewer overrides the AI's decision, that fact should be recorded, along with the reason. This creates a record that can be audited and helps identify when the model is systematically wrong.

Third, they should ensure that every claimant has access to human review. This means making the appeal process clear in policy documents and on the insurer's website. It also means not closing a file until the claimant has been explicitly offered the opportunity to speak with a human.

Fourth, they should publish transparent model cards for regulators. These cards would describe the variables used, the training data, and the model's known limitations. This transparency would build trust and reduce the likelihood of regulatory action.

These changes come with costs. Bias testing and model documentation require specialized expertise and can slow down the underwriting process. But the costs of a single high-profile mistake, in terms of legal fees, fines, and reputational damage, can be far greater. The teacher's case is a cautionary tale.

For Consumers: Read the Fine Print

Consumers can also take steps to protect themselves. When purchasing a disability or life insurance policy, ask whether the underwriting is automated. If it is, ask what factors the model considers and whether you can request a human review of the decision.

If your claim is denied, do not accept the decision at face value. Request a written explanation, and ask for a review by a human underwriter. Many insurers have processes in place, but they may not be prominently advertised.

Keep detailed medical records that align with the policy's definitions. If your condition is not well-documented, the AI may not be able to assess it accurately. A paper trail can help a human reviewer understand your case.

Know your state's appeal deadlines. Most states require insurers to respond to appeals within a certain timeframe, and missing a deadline can forfeit your rights. A quick internet search can tell you what those deadlines are.

Outcomes vary by jurisdiction. Some states have strong consumer protections, while others are more insurer-friendly. If you are considering a claim, it may be worth consulting an attorney who specializes in disability insurance.

Industry Responses and Counterarguments

Not all industry observers agree that the teacher's case signals a systemic problem. Some actuaries argue that AI models, despite occasional errors, are on balance more accurate than human underwriters, who are prone to inconsistency and fatigue. They point to studies showing that machine learning models can predict disability durations more accurately than traditional methods, reducing overall claim costs and enabling insurers to offer lower premiums.

Proponents of AI underwriting also note that human underwriters have their own biases, often unconscious, and that the use of standardized algorithms can actually reduce discrimination if designed carefully. The key, they argue, is not to abandon AI but to improve its oversight.

Others in the industry caution against overregulation. They warn that requiring extensive bias testing and model documentation could raise compliance costs, which would be passed on to consumers in the form of higher premiums. Smaller insurers, in particular, might struggle to meet new requirements, potentially reducing competition in the market.

Yet the teacher's case illustrates a fundamental tension: the very efficiency that makes AI attractive also makes it opaque. When a human underwriter makes a mistake, the claimant can often get an explanation. When an AI makes a mistake, the claimant may not even know that a model was involved. This asymmetry in information is a growing concern for consumer advocates.

Some carriers are experimenting with explainable AI, which uses simpler models that are easier to interpret, or with post-hoc explanations that attempt to clarify why a decision was made. These approaches are promising, but they are not yet widespread. The trade-off is often a slight reduction in predictive accuracy, which some carriers are unwilling to accept.

The debate is unlikely to be resolved soon. As AI models become more sophisticated, the potential for bias may increase even as the potential for accuracy improves. The teacher's case serves as a reminder that the ultimate goal of insurance is to provide financial security, and that technology should serve that goal, not undermine it.

For now, the burden falls on regulators to ensure that AI is used fairly, and on consumers to be vigilant. The teacher's colleague, who never got her review, is a reminder of what happens when the system fails. Her case, though less publicized, is equally important in the ongoing conversation about the future of insurance.

This article is for informational purposes only and does not constitute professional advice. Always consult a qualified professional for guidance on your specific situation.