AI in healthcare: applications, limits and accountability

You have probably heard that artificial intelligence will transform medicine. Part of that is already happening: there are systems that help read mammograms, summarize consultations and answer health questions in plain language. But the distance between a good test result and a real benefit for the patient can be large.

This article explains where AI shows up in healthcare, what peer-reviewed studies show about its limits and how accountability is distributed in Brazil. It is not medical advice. For any decision about your health, see a licensed professional.

Where AI shows up in healthcare

The World Health Organization (WHO) grouped the uses of so-called large multi-modal models (LMMs) into five categories. These are generative AI models capable of receiving and producing more than one type of data, such as text and images. According to the WHO guidance of January 2024, the anticipated uses are:

• diagnosis and clinical care, such as answering patients’ written questions;
• patient-guided use, such as researching symptoms and treatments;
• administrative tasks, such as documenting and summarizing consultations;
• medical and nursing education, including simulated patient encounters;
• scientific research and drug development.

Beyond generative AI, there is an older and more regulated category: software that analyzes clinical data for a specific purpose, such as flagging suspicious areas in an imaging exam. In the United States, the FDA, the agency that regulates food and drugs, maintains a public list of authorized AI-enabled medical devices. Most entries are in radiology. The FDA itself says it is still studying how to identify, in the list, devices based on foundation models, the family of models to which LLMs belong.

The difference matters to you. A tool designed and validated for a narrow task is one thing. A general-purpose chat assistant, trained to answer questions on any subject, is another (see what LLMs are).

What the evidence shows when the task is narrowly defined

One of the most cited examples is the MASAI trial, conducted in Sweden. More than 80,000 women in breast cancer screening were randomly assigned to two groups: AI-supported mammogram reading or traditional double reading by two radiologists.

In the safety analysis published in Lancet Oncology in August 2023, the AI group detected 6.1 cancers per 1,000 women, compared with 5.1 per 1,000 in the traditional group. The radiologists’ reading workload fell by 44.3%. The rate of women recalled for additional tests was similar: 2.2% with AI and 2.0% without.

Two details prevent overstated readings. First, AI did not replace the doctors: it helped decide which exams needed more attention. Second, the study’s primary outcome, the rate of cancers that appear between one screening and the next, still depended on longer follow-up when this analysis was published. The authors presented the result as evidence of safety, not as definitive proof of benefit.

Where AI fails

Performance that does not carry over elsewhere

In 2021, researchers at the University of Michigan tested, with their own data, a commercial sepsis prediction model used in hundreds of American hospitals. Sepsis is the body’s severe response to an infection. The study, published in JAMA Internal Medicine, analyzed 38,455 hospitalizations.

The model failed to identify 67% of patients who developed sepsis and generated alerts for 18% of all hospitalizations. The discrimination measure, called AUROC (ranging from 0.5, equivalent to chance, to 1, a perfect score), was 0.63. The authors concluded that performance was poor and questioned widespread adoption without external validation. The lesson applies to any tool: a good number in one hospital does not guarantee the same number in another.

Bias inherited from the data

A study published in Science in 2019 analyzed a commercial algorithm used to select patients for high-risk care management programs in the United States. At the same risk score, Black patients were sicker than white patients.

The cause lay in the choice of target. The algorithm predicted healthcare costs, not healthcare needs. Because access to care was unequal, less money was spent on Black patients with the same needs. According to the authors, correcting this distortion would raise the share of Black patients referred for additional help from 17.7% to 46.5%. Nobody programmed the bias on purpose. It came from a shortcut in the data.

The model’s knowledge does not reach the user intact

Language models tend to perform well on medical exams. A study led by Andrew Bean and colleagues showed why that is not enough. In an April 2025 preprint, 1,298 participants received ten medical scenarios and were randomly assigned to use one of three language models or a source of their choice.

On their own, the models identified the correct conditions in 94.9% of cases. When people used these models, relevant conditions were identified in fewer than 34.5% of cases, and the appropriate course of action, such as going to the emergency room or booking an appointment, in fewer than 44.2%. The result was no better than the control group’s. The authors point to the interaction between person and machine as the central obstacle and recommend testing with real users before any public use.

There is also the risk of the model confidently stating something false, a phenomenon explained in AI hallucinations. Among the risks of LMMs, the WHO lists false, inaccurate, biased or incomplete responses, low-quality training data, cybersecurity failures and automation bias: the tendency to trust the machine too much and let errors slip through.

Who is accountable for decisions in Brazil

Federal Council of Medicine (CFM)

CFM Resolution No. 2,454/2026, issued by Brazil’s Federal Council of Medicine and published on February 27, 2026, regulates the use of AI in medicine. It takes effect 180 days after publication. The key points for patients:

• physicians must use AI only as a support tool and remain ultimately responsible for clinical, diagnostic, therapeutic and prognostic decisions;
• delegating to AI the communication of diagnoses, prognoses or treatment decisions without human mediation is prohibited;
• you have the right to be clearly informed when AI is used as relevant support in your care, and you may refuse that use;
• the use of AI as decision support must be recorded in the medical record;
• institutions must classify systems by risk (low, medium, high or unacceptable) and maintain governance, auditing and monitoring.

The resolution also protects physicians’ autonomy. They cannot be required to follow AI recommendations automatically and should not be penalized for not following them, provided they act in accordance with technical and ethical standards.

Anvisa

When software has a medical purpose, it may be classified as software as a medical device (SaMD) and must be cleared by Anvisa, Brazil’s health regulatory agency, under RDC No. 657/2022. Wellness software, such as fitness apps, and purely administrative systems fall outside this classification.

In its Q&A document on the rule, from September 2022, Anvisa explains that the use of AI does not change the classification: what counts is the intended use. Manufacturers of AI software must describe the databases used for training and verification, with their origin, size and statistical justification. The agency also noted that, at that date, there was no cleared product with continuous-learning AI, that is, AI whose code keeps changing after reaching the market.

International principles

In 2021, the WHO published six principles for AI in health (WHO, June 2021): protecting human autonomy; promoting well-being, safety and the public interest; ensuring transparency and explainability; fostering responsibility and accountability; ensuring inclusiveness and equity; and promoting AI that is responsive and sustainable. The organization warns against overestimating the benefits of AI and notes that systems trained on data from high-income countries may perform poorly in other settings.

How to use this information

In the editorial team’s assessment, a few questions help separate responsible use from empty promises, whether you are a patient, a manager or a professional:

• What task was the tool validated for? A system tested to support mammogram reading has not been tested for anything else.
• Was it validated outside the place where it was built? The sepsis model case shows why this matters.
• Does the data represent the population being served? The Science study shows how a poorly chosen target creates inequality.
• Is there a professional responsible for the final decision? In Brazil, the CFM requires this in medicine.
• Is the product cleared, when it has a medical purpose? That is Anvisa’s requirement for software that qualifies as a medical device.

If you use a chat assistant to understand a term or prepare for an appointment, treat the answer as a starting point, never as a diagnosis. The criteria in how to evaluate AI responses help you check what the model said. Symptoms, tests and treatments should be discussed with a healthcare professional.

Key takeaways

AI already shows concrete results in narrow, well-tested tasks, such as support for breast cancer screening. The same studies indicate where it fails: performance that does not carry over to another hospital, biases inherited from the data and a wide gap between being able to pass an exam and helping a real person.

That is why Brazilian and international rules converge on one point: AI supports, but the decision and the accountability remain human. For you, this means the right to know when it is used in your care and to refuse that use.