Microsoft AI diagnostic tool hits 83% vs 36% for generalist doctors

Key takeaways

• Microsoft's AI diagnostic system, MAI-DxO, reached 83% accuracy on 304 difficult New England Journal of Medicine (NEJM) case records in a peer-reviewed study published in Science on October 22, 2026, according to Microsoft AI.
• Generalist physicians in the same setup scored 22% without reference material and 36% when allowed to use textbooks and the internet, Microsoft said.
• The study's own data suggest most of the gain comes from the underlying frontier models such as OpenAI's GPT-5 and GPT-5.5, with Microsoft's orchestration layer adding a smaller boost on top, according to TechCrunch's reading of the paper.
• The test used curated, retrospective case records rather than real patients. Specialists were not in the comparison, and the system is not cleared for clinical use.

What the Microsoft AI diagnostic study in Science found

Microsoft's AI diagnostic system, MAI-DxO, correctly diagnosed 83% of 304 difficult NEJM case records in a peer-reviewed study published in Science on October 22, 2026, against 36% for generalist doctors allowed to use textbooks and the internet, Microsoft AI said.

The system is called the Microsoft AI Diagnostic Orchestrator, or MAI-DxO. It does not use a new medical model. Instead, it coordinates several AI agents built on existing large language models. They act like a virtual panel of doctors: they ask questions, order tests, debate possible diagnoses over several rounds, and keep track of what the tests would cost.

The researchers tested it on a benchmark called SDBench (Sequential Diagnosis Benchmark). Each case is revealed step by step. The system, or the doctor, starts with a short presentation and must request further history, examination findings, or test results before committing to a diagnosis. The benchmark scores accuracy and the estimated cost of the tests requested.

The headline numbers

Microsoft AI's announcement gives these results for the published study:

Source: Microsoft AI, "Building Medical Superintelligence," October 22, 2026. Microsoft did not break down the physician results for the emergency department cases in its announcement.

Microsoft says the orchestrator was "more than twice as accurate as physicians using textbooks and the internet." It also says no other human or AI approach in the study was both more accurate and cheaper than MAI-DxO. The cost-optimised setting reached 81% on the NEJM cases at less than half the cost of the systems it was compared with, according to the company.

How the results compare with the 2025 preprint

The first version of the work, "Sequential Diagnosis with Language Models," was posted on arXiv on June 27, 2025, and revised on July 2, 2025. That version paired MAI-DxO with OpenAI's o3 model. It reported 80% accuracy in the default setting and 85.5% in the maximum-accuracy setting. The 21 practising US and UK physicians in that study averaged 20%.

The preprint figures drew criticism at the time because the doctors were not allowed to consult colleagues, textbooks, or generative AI. Microsoft's own 2025 announcement said this was meant to measure "raw" human performance, and listed it as a limitation.

The Science version changes that comparison. It adds a second physician group that could use reference material, and that group scored 36%. The headline AI figure is now 83% with GPT-5, slightly below the 85.5% peak in the preprint. The gap between machine and doctor is still large, but it is smaller than the roughly fourfold difference reported in 2025.

Sources: arXiv 2506.22405; Microsoft AI announcements of June 30, 2025 and October 22, 2026.

The models do most of the work

The most important caveat comes from the study's own data. TechCrunch, which reviewed the paper, reported that GPT-5 alone with a short, basic prompt scored 78.9% on the complex cases, at about $8,400 per case in estimated test costs. Adding MAI-DxO lifted accuracy by roughly four points and cut the estimated cost by about $3,000 per case.

With GPT-5.5, which OpenAI released in April 2026, the orchestrator raised complex-case accuracy from 72% to nearly 87% at a slightly lower cost, according to TechCrunch. The boost was largest for weaker models. Google's Gemini 2.5 Flash scored 46.4% on its own and nearly 69% with MAI-DxO.

On the everyday emergency department cases, the orchestrator added little accuracy or cost saving for either GPT-5 or GPT-5.5, TechCrunch reported. Both GPT-5.5 configurations scored about 85% at similar cost. TechCrunch also noted that the gap between weaker and stronger models was larger than the improvement MAI-DxO provided. In other words, the choice of model mattered more than the orchestration.

A Microsoft spokesperson told TechCrunch that the orchestration "helps with the broader challenge of optimization both accuracy and cost." Microsoft's announcement says the method held up across newer models without being changed, which suggests the approach is "not tied to one model generation."

What the study did not test

• Real patients. The NEJM cases are curated teaching records with known final diagnoses, and the emergency cases come from a research database. No patient was diagnosed or treated by the system.
• Specialists. The human comparison used primary care doctors and general internists. Hard cases like these would usually go to specialists or physician teams, which were not tested, TechCrunch noted.
• Clinical approval. Microsoft's 2025 announcement said the system was "not yet approved for clinical use" and would need safety testing, clinical validation, and regulatory review. The October 2026 announcement describes the work as research and a planned study, and makes no claim of regulatory clearance.
• Full reproducibility. Microsoft says it is open-sourcing SDBench, including its gatekeeper, judge, and cost-estimation code, but only for a selection of the NEJM cases.

Suleyman's "medical superintelligence" pitch

Suleyman told TechCrunch he expects Microsoft's next model to "make really enormous leaps here." He said the company wants "the absolute number one state-of-the-art medical clinical decision support products." Dominic King, vice president of health at Microsoft AI, said: "This technology is ready for real-world evaluations."

Microsoft says the next step is a prospective study with Beth Israel Deaconess Medical Center in Boston. In that study, patients will interact with the system directly, and its conclusions will be compared with physicians' independent assessments. The company is also working with Mayo Clinic on a healthcare foundation model. TechCrunch reported that a release is planned for next year. A companion paper, "The Building Blocks of Medical Superintelligence," appears in BMJ Digital Health and AI, according to Microsoft.

Microsoft is not alone. Google and OpenAI have made similar claims about their own medical AI, TechCrunch noted. The models MAI-DxO runs on best are OpenAI's, which shows how closely the AI labs' research is linked.

What it means

The peer-reviewed result is stronger evidence than the 2025 preprint. The doctors had access to references, the test set includes ordinary emergency cases, and the paper reports results across models from OpenAI, Google, Anthropic, Meta, and DeepSeek. The conclusion is narrower than the "AI beats doctors" headline suggests, though. On curated paper cases, frontier language models already outperform generalists working alone. Microsoft's orchestration makes them somewhat more accurate and cheaper, mostly on the hardest cases and with weaker models.

For hospitals and regulators, the open question is whether these gains survive contact with real patients. Real patients have incomplete histories, physical examinations, and their own preferences. The Beth Israel study is the first test of that. Until results are published, MAI-DxO remains a research system and not a clinical product. For readers following the wider AI agent race, the result is also a reminder that orchestration and prompting techniques add less as the underlying models improve. Bullquill's guides to building AI agents with tool calling and retrieval-augmented generation explain the same building blocks at a smaller scale. Our reports on OpenAI's agent monitoring and Gemini 4 Argon benchmarks cover how fast the base models are moving.

Sources

• Science: Microsoft AI sequential diagnosis paper (DOI 10.1126/science.aeh7930), October 22, 2026
• Microsoft AI: Building Medical Superintelligence (October 22, 2026)
• TechCrunch: Microsoft's Mustafa Suleyman says its AI outperforms doctors on hard cases (October 22, 2026)
• arXiv: Sequential Diagnosis with Language Models (2025 preprint)
• Microsoft AI: The Path to Medical Superintelligence (June 30, 2025)