I think you'd be incredibly surprised how often charts are super, super incomplete or wrong. Like "pt has no pancreas and presented with pain and weeping from a 6yo pancreatectomy scar" but the chart doesn't mention the surgery or the entire missing organ wrong. Like "pt is a twin whose sibling died traumatically of cancer in front of them a year ago and presents with probable hypochondria about cancer" but the chart doesn't mention any family history wrong. Like "lifelong history of severe cognitive impairment substantiated by a psych eval; attended annual physical before being sent to imaging for head trauma because of observed impairment" but the chart doesn't mention cognition (someone was too polite to note it) nor the psych eval (records sharing wasn't allowed) wrong.

Those are a very few examples off the top of my head. I worked in EMR. I don't know shit about medicine, but man, do I know a lot about the complaints physicians and their staff send when they think it's the records system's fault that the chart was wrong or missing info.

In a big chunk of cases, the MD/NP/whatever's in-person role is determining what's not on the chart so that they can then ask appropriate follow-up questions. Given the massive range of possible dx for a given issue, and how much of getting the right dx doesn't have to do with probabilities/numbers of similar patients with the same symptom:dx data that'd be in the training set, I have major doubt that an LLM can appropriately intuit or appropriately question in order to diagnose.