Part thirteen of Healthcare in Retirement, and a turn in the road. For twelve parts I have written about one tricky Medicare situation at a time. This week I want to write about the tool more and more of you are using to make sense of those situations, because I keep hearing the same three sentences about it. I am not an advisor. I am an engineer, and I build and run an AI assistant every working day. It drafts, it checks, it books, and it shows me where it got each thing. So when I hear people my readers' age describe an AI that makes up sources, gets facts wrong, and argues, I recognize the tool they are describing. I stopped using that one a while ago. This is my perspective on the gap between the AI people tried and the AI that exists now, and how to close it without taking anyone's word for it, including mine.
Who is actually using this?
More of you than the stereotype allows. Pew Research surveyed 5,119 adults in February 2026 and found that 23% of adults 65 and older have used an AI chatbot, though only 7% use one daily and only 6% describe themselves as confident with one. AARP's technology survey of adults 50 and older, fielded in the fall of 2025, found that "AI usage has nearly doubled, rising from 18% in 2024 to 30% in 2025." So roughly one in four of you has tried it, most of you were not impressed enough to make it a habit, and almost none of you feel sure-footed. That matches what I hear. It also matches the tool most people tried, which is the point of this article.
Why does my AI make up sources?
Because the one you tried was answering from memory. An AI language model is built from an enormous amount of text, and when you ask it a question it produces the most likely next words. If nothing connects it to the live internet, it cannot look anything up. It can only recall, and when it recalls a source it may assemble a plausible-looking title, author, and date the way it assembles any other sentence. That is where the fake citations came from, and in 2023 that was every assistant. What changed is that the major assistants added live web search with links you can click. One of them described its March 2025 release this way: when it uses the web, "it provides direct citations so you can easily fact check sources." The others added the same capability in the same window, from late 2024 into 2025. A source you can open is a different thing from a source the model remembers. The test takes one minute. Ask any Medicare question, then type: "Give me the link to the government page you are relying on." Open the link. Does the page exist, and does it say what the assistant said? If the answer is no, you are holding the old tool, or the old setting, and nothing it tells you about Medicare should move money until a human checks it.
Is the AI wrong about Medicare?
Less often than people are, on the general questions, and I did not expect that. In June 2024, three researchers put 77 real Medicare questions to four assistants and published the results in The Gerontologist. The two chat models scored 97.4% and 93.4% correct. The beneficiaries those questions were originally written for scored 55.4% overall. The two voice assistants, the ones people talk to from across the room, scored 30.3% and 2.63%. Those were 2024 versions and every one of them has changed since, so hold the numbers loosely. But the shape of the finding has held up in my own use: a modern chat model knows what a Part D penalty is better than most of us do, and the little speaker on the counter does not. Emily Langston, Neil Charness, and Walter Boot wrote that paper, and their title was a question, "Are Virtual Assistants Trustworthy for Medicare Information?" Their answer was: depends entirely which one.
Now the honest half. Newer is not automatically more accurate. In April 2025 the most widely used assistant's maker published test results showing its newer reasoning model invented facts about people 33% of the time on one benchmark, against 16% for the model it replaced, and said more research was needed to understand why. In January 2026 the biggest search engine removed its AI summaries for certain medical searches after a newspaper investigation found the summaries gave "normal" lab ranges that did not account for age or sex. These are current products from the largest companies in the field. The problem got smaller. It did not go away. So the second test: ask the assistant, "What is the Medicare Part B premium for 2027?" The correct answer, today, is that nobody knows. Medicare announced the 2026 premium of $202.90 on November 14, 2025, and the 2027 figure will arrive on roughly the same schedule. An assistant that tells you the 2027 number with a straight face is guessing, and an assistant that says "that has not been announced yet, here is the 2026 figure and where it came from" is the one you want.
Why does it argue with me, or agree with everything I say?
Both complaints are real and they are opposite failures of the same dial. On April 29, 2025, the most widely used assistant's maker rolled back an update within days of shipping it, because, in the reporting of VentureBeat's Carl Franzen, the assistant "began offering uncritical praise for virtually any user idea, no matter how impractical, inappropriate, or even harmful." The company's own explanation was that it had leaned too hard on short-term thumbs-up signals from users and built a tool that "leaned too far into affirmation without discernment." Read that again with Medicare in mind. An assistant that agrees with you is the dangerous one. If you tell it your Part B has no premium because a neighbor said so, you need it to say no, and to show you the page. That is not arguing. That is the job. The old failure was an assistant that dug in on a wrong answer with no source. The current design goal, at least on paper, is an assistant that pushes back with one. Third test: tell it something you know is false about Medicare and see whether it corrects you, politely, with a link. If it just agrees, put it down.
How do I make it show its work?
The three tests above are the habit, and the habit matters more than the tool. Ask for the source. Open the source. Notice when it should say "not announced yet." I would add one more rule that I use every day: an answer I cannot trace is a draft, not a fact. In my work the assistant hands me a paragraph and a link, and I open the link before the paragraph goes anywhere. That is the whole difference between people who get burned by these tools and people who get real work out of them. It is not age and it is not intelligence. It is whether anybody checks before the answer counts. In Medicare, an unchecked answer has a price that this series has already worked out: a Part D late enrollment penalty runs for life, and the Medigap door that opens at 65 locks behind you. Nobody should learn either one from an assistant that was guessing.
So which AI should I use?
I am not going to name products, and not only because they change every few months. The category is what matters. Use an assistant that can search the live web and shows you links. Turn that feature on if it is a setting. Ask it Medicare questions the way you would ask a well-read friend: to explain a letter, to define a term, to draft the questions you will bring to a professional. Do not ask it which plan to pick, because the honest answer depends on your county, your doctors, your prescriptions, and next year's plan list, and the rest of this week is about exactly what an assistant cannot know about you no matter how good it gets.
What a Certified Medicare Planner® does differently
They already do the three tests, on you, for a living. They will not tell you a 2027 number that has not been published. They pull the government page, not their memory of it, and they know which page. And they push back, with the source, when a client arrives certain of something a neighbor or a chatbot said. What they add that no assistant can is the part that is not on the internet: your actual doctors in next year's directory, your actual prescriptions at your actual pharmacy, and the plan list for your county the day it goes live. Bring them the answer the AI gave you. They love a client who did the reading. The advisors are at 602-281-3898.
Friday, October 2. Foothills Library, Glendale, 10:30 AM. Mustang Library, Scottsdale, 2:30 PM. No cost. Register at 123easymedicare.com/medicare-workshop or call (877) 220-1089.
Next in Healthcare in Retirement: the seven things no AI can know about your Medicare, however good it gets. One of them is a number that is public today and one is a number that will not exist until November, and an assistant that answers both with the same confidence just told you everything you need to know about it.
This event is presented by 123EasyMedicare, a brand of American Retirement Advisors, an independent, private organization. It is not sponsored by, endorsed by, or affiliated with Medicare, the Centers for Medicare & Medicaid Services, the Social Security Administration, or any government agency. The libraries are not sponsors of, and are not affiliated with, this event. Educational only; not tax, legal, or insurance advice.
You Finished the Series
Up next: Both Ends of the Table →A seven-part series on the largest wealth transfer in history — for the families on both sides of it.
Disclaimer: This article is for educational purposes only. It is not sponsored, endorsed, or otherwise representative of Medicare or the federal Medicare program. American Retirement Advisors is not a government agency. For official Medicare information, visit medicare.gov or call 1-800-MEDICARE (1-800-633-4227).