Researchers find that following 22% of Copilot’s medicine advice would kill you

If you’re in need of medical help, asking an AI chatbot for advice is not the way to go about it. If you need some convincing on why it’s such a bad idea, a recent study into Copilot’s medical prowess found it quite lacking. In fact, almost a quarter of the responses it gave could put you in a far worse state than before you took any medicine.

Researchers find that Copilot does not give good medical advice

In a paper published on Scimex titled “Don’t ditch your human GP for Dr Chatbot quite yet,” researchers asked 10 frequently asked questions about each of America’s 50 most prescribed drugs, generating 500 answers total. The researchers then went over the responses and graded how medically accurate they are. The AI got an average completion score of 77%, with the worst example being only 23% complete.

So, how accurate was Copilot in giving advice about medicine? The results were pretty scary:

Only 54% of answers agreed with the scientific consensus, the experts say. In terms of potential harm to patients, 42% of AI answers were considered to lead to moderate or mild harm, and 22% to death or severe harm. Only around a third (36%) were considered harmless, the authors say.

As such, if you need any medical advice, turning to an AI chatbot has a chance to do more harm than good. It’s best to instead consult someone who’s a professional and get their opinion instead. In the meantime, check out all the new stuff Copilot can actually do and leave the complicated stuff to the people with training.