ChatGPT Does As Well As Humans Answering Genetics Questions, Study Finds

ChatGPT did as well as human respondents on multiple-choice questions related to genetics, though sometimes it gave different answers to the same question when asked it more than once, a new study in the European Journal of Human Genetics reports. A pair of researchers from the National Human Genome Research Institute asked ChatGPT, a large-language model, 85 questions that had previously been posed via the social media platforms Twitter and Mastodon. All questions, they note, were posed in 2021 or later, as ChatGPT was trained using earlier data. Overall, ChatGPT was 68.2 percent accurate, while human respondents were 66.6 percent accurate, and both groups performed better on questions that were focused on memorization rather than critical thinking. But the researchers also found that when the same questions were posed to ChatGPT again, for 16.5 percent, it gave a different answer the second time around. The researchers write that they "were impressed with ChatGPT’s performance" but caution that it will give incorrect and inconsistent answers while supplying plausible-sounding explanations. "In medical or other high-stakes settings, this is concerning," they add.