ChatGPT shows 'impressive' accuracy in clinical decision making

by Mass General Brigham

Credit: Pixabay/CC0 Public Domain

A new study led by investigators from Mass General Brigham has found that ChatGPT was about 72% accurate in overall clinical decision making, from coming up with possible diagnoses to making final diagnoses and care management decisions.

19 sept 2023--The large-language model (LLM) artificial intelligence chatbot performed equally well in both primary care and emergency settings across all medical specialties. The research team's results are published in the Journal of Medical Internet Research.

"Our paper comprehensively assesses decision support via ChatGPT from the very beginning of working with a patient through the entire care scenario, from differential diagnosis all the way through testing, diagnosis, and management," said corresponding author Marc Succi, MD, associate chair of innovation and commercialization and strategic innovation leader at Mass General Brigham and executive director of the MESH Incubator.

"No real benchmarks exists, but we estimate this performance to be at the level of someone who has just graduated from medical school, such as an intern or resident. This tells us that LLMs in general have the potential to be an augmenting tool for the practice of medicine and support clinical decision making with impressive accuracy."

Changes in artificial intelligence technology are occurring at a fast pace and transforming many industries, including health care. But the capacity of LLMs to assist in the full scope of clinical care has not yet been studied.

In this comprehensive, cross-specialty study of how LLMs could be used in clinical advisement and decision making, Succi and his team tested the hypothesis that ChatGPT would be able to work through an entire clinical encounter with a patient and recommend a diagnostic workup, decide the clinical management course, and ultimately make the final diagnosis.

The study was done by pasting successive portions of 36 standardized, published clinical vignettes into ChatGPT. The tool first was asked to come up with a set of possible, or differential, diagnoses based on the patient's initial information, which included age, gender, symptoms, and whether the case was an emergency.

ChatGPT was then given additional pieces of information and asked to make management decisions as well as give a final diagnosis—simulating the entire process of seeing a real patient.

The team compared ChatGPT's accuracy on differential diagnosis, diagnostic testing, final diagnosis, and management in a structured blinded process, awarding points for correct answers and using linear regressions to assess the relationship between ChatGPT's performance and the vignette's demographic information.

The researchers found that overall, ChatGPT was about 72% accurate and that it was best in making a final diagnosis, where it was 77% accurate. It was lowest-performing in making differential diagnoses, where it was only 60% accurate. It was only 68% accurate in clinical management decisions, such as figuring out what medications to treat the patient with after arriving at the correct diagnosis.

Other notable findings from the study included that ChatGPT's answers did not show gender bias and that its overall performance was steady across both primary and emergency care.

"ChatGPT struggled with differential diagnosis, which is the meat and potatoes of medicine when a physician has to figure out what to do," said Succi. "That is important because it tells us where physicians are truly experts and adding the most value—in the early stages of patient care with little presenting information, when a list of possible diagnoses is needed."

The authors note that before tools like ChatGPT can be considered for integration into clinical care, more benchmark research and regulatory guidance is needed. Next, Succi's team is looking at whether AI tools can improve patient care and outcomes in hospitals' resource-constrained areas.

The emergence of artificial intelligence tools in health has been groundbreaking and has the potential to positively reshape the continuum of care. Mass General Brigham, as one of the nation's top integrated academic health systems and largest innovation enterprises, is leading the way in conducting rigorous research on new and emerging technologies to inform the responsible incorporation of AI into care delivery, workforce support, and administrative processes.

"Mass General Brigham sees great promise for LLMs to help improve care delivery and clinician experience," said co-author Adam Landman, MD, MS, MIS, MHS, chief information officer and senior vice president of digital at Mass General Brigham.

"We are currently evaluating LLM solutions that assist with clinical documentation and draft responses to patient messages with focus on understanding their accuracy, reliability, safety, and equity. Rigorous studies like this one are needed before we integrate LLM tools into clinical care."

More information: A Rao et al., Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow: Development and Usability Study, Journal of Medical Internet Research (2023). DOI: 10.2196/48659

 

How artificial intelligence gave a paralyzed woman her voice back

by Stanford University Medical Center

A research participant in the Dr. Edward Chang’s study of speech neuroprostheses, is connected to computers that translate her brain signals as she attempts to speak into the speech and facial movements of an avatar on Monday, May 22, 2023, in El Cerrito, Calif. At left is UCSF clinical research coordinator Max Dougherty. Credit: Noah Berger

Pat Bennett's prescription is a bit more complicated than "Take a couple of aspirins and call me in the morning." But a quartet of baby-aspirin-sized sensors implanted in her brain are aimed at addressing a condition that's frustrated her and others: the loss of the ability to speak intelligibly. The devices transmit signals from a couple of speech-related regions in Bennett's brain to state-of-the-art software that decodes her brain activity and converts it to text displayed on a computer screen.

19 sept 2023--Bennett, now 68, is a former human resources director and onetime equestrian who jogged daily. In 2012, she was diagnosed with amyotrophic lateral sclerosis, a progressive neurodegenerative disease that attacks neurons controlling movement, causing physical weakness and eventual paralysis.

"When you think of ALS, you think of arm and leg impact," Bennett wrote in an interview conducted by email. "But in a group of ALS patients, it begins with speech difficulties. I am unable to speak."

Usually, ALS first manifests at the body's periphery—arms and legs, hands and fingers. For Bennett, the deterioration began not in her spinal cord, as is typical, but in her brain stem. She can still move around, dress herself and use her fingers to type, albeit with increasing difficulty. But she can no longer use the muscles of her lips, tongue, larynx and jaws to enunciate clearly the phonemes—or units of sound, such as "sh"—that are the building blocks of speech.

Although Bennett's brain can still formulate directions for generating those phonemes, her muscles can't carry out the commands.

Rather than train the AI to recognize whole words, the researchers created a system that decodes words from phonemes. These are the sub-units of speech that form spoken words in the same way that letters form written words. "Hello," for example, contains four phonemes: "HH," "AH," "L" and "OW."

Using this approach, the computer only needed to learn 39 phonemes to decipher any word in English. This both enhanced the system's accuracy and made it three times faster.

On March 29, 2022, a Stanford Medicine neurosurgeon placed two tiny sensors apiece in two separate regions—both implicated in speech production—along the surface of Bennett's brain. The sensors are components of an intracortical brain-computer interface, or iBCI. Combined with state-of-the-art decoding software, they're designed to translate the brain activity accompanying attempts at speech into words on a screen.

About a month after the surgery, a team of Stanford scientists began twice-weekly research sessions to train the software that was interpreting her speech. After four months, Bennett's attempted utterances were being converted into words on a computer screen at 62 words per minute—more than three times as fast as the previous record for BCI-assisted communication.

"These initial results have proven the concept, and eventually technology will catch up to make it easily accessible to people who cannot speak," Bennett wrote. "For those who are nonverbal, this means they can stay connected to the bigger world, perhaps continue to work, maintain friends and family relationships."

Approaching the speed of speech

Bennett's pace begins to approach the roughly 160-word-per-minute rate of natural conversation among English speakers, said Jaimie Henderson, MD, the surgeon who performed the surgery.

"We've shown you can decode intended speech by recording activity from a very small area on the brain's surface," Henderson said.

Henderson, the John and Jean Blume-Robert and Ruth Halperin Professor in the department of neurosurgery, is the co-senior author of a paper describing the results, published Aug. 23 in Nature.

His co-senior author, Krishna Shenoy, Ph.D., professor of electrical engineering and of bioengineering, died before the study was published.

Frank Willett, Ph.D., a Howard Hughes Medical Institute staff scientist affiliated with the Neural Prosthetics Translational Lab, which Henderson and Shenoy co-founded in 2009, shares lead authorship of the study with graduate students Erin Kunz and Chaofei Fan.

In 2021, Henderson, Shenoy and Willett were co-authors of a study published in Nature describing their success in converting a paralyzed person's imagined handwriting into text on a screen using an iBCI, attaining a speed of 90 characters, or 18 words, per minute—a world record until now for an iBCI-related methodology.

In 2021, Bennett learned about Henderson and Shenoy's work. She got in touch with Henderson and volunteered to participate in the clinical trial.

How it works

The sensors Henderson implanted in Bennett's cerebral cortex, the brain's outermost layer, are square arrays of tiny silicon electrodes. Each array contains 64 electrodes, arranged in eight by eight grids and spaced apart from one another by a distance of about half the thickness of a credit card. The electrodes penetrate the cerebral cortex to a depth roughly equaling that of two stacked quarters.

The implanted arrays are attached to fine gold wires that exit through pedestals screwed to the skull, which are then hooked up by cable to a computer.

An artificial-intelligence algorithm receives and decodes electronic information emanating from Bennett's brain, eventually teaching itself to distinguish the distinct brain activity associated with her attempts to formulate each of the 39 phonemes that compose spoken English.

It feeds its best guess concerning the sequence of Bennett's attempted phonemes into a so-called language model, essentially a sophisticated autocorrect system, which converts the streams of phonemes into the sequence of words they represent.

"This system is trained to know what words should come before other ones, and which phonemes make what words," Willett explained. "If some phonemes were wrongly interpreted, it can still take a good guess."

Practice makes perfect

To teach the algorithm to recognize which brain-activity patterns were associated with which phonemes, Bennett engaged in about 25 training sessions, each lasting about four hours, during which she attempted to repeat sentences chosen randomly from a large data set consisting of samples of conversations among people talking on the phone.

An example: "It's only been that way in the last five years." Another: "I left right in the middle of it."

As she tried to recite each sentence, Bennett's brain activity, translated by the decoder into a phoneme stream and then assembled into words by the autocorrect system, would be displayed on the screen below the original. Then a new sentence would appear on the screen.

Bennett repeated 260 to 480 sentences per training session. The entire system kept improving as it became familiar with Bennett's brain activity during her speech attempts.

The iCBI's intended-speech translation ability was tested on different sentences from those used in the training sessions. When the sentences and the word-assembling language model were restricted to a 50-word vocabulary (in which case the sentences used were drawn from a special list), the translation system's error rate was 9.1%.

When the vocabulary was expanded to 125,000 words (large enough to compose almost anything you'd want to say) the error rate rose to 23.8%—far from perfect, but a giant step from the prior state of the art.

"This is a scientific proof of concept, not an actual device people can use in everyday life," Willett said. "But it's a big advance toward restoring rapid communication to people with paralysis who can't speak."

"Imagine," Bennett wrote, "how different conducting everyday activities like shopping, attending appointments, ordering food, going into a bank, talking on a phone, expressing love or appreciation—even arguing—will be when nonverbal people can communicate their thoughts in real time."

The device described in this study is licensed for investigative use only and is not commercially available. The study, a registered clinical trial, took place under the aegis of BrainGate, a multi-institution consortium dedicated to advancing the use of BCIs in prosthetic applications, led by study co-author Leigh Hochberg, MD, Ph.D., a neurologist and researcher affiliated with Massachusetts General Hospital, Brown University and the VA Providence (Rhode Island) Health care System.

More information: Edward Chang et. al., A high-performance neuroprosthesis for speech decoding and avatar control, Nature (2023). DOI: 10.1038/s41586-023-06443-4 www.nature.com/articles/s41586-023-06443-4

Francis Willett et. al., A high-performance neuroprosthesis, Nature (2023). DOI: 10.1038/s41586-023-06377-x www.nature.com/articles/s41586-023-06377-x

Nick F. Ramsey et al, Brain implants that enable speech pass performance milestones, Nature (2023). DOI: 10.1038/d41586-023-02546-0 , www.nature.com/articles/d41586-023-02546-0

 

A framework of biomarkers for brain aging

by Higher Education Press

Framework of biomarkers for brain aging. Credit: Aging Biomarker Consortium

China and the world are facing severe population aging and an increasing burden of age-related diseases. Aging of the brain causes major age-related brain diseases, such as neurodegenerative diseases and stroke. Identifying biomarkers for the effective assessment of brain aging and establishing a brain aging assessment system could facilitate the development of brain aging intervention strategies and the effective prevention and treatment of aging-related brain diseases.

19 sept 2023--Thus, experts from the Aging Biomarker Consortium (ABC) have combined the latest research results and practical experience to recommend brain aging biomarkers and form an expert consensus during the seminar held on March 12, 2023 at Tongji University, aiming to provide a basis for assessing the degree of brain aging and conducting brain-aging-related research with the ultimate goal of improving the brain health of elderly individuals in both China and the world.

Biomarkers reflecting brain aging were recommended, with the aim of addressing clinical issues such as "How biologically old is the individual now?", "How fast is the individual aging?", and "How far is the individual from age-related brain diseases?" The related consensus of brain aging has been published in Life Medicine.

Brain aging involves multi-dimensional and multi-scale changes in molecules, cells, tissues, organs, the whole body, and groups of individuals. Therefore, multi-dimensional and multi-scale information needs to be integrated to accurately evaluate the state of brain aging. Brain aging biomarkers refer to markers that can accurately reflect "real age," brain structure and brain function, which can be used to determine the degree of brain aging and evaluate the effect of anti-aging interventions.

In recent years, remarkable progress has been made in exploring the biological age of the brain. Considering the accessibility and convenience of clinical procedures, this consensus screens brain aging biomarkers from the three dimensions including brain function, imaging and body fluids for reference in clinical work and follow-up studies.

Members of the ABC first identified a list of key issues related to the biomarkers for brain aging through online collaboration based on available publications and the research of the ABC members in this paper. The identified biomarkers were further discussed at a face-to-face meeting to reach a consensus.

All recommendations were fully reviewed and discussed among the members of the ABC to allow diverse perspectives and considerations for this consensus. And this consensus follows the internationally accepted conventions for expressing the level of evidence and strength of recommendations. Further validation of these markers in different age groups is needed.

More information: Yu-Juan Jia, et al, A framework of biomarkers for brain aging: a consensus statement by the Aging Biomarker Consortium, Life Medicine (2023). DOI: 10.1093/lifemedi/lnad017

 

Researchers publish editorial on epigenetic aging in oocytes

by Impact Journals LLC

Credit: Pixabay/CC0 Public Domain

A new editorial paper titled "Epigenetic aging in oocytes" has been published in Aging.

19 sept 2023--Aging-related phenotypes span many different tissues and cell types, and start to occur at different ages—a different typical age for every cell type. In their new editorial, researchers Peera Wasserzug-Pash and Michael Klutstein from The Hebrew University of Jerusalem discuss one of the earliest occurring aging events in the human body, which is the beginning of female reproductive aging and deterioration. The clinical cutoff for advanced maternal age (AMA), a condition associated with poor reproductive outcomes, is 35 years old.

According to the research, "The early onset of reproductive aging poses a significant challenge to clinicians since a global consistent increase in maternal age at first birth has occurred in recent decades, effectively shortening the available time window for reproduction."

As the rate of patients with advanced maternal age rises, and with it, the number of patients in fertility clinics, so does the necessity for a fundamental understanding of the reproductive aging process. In recent years, it has been established that there is a substantial dominating influence of oocyte quality loss on age-related fertility decline. This is best demonstrated by the rise in IVF success rates in reproductively aged women when they receive an egg donation from a younger woman. Oocyte quality loss is characterized by diminished cellular function and an increased occurrence of chromosomal nondisjunctions.

"Our recent publication addresses the question of additional, epigenetic mechanisms that lead to the occurrence of age-related oocyte quality loss," the paper states.

More information: Peera Wasserzug-Pash et al, Epigenetic aging in oocytes, Aging (2023). DOI: 10.18632/aging.204976

 

ChatGPT is debunking myths on social media around vaccine safety, say experts

by Taylor & Francis

Credit: Pixabay/CC0 Public Domain

ChatGPT could help to increase vaccine uptake by debunking myths around jab safety, say the authors of a study published in the journal Human Vaccines & Immunotherapeutics.

19 sept 2023--The researchers asked the artificial intelligence (AI) chatbot the top 50 most frequently asked COVID-19 vaccine questions. They included queries based on myths and fake stories such as the vaccine causing long COVID.

Results show that ChatGPT scored 9 out of 10 on average for accuracy. The rest of the time it was correct but left some gaps in the information provided, according to the study.

Based on these findings, experts who led the study from the GenPoB research group based at the Instituto de Investigación Sanitaria (IDIS)—Hospital Clinico Universitario of Santiago de Compostela, say the AI tool is a "reliable source of non-technical information to the public," especially for people without specialist scientific knowledge.

However, the findings do highlight some concerns about the technology such as ChatGPT changing its answers in certain situations.

"Overall, ChatGPT constructs a narrative in line with the available scientific evidence, debunking myths circulating on social media," says lead author Antonio Salas, who as well as leading the GenPoB research group, is also a Professor at the Faculty of Medicine at the University of Santiago de Compostela, in Spain.

"Thereby it potentially facilitates an increase in vaccine uptake. ChatGPT can detect counterfeit questions related to vaccines and vaccination. The language this AI uses is not too technical and therefore easily understandable to the public but without losing scientific rigor.

"We acknowledge that the present-day version of ChatGPT cannot substitute an expert or scientific evidence. But the results suggest it could be a reliable source of information to the public."

In 2019, the World Health Organization (WHO) listed vaccine hesitancy among the top 10 threats to global health.

During the pandemic, misinformation spread via social media contributed to public mistrust of COVID-19 vaccination.

The authors of this study include those from the Hospital Clinico Universitario de Santiago which the WHO designated as a vaccine safety collaborating center in 2018.

Researchers at the center have been exploring myths around vaccine safety and medical situations that are falsely believed to be a reason not to get vaccinated. These misplaced concerns contribute to vaccine hesitancy.

The study authors set out to test ChatGPT's ability to get the facts right and share accurate information around COVID vaccine safety in line with current scientific evidence.

ChatGPT enables people to have human-like conversations and interactions with a virtual assistant. The technology is very user-friendly which makes it accessible to a wide population.

However, many governments are concerned about the potential for ChatGPT to be used fraudulently in educational settings such as universities.

The study was designed to challenge the chatbot by asking it the questions most frequently received by the WHO collaborating center in Santiago.

The queries covered three themes. The first was misconceptions around safety such as the vaccine causing long COVID. Next was false contraindications—medical situations where the jab is safe to use such as in breastfeeding women.

The questions also related to true contraindications—a health condition where the vaccine should not be used—and cases where doctors must take precautions, for example, a patient with heart muscle inflammation.

Next, experts analyzed the responses then rated them for veracity and precision against current scientific evidence, and recommendations from WHO and other international agencies.

The authors say this was important because algorithms created by social media and internet search engines are often based on an individual's usual preferences. This may lead to "biased or wrong answers," they add.

Results showed that most of the questions were answered correctly with an average score of nine out of 10 which is defined as "excellent" or "good." The responses to the three question themes were on average 85.5% accurate or 14.5% accurate but with gaps in the information provided by ChatGPT.

ChatGPT provided correct answers to queries that arose from genuine vaccine myths, and to those considered in clinical recommendation guidelines to be false or true contraindications.

However, the research team does highlight ChatGPT's downsides in providing vaccine information.

Professor Salas, who specializes in human genetics, concludes, "Chat GPT provides different answers if the question is repeated 'with a few seconds of delay.'

"Another concern we have seen is that this AI tool, in its present version, could also be trained to provide answers not in line with scientific evidence.

"One can 'torture' the system in such a way that it will provide the desired answer. This is also true for other contexts different to vaccines. For instance, it might be possible to make the chatbot align with absurd narratives like the flat-earth theory, deny climate change, or object to the theory of evolution, just to give a few examples.

"However, it's important to note that these responses are not the default behavior of ChatGPT. Thus, the results we have obtained regarding vaccine safety can be probably extrapolated to many other myths and pseudoscience."

More information: Chatting with ChatGPT to learn about safety of COVID-19 vaccines—a perspective, Human Vaccines & Immunotherapeutics (2023). DOI: 10.1080/21645515.2023.2235200. www.tandfonline.com/doi/full/1 … 1645515.2023.2235200

 

Overactive bladder symptoms common with pelvic organ prolapse

by Lori Solomon

Seven in 10 women with pelvic organ prolapse (POP) report overactive bladder (OAB) symptoms, according to a study published online July 13 in the International Journal of Women's Health.

19 sept 2023--Komkrit Aimjirakul, M.D., from Mahidol University in Bangkok, and colleagues examined the prevalence and risk factors for OAB symptoms in women with POP and compared the improvement of OAB symptoms among treatment groups: pelvic floor exercise, pessary, and surgery. The analysis included 754 patients seen at a urogynecology clinic (2016 through 2020).

The researchers found that the incidence of OAB symptoms was 70 percent and two-thirds of patients (65 percent) reported moderate-to-severe bother. Significant risk factors included: the lowest points of the anterior wall (odds ratio [OR], 0.60; P = 0.01), longer perineal body (OR, 0.78; P = 0.02), and previous vaginal delivery (OR, 2.10; P = 0.02). Just over one-third of women experienced improvement in OAB symptoms (36.6 percent) with treatment. Pessary (OR, 1.40; 95 percent confidence interval [CI], 0.94 to 2.07; P = 0.10) and surgery (OR 1.30; 95 percent CI, 0.80 to 2.12; P = 0.28) trended toward a larger effect compared with pelvic floor exercise.

"This information can be useful for patient counseling and to guide patient expectations," the authors write. "Other treatment modalities should be considered in women whose OAB symptoms persist despite POP correction."

More information: Komkrit Aimjirakul et al, A Retrospective Cohort Study on the Prevalence, Risk Factors, and Improvement of Overactive Bladder Symptoms in Women with Pelvic Organ Prolapse, International Journal of Women's Health (2023). DOI: 10.2147/IJWH.S413670

 

Exercise-induced hormone irisin may reduce Alzheimer's disease plaque and tangle pathology in the brain

by Massachusetts General Hospital

Credit: Rice University

Researchers who previously developed the first 3D human cell culture models of Alzheimer's disease (AD) that displays two major hallmarks of the condition—the generation of amyloid beta deposits followed by tau tangles—have now used their model to investigate whether the exercise-induced muscle hormone irisin affects amyloid beta pathology.

19 set 2023--As reported in the journal Neuron, the Massachusetts General Hospital (MGH)–led team has uncovered promising results suggesting that irisin-based therapies might help combat AD.

Physical exercise has been shown to reduce amyloid beta deposits in various mouse models of AD, but the mechanisms involved have remained a mystery.

Exercise increases circulating levels of the muscle-derived hormone irisin, which regulates glucose and lipid metabolism in fat tissue and increases energy expenditure by accelerating the browning of white fat tissue.

Studies have revealed that irisin is present in human and mouse brains and that its levels are reduced in patients with AD and in mouse models of the condition.

To test whether irisin plays a causal role in the link between exercise and reduced amyloid beta, Se Hoon Choi, Ph.D. and Eun Hee Kim, Ph.D., of the Genetics and Aging Research Unit at MGH, along with additional research colleagues applied the hormone to their 3D cell culture model of AD.

"First, we found that irisin treatment led to a remarkable reduction of amyloid beta pathology," says Choi. "Second, we showed this effect of irisin was attributable to increased neprilysin activity owing to increased levels of neprilysin secreted from cells in the brain called astrocytes."

Neprilysin is an amyloid beta–degrading enzyme that has been found to be elevated in the brains of mice with AD that were exposed to exercise or other conditions leading to reduced amyloid beta.

The researchers uncovered even more details about the mechanisms behind irisin's link to reduced amyloid beta levels. For example, they identified integrin αV/β5 as the receptor that irisin binds to on astrocytes to trigger the cells to increase neprilysin levels.

Furthermore, they discovered that irisin's binding to this receptor causes reduced signaling of pathways involving two key proteins: extracellular signal-regulated kinase (ERK) and signal activator of transcription 3 (STAT3). Reduced ERK-STAT3 signaling was critical for irisin-induced enhancement of neprilysin.

Previous studies have shown that in mice, irisin injected into the blood stream can make its way into the brain, making it potentially useful as a therapeutic.

"Our findings indicate that irisin is a major mediator of exercise-induced increases in neprilysin levels leading to reduced amyloid beta burden, suggesting a new target pathway for therapies aimed at the prevention and treatment of Alzheimer's disease," says Rudolph Tanzi, Ph.D., a senior author of the study and director of the Genetics and Aging Research Unit.

Additional co-authors include Hyeonwoo Kim, Mark P. Jedrychowski, Grisilda Bakiasi, Joseph Park, Jane Kruskop, Younjung Choi, Sang Su Kwak, Luisa Quinti, Doo Yeon Kim, Christiane D. Wrann, and Bruce M. Spiegelman.

More information: Se Hoon Choi & colleagues, Irisin reduces Amyloid-? by inducing the release of neprilysin from astrocytes following downregulation of ERK-STAT3 signaling, Neuron (2023). DOI: 10.1016/j.neuron.2023.08.012. www.cell.com/neuron/fulltext/S0896-6273(23)00623-2