Use of machine learning models to identify National Institutes of Health-funded cardiac arrest research.

Abstract

To compare the performance of three artificial intelligence (AI) classification strategies against manually classified National Institutes of Health (NIH) cardiac arrest (CA) grants, with the goal of developing a publicly available tool to track CA research funding in the United States. Three AI strategies-traditional machine learning (ML), large language model (LLM) zero-shot learning, and LLM few-shot learning-were compared to manually categorized CA grant abstracts from NIH RePORTER (2007-2021). Traditional ML used a regularized logistic regression model trained on embedding vectors generated by OpenAI's text-embedding-3-small model. Zero-shot learning, using GPT-4o-mini, classified grants based on task descriptions without labeled examples. Few-shot learning included six example grants. Models were evaluated on a balanced 20% holdout test set using accuracy, precision (positive predictive value), recall (sensitivity), and F1 score (harmonic mean of precision and recall). Out of 1,505 grants categorized, 378 (25%) were identified as CA research, yielding 302 grants in the holdout test set, 76 of which were CA research. The few-shot approach performed best, achieving the highest accuracy (0.90) and the best balance of precision and recall (F1 score 0.82). In contrast, traditional ML had the lowest accuracy (0.87) and the highest precision (0.89) but suffered from poor recall, with approximately 2.5 times more false negatives than either generative approach. The zero-shot approach outperformed traditional ML in accuracy (0.88) and recall (0.86) but had lower precision (0.72). AI can rapidly identify CA grants with excellent accuracy and very good precision and recall, making it a promising tool for tracking research funding.