Using Small Language Models to measure Nursing Key Performance Indicators from Patient Stories

In the broader effort to humanize healthcare, person-centred nursing Key Performance Indicators (KPIs) have been proposed to evaluate care quality from the patient’s perspective. Patient stories offer a rich, qualitative source for understanding care experiences, highlighting what matters most to individuals and revealing both positive and negative aspects of care. However, these narratives are typically unstructured and require expert review to identify relevant KPIs and assess associated sentiments. This study investigates the use of Small Language Models (SLMs) to automate the extraction of KPIs and sentiment classification from patient stories. We demonstrate that a fine-tuned BERT model can effectively perform both tasks, achieving a Cohen’s kappa agreement score of 0.56, comparable to the inter-rater agreement observed between two human expert annotators (0.58) vs a baseline agreement of 0.15 using GPT-4 and 0.34 with GPT-5. The obtained macro-average F1-Score on the KPIs was 76.65% whereas the F1-score of GPT-4 and GPT-5 were 26.54% and 44.78% respectively. These findings suggest that SLMs hold promise for supporting scalable, person-centred evaluation of healthcare experiences.