Design and validation of a scoring tool for performance measurement in the management of anaphylaxis under general anaesthesia

Abstract

Introduction
Competent management of anaphylaxis under general anaesthesia is a core anaesthetic skill and reliable assessment of performance is essential. We aimed to create a scoring tool which was reliable and valid.
Methods
The modified Delphi technique was used to produce a checklist of weighted tasks. 2 groups (8 junior and 8 senior anaesthetic trainees) undertook an anaphylaxis simulation and a panel of raters scored their performance using the tool.
Results
The Delphi process reached concordance after 2 rounds (Kendall W 0.75, p<0.001), producing a checklist of 22 weighted tasks. Internal consistency was excellent (Cronbach’s alpha 0.91-0.96). Total inter-rater reliability was high (ICC=0.98, 95% CI .96-0.99) and the majority of tasks independently showed good reliability. Principal component analysis indicated that the tool could be used reliably with a single rater. Mean total scores for junior vs senior, showed no significant difference (p>0.05).
Conclusions
The Delphi technique was effective and efficient. The checklist was highly reliable, suggesting it could be used for both formative and summative assessment. An unexplained variance of <5% with a single rater, allows deployment with limited resources. High reliability may also represent comprehensive sampling of the construct, and supported by Cronbach’s alpha values, indicates good content validity. Failure to demonstrate a performance difference, between juniors and seniors,  limits interpretation of construct validity. This could be resolved in future by increasing the experience gap. Given the resource implications of validating assessment tools for myriad specific scenarios, we instead suggest the development of a validated and standardised Delphi toolkit.