The effect of anchors on reliability of endoscopic tremor ratings

Objectives/Hypothesis: The purpose of this study was to determine the effects of anchors and training on intrarater and inter-rater reliability for visual-perceptual, endoscopic tremor ratings. Study Design: Prospective cohort study. Methods: Nasoendoscopy recordings of 10 participants with a diagnosis of essential voice tremor were evaluated by five voice specialists using the Vocal Tremor Scoring System. Ratings were performed before, immediately after, and 4 weeks after implementation of a training program with anchor stimuli. Immediate and long-term post-training ratings were performed with simultaneous use of anchor samples for each rating. Results: Intrarater reliability showed significant improvement from pretraining to immediate and long-term post-training. Mean correlation coefficients (Spearman's rho) increased from 0.71 at pretraining to 0.84 and 0.90 at immediate and long-term post-training, respectively. Inter-rater reliability was not affected by training with anchors, with mean correlation coefficients ranging from 0.62 at pretraining to 0.58 and 0.64 at immediate and long-term post-training, respectively. Conclusions: Consistent, reproducible ratings are critical for the interpretation and comparison of endoscopic tremor data. Reliability findings from this study indicate that the use of anchor samples as referents for making ordinal judgments about the severity of tremor in oropharyngeal and laryngeal regions was helpful for improving internal standards and consistency but less useful for calibrating across different raters. Level of Evidence: 4 Laryngoscope, 2016 127:411–416, 2017.