Many rare diseases lack the well-characterized endpoints that we associate with outcome measures in more common diseases and disorders. Given that most clinical trials for rare diseases are small and often rely on assessments that are not designed specifically for the disease, it is important that every assessment is administered and scored correctly. In trials conducted in very small populations, missing, erroneous, or invalid data can have an outsized impact.

In an effort to determine the types and frequency of errors made in these types of trials, one of VeraSci’s scientific experts reviewed assessments from 135 visits with 37 patients in international clinical trials for mucopolysaccharidoses (MPS). MPS are a group of rare, inherited lysosomal storage disorders that can lead to neurological symptoms, including impaired cognition, difficulties in language and speech, behavioral abnormalities, sleep problems, and seizures.

We reviewed paper-based cognitive and adaptive behavior assessments that are sensitive to neurocognitive changes in MPS disorders, including the Vineland Adaptive Behavior Scale, Second Edition (VABS-II), Bayley Scales of Infant Development, Third Edition (BSID-III), and the Kaufman Assessment Battery for Children, Second Edition (KABC-II), to quantify the scoring and administration errors. Test administrators for the trials were Ph.D.-level neuropsychologists that had been trained previously. For the VABS-II, more than half of the assessments had no errors.

Of those with errors, the most common errors were summation errors, followed by table look up errors. A small number failed to follow the basal/ceiling rules. For the BSID-III, 45% had no errors. The most common error type was a double basal error, followed by failure to follow basal/ceiling rules and summation errors. A smaller number of KABC-II assessments were reviewed but scoring errors and incorrect administration of subtests were the most common errors. The impact of these errors was considerable. In some instances, the impact was an inaccurate calculation of endpoints. What can be done? We have a few recommendations:

  • A central rater can play a pivotal role in rare disease trials. Central raters can identify errors so that they can be corrected during the study.
  • Data quality review and remediation can have significant positive effects on signal detection.
  • Computer-based scoring programs can eliminate summation errors.
  • Rater training can be improved. In particular, we recommend explicit training on the double basal guideline for the BSID-III and provision of clear scoring guidelines for the KABC-II subtests.

We would like to express our appreciation to the sponsors who allowed us to review their assessments and trial data and the children and families who participated in this important research. Do you have questions about how to apply these lessons to your next clinical trial? Contact us.

References and Additional Information

Sanchez, CE, Kahn, A, Stern, L, Atkins, AS, Snyder, H, & Keefe, RSE (2018, February). Common Rater Errors for Assessment in Pediatric Rare and Orphan Diseases. Poster presented at the Annual Scientific Meeting of the International Society for CNS Clinical Trials and Methodology, Washington, D.C.

Shapiro, E. G., Escolar, M. L., Delaney, K. A., & Mitchell, J. J. (2017). Assessments of neurocognitive and behavioral function in the mucopolysaccharidoses. Molecular genetics and metabolism, 122S, 8–16. doi:10.1016/j.ymgme.2017.09.007