As the COVID-19 pandemic halted many in-person clinical trial visits, sponsors were faced with difficult decisions about whether to delay, cancel, or modify their on-going clinical trials. One option for some trials was to replace in-person visits with remote or virtual visits. Here, we will share VeraSci’s experience helping one of our sponsors with the transition to remote visits during the pandemic.

Background 

The trial in question was a large, global, phase II trial in schizophrenia. The pandemic related lockdown began about two years into the trial and dramatically impacted the ability to conduct in-person assessments at numerous sites. One of the key assessments used in the trial was the Brief Assessment of Cognition (BAC). The BAC is a performance-based outcome measure typically administered on a tablet by a rater during a patient’s visit to the clinic. The challenge was to develop a method of remote administration to be used during the pandemic.

Considerations at Multiple Levels

Making the switch to remote visits was a time-sensitive, multifaceted problem. Chief among our concerns was whether the assessment could be performed in a way that maintained scientific integrity. It was important to ascertain whether conducting the assessment through a telehealth visit would preserve the validity of the test. Regulatory considerations were equally important, however, and regulatory guidelines were emerging and evolving rapidly at the time. VeraSci was in close contact with regulatory agencies, and Dr. Rich Keefe, VeraSci CEO, was closely involved in the organization of the International Society of CNS Clinical Trials (ISCTM) Working Group meetings used to disseminate information about remote assessments from FDA and EMA to industry. We worked closely with Dr. Luca Pani, VeraSci VP of Regulatory Strategy & Market Access Innovation, to make sure our plans were as aligned as possible with emerging regulatory guidelines. There also were important data privacy and security considerations. It was essential to select solutions that met data privacy and security requirements, as well as ensuring patients’ privacy needs were being met. Finally, there was an array of practical concerns that needed to be addressed, such as technology, connectivity, training, testing materials, and data management.

Modifying the BAC for Remote Administration

Preparing the BAC for remote administration required a subtest-by-subtest consideration of what could be done in a way that was both practical and scientifically valid. Some of the biggest challenges were the Symbol Coding and Token Motor tasks, which required the use of a touchscreen. We considered several options for implementing these two tasks but ultimately decided to eliminate those subtests to meet the timeline requirements. We determined that the remainder of the subtests could be administered remotely with modifications. Using a telehealth format that incorporates videoconferencing technology was vital because it allowed the rater to monitor the patient’s environment and behavior during testing. Importantly, the modified assessment still enables the rater to compute a valid composite score.

BAC Remote Administration Deployment

The first step in deploying the BAC for remote administration was to develop the materials and procedures needed. This included developing a rater training process, developing technical support procedures, creating an administration manual, and modifying data entry procedures as well as identifying, qualifying, and procuring a secure platform for videoconferencing.

Remote Administration Deployment Process

The next step in deployment was rater training. This involved identifying the sites and raters that needed to perform remote administration of the BAC, distributing necessary materials to conduct the assessment, and distributing telehealth platform credentials.

Finally, as we began rolling out remote administration, we took a number of steps to make sure everything went smoothly. We provided on-going rater and technical support, conducted rater debriefing calls, supplied full data review with feedback to raters, and we evaluated rater and patient surveys. We used all the information we collected as feedback to refine our processes.

Results

As this study is currently on-going, we have been collecting formal and informal feedback from raters and patients. Overall, the responses have been very positive. Patients and raters have appreciated feeling safe while still doing the assessments. Knowing that we are concerned about their safety has led to an increased sense of engagement. We have not had anyone express privacy concerns. We have had some feedback from raters that there is a learning curve to managing both the technology for the assessment and the videoconferencing technology simultaneously. Most have felt that with practice, over time, they have become more comfortable. Most importantly, raters have reported that they do not believe there is any compromise to the validity of the testing or the data.

For more information about using remote assessments in your next clinical trial

As the COVID-19 pandemic made in-person visits difficult or impossible, sponsors and CROs turned to remote visits and remote assessments to keep trials running. This is an approach to which several regulators have signaled interest and VeraSci has endorsed. After careful consideration, we’ve moved forward with transitioning multiple on-going trials to remote assessments.

Increasingly, however, we’ve noticed a trend with some in our industry suggesting remote assessments can be added as an option in protocols to use any time a resurgence of COVID-19 happens to avoid missed visits or protocol deviations. These suggestions and recommendations ignore the complexities of implementing remote assessments in a way that will be acceptable by regulatory guidelines. While regulators have shown a willingness to be flexible during the pandemic, the validity of the data must be maintained. Downplaying the complexities involved in properly implementing remote assessments creates a risk that your trial data will not be considered acceptable to regulators.

How to Decide If Assessments Are Appropriate for Remote Administration

Not all assessments are suitable for remote administration. Each assessment needs to be considered individually in the context of the trial, and some may require modifications. Patient Reported Outcomes (PROs) can almost always be administered remotely because, by definition, they require patients to provide responses without external support. Many clinical interviews with patients and observers that do not involve any physical assessments are also appropriate for remote assessment based on reasonable clinical judgment. For some assessments, such as those that involve the assessment of non-verbal emotional expression, video is essential for an accurate assessment. In other cases, these can be completed by phone. In all cases, the appropriateness of remote assessment of clinical interviews relies on participants having a private space for completing the interview.

Traditional performance-based assessments (PerfOs) such as tests of cognition, motor function, and functional capacity can often be administered remotely, but typically require some modification. The level of modification varies widely, ranging from none needed for a completely verbal assessment to extensive modifications for assessments involving manipulation of objects. In some cases, the need for object manipulation makes remote administration impossible. Specific timing needs can also impact the feasibility of remote assessment. It may be necessary to send materials to subjects in advance of the session or to have someone with the subject to assist with administration. Similar to clinical interviews, these assessments often require videoconferencing. These assessments also require the subject to have a private space free of distraction and have the additional burden of the electronic equipment and often the data plan.

Scales involving a physical exam that include touching the subject, such as assessing rigidity or strength, are the most significant challenge for remote administration. For scales that include these elements, it may be possible to do a partial administration that does not include these items. For example, proxy scales for the assessment of medication side effects have been utilized by eliminating items that require hands-on assessment. Decisions regarding remote administration of scales with these elements are the most complex and require thorough consideration of the protocol, importance of those measurements, and the impact of not assessing them on the scale’s validity.

The appropriateness of remote administration of any assessment may require input from those who developed the instrument. During the COVID-19 era, some authors of tests or rating scales have weighed in on the appropriateness of their scales for remote administration. Some authors have provided specific suggestions for modification, and some have provided specific guidance indicating that their scale is not appropriate for remote administration. We always recommend consulting with the authors of an assessment to get their input on any modifications under consideration. A thorough literature search may also be necessary to determine whether there is any precedent for remote administration and if any validation research has been completed.

Switching Between Modes of Administration

We’ve also seen recommendations to allow individual subjects “ping-pong” between remote and in-clinic assessments, meaning that they could move back and forth between remote and in-clinic assessments multiple times based on conditions at the time of each visit. Although no rule currently exists, there is no regulatory precedent which introduces some risk for sponsors moving forward with this approach. Switching back and forth between remote and in-person assessments would require a thoughful justification for the multiple changes to allow for data interpretability in the context of, among others, the ICH E9 addendum framework.

It will be essential for investigators to document why they chose to use remote assessments because many remote assessments are not supported by good evidence of equivalency with the original measures.

Assessments Needed to Demonstrate Safety

Assessments which are designed to detect safety signals require special attention. When it comes to safety related assessments, it is likely better to err on the side of collecting incomplete data rather than collecting no data. Even if an assessment could only be partially administered or would rely on a patient to tell the clinician about something the clinician would normally observe directly, it may be better than receiving no safety information at all.

Data Analysis Considerations

Finally, when it comes time to analyze the data, special care must be taken to account for changes in the mode of administration appropriately. Sponsors will still be expected to follow existing regulatory guidance for the statistical analysis of clinical trial data, particularly as it relates to intercurrent events, estimands, and intent-to-treat principles as well as proper handling of missing data.

Do you have questions about adding remote assessments to a new or on-going trial? Contact us to discuss and get support in designing your trial.

 

The Expanded Disability Status Scale (EDSS) is one of the most widely used endpoints in multiple sclerosis (MS) research and in clinical trials of investigational treatments for MS. The EDSS is an elaborate scoring system applied to a neurological examination that results in a score that quantifies the level of disability in patients with MS. It is a validated measure that is recognized for its ability to track MS progression over time.

One of the drawbacks of traditional implementations of the measure is that it is difficult to score and is prone to scoring and data quality issues. VeraSci worked with Neurostatus to develop an eCOA version of the EDSS that would be functionally identical to the paper version while providing the improvements in data quality and ease of administration associated with eCOA systems. Neurostatus-EDSS on the Pathway eCOA platform contains automatic score algorithms that help reduce rater error and improve data quality.

During the neurological exam, the rater taps the responses for sub-scores and functional system scores on the chapter pages as they progress through the assessment. The synopsis page, a feature available from any chapter page, allows the rater to track their progression throughout the administration.

Once all the functional system scores and ambulation scores have been assigned, the EDSS score is determined. At the end of the administration of the assessment, raters request algorithmic feedback on their scores. The algorithmic feedback indicates any discrepancies or inconsistencies in the data by color-coding the data in question on the synopsis and chapter pages. Raters can modify sub-scores and chapter scores flagged in the system, resulting in fewer inconsistencies. Expert data reviewers from Neurostatus review the assessment and query any questionable entries not automatically flagged by the system.

Are you interested in learning more? Watch a brief demo of the Neurostatus-EDSS video below.

We recently wrote about the first two special meetings of the Innovative Technologies Working Group of the International Society for CNS Clinical Trials and Methodology (ISCTM) to discuss challenges related to implementing remote assessments in CNS clinical trials in the context of COVID-19 precautions. These calls provided an open forum for discussion between FDA, industry, academia, and clinical trial sites. There has since been a third call, which also included participants from EMA sharing their views. The call focused on three key topics.

Statistical Considerations for Trials Using Two Different Assessment Methods (In-person vs. Remote)

The first key topic was a discussion of statistical analyses in trials using both in-person and remote assessments. The working group noted that in most cases, remote and in-person assessments could not be considered interchangeable. This discussion addressed the following aspects of this challenge:

    • Making adjustments based on the heterogeneity remote assessments add to the data
    • Adjusting the sample size because of the heterogeneity remote assessments add to the data
    • Handling inter-site variability in the percentage of assessments conducted remotely
    • Whether to stay with remote assessments or transition to in-person assessments once restrictions ease for trials where baseline assessments were administered remotely
    • Missing data handling for assessments that cannot be done remotely

Panelists on the call stressed the difficulty of making universal recommendations for study design and statistical analysis plans and urged sponsors to work directly with regulators on specific program plans. This applies to the two other topics discussed on the call as well. They also noted the importance of documenting what data was missing or collected differently.

Acceptability of Revising Some Elements of Performance-Based Outcome Measures to Allow for Remote Assessment

The second topic was a discussion about the acceptability of changing some elements of outcome measures, like cognitive tasks or motor skills tasks, to allow for remote assessment. The following points were covered:

    • Whether protocol amendments are needed to address issues resulting from the COVID-19 situation, such as capturing new mental health problems not included in the original protocol?
    • Whether amendments are needed to allow the use of remote assessments that have been shown to correlate adequately with the original measures
    • The necessity of amending scales to reflect the effect of social distancing measures during the COVID-19 period
    • The process and data requirements for the validation of remote assessment tools intended to be used instead of in-person assessment
    • Whether imputation methods can be used to account for the effect of the COVID-19 pandemic on primary objectives
    • Whether a treatment effect in the absence of the COVID-10 pandemic can be used

Methodological Considerations Related to the Effects of COVID-19 on Patient Symptoms

The final topic was a discussion of methodological considerations related to the effects of COVID-19, such as isolation, psychosocial stressors, and changes in routine, on patient symptoms like mood and anxiety. The issues covered included:

    • If certain patient symptoms (e.g., mood, anxiety) should be considered an increase or a new baseline
    • Using separate assessments for pre-COVID and post-COVID periods
    • Including testing and COVID-19 status of to substantiate differences in observed results
    • Whether guidance can be provided to raters for completion of assessments in which clearly COVID-19 related factors are excluded and how these results would be used in a sensitivity analysis
    • Incorporating scales that specifically measure the impact of life events into clinical trials

A recording of the full discussion is available here.

The ISCTM Innovative Technologies Working Group is co-chaired by Mike Davis, FDA Division of Psychiatry Products Office of New Drugs, and Rich Keefe, CEO at VeraSci.

On Tuesday, September 22, ISCTM will hold a Virtual Hot Topics session from 10:30-11 ET on “Regulatory viewpoints from EMA and FDA on contemporary challenges and strategies for conducting and analyzing clinical trials in the COVID-19 era.” The session will be chaired by Dr. Keefe from VeraSci and Dr. Davis and Dr. Valentina Mantua from FDA. It will include several panelists from FDA and EMA.

Do you have questions about using remote assessments for your ongoing study?

VeraSci is offering sponsors a complimentary consultation to discuss options for remote assessment as well as other challenges related to the impact of COVID-19 on their studies.

Preliminary Feasibility and Validity Data in Older Adults with and without Subjective Cognitive Decline

This blog series shares what’s happening day-to-day in our Innovation Lab. The VeraSci Innovation Lab was created to pair scientific knowledge with cutting-edge technology to improve the tools and measurements available for use in clinical trials.

Background

VeraSci’s Innovation Lab has been investigating use of a wearable and removeable shoe insole to assesses gait and related measures in aging individuals. We recognize that streamlined assessment of gait in real-world settings has the potential for widespread utility in clinical trials across CNS disorders including Alzheimer’s disease, Parkinson’s disease, and multiple sclerosis, among others. We recently presented preliminary findings from an ongoing study at the American Society of Clinical Psychopharmacology (ASCP) 2020 Annual Meeting.

Introduction

The integration of wearable devices into regulated clinical trials for the assessment of cognition and functioning continues to be a topic of interest. If appropriately implemented, measurements from validated wearable technologies have the potential to revolutionize the conduct of clinical trials by facilitating the development of site-less clinical trial designs. In order to be truly informative, however, endpoints collected by these devices must undergo the same clinical and technological validation process required by other currently accepted tools. This study examines the reliability of industry-grade pressure sensing insoles for the real-time capture of gait measures of older adults, and the relationship between gait measures and standard measures of cognition and functioning.

Participants & Study Design 

Recruitment for this study is ongoing.  At the time of the analysis, 28 adults age 55 and older had completed the study, including 19 healthy controls and 9 individuals with subjective cognitive decline. Subjective cognitive decline was determined by a total score greater than or equal to 4 on the Cognitive Functional Instrument (CFI). The study design includes:

  1. A baseline assessment, which involves data collection from the insoles during four standardized walking tests as well as completion of a cognitive test battery
  2. A one-week period of data collection from the insoles during participants’ daily lives
  3. A one-week follow-up laboratory assessment, which involves data collection from the insoles during the same four standardized walking tests used during the baseline assessment

Measures, Assessments and Analysis 

The study utilized measures of gait, functional capacity, and cognition. The insoles produce measures that characterize a subject’s gait across a gait cycle as well as within the gait cycle, such as the ground force reaction, cycle time, and mean stride length. Functional capacity was assessed using the Virtual Reality Functional Capacity Assessment Tool (VRFCAT). The VRFCAT uses a realistic simulated environment to recreate routine activities of daily living like meal planning and preparation, using transportation for getting places, shopping for groceries, and handling money. Cognition was assessed using the Brief Assessment of Cognition (BAC).

Results & Discussion

The 500-foot walk test generally produced more reliable gait measures than the other walking assessments. The shorter walking tests were considerably more variable. This is consistent with previous findings that the reliability of gait measures increases with longer walking assessments.

Several gait measures significantly correlated (no correction for multiple comparisons) with functional capacity and cognitive measures. Double support time (the time during which both feet are on the floor) and single leg stance duration (time that a given foot was in contact with the ground) were both significantly correlated with the VRFCAT (r=.45 to .54) and with the BAC subtests letter fluency (r=-.48 to -.53) and delayed cued recall (r=-.43 to -.52).

Prior research has demonstrated that older adults with subjective cognitive decline show impaired performance on both functional capacity assessed by the VRFCAT and on objective cognition tasks in the BAC. Older adults with subjective cognitive decline are at increased risk for Alzheimer’s disease. In conjunction with previously reported research, the preliminary results of this ongoing study suggest gait variables may provide sensitive tools for identifying older adults at-risk for developing Alzheimer’s disease and for evaluating responsiveness to treatment.

Download the ASCP 2020 Poster

Placebo response is a potentially devastating component of CNS trials, particularly those with subjective patient-reported outcomes and high levels of interaction with clinical staff. In pain, depression, and other psychiatric studies using subjective primary endpoints, high placebo rates contribute substantially to the late-stage failure of promising treatments.

VeraSci is now offering a scientifically-backed instrument, the Placebo Control Reminder Script (PCRS), through the Pathway eCOA system. The PCRS, developed by the Hassman Research Institute, is a brief script read to participants immediately before administering study assessments. It educates and reminds them about placebo response factors and has been shown to decrease placebo response.

Data from a study of the PCRS were presented at the 2020 Annual Meeting of the American Society of Clinical Psychopharmacology (ASCP). The study examined the PCRS in subjects with major depressive disorder or with a psychotic disorder who had at least moderate depression. Subjects were randomized to either the intervention group or the control group. The intervention group was read the PCRS before administering the baseline and each subsequent assessment. The PCRS was not read to the control group. Subjects were told they had a 50% chance of receiving either the placebo or an active drug, but all subjects received a placebo. Subjects in the intervention group reported significantly smaller reductions in depression than the control group (i.e., a smaller placebo response). The intervention group also reported fewer adverse events. These findings indicate the PCRS is a useful tool to mitigate elevated placebo response rates that are often seen in failed late phase trials.

The PCRS is easy to implement. It doesn’t take long to administer, the language in the script is straightforward and easy to understand, and the administration is streamlined within Pathway. PCRS and Pathway provide a scientifically-backed mechanism for mitigating placebo response in a way that doesn’t significantly increase the burden on clinical sites or subjects.

To learn more about placebo response mitigation, download our whitepaper.

 

References

Cohen, E.A., Hassman, H.H., Walling, D.P., Wyka, K., Horan, W.P., Keefe, R.S.E., Grindell, V.M., Glass, S.J., Ball, R.R., Styczynski, J., Lobb, J.M., Ereshefsky, L. (2020, May) The Placebo-Control Reminder Script in Depression and Psychosis Trials: An Antidote for the Placebo and Nocebo Response. The American Society of Clinical Psychopharmacology (ASCP), Virtual Meeting

Cohen, E.A., Hassman, H.H., Walling, D.P., Hoover, S., Wyka, K. Ball, R.R., Joseph, A.V., Lobb, J.M., Hazzard-Rudolph, D., Ereshefsky, L. (2018, November) A First-Time Investigation of a Subject Intervention to Reduce the Placebo and Nocebo Effects: A Multicenter, Randomized, Single-Blind, All Placebo Study of a Placebo-Control Reminder Script for Subjects with Major Depression. CNS Summit, Boca Raton, FL