Using Electronic Patient-Reported Outcomes (ePRO) for the First Time: Lessons Learned
The importance of patient-reported outcomes (PROs) is evidenced by their increased use in clinical trials and in drug and device label claims. Here Darja Turner shares her experiences using an electronic PRO tool for the first time.
In many cases, the best way to measure symptoms and functional status is through insights provided by the patient, but few people know how to go about integrating ePRO (electronic patient-reported outcomes) into their clinical trials. I completed a Phase 2 clinical trial using electronic an ePRO for the first time and identified the key decision points, responsibilities and lessons learned for incorporating ePRO into our clinical trial practices.
ePROs improve data capture, quality andtransparency, but few people are experts at integrating ePRO into their clinical trials. Here are my “lessons learned” for integrating ePRO in a Phase 2 clinical trial, along with suggestions on how to plan and use these insights for future projects.
Study Protocol and ePRO
FDA Guidance for Industry, Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims states, “If a patient diary or some other form of unsupervised data entry is used, the clinical trial protocol is reviewed to determine what steps are taken to ensure that patients make entries according to the clinical trial design and not, for example, just before a clinic visit when their reports will be collected.”1 This is a critical requirement!
The protocol in this proof of concept trial required patients to complete a daily questionnaire in the evening between the hours of 6:00 PM and 11:59 PM using a visual analogue scale (VAS) score as the primary data endpoint. Other protocol features included:
-
A complex algorithm that analyzed patient criteria for enrollment and randomization (masked from both sites and monitors):
- The ePRO instrument calculated eligibility from data collected from the patient. The algorithm processed patient data and once the algorithm calculations were complete, eligible patients were enrolled and provided with a bi-weekly schedule of study visits.
Three different ePRO questionnaires were due for completion at different collection times:
- Daily, between 6 PM and 11:59 PM;
- Bi-weekly at every visit, activated at every visit and available until 11:59 PM; and,
- The ePRO collected patient diaries using either US English or US Spanish.
Benefits of Using an ePRO
There are numerous benefits to using PROs, and very clear advantages of ePROs over paper-based tools. Although additional training for study sites is required with ePRO, the use of this technology offers cleaner data capture so study coordinators can spend more time with their study participants and potentially enjoy higher patient compliance. Electronic PRO eliminates missed, illegible, or illogical responses.
Study coordinators are exempt from manual calculations, and from spending precious time reviewing or deciphering patients’ handwritten diaries.
Efficiencies are realized with accelerated data availability from electronic capture, including the elimination or reduction of missing data. Data collected by an ePRO is in real-time within the context of the patient’s life. The ePRO collection of diaries and questionnaires improves the data quality for analysis and trial management.
Depending on the study design, patients cannot backfill or forward-fill diaries, because questionnaires can be programmed to appear and disappear at set times.
Here’s how it works: A time and date stamp is created when the subject logs in, so an audit trail is created. Time and date stamps are “proof” that patients complied with the protocol and completed the questionnaire during the intended time. In my study, patients were asked to choose to receive their instruction in English or Spanish, whichever language was most suitable to them, thereby eliminating the need to estimate how many questionnaires to print in each language for every site.
Selecting an ePRO System: Assessing Advantages
Careful upfront planning can save time and money during study execution. With less than two months to set the entire trial up, I selected two electronic systems to collect study data: an interactive web response system (IWRS) and an ePRO system. Study planners noted that in their experience, integrating IVRS/IWRS and ePRO resulted in fewer errors, minimized the number of data clarification forms (DCFs), and allowed better control of keeping study costs aligned with budgets. Each technology system captured the applicable study data.
If the patient met the eligibility criteria, the investigator logged into the IWRS and received a study patient number. The patient number was manually entered into the ePRO device.
When the patient number was manually entered on the ePRO mobile device, the device was then given to the patient. It was used to collect patient data in transit.
We chose an ePRO system that offered a number of advantages to address the complexity of this trial. The ePRO system provided to study participants included mobile eDiary device, a Wireless Pak for transmission, and in case the wireless transmission failed, a TeleCradle for analogue transmission. The mobile eDiary device was portable and lightweight, with a compact screen.
Two devices were ordered for each of the investigative sites since expectations were to have no more than two active patients enrolled at each site at any one time. The working plan was that if one subject did not meet the criteria and had to be discontinued, the device would be reassigned to a new patient. The same held true for those that finished the trial; once a patient completed the study the device would be reassigned to a new study patient meeting enrollment criteria. However, the initial plans were inadequate and it became necessary to re-order additional devices; in some cases, many more than two per site.
The study sites had several responsibilities for monitoring patients once they were enrolled. These included the daily checking of patient compliance through the online reporting portal and contacting any patients at risk of non-compliance, or just needing to charge the batteries of their device. During training, site investigators or study coordinators were required to demonstrate an understanding of how to work with the eDiary and to answer questions about the device (or, know when to refer questions to the vendor’s Support Center). Each patient was expected to receive 15 to 30 minutes of training from the study coordinator on how to use the eDiary.
Initially, many patients called the vendor’s Support Center for eDiary training and questions, either because they didn’t remember some of the instructions once they had departed from the study site or the initial training wasn’t intensive enough. The solution was to develop a study-specific Troubleshooting Guide to address the most common questions, with the aim of reducing the Support Center call volume. It took approximately 10 days while waiting for the guide to be approved by the IRB.
Once patients received their ePRO device from the study coordinators or investigator, they were required to complete their diaries each day for two weeks. At the first visit following the initial screening, patient compliance was assessed with the touch of the button on the ePRO LogPad. With three separate questionnaires distributed at different points within the study, the potential for human error (and disseminating an incorrect questionnaire) was real. The ePRO system provided the right questionnaire to each patient at the right time and helped mitigate the potential for human error. If a patient withdrew early or completed the study, a final questionnaire was provided at the time of study withdrawal or completion.
Study sites were further supported by PHT with 5 key elements:
- An online reporting portal: For the study coordinator to monitor patient compliance.;
- An eDiary training kit: For study coordinators this was included that showed how to use the ePRO device with instructions for training their patients. Each patient was allowed to practice on the training eDiary before being issued their own;
- A 20-25 page site support guide: Specific to the Phase 2 trial. It included directions on how to log into the online reporting portal, the expectations of each study visit, and how to train and work with participants for ePRO study success;
- Documents generated by our study team: Providing instruction on how to use the ePRO system were approved by the study Institutional Review Board (IRB). Power adaptors, power cords, a universal pictorial step-by-step “getting started” card, and support materials were supplied by the vendor; and
- A study archive of all data collected at the site: Furnished at the end of the trial.
Items to Consider: Where We Erred and Some “Lessons Learned”
Human Error
Data entered into and analyzed by the eDiary determined eligibility, randomization and study medication kits. Since a single patient number connected the IWRS to the eDiary, it was critical that the Sponsor Clinical Trial Manager ensure that the correct patient number had been transcribed from one system to the other. Discrepancies can only be resolved with Data Clarification Forms (DCFs).
Lesson Learned: When two technology applications are used, such as IWRS/IVRS and ePRO, enable them to transmit data between the systems.
High and Low Enrolling Study Sites
Don’t forget that the success of any clinical trial is largely determined by the investigative sites that enroll patients, so have adequate ePRO devices when and where they are needed. Every study has high performing sites and low performing sites, and this trial was no exception. Costs add up quickly when ePRO devices must be shipped urgently overnight. Additional devices need to be on hand for keeping the high performing sites stocked.
Lesson Learned: Order your ePRO devices wisely in the initial contract to keep trials and budgets managed and on track.
Data Transmission
Find out up front if the patient lives in an area that supports wireless transmission of data. There are areas in the US (and other countries as well if you are considering using ePRO for global trials) that do not have adequate cellular reception to support wireless transmission. It’s best to provide each ePRO device with a choice of transmission methods, both digital and analog just in case. Having options avoids downstream frustrations for both patients and the investigative sites. However, in this experience, even on the occasions when disruptions with transmission did occur, the data were retained in the PHT LogPad, so no study data was lost. In those circumstances, the investigative site transmitted data during the bi-weekly patient visits.
Lesson Learned: Ensure whatever ePRO device you select retains study data until transmission is possible.
Site Support and Training
Despite carefully scripted training and mentoring for our sites, reality sometimes trumps expectations. The vendor’s Support Center received a higher call volume than expected for general questions and ePRO training. Study sites should check StudyWorks on a regular basis.
Site-specific emails went out to each site every week to identify patients where follow up with study coordinators was warranted (i.e. compliance, missed entry, low battery status, etc.). Additional ancillary training was provided throughout the course of the trial to address concrete, not hypothetical, questions arising from hands-on experience. Summaries of key messages were disseminated in bi-weekly newsletters so sites could learn from each other. We also scheduled regular “town-hall” meetings where, whoever wanted to, could call in at a specified hour and ask questions, talk or share experiences with others. Our efforts paid off with very high compliance across all sites.
Lesson Learned: Schedule at least two hours for an ePRO Investigator meeting so everyone has plenty of time for hands-on training. Consider refresher training for sites and monitors to bridge any gap between the Investigator Meeting and first patient in. Stay involved. Ultimately it’s the sponsor’s responsibility to keep sites and patients engaged as active participants.
Plan for Success
Here are some additional suggestions to successfully plan for an ePRO study:
The ePRO Vendor
Select your ePRO vendor early and take time to understand their processes, document flows, and timelines. Get to know your Project Manager and Coordinator to establish who is responsible for what at each stage of the study, so handoffs are smooth. Leverage the ePRO vendor’s years of operational experience by including them during protocol writing and all planning phases so reasonable endpoint data and instruments (according to FDA guidelines) are used for protocol development.
Any modification to existing questionnaires requires more time to prove fit for purpose. If a combination of collection solutions is used, such as IWRS and ePRO, consider integrating these devices to eliminate the chance of manual errors. This will deliver fewer errors, reducing frustration and costs. However, some integrations may increase the trial timeline, so plan accordingly.
The Clinical Trial Team
Assemble the core team early. Your data management, site management, and biostatistics colleagues need the same understanding of timelines as you to facilitate rapid approval of documents such as Confidential Disclosure Agreements (CDAs), ePRO contract, Statement of Work (SOW), Data Summary (DS) and Data Transfer (DT) Specifications, and User Acceptance Testing (UAT).
Review the ePRO Requirements Document carefully, as this determines what the trial will look like, and all that is required at each step along the way. Any change to this document later, even a minor modification, has an impact on the trial timeline. Changes require that the ePRO vendor program and test the trial changes in all languages.
While every document is important, the Requirements Document is the most important.
More extensive protocol amendments can disrupt budgets and timelines. A thorough Requirements Document review allows for solid initial software deployment and avoids subsequent updates. Test the ePRO device extensively during the user acceptance testing, and document the findings. These test results will be needed later for the test data transfer.
The Investigator Sites
It’s important to confirm and verify each site’s mobile network and signal strength before determining which transmission devices to ship to the site for an ePRO study. If possible, select “technology-loving” sites and study coordinators. If the site is ePRO naïve, be sure the study coordinator is familiar with electronic devices (especially computers and mobile phones). Consider adding an extra study coordinator if the study coordinator is not comfortable with technology.
Study coordinators are training the patients enrolled in the trial, so plan adequate time (preferably 2 hours) for the investigator meeting. The better they know how to use the eDiary, how to charge it and transmit data, the better the patients will be trained. Better training means higher compliance.
Keep sites involved and informed. Remind them to check their online reports daily. It is important to monitor the eDiary battery level and patient compliance. Remind study coordinators that the sponsor can review a Support Center report. Consider publishing a monthly newsletter featuring anonymous questions/answers about the trial, or the most frequent questions addressed by the vendor’s Support Center. This is a very effective reminder for sites that are reluctant to ask questions of the sponsor.
Conclusion
The keys to ePRO success are to:
1. Plan early
2. Pick an experienced ePRO vendor as a guide
3. Properly educate and allow hands-on training for users
4. Anticipate some human error
5. Provide on-going training and communication of best practices to the sites
Ideally, it takes six-eight weeks to set up and troubleshoot an ePRO system, not counting legal review, so allow plenty of time for contingencies.
Undoubtedly there is more upfront work required to use an ePRO System but in the end proper planning can save time, money and frustration. An experienced ePRO vendor will be familiar with data transfers, communication, training and integration of the ePRO with IWRS/IVRS, so it’s important to pick a knowledgeable partner. As more outcomes data are required for claims by regulatory agencies and payers, ePRO is the way of the future.
1US Dept. of Health and Human Services. Guidance for Industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. Available at: www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guid.... 2009: Accessed August 1, 2012.
About the Author
Darja Turner, lic.phil.I, Clinical Psychologist, studied Clinical Psychology at the University of Basel where she completed her training in 2001. Her experience includes management of numerous trials including the first ePRO trial of a leading pharmaceutical company. Ms. Turner’s views and opinions expressed are personal and should not be attributed to any organization. She can be reached at D.Turner@gmx.ch