Transcription
Clinical Trial MethodologyuctingologyAn evidence-based approach to conductingclinical trial feasibility assessmentsFailure to enroll patients is now one of the leading causes of clinical trial delays – andeven of trial failures. By using a data-driven approach to determining trial feasibility,drug sponsors can specify with a high degree of confidence exactly how long it will taketo fulfill a study’s patient quota. Today, there are rich sources of data that can informthe trial forecast, as well as sophisticated modeling tools to test ‘what if’ scenariosand establish confidence levels for the results. Using these resources, sponsors canaccurately forecast enrollment timelines, flag potential risks to the plan and identifyvariables that can be adjusted to influence the timeline.Otis JohnsoninVentiv Health Clinicalotis.johnson@inventivhealth.comKeywords: clinical trial enrollment forecasting feasibility patient enrollment simulatingenrollmentSponsors of drugs in development and theContract Research Organizations (CROs)working on their behalf are often caught offguard by the difficulties they experience inenrolling subjects into trials. Indeed, timelinesfor the majority of trials must be extended inorder to fulfill their patient quotas. As thetraditional approach to forecasting clinicaltrial timelines – relying on investigators’ estimates – is failing sponsors, it is time to takea new approach. We propose a comprehensive methodology for researching recruitmentfeasibility that incorporates patient insights,draws on commercially available data anduses statistical modeling to improve thepredictability of patient enrollment.Patient enrollment statisticsThe biopharmaceutical industry has long operated on the belief that enrolling patients in clinical trials is easy enough, provided that the protocol is right and that the best investigator siteshave been enlisted. The facts, however, suggestthat although this might have been the case insimpler times, it is no longer so. Consider that: 11% of selected sites never enroll a singlepatient [1] ;10.4155/CLI.14.139 2015 Future Science Ltd 48% of all sites underperform, meaningthat they do not deliver the number ofpatients they expect [2] ; Sponsors’originaltimelinesforPhase II–IV studies usually end updoubling in order to meet the desiredenrollment levels [2] ; Nearly 80% of clinical trials fail to meettheir enrollment timelines [3] ; Although results vary widely by therapeutic area, on average, only half ofall patients screened complete clinicaltrials [2] .Many enrollment forecasts are faulty,causing sponsors to embark on trials with afalse sense of what they can accomplish in agiven time frame. The resulting deviationsfrom enrollment forecasts have a significantnegative effect on research costs and corporate revenues, and in many cases, the trialbudget gets larger. The daily operationalcosts of running a trial are an estimatedUS 37K, and any timeline extension resultsin additional research costs that were notaccounted for in the trial budget [4] . WhatClin. Invest. (Lond.) (2015) 5(5), 491–499part ofISSN 2041-6792491
Clinical Trial MethodologyJohnsonis more, the opportunity costs of delayed market entryare an estimated 600K to 8M a day [5] .Traditional feasibility practicesThe difficulty manufacturers have in bringing trialsin on time and on budget is not for lack of planning.The fault lies in the fact that the plans are generallybased on unreliable or partial information. Once theyhave developed a recruitment plan, sponsors attemptto confirm that their timelines are realistic. They routinely conduct ‘feasibility exercises’ that involve working with their CRO and sites (traditional research sites,site networks and integrated health systems) to answerthe following types of questions: What regulatory hurdles can we expect to encounter in the countries being considered for the study? Is the study design consistent with the standard ofcare in the various regions? Is there a large enough patient population to justifyinclusion of particular countries and sites? Are the sites able to get the right infrastructure andstaff in place to support the trial? How many patients can a given site enroll in aspecified time frame?Sponsors, or their designated CROs, then processthe enrollment information from the surveys to establish enrollment plans. Often, these plans are simplybased on extrapolation of a mean enrollment rate todetermine the number of sites needed and the expectedenrollment duration. This approach does not considervarious factors that can affect enrollment timelines.Chief among these often-overlooked factors are that11% of sites that do not enroll any patients [1] and reliance on just the midpoint to make projections willskew the results.In the end, the sponsor wants confirmation thatits plan is realistic and will result in a successful trial.The CRO wants to be sure that it can comply withthe sponsor’s expectations, enrolling the trial withinthe deadline. And sites want to ensure that they candeliver quality results, and, through their contact withsponsors and CROs, want to learn of upcoming trialsof potential interest.Because most sponsors primarily rely on investigator sites to carry out trial recruitment, they, quitelogically, assess recruitment feasibility by surveyingsites on their projected enrollment capability. Theyask for input via questionnaires on how many patientswho fit the study criteria an investigator could expectto recruit for an upcoming trial. Companies then,492Clin. Invest. (Lond.) (2015) 5(5)typically, temper the results with their own intuitionand judgment to estimate how long it will take torecruit the necessary number of target patients. Thistempering is necessary because investigators tend tooverestimate their enrollment capability, hoping theywill appear more attractive to sponsors and CROs andget selected for the trial. How well this approach worksdepends on the experience of those involved. At times,it may produce a valid estimate, but the above statistics on enrollment performance suggest that it usuallydoes not.Limitations of relying on feedback frominvestigatorsThere is nothing wrong with surveying investigators togather input into study feasibility; it is simply that thisstep alone is inadequate to produce accurate enrollment forecasts consistently. The reasons are many,some of which have to do with the validity of sites’responses, since: Busy physicians do not tend to run queries againsta patient database to answer sponsors’ questionsabout available patients with any precision; theysimply provide a rough estimate; Physicians cannot divine the future, but at bestcan only make an educated guess as to how manypatients they would be able to furnish; Sites tend to be overly optimistic and to overcommit when it comes to recruiting patients – asobserved by Dr. Louis Lasagna many years ago andnow dubbed ‘Lasagna’s Law.’ This is not ascribedto any intent to deceive, but rather to do what isbest for their patient. If the proposed study couldbe advantageous to their patients, investigatorshave a vested interest in estimating their enrollment potential on the high side so that they will beselected for participation (see Figure 1).Other limitations stem from the way sponsors handlethe information they get from sites. For instance: Sponsors routinely put too much credence in whatinvestigators tell them. Experience suggests thatinvestigator estimates are often inflated by as muchas 75%; Often, sponsors do not dig deep enough in askingsites about their enrollment potential, or do notnecessarily understand what sites take into accountwhen providing their answers. One site may estimateits enrollment potential based on the total number ofpatients it sees in the therapeutic area while anotherfuture science group
An evidence-based approach to conducting clinical trial feasibility assessments There is a tendency to assume (incorrectly) thatall sites will recruit patients, beginning at thesame time, and at the same rate. In actuality, thisis quite variable, as there are differences in ethicscommittee review and approval timelines, differences in site initiation timelines by country, differences in access to patients and differences inallowable patient recruitment methods by regionand country.Conducting due diligenceA methodical, multistep process that uses all availableinformation (from internal and external sources) canbe used to predict the probability of enrollment successin a specific time frame, given certain variables, with ahigh degree of accuracy. The recommended approachincorporates information gleaned from physician surveys as only one piece of a much broader, more intensedue diligence process that considers the patient perspective, taps commercially available databases anduses statistical modeling.The ideal time to begin this work is during thepreparation of the Clinical Development Plan (CDP).Thus, findings can be incorporated into the plan whilethere is still time to influence the protocol.A comprehensive feasibility assessment involvesgathering information as detailed below and thenusing statistical modeling to incorporate and manage itall. In reality, the following steps are performed in thisgeneral order, although not in quite as linear a fashion.The due-diligence stage of trial planning involves: Gauging patient availability; The first step in the process is to estimate the number of patients who will be eligible to participate inthe trial. This calculation will be based on a breakdown, by country, of the incidence/prevalence ofthe disease and competing trials in the same therapeutic area; the study inclusion/exclusion criteria;the treatment guidelines and procedures; Electronic Medical Records (EMRs), prescription,and/or integrated medical claims databases shouldbe mined to understand how many patients arebeing treated for the particular disease and wherethey are located (see Figure 2). These databases canbe searched by diagnostic and procedure codes, aswell as by various other inclusion/exclusion criteria. One caveat is that the information contained inthese databases is not always sufficient to evaluatefuture science groupType II diabetes on metforminEstimated patientsenlrolled (n)may consider how many patients are likely to meetthe trial’s specific inclusion/exclusion criteria;Clinical Trial Methodology25002,0002000150010005002000Dr. A15040Dr. BDr. CDr. D300Dr. EInvestigatorFigure 1. Sample investigator estimates of patientswith Type II diabetes, taking metformin. Fiveinvestigator sites reported the number of patients thatthey believed fit the eligibility criteria for a trial foran antidiabetes treatment requiring that patients bestabilized on metformin. Their estimates vary widely,and taking an average of these estimates (538 patientsper site) will be very misleading to study planners. Amore realistic approach is to use the median of 200, orto disregard the 2000 as a potential outlier if additionalfollow-up information cannot be obtained from theinvestigator reporting this information. The enrollmentdata set should then be processed to examine themedian (175) and 25th to 75th percentile range(68–275) to develop a good understanding of patientaccess across the investigators.patient counts based on a study’s particular inclusion/exclusion criteria. For instance, it may not bepossible to tell from EMRs a patient’s predictedFEV1 (forced expiratory volume in 1 second – alung function measurement), as this information isoften captured on a separate device and may not getuploaded to the central EMR system; Gathering the patient perspective; At an early stage in the trial planning process, itis important to understand how the target patientpopulation would respond to the protocol requirements. What would patients find appealing orobjectionable about the approach? What motivational drivers would influence their participationdecision? It is possible, for instance, that someaspect of the protocol could pose an unforeseenemotional or logistical hurdle for patients and be arisk to enrollment. Taking the time to assess this isin complete harmony with the US FDA’s ‘PatientFocused Drug Development’ initiative that ‘aimsto more systematically gather patients’ perspectiveson their condition’ [6] ; A combination of primary research and social mediamonitoring should be used to validate assumptionsmade about patient attitudes, beliefs and behaviorsso that the protocol can be evaluated through apatient’s lens. When discovered early, any possiblewww.future-science.com493
Clinical Trial MethodologyJohnsonPrevalence 9329%0.9330%–1.0925%1.0920%–2.2789%Figure 2. Incidence of Type I diabetes excluding patients with chronic obstructive pulmonary disease. Illustratesthe incidence of patients in the target population (Type I diabetes, excluding patients with chronic obstructivepulmonary disease), by location.Reproduced with permission from [8] .For color images please see online: .14.139risks can either be mitigated or factored into enrollment estimates. For example, in the early stages ofplanning one study in overactive bladder, patientfeedback had a significant influence on the way thesponsor proceeded. In a focus group, patients statedvery strongly that they would not subject themselves to an ophthalmology procedure that requiredthem to sit in a dark room for 30 min. The protocolwas actually canceled based on this feedback; 494Selecting countries with the most potential. Thenext logical step is to determine the optimal mixof countries based on their potential for enrollingpatients. This is a matter of blending the information gathered above on patient counts, analogoushistorical trials and competing trials with information on where sites operate by therapeutic area,indication and specialty. A number of commercially available databases can reveal how many siteshave the relevant experience, by country;The number of available patients can be affected bythe number of competing trials in the same indication in a given country. The global, competitivelandscape can be surveyed using a combination ofclinical trial registries, publications and subscription databases to identify areas where competingtrials are/are not likely to make recruiting difficult,while targeting areas with an adequate number ofexperienced investigators. Another valuable pieceof intelligence can be gained by gathering intel-Clin. Invest. (Lond.) (2015) 5(5)ligence on comparable historical trials. Visualizations, such as the ones below in Figures 3 & 4 canillustrate for trial planners where sites meetingbasic criteria are most concentrated [7] . These visualizations and other supporting analyses are thendocumented in a country-level decision matrixsimilar to Table 1 to facilitate selection of the mostappropriate countries for a particular trial; Identifying the best investigators/sites. A majorfactor in enrollment success is site selection. Companies tend to turn repeatedly to a limited numberof sites with which they have prior experience – anapproach that often leads to disappointing results.A better approach is to use research and analytics totarget the right sites that have:–– The ability to enroll the most patients, giventheir proximity to patients;–– Determining sites’ proximity to patients isa matter of overlaying a database of clinicalinvestigator listings (such as the FDA’s 1572database in the USA) onto a patient heat map,similar to that depicted in Figure 2. The resultidentifies investigators who are colocated withconcentrations of patients who meet the studycriteria, making them prime targets, at least interms of their access to patients;–– A history of strong performance, both in termsof recruitment and data quality;future science group
An evidence-based approach to conducting clinical trial feasibility assessments–– Evaluating sites based on their prior performance can involve, at the most basic level,reviewing historical data on analogous products from within a company’s clinical trialmanagement system. It should also includereviewing public sources such as www.clinicaltrials.gov. As internal sources are usually limited in scope and public sources are limited indetail, the ideal solution is to tap commercialdatabases maintained expressly for this purposeas well. Through such a service, investigatorsare assigned an objective, composite performance score based on the number of trials theyhave participated in and their past enrollmentperformance, to include initiation periods,screening rates, and failure rates;–– The required capabilities in terms of staff,experience, infrastructure and equipment;Most protocols will require that sites have specific capabilities, whether it be expertise in thetherapeutic area, familiarity with a particulartechnology or access to specialized equipment.Analytical tools are available to profile sites ona range of dimensions, including their researchactivity, infrastructure, personnel, initiationtimelines and access to the targeted patientpopulation;–– Sufficient capacity without competing trials;–– This can best be established by using the technique that sponsors have traditionally relied onto the exclusion of most others: sending questionnaires to sites. Once sites have been identified that appear to meet the above criteria, theycan be surveyed as to their availability in theproposed time period. Whether they are participating in competing trials can also be verifiedthough publicly available and subscription trialintelligence databases.Note that conducting this research by mining available databases does not preclude the need to surveysites. However, it does mean that the questionnairescan be much shorter, reducing the burden on site personnel. It also is a much more evidence-based approachthat overcomes the inherent bias in site-reported information and considers sites’ applicability across multiplefactors that affect performance.Simulating enrollment ratesThe findings from the above research should then beconverted to quantitative inputs that are run throughan assumption-based enrollment model as a way offuture science group1InvestigatorsClinical Trial Methodology13Figure 3. Investigator availability at a glance.Investigator availability for Type II diabetes mellitusfrom a site intelligence database. This suggests thatideal locations for the study based on investigatoraccess (one of several country selection parameters)include parts of Asia, South America, Europe and theUSA, given the clustering of sites experienced in thisspecific indication.Reproduced with permission from [7] .determining if the right planning parameters are inplace for the trial. The specific statistical technique,Monte Carlo Simulation, calculates the probabilitythat a particular outcome will occur based on a givenaction or set of assumptions. It considers the selectedcountry and site distribution and works by assigning arange of values to each input variable – such as screenfailure and site failure expectations, patient availability,enrollment performance and the effect of patient outreach campaigns – and then running a series of virtualtrial simulations. Typically, the software will run 1000Indication-specific investigators by 800Investigators (n)9489401000Figure 4. Number of indication-specific investigators bycountry. This figure shows availability of investigatorsexperienced in the indication by country, a key datapoint in country selection decisions. It can be used toselect the European countries best suited for the trialbeing planned.www.future-science.com495
Clinical Trial MethodologyJohnsonTable 1. Country decision matrix†.CountryRelevant trial Competing accessInitiationtime lineEnrollment co†Populate with data from due diligence reaseach of internal and external data sources.to 5000 simulations, with each using a different set ofrandom values from within the range.This type of assumption-based modeling is especially useful for program planning, as multiple scenarios can be modeled. The results are then displayed as adistribution chart showing the probability of meetingenrollment targets for each scenario (see Figure 5).eralofThis level of precision in enrollment forecasting hasnever been possible before and stands to improve thetrial enrollment process dramatically by setting 080.060.040.020.0010152025Enrollment period (months)30Figure 5. Enrollment probability by month. A MonteCarlo Simulation can be used to model the probabilityof enrolling a trial within a given time period. Theoutput of this particular simulation reveals that witha given set of variables (specific countries and sites),there is only a 10% probability of enrolling the trialin 9.7 months and a 90% probability of doing so in 18months. If the goal is to enroll the trial in 10 months,certain variables will have to change for there to be astrong probability of success.496Clin. Invest. (Lond.) (2015) 5(5)expectations from the outset. This model is now in routine use, and we are monitoring studies going throughthe process and will have results to report in a year.Case study: retrospective analysisA leading global pharmaceutical company undertaking a Phase II clinical trial of an osteoarthritis therapyengaged its CRO to randomize 200 patients into thetrial. The pre-determined plan called for completingpatient enrollment in 8 months, using 25 sites thatthe sponsor had selected based on its past experiencewith them.After several months of sluggish enrollment, thesponsor decided to implement its contingency planand added eight rescue sites. The maneuver was bothexpensive and time consuming. Activating each additional site cost 160K, plus monthly maintenancecosts of 1.5K per site. Despite the money spent,recruitment remained suboptimal. It ultimately took11.5 months to enroll all 200 patients, 3.5 monthslonger than planned. Of the 33 total sites, four hadrecruited no patients.Although the sponsor did not have the benefit ofplanning its trial with the information and analyticsproposed here, it is possible to demonstrate retrospectively how doing so might have changed the outcome.When the original parameters for the trial were putthrough the simulation model described above, theresults explained why the trial had not gone as planned:there had been only a 20% probability of success inusing the selected 25 sites to recruit 200 patients in8 months.To test the power of the evidence-based approach tosite selection, analysts, retrospectively – and withoutknowledge of the sponsor’s original list of sites – produced a fresh list of recommended sites. They minedthe databases described above and ranked sites basedon their proximity to target patients and objective ratings of their past performance. Only 15 of the 33 sitesthat the sponsor had involved in the study were amongthe top sites that surfaces as a result of this research.future science group
An evidence-based approach to conducting clinical trial feasibility assessmentsThese 15 sites had a proven history of randomizingthree times as many patients with significantly fewerscreen failures than those not selected.The simulation model was run again, this time withbetter planning parameters and a list of sites drawnfrom the database research. The output from the simulation exercise indicated that the sponsor would havehad an 85% probability of successfully enrolling thetrial in 11.5 months using 29 sites selected through theprocess. This is how long it actually took the sponsor.Unfortunately, because the sponsor did not turn to thebest sites, it had to involve 33 (rather than 29), and itmissed its enrollment deadline by 3.5 months (Table 2) .Conservatively, this company spent close to one million dollars in unbudgeted funds to rescue this trialand, perhaps even more important, had to reforecastits revenue projections for the product, given its delaytoward approval. Had the sponsor gone about thefeasibility exercise in the way recommended here, thecompany would likely have been spared the expense,disappointment, and disruption of not meeting its goals.Tips for successStudy planners wishing to apply an evidence-basedapproach to evaluating a study’s feasibility should: Begin the feasibility assessment before the protocol is finalized. Ideally, it should begin during thecreation of the CDP; Take a multidisciplinary approach to performingthe feasibility assessment. The best teams includemembers with experience in Clinical Informatics and Analytics, Medical Affairs, the giventherapeutic area and Clinical Operations; Work closely with the medical director alignedwith the therapeutic area to ensure that they have aClinical Trial Methodologycomprehensive understanding of the target patientpopulation; Include patient insights early in the process to ruleout any risk to enrollment stemming from patientattitudes, beliefs and behaviors; Rely on objective data drawn from sourcesother than sites themselves when assessing sites’enrollment potential; Gather information from sites as part of the process, but do not automatically take their enrollmentestimates at face value. Delve deeper when askingabout their patient population and factor their estimates into a broader analysis that incorporates datafrom several sources; Take advantage of statistical modeling to assessoperational parameters and gain confidence in thechosen options; Avoid ‘planning at the midpoint.’ When sitesrespond to feasibility questionnaires with wildlydifferent estimates, working with an averageor mean is going to be misleading. Rather, it ismore accurate to look at the distribution curve ofresponses, decide how to handle outliers and consider the full performance distribution in yourplanning process. This will produce a much morerealistic idea of sites’ enrollment potential; Flag potential risks during the feasibility study, andbrainstorm ways to mitigate them; Use technology to the fullest and continue to evaluate emerging technology as new innovations andmethodologies are always becoming available.Table 2. Comparison of recruitment projections.ScenarioEnrollment time line Number of sites(months)Probability ofsuccess (%)Sponsor’s original trial enrollment plan82520Sponsor’s actual results11.533 (four of whichrecruited no patients)Sponsor’s projected performance(calculated retrospectively) with evidencebased site selection11.52985In a real-life exercise, a company’s projected patient enrollment was modeled using Monte Carlo Simulation. The results indicated that thecompany had only a 20% chance of succeeding within its established time line, given the sites it had selected. In fact, the company did notmeet its trial goals. It needed an extra 4 months and four rescue sites to complete enrollment. The same model indicated that the companywould have had an 85% chance of meeting its enrollment target if it had worked with a set of 29 sites selected based on database researchand different criteria, and if it had allotted 4 months longer for the process.future science groupwww.future-science.com497
Clinical Trial MethodologyJohnsonFuture perspectiveRecognition of the value of a data-driven approach toperforming feasibility analyses is spreading, and therace is on among industry vendors to develop the mostcomprehensive and accurate feasibility solution. Interest in data-driven feasibility assessment as a businessis also being fueled by the growing availability of thedata itself; access to structured and unstructured datafrom public sources is expanding. Competitors in thisthriving business will include established companies aswell as start-ups.A key area for exploration of value in this space willbe social media. The explosion of online forums forpatients and caregivers will make it easier to understand and monitor the intensity of activity arounda disease in relation to geography. Also, as a resultof more industry consortia and sponsors’ interestin sharing data on sites, more information will alsobecome available to study planners about sites’ infrastructure and performance. Consequently, sponsors will need to rely less on subjective informationsupplied by investigators.Executive summaryPatient enrollment statistics The majority of clinical trials deviate from their forecast due to difficulties in enrolling patients in the prescribedtime frame. The problem is not a lack of planning, but rather of planning based on faulty and/or incomplete informationgleaned from investigators about how many patients they can furnish.Traditional feasibility practices & limitations of relying on feedback from investigators Investigators overestimate how many eligible patients they see. Surveying investigators, as has been done traditionally, should not be the main source of information on whichrecruitment plans are based, but one among many.Conducting due diligence With the advent of commercial databases on patient health and investigators’ performance, sponsors can conducta much broader, evidence-based due diligence process to determine if the job can be done on time, where itshould be done and by whom. The first step is to assess patient availability, by country. Electronic Medical Records, prescription, and integratedmedical claims databases can be mined for basic patient counts. These should be adjusted to reflect competitionfrom existing trials in related therapeutic areas. The result is a heat map showing the geographic concentration oftarget patients. Primary research and social media listening should be undertaken to gather the patient perspective and ensurethat the pr
An evidence-based approach to conducting clinical trial feasibility assessments Clinical Trial Methodology may consider how many patients are likely to meet the trial's specific inclusion/exclusion criteria; There is a tendency to assume (incorrectly) that all sites will recruit patients, beginning at the same time, and at the same rate.
