• Working Title: “Thresholds to invest: manufacturers’ decisions to sponsor real world comparative trials.”
  Authors:  Anirban Basu, PhD and David Meltzer, MD, PhD, University of Chicago
  [Abstract]

• Working Title: “The world may be flat, but is the prior?  The use of empirically informative priors in comparative-effectiveness research.”
  Authors:  David J. Vanness, PhD and TBA
  [Abstract]

• Working Title: “Getting to yes: how much additional evidence do payers really need to make a coverage decision?”
  Authors:  Bryan Luce, MBA, PhD, David J. Vanness, PhD and TBA
  [Abstract]

• Working Title: “The role of dynamic predictive simulation in guiding the pragmatic, adaptive trial.”
  Author:  J. Jaime Caro, MDCM, FRCPC, FACP
  [Abstract]

• Working Title: "The Landscape of Pragmatic Comparative Effectiveness Trials."
  Authors: Rebecca Singer Cohen, MPP; Bryan R. Luce, PhD, MBA; TBA
  [Abstract]

• Working Title: "The Role of the Pharmaceutical Industry in the Completion of Pragmatic Trials."
  Authors: TBA
  [Abstract]  

Real world comparative trials are costly, both in terms of money and time.  They also risk generating adverse information (from the perspective of the manufacturer) that may deter manufacturers from investing.  However, payers both in the U.S. and abroad are increasingly demanding real-world evidence in order to provide coverage for their products.  Furthermore, with the recent announcement of $1.1 billion in federal funding for comparative effectiveness research (CER) as part of the American Recovery and Reinvestment Act of 2009, the returns to manufacturers for not undertaking CER are likely to change.  In this changing environment, characterizing manufacturers’ incentives and disincentives to invest in CER will be crucial to designing the most effective framework to stimulate and regulate the generation of this important evidence.  In this work, the authors will develop formal economic models calculating just how much does cost, time and/or risk need to be decreased for a manufacturer to fund CER that would meet the evidence thresholds for potential payers. 

The paper will begin by discussing payers’ demand for information in the context of both coverage decisions and price negotiations.  Both single-payer and multiple payer markets will be considered, as will short run versus long run effects.  A formal model will then be developed to account for the implications of comparative effectiveness information on a manufacturer’s expected revenue and incentives to invest, accounting for market competition, payers’ demand for information and cost of trial.  Specific attention will be given to pre-study market share, price of products and time to patent expiration.  The paper will also consider the potential existence (and discovery) of patient subgroups of varying sizes where one product is truly superior, equal or inferior compared to competitor.  

The paper will use a Bayesian framework in which prior information on the likelihood of the trial generating true results is minimally informative, therefore generating the greatest expected social value of CER.  Posterior information will characterize the probability of true results that are acceptable to payers (either in a discrete sense with respect to coverage, or a continuous sense with respect to price).  Given the anticipated posterior, and both the cost and duration of the trial required to attain that posterior, the paper will then predict anticipated market share and prices post-trial.  Counterfactual expected net revenues for a manufacturer under both the decision to invest or not to invest in the trial will be constructed, accounting for similar decisions made by competitor.  From those counterfactuals, it will be possible to calculate changes in costs, duration and/or risk of losing market share required for a manufacturer to invest in the trial.  The paper will conclude with numerical values for these changes using specific cases. 

Follow-on work may include empirical conjoint surveys to elicit from potential sponsors their funding-decision “tipping point” relative to the dynamic cost-time-risk paradigm.

Perhaps the most-repeated criticism of Bayesian analysis is the requirement of a prior distribution.   Decision-makers struggling with first, second and sometimes third-order uncertainty are not surprisingly concerned by introducing a fourth level – uncertainty over which prior to choose.  Work by Jeffreys (1946) on non-informative priors and Bernardo (1979) on reference priors standardized methods for determining priors, but with a degree of agnosticism toward prior knowledge that is contrary to the nature of Bayesian analysis insofar as it is designed to inform decisions made in the real world (see Kass and Wasserman 1994).  Laplace’s principle of “insufficient reason” may make sense behind a philosophical veil of ignorance, but why play dumb in the real world?

An initial task of this effort will be to review the historical experience of previous comparative effectiveness trials as well as potential present or future such candidates, with the aim of determining the extent to which a legitimate fully-informed evidence base (i.e. Bayesian prior”) was/is present; and to gauge the extent to which it could have/could affect the design and efficiency of the trial.  The paper will also argue for a rigorous and repeatable method of incorporating empirical information into priors used for the design of comparative effectiveness research (CER).  Manufacturers are acutely aware of the time, cost and risk associated with designing CER trials because they have the potential to affect coverage decisions (see Working Paper #1).  To make these decisions, large governmental payers such as NICE in the UK and Medicare in the U.S. are increasingly relying on systematic review and meta-analysis, using either Bayesian or classical approaches.  When planning CER trials, it should be in the best interest of whoever intends to pay for the research (manufacturer or government or both) that each additional dollar spent will generate the maximal amount of actionable information. 

A formal economic model will be constructed that presumes a post-trial coverage decision will be made on the basis of a meta-analysis of effectiveness results, including the CER trial being designed.  We will demonstrate analytically that the expected net benefit of undertaking such a CER trial designed using a meta-analytically generated prior will more accurately predict the actual net benefit than a CER trial designed using a non-informative prior.  A simulation-based analysis will then be undertaken, using a real-world meta-analysis and simulated CER trials.  We will model various payer decision-rules (i.e., modeling different ways that payers actually “Get to Yes” – see Working Paper #3). 

While use of prior information may not be able to guarantee a cheaper or shorter trial (although, on balance, one would expect some efficiency to be gained), we believe it will be possible to show that using a meta-analytic prior will reduce risk by making the “go vs. no-go” decision more accurate.  We further believe that this line of analysis will identify the potential for trial cost-sharing arrangements between government and manufacturer that would maximize expected social net benefit.

“…since logic is concerned with implications among propositions, many have thought it natural to extend logic by setting up criteria for the extent to which one proposition tends to imply, or provide evidence for, another.  It seems to me obvious, however, that what is ultimately wanted is criteria for deciding among possible courses of action…” (Savage 1954, p. 6) 

Whether the criteria Savage speaks about in the first part are classical notions of “confidence” or Bayesian concepts of “credibility,” these are concerns mostly of the analyst.  In the real world, decision makers must act on information.  Regardless of the level of confidence in a proposition, a decision often must be made.  

In the field of health technology assessment, much methodological work has recently focused on quantifying the “value of information” (VOI.  In VOI, information only has value when it changes a decision.   The expected net benefit of sampling, for example, compares the cost of acquiring one more observation to the benefit of the information that observation is expected to yield in terms of improving a decision.  Again, these are more concerns of the analyst.  Although the language and mathematics of VOI theory can be daunting, the issue is a common, every day problem faced by health care decision makers, who intuitively weigh incoming evidence relative to existing evidence and, at some point, tips the decision one way or another.  This exploratory paper will help to serve as a translation from analytic to executive language and back again – helping analysts to design comparative effectiveness research (CER) that maximizes VOI relative to the naturally-occurring…yet elusive…evidence thresholds apparent in real world decision-making. 

This paper will consider several criteria for evaluating propositions, including classical and Bayesian hypothesis tests of superiority and non-inferiority (with various thresholds for Type I and Type II error), posterior expected net benefit and acceptability.  These criteria, and their implications, will be translated into scenarios using lay language and presented in one or more semi-structured interviews with panels of real-world public and private purchasers.   The panel’s assessment of these criteria will be explored using content analysis.  Panel members will also be asked to participate in conjoint analysis exercises designed to elicit the payer’s willingness to pay (WTP) for additional information in a variety of scenarios where hypothetical coverage decisions must be made.  Scenarios with equivalent observed data but different presentations (i.e., Bayesian vs. Classical; hypothesis testing vs. expected value) will be presented.  We will also consider how WTP for additional information varies with other factors, such as the size of the population (and budget) at risk, political sensitivity, public and market perceptions, etc.

Clinical trials are usually designed to have the “statistical power” necessary to test a meaningful hypothesis with a suitably low probability of Type I error.  But many times, this process is based on expert opinion about both what constitutes a “meaningful” hypothesis and the variability in outcomes that determines power to test the hypothesis under a particular design.  Hundreds of millions of dollars are committed to clinical trial designs and implementations with very little sense of the chances for success, of what might have an impact on these and to what extent.  Furthermore, most of these designs are inflexible and unable to incorporate information as it becomes available to update the forecasts and assess possible changes in the direction of the trial. With the advent of “adaptive” trial designs, additional variables come into play and the situation is even more uncertain (though possibly beneficially so).

In this working paper, we will examine how simulation techniques — widely used in other fields — can be applied to the design and conduct of adaptive clinical trials to help lay out the alternatives and to predict quantitatively their implications. The paper will demonstrate how well designed simulations can closely replicate the proposed trial and incorporate the best available evidence to help predict what might happen, how sensitive this is to the assumptions that are made at that stage and help guide the development of decision rules to adapt to the expected incoming evidence.  The predictions can cover the clinical results, the likelihood of a successful trial, the drivers of those outcomes and the effect of various design and implementation choices (e.g., how often patients are seen, inclusion and exclusion criteria, measures used to document outcomes, and so on). The paper will also demonstrate how simulation can also be used to estimate the costs of the trial and how these change according to the options selected.  For adaptive trials, it will be possible to use simulations on an ongoing basis to predict value of information and alter the trial design or recruitment accordingly to maximize the decision-making value of each additional observation.  It is impossible to be certain about the impact of all the decisions but it is surely much better to quantify these and examine them rather than going in with only a nebulous sense of what might happen. 

In today’s parlance, comparative effectiveness research transcends systematic reviews, and the nation has an interest in funding pragmatic comparative clinical trials. Comparative effectiveness research has undergone a trajectory of activity (from industry research to legislative proposals) that has culminated most recently with an additional $1.1 billion in economic stimulus funding to support this kind of research. Specifically, $400 million has been designated for use by the National Institutes of Health for comparative clinical trials; and therein lies an unusual opportunity.

In the past, comparative clinical trials such as ENHANCE, CATIE, and ALLHAT have been expensive, have taken many years, and even somewhat controversial, for instance in terms of relevance as well as in lasting impact on health care. It is important to note, however, that although these trials have been criticized, the comparative nature of the trial design succeeded in uncovering knowledge we would not have known otherwise. As we embark on a new era of rising evidentiary standards coupled with an increasing need for real-world evidence, it is instructive to learn the extent to which pragmatic comparative clinical trials are being pursued. In other words, what does the comparative effectiveness real-world trial landscape look like?

The intent of this paper is to try to understand the extent to which the research community (industry, academic, etc) is pursuing and funding pragmatic comparative clinical trials. We surveyed recent and ongoing clinical trials that fit the comparative effectiveness model and demonstrate real-world or pragmatic elements.

Two searches were conducted. First, a literature search was conducted using PubMed with pre-determined search terms (such as “pragmatic”). Articles were then hand sorted to find trials that were 1) comparative/ head-to-head, 2) Phase IV or later, and 3) include a real-world element. The second search reviewed trial results from Clinicaltrials.gov using similar key search terms. Results were again hand-sorted to find trials that were 1) comparative/ head-to-head, 2) Phase IV or later, and 3) include a real-world element.

Both searches uncovered over one hundred articles or trials, and yet we judged that only a third of these were directly relevant to our topic of interest (pragmatic comparative clinical trials). Generally, few results surfaced for pragmatic comparative clinical trials conducted prior to 2000, and the vast majority have been conducted since 2004. Approximately 2/3 of studies are for procedures, therapies, or treatments other than pharmaceutical interventions. Additionally, although the practice-setting is common for clinical trials, the term “pragmatic” lacks a standard definition; indeed, the range of use for this term provides a great deal of confusion. Overall, 34 clinical trials were found that appeared to be pragmatic comparative effectiveness clinical trials.

Many issues remain, however, that both prevent a precise understanding of the pragmatic comparative effectiveness clinical trial landscape, and that may also create obstacles to future integration and adoption of relevant methodologies. Clinicaltrials.gov, for instance, is an imperfect tool for understanding ongoing clinical trials because of the specific nature of the requirements for registration [42 U.S.C. 282(j)(3)(A)]. Additionally, as mentioned above, the lack of standard terminology and consistent descriptors prevent a natural trend toward conducting trials of this nature.

In summary, it is rare to encounter pragmatic designs or methods in Phase IIIb or IV trials. Of those trials that may be considered to display some element of being “pragmatic”, the vast majority have surfaced in the last 5-8 years.

This paper will analyze the role that pharmaceutical companies have played in the conceptualization and conduct of pragmatic comparative trials of pharmaceutical products over time. (Complete abstract forthcoming)