Letter to Comptroller General of the United States David M. Walker

September 5, 2006
The Honorable David M. Walker
Comptroller General of the United States
Government Accountability Office
441 G Street, N.W.
Washington, D.C. 20548

Dear Mr. Walker:

Recent investigations into the Food and Drug Administration's (FDA) approval of the drug Ketek have raised broader questions about the FDA's and, specifically, the Office of Antimicrobial Products' acceptance of so-called "non-inferiority" trials as proof of the effectiveness of other antibiotics. Ketek's effectiveness was established on the basis of "non-inferiority" studies, which many experts believe are inappropriate for studying some of the indications for which Ketek was approved. This letter asks that the Government Accountability Office (GAO) evaluate FDA's reliance on these studies.

The most straightforward manner of establishing effectiveness is to compare the patient outcome when taking the medication to the patient outcome when taking a placebo. In some cases, however, this is neither possible nor ethical-if the likely outcome of taking a placebo would be death or other serious adverse health impacts, then establishing the effectiveness of a new medication must be done via some other means. One of these means involves the use of a "non-inferiority" study, in which one group of patients is given a medication of known effectiveness and a second group is given the medication for which effectiveness is sought to be established. The new medication's effectiveness can be established if it performs in a similar manner (within some statistical range) to the known medication.

Over the years, concerns have been raised about the use, design and limitations of such non-inferiority trials.[1],[2] For example, a 1998 International Conference on Harmonization (ICH) Guidance states that there are "well known difficulties" associated with non-inferiority trials that "relate to the implicit lack of any measure of internal validity (in contrast to superiority trials)." [3] That guidance further states that non-inferiority trials are "not conservative in nature, so that many flaws in the design or conduct of the trial will tend to bias the results towards a conclusion of equivalence."[4] Another ICH Guidance describes the specific circumstances under which effectiveness can be established by non-inferiority trials: "for a finding of non-inferiority to be interpreted as showing efficacy, the trial needs to have had the ability to distinguish effective from less effective or ineffective treatments [called assay sensitivity] … if a trial is intended to demonstrate efficacy by showing a test treatment to be non-inferior to an active control, but lacks assay sensitivity, the trial may find an ineffective treatment to be non-inferior and could lead to an erroneous conclusion of efficacy."[5] Thus, the limitations of non-inferiority trials and the potential for bias in the results have been well-documented. If careful attention is not given to the design of non-inferiority trials, then the trial may not be able to accurately assess effectiveness and may provide potentially misleading results.

We believe that these issues of trial design and process are critical to protecting the public from unnecessary exposure to safety risks from drugs with questionable effectiveness. Therefore, we request that the Government Accountability Office (GAO) evaluate the FDA's oversight of and reliance on non-inferiority trials to establish effectiveness.

We request that GAO's evaluation of FDA's reliance on non-inferiority trials be conducted in two phases. In the first phase, we request that the GAO compile a document that responds to the following questions:

1. In the past 10 years, how many products has the FDA Office of Antimicrobial Products approved that have established effectiveness on the basis of non-inferiority studies? Please provide a list of all of those products. For each product please provide:
a. The indication for which the product was approved; whether this indication is or is not serious and life threatening (pursuant to the FDA's definition contained in 21 CFR 312.81);
b. The sponsor of the NDA;
c. The date on which the product was approved;
d. The name(s) of the active control drug(s) or comparator(s);
e. Whether the comparator was approved in the U.S.;
f. The margin used in the trial;
g. The treatment difference between the active control and the test drug and the associated confidence intervals;
h. Any groups or subgroup analyses included in labeling;
i. Whether the active control drug used to establish non-inferiority for each medication was itself approved on the basis of a placebo study or other superiority trial, or if it too was approved on the basis of non-inferiority;
j. A copy of the explanation contained in the applicant's analysis of the study for why the results of the non-inferiority trials could be believed to assure the effectiveness of the drug (as required by 21 CFR 314. l26(b)(2)(iv)) and, if the required explanation was not included in the analysis of the study, an explanation as to why it was not included; and
k. A copy of any analysis(es) by FDA staff relevant to whether the non-inferiority trial or trials were adequate to establish the effectiveness of the new drug.

2. In the past 10 years, what proportion of products approved by the Office of Antimicrobial Products for which effectiveness trials were required established effectiveness on the basis of non-inferiority trials vs. on the basis of placebo controlled or other superiority trials?

In the second phase of GAO's work in response to our request, we ask that GAO prepare a report containing GAO's evaluation of the FDA's oversight over and reliance on non-inferiority trials, including any findings and recommendations. We request that this analysis provide the following information:
1. A document published by the Division of Anti-Infective Drug Products in October 26, 1992, called "Points to Consider: Clinical Development and Labeling of Anti-Infective Drug Products"[6] indicates that the FDA had concerns about the design and use of non-inferiority trials:

"… the so-called 'bio-creep' phenomenon is always of concern to the Agency. This phenomenon is the selection of successively less effective comparator agents, which individually fit a statistical confidence interval relative to the product to which it was compared. This process, over time, may result in the presumed 'equivalence' of statistically and clinically inequivalent products. Also, the recognized effectiveness of certain products changes with time due to alterations in resistance patterns and development of new knowledge. Constraints imposed by FDA staffing, regulatory requirements, and product manufacturers often hinder the rapid re-labeling of approved products. In order to prevent the occurrence of 'bio-creep' and the selection of 'approved,' yet inappropriate, comparator agents - especially when the selected comparator agent was itself approved on the basis of equivalence in active-controlled trials - we advise applicants to discuss comparator agents with the Division, if they have any doubts, prior to the initiation of their clinical development program. Products establishing equivalence to less effective products should have such information readily available to physicians in the product label. Promotion of such products should also include balanced information regarding the data upon which the product was approved for marketing."[7]

The following questions are based on the concerns raised in that document:
a. Please evaluate the extent to which the current process at FDA allows for or protects against "bio-creep" and whether trials used as the basis of establishing effectiveness had the potential to mislead and indicate "equivalence" of statistically and clinically inequivalent products.
b. What steps has the FDA taken to prevent "the occurrence of 'bio-creep' and the selection of 'approved,' yet inappropriate, comparator agents - especially when the selected comparator agent was - approved on the basis of equivalence in active-controlled trials"?
c. To what extent does the FDA re-label approved antibiotic products because of changes in effectiveness due to "alterations in resistance patterns and development of new knowledge"?
d. What "constraints imposed by FDA staffing, regulatory requirements, and product manufacturers" hinder the re-labeling of approved products to account for resistance and development of new knowledge?
e. Do products that establish "equivalence to less effective products" have such information readily available to physicians in the product label?
f. Does promotion of products that establish "equivalence to less effective products" include balanced information regarding the data upon which the product was approved for marketing"?

2. A list of all drug products approved by the Food and Drug Administration since October 26, 1992, for which effectiveness was established on the basis of non-inferiority trials vs. on the basis of placebo-controlled or other superiority trials. Please provide:
a. The indication(s) for which each of these drugs was approved;
b. The medical reviewing division within CDER that approved the drug; and
c. The justification for use of this trial design that was provided, as required by 21 CFR 314.l26(b)(2)(iv).

3. According to FDA staff, an internal regulatory briefing was held in July 2005 to discuss issues related to the use of non-inferiority trials for indications such as acute exacerbation of chronic bronchitis (AECB).
a. Was consensus reached at this briefing regarding the use of non-inferiority trials to demonstrate effectiveness?
b. What actions, if any, is the FDA taking in response to those conclusions?
c. Since the briefing, how many NDAs using non-inferiority trials to establish drug effectiveness has the FDA accepted for review?

4. An analysis of whether FDA's acceptance of non-inferiority trials to establish drug effectiveness adheres to the principles for such trials that the agency has set forth in regulations, guidance, and related documents, including 21 CFR 314.126, and FDA guidance documents implementing ICH guidance E3 ("Structure and Content of Clinical Study Reports"), E9 ("Statistical Principles for Clinical Trials"), and E10 ("Choice of Control Group and Related Issues in Clinical Trials"). For example, the FDA guidance implementing E9 states:

"There are well-known difficulties associated with the use of the active control equivalence (or noninferiority) trials that do not incorporate a placebo or do not use multiple doses of the new drug. These relate to the implicit lack of any measure of internal validity (in contrast to superiority trials), thus making external validation necessary. The equivalence (or noninferiority) trial is not conservative in nature, so that many flaws in the design or conduct of the trial will tend to bias the results towards a conclusion of equivalence. For these reasons, the design features of such trials should receive special attention and their conduct needs special care. For example, it is especially important to minimize the incidence of violations of the entry criteria, noncompliance, withdrawals, losses to follow-up, missing data, and other deviations from the protocol, and also to minimize their impact on the subsequent analyses.

"Active comparators should be chosen with care. An example of a suitable active comparator would be a widely used therapy whose efficacy in the relevant indication has been clearly established and quantified in well-designed and well documented superiority trial(s) and that can be reliably expected to exhibit similar efficacy in the contemplated active control trial. To this end, the new trial should have the same important design features (primary variables, the dose of the active comparator, eligibility criteria, and so on) as the previously conducted superiority trials in which the active comparator clearly demonstrated clinically relevant efficacy, taking into account advances in medical or statistical practice relevant to the new trial.

"It is vital that the protocol of a trial designed to demonstrate equivalence or noninferiority contain a clear statement that this is its explicit intention. An equivalence margin should be specified in the protocol; this margin is the largest difference that can be judged as being clinically acceptable and should be smaller than differences observed in superiority trials of the active comparator. For the active control equivalence trial, both the upper and the lower equivalence margins are needed, while only the lower margin is needed for the active control noninferiority trial. The choice of equivalence margins should be justified clinically."[8]

Thank you very much for your consideration of this request. We ask that the GAO staff schedule a meeting with our staffs to discuss this request and to work out an appropriate schedule for submitting a response.

Sincerely,

JOHN D. DINGELL CHARLES GRASSLEY

EDWARD MARKEY HENRY WAXMAN

BART STUPAK

http://markey.house.gov/index.php?option=com_content&task=view&id=2084&Itemid=141