Payment for drugs prescribed outside of hospitals is not included within the basic set of healthcare services covered by Medicare in Canada. Accordingly, individual provinces and territories developed their own separate public insurance schemes for drug coverage. The federal government also runs drug plans for members of the military and for native Canadians, among others. All of these plans use a positive formulary (ie, a list of drugs that are paid for), with recommendations for which drugs to include coming from an expert advisory committee.

The presence of multiple committees with different criteria for decision making and with different timetables resulted in a sometimes-chaotic situation. A drug may have been approved for reimbursement in one province early on without any restrictions for use, whereas in another province the same drug would not be listed at all. Furthermore, pharmaceutical companies had to submit the same information to be reviewed by different federal and provincial committees.¹ In 2003 the Common Drug Review (CDR) was established to coordinate reviews for all plans except the one operated by the province of Quebec. CDR reviews applications for coverage for new chemical entities, new combination products, and most recently new indications for existing products.

Pharmaceutical companies wishing to have a drug listed on Canadian formularies submit clinical and pharmacoeconomic information to CDR. Then, CDR contracts a team that prepares a clinical review, including a systematic review of all relevant published and unpublished randomized controlled trials. In addition, CDR examines and critiques the manufacturer’s pharmacoeconomic evaluation. The review team then prepares a report for the Canadian Expert Drug Advisory Committee (CEDAC), a body appointed by the Board of Directors for the Canadian Agency for Drugs and Technologies in Health (see http://www.cadth.ca/index.php/en/cdr/committees/cedac). CEDAC uses the review to assess the clinical and economic value of the product and then makes a recommendation about listing, taking into consideration the medication’s effectiveness, safety, and cost-effectiveness compared with existing therapies.² CDR recommendations are not binding on any of the participating drug reimbursement plans, which are free to make their own funding decisions.³ Therefore, provincial differences in listings may persist.

CDR recommendations have come in for considerable criticism from a number of quarters, specifically drug manufacturers and patient groups.² CDR reviewers are said to be deficient in expertise, and the review process is said to lack fairness, objectivity, and transparency.⁴ The pharmaceutical companies are especially critical of CDR’s reliance on cost-effectiveness criteria in making its recommendations, charging that this process “leaves Canadian physicians with restricted choices and does little to help patients who may not respond well to an existing therapy.”⁵ This theme of patients being denied access to important new medications is echoed by patient groups.⁶

In assessing these concerns, it is useful to look at how CDR compares with other international agencies that are carrying out similar work. This study was undertaken to compare the recommendations CDR has made with recommendations made by similar bodies in other countries, specifically looking at whether CDR imposes more restrictions on access than other agencies that also use a combination of clinical and pharmacoeconomic evidence in their decision making.

METHODS
All recommendations made by CDR and posted on its Web site from September 2003, when it started receiving submissions, until September 30, 2006, were accessed from the organization’s Web site (http://www.cadth.ca/index.php/en/cdr/search?&status=complete&order_field=drug_name). For each drug, a single author abstracted the generic name, the primary indication, and the funding recommendation. If a drug was evaluated more than once, only the most recent recommendation was used. In addition, for each drug, we downloaded a document that contained a brief summary (usually less than 1 page) of the reasons for the recommendation. Because knowledge about a drug may change over time, the date of the recommendation (month and year) also was recorded.

Other agencies making funding recommendations were included, provided that they met the following criteria:

• They were located in a developed country, operationally defined as membership in the Organization for Economic Cooperation and Development.
• Information about the decision was available in English or French.
• Pharmacoeconomic studies were used in deciding whether or not to recommend funding drugs.
• Reports of recommendations were posted on the agency Web site, and the reports included the date on which the recommendation was made and the indication(s) for use of the drug.
• Recommendation categories had to be more than dichotomous (yes/no); for example, a range of decisions must include an option such as “fund with restrictions.”
• Recommendations were made on at least 50% of the drugs considered by CDR.

Based on the above criteria we identified 2 agencies: Pharmaceutical Benefits Advisory Committee (PBAC, Australia) and Scottish Medicines Consortium (SMC, Scotland). Both PBAC and SMC operate in a manner broadly consistent with the way CDR functions in terms of the types of evidence that they require and the way that they evaluate that evidence.^7-9

The PBAC (http://www.health.gov.au/internet/wcms/publishing.nsf/Content/pbac-outcomes-by-meeting) and SMC (http://www.scottishmedicines.org.uk/smc/homepage.html) Web sites were searched by the same author from the time they began posting recommendations (the start of 1999 for PBAC; 2002 for SMC) until September 2006 for recommendations about the same drugs for the same indication as those considered by CDR. Recommendations about the drugs were recorded, along with the dates those recommendations were made (month and year). From December 1999 to March 2003, PBAC’s Web site only contained information about drugs receiving positive recommendations. Once again, if a drug had been reviewed multiple times, the most recent recommendation was used.

We downloaded documents containing information about the studies that the company submitted in support of an application for funding, the agencies’ analysis of this information, and the reasons for their recommendations. Detailed documents containing this information were available for drugs approved by PBAC from 2005 onwards and for SMC from December 2004 onwards. Prior to these dates, only very abbreviated information on reasons for recommendations was available on the PBAC Web site and none was available on the SMC Web site. Clinical and pharmacoeconomic reasons for decisions were abstracted from these documents by a single author.

CDR has 4 different types of recommendations: (1) unrestricted listing, (2) list in a manner similar to other drugs in the class, (3) list with criteria, and (4) do not list. One of these recommendations, “list in a manner similar to other drugs in the class,” is difficult to compare with recommendations of other agencies, as it can mean different things in different provinces and for different drugs. In some provinces, it could mean list with criteria if other drugs in the class also have criteria attached to their listing; in other provinces, it could mean an unrestricted listing. Therefore, we carried out a sensitivity analysis, first assigning all drugs with this recommendation to “list with criteria” and then to “unrestricted listing.”

PBAC makes 4 types of recommendations: (1) unrestricted listing, (2) restricted benefit, (3) authority required (the equivalent of prior approval), and (4) do not list. SMC makes the following recommendations: (1) accepted for use, (2) accepted for restricted use, and (3) not recommended for use.

For the comparison between CDR and PBAC, we collapsed PBAC categories “restricted benefit” and “authority required” into a single category as these 2 categories, taken together, are equivalent to CDR category “list with criteria.” For the comparison between CDR and SMC, we used the 3 different categories from each agency.

We performed 2 different comparisons on the recommendations from CDR and PBAC for the drugs in common and did the same for recommendations from CDR and SMC for the common drugs. The first comparison in each case used χ² analysis to examine whether differences in the percentage of drugs in each of the categories were greater than would be expected by chance. In other words, did CDR and PBAC put equal proportions of drugs into each of the 3 categories? Then we compared recommendations on individual drugs for CDR and PBAC, and for CDR and SMC using weighted kappa scores. Kappa scores measure whether there is more or less agreement between different agencies’ recommendations than would be expected by chance. Levels of agreement were graded in accordance with the recommendations of Landis and Koch.¹⁰ In the case of discordant recommendations about individual drugs, we examined whether there was a difference in time when the agencies made their recommendations. If there was 52 weeks or less difference in the times of the recommendations, we assumed that the agencies would have had essentially the same level of evidence available on the drug’s effectiveness and safety. Finally, where CDR recommendations were discordant from those of either PBAC or SMC, we abstracted and compared the clinical and pharmacoeconomic reasons for the agencies’ recommendations.

All analyses were done twice—once with CDR drugs recommended as “list in a similar manner to other drugs in class” put in the “list with criteria” category and a second time with these drugs in the “unrestricted listing” category. Differences in the way that drugs with the CDR recommendation “list in a similar manner to other drugs in class” were categorized did not affect the analysis, and here we report only our findings when these drugs were put into the “list with criteria” category. Finally, we also compared recommendations on individual drugs for PBAC and SMC.

Statistical analysis was done using StatView 5.0 for Windows (SAS Institute, Cary, North Carolina).

RESULTS
Until the end of September 2006, CDR made 51 recommendations on 47 drugs. Four drugs were evaluated twice. (See Table 1 for a list of drugs and indications and which other agencies evaluated these drugs.) CDR recommended 2 (4.3%) drugs for unrestricted listing, 23 (49.0%) for conditional listing, and recommended against listing 22 (46.8%) drugs. PBAC and SMC made recommendations about 31 and 29 of these products, respectively. Twenty-two drugs were assessed by all 3 agencies. Out of the 9 products assessed only by CDR, the recommendations were unrestricted listing (1 drug), list in a similar manner to other drugs in class (1 drug), list with restrictions (1 drug), and do not list (6 drugs). All of the 6 drugs rejected by CDR were deemed not to be cost-effective. CDR also cited negative cost-effectiveness and/or high costs as either the sole reason or 1 of the reasons for recommending against listing the other 16 unlisted drugs. Similarly, cost-effectiveness considerations were cited by PBAC and SMC in their recommendations to reject 9 of 9 drugs (PBAC) and 8 of 8 drugs (SMC).

There was no statistically significant difference in the percentage of drugs assigned to each of the 3 categories (unrestricted listing, list with criteria, do not list) when CDR was compared with either PBAC (P = .3329) or SMC (P = .0571).

CDR and PBAC made the same recommendations for 24 of the 31 drugs (concordance 77.4%) (Table 2). The weighted kappa was 0.4386 (95% confidence interval [CI] = 0.1246, 0.7526), indicating a moderate level of agreement. CDR and SMC agreed on the recommendations for 14 of 29 drugs (concordance 48.3%) (Table 3). The weighted kappa was 0.192 (95% CI = 0, 0.426), indicating a poor level of agreement.

Table 4 shows all 22 drugs assessed by all 3 agencies and their recommendations. The recommendations are concordant in only 9 cases. When 2 agencies agreed on a recommendation and a third disagreed, in 7 cases it was SMC that was discordant versus 3 and 2 cases each of PBAC and CDR, respectively. In 1 case (insulin glargine), all 3 agencies made different recommendations.

Detailed PBAC documents were available in 4 cases where there were discordant recommendations between CDR and PBAC; for CDR and SMC discordancies there were detailed SMC documents for 5 drugs. The eAppendix Table (available at www.ajmc.com) summarizes the clinical and pharmacoeconomic reasons given by each agency for its recommendations. In general, the analysis of the clinical data was similar, but there were some exceptions. For example, SMC’s evaluation of atomoxetine was somewhat more positive than CDR’s. The main differences in evaluations were in the pharmacoeconomic data. For instance, SMC was quite supportive of the data for pegaptanib, whereas CDR was largely negative. In one case, mycophenolate sodium, despite similar analyses of the clinical and pharmacoeconomic information, CDR and SMC made different recommendations (CDR—list with criteria; SMC—recommended for use).

There were 7 discordant recommendations between CDR and PBAC. In only 1 case was there more than 52 weeks difference between the recommendations of the 2 agencies. For the discordant recommendations between CDR and SMC, in 6 of 15 cases there was more than 52 weeks difference.

PBAC and SMC made similar recommendations for 11 of 22 drugs with a weighted kappa of 0.1705 (95% CI = 0, 0.4407), indicating poor agreement.

DISCUSSION
Among the drugs evaluated by CDR and 1 or both of the other agencies, our results failed to show a difference between CDR and the other 2 agencies with respect to the proportion of drugs recommended for full listing, listing with conditions, or rejection, although the comparison between CDR and SMC did approach significance. The absence of a significant difference may reflect the fact that the agencies had a relatively small number of drugs in common.

When concordance between recommendations concerning individual drugs was evaluated, there was poor to (at best) moderate agreement between CDR and the other 2 agencies, as well as between PBAC and SMC. This low level of agreement between CDR and the other agencies seemed primarily to be due to analyses of the pharmacoeconomic data by the different agencies. These differences may reflect both the quality of the pharmacoeconomic studies submitted to the agencies and a number of different factors that go into these studies, including differences in the proposed price of the product, the price and effectiveness of competing products in the national markets, and the cost of hospitalization and physician visits. In addition, other considerations that may have accounted for differences in recommendations are the prevalence of the disease that the drug is designed to treat, the seriousness of the medical condition, the perceived need for the treatment, the composition of the panel making the recommendation, and the scientific rigor and relevance of evidence for comparative safety and effectiveness.^11,12

When all 3 agencies made recommendations about a drug and there was agreement between 2 agencies and 1 disagreed, it was more likely that SMC would be the discordant agency.

In a minority of cases there was a considerable gap in time (more than 52 weeks) when the agencies made their recommendations. Therefore, the difference in recommendations may have been due to differences in drug safety and effectiveness evidence that was available for evaluation. This possibility is speculation on our part because we have details only about the information submitted to PBAC since 2005 and to SMC since December 2004. Such details are completely lacking for CDR. Furthermore, we have data from CDR only on the time between when the agencies received the files and when they made their recommendations.

Amegatse and colleagues looked at a smaller sample of drugs evaluated by the CDR and compared the organization’s recommendations with those from PBAC and SMC; they reported 86% congruence between CDR and PBAC and 46% congruence between CDR and SMC.¹³ However, their analysis was only published in abstract version, so we are unable to compare their methodology with ours.

Canada’s Research-Based Pharmaceutical Companies (Rx&D) also have examined the status of 50 drugs evaluated by CDR and concluded that CDR recommends reimbursement for fewer drugs than France, Sweden, Switzerland, and the United Kingdom and more drugs than Australia and New Zealand.5 The methodology used by Rx&D is poorly described; therefore, the comparisons it makes with CDR were difficult to interpret.

Our conclusions about CDR recommendations compared with recommendations from other agencies were limited to those agencies that we were able to study. Other agencies that did not post information in English or French may have performed evaluations similar to those of CDR and reached different conclusions. The terminology that SMC uses has evolved over time, and it was not always possible to look back and compare different sets of advice. In Australia, companies may apply for a restricted listing as opposed to an unrestricted one. It is unclear whether the same situation exists in Canada or Scotland. The extent to which an application for a restricted listing affects an agency’s final recommendation is unknown. In some instances, we have significantly oversimplified an agency’s history with some drugs. For instance, insulin glargine was rejected a number of times by PBAC and only given unrestricted listing after the company agreed to a price reduction.¹⁴ Finally, negative recommendations about reimbursement status are not static; companies are able to resubmit applications to all 3 agencies, and changes in either clinical information or pharmacoeconomic analyses can lead to a reversal in a recommendation. For example, since we completed our study, CDR has recommended ciprofloxacin/dexamethasone otic suspension; PBAC has recommended cinacalcet, erlotinib, memantine, and tipranavir; and SMC has recommended omalizumab. Therefore, our results should be seen as a picture at a single point in time.

Our results show the importance of local factors in determining whether or not drugs are reimbursed by formularies, and we believe that this conclusion is applicable to both private and public payers. When considering whether to reimburse a medication, decision makers need to decide what pharmacoeconomic considerations are most relevant to the plan that they are responsible for and evaluate the analyses in that light. It is to be expected that different plans or programs will make different decisions about the same drugs.

In conclusion, although CDR’s recommendations often differ from those of other comparable agencies for the same drug with the same indication, these differences appear to be based on pharmacoeconomic evaluations and likely reflect discrepancies between countries in national markets and health systems.