Matching the Problem with the Methodology: The Failure of the EC’s PatQual Report

Guest Post by Ronald Mann, Professor of Law at Columbia Law School

There's a lot of press lately for PatQual, the EC's massive "Study on the Quality of the Patent System in Europe." To read the press clippings, you'd think the European Commission had produced the definitive work on how to assess the output of government patent offices. But a closer look suggests this study falls far short of the standards for which academics or serious policy analysts strive.

To start with the most obvious problem, the standards of the data analysis are far below what would pass in this country, much less in the quantitatively sophisticated venues of European universities. Although the 194-page report includes more than 100 figures and tables, with detailed and intricate cross-tabulations that slice numerous survey responses across metrics like the business sector of the respondent, the size of the company's patent portfolio, or the size of the company, not a single one of the graphics mentions the number of companies responding to any particular inquiry, and only once in the entire tome do the authors mention that a mere 221 companies and 98 Universities responded. The small n means that the bulk of the report describes "important" findings about the opinions of important sectors of the EU based on the opinions of a handful of companies.

To give one of the starker examples, Table 4 discusses answers about the relative importance of metrics of patent quality by firm size. The column for medium firms (between 50 and 250 employees) reflects the responses of less than 25 companies – out of presumably tens of thousands of such firms across the EU. Given the thin description of the sample in the report, it is entirely possible that all of those firms are located in a single country, or even a single city.

For my own purposes, the most important subjects that the report addresses are the comparative quality of the EU and US systems, questions that are particularly important now as the US and EU offices attempt to establish work-sharing arrangements. To its credit, the report does a good job of discussing the multi-faceted nature of patent quality from a user's perspective, concluding that it includes some combination of timeliness, reliable validity, and cost effectiveness. And it makes some sense to use a survey to analyze the relative importance different users attribute to those different quality metrics. Even there, though, we would expect some attention to sample size and selection bias (the respondents were identified primarily through trade associations, which presumably have their own particular axes to grind on the issues that the report addresses).

But it makes no sense to use surveys to compare how the different systems are doing on those metrics. So, for example, the report goes out of its way to emphasize how poorly the U.S. office is doing on timeliness as compared to the EU – 81% of respondents rate the EU well on timeliness but only 51% rate the US well. But why should we care about a survey on timeliness, when actual data is available? Although reasonable minds could differ on exactly what the right metric for timeliness is, how comparable the two systems are and what the optimal pendency time would be, it is easy to obtain reliable quantitative data on the existing situation. And the data show pretty definitively that however concerned the PTO and US patenters are about pendency time in the US, it is even longer in the EU (almost 50% longer in fact).

So what do we get from a survey purporting to show that the EU is doing much better on timeliness? We learn that if you ask people in the EU, they think their system is better than ours. Presumably if you asked patentees in the US, where Dave Kappos has made his attack on the backlog a highly visible affair, you'd get the same answer.

But that only shows the limits of survey evidence. It is just as easy to produce a poorly designed survey that "proves" things that are demonstrably false as it is to produce a poor case study that emphasizes poorly chosen and unrepresentative anecdotes. As empirical scholars, it is important to ensure that the data on which policymakers make important choices is as reliable as it should be. And the PatQual study, sad to say, surely fails any reasonable baseline of reliability.

26 thoughts on “Matching the Problem with the Methodology: The Failure of the EC’s PatQual Report

  1. 26

    Indeed, Ned. In my previous comment I already smelled something rotten in the USPTO’s pendency figures, and Dennis’ recent post confirms my worst suspicions. Restarting the clock at each RCE is a pretty shameless manner of inflating productivity figures while deflating those related to pendency.

  2. 25

    Max, you may have seen this from me before, but I think the greatest advance of the Romans was the Rule of Law, not their military organization (great) or their civil engineering (great). The rule of law redid all civilization for the better. For only with law does one have rights.

    Where societies are ruled by the leader, the boss or the unelected committee, one relies on their “wisdom,” their sense of justice and their lack of corruption. However, rulers with these attributes are few and far between.

    We in the West are lucky to be descendants of Rome. Lucky indeed.

  3. 24

    Ned, I’m very grateful that I am one of the very few human beings that lives somewhere where the “Rule of Law” actually exists. The Rule of Law is about the most precious thing a human society can have. It scares me how few people grasp that.

    I imagine that trying to explain the concept to a citizen of, say, the PRC, would be well-nigh impossible.

  4. 23

    “But it makes no sense to use surveys to compare how the different systems are doing on those metrics. So, for example, the report goes out of its way to emphasize how poorly the U.S. office is doing on timeliness as compared to the EU – 81% of respondents rate the EU well on timeliness but only 51% rate the US well. But why should we care about a survey on timeliness, when actual data is available? Although reasonable minds could differ on exactly what the right metric for timeliness is, how comparable the two systems are and what the optimal pendency time would be, it is easy to obtain reliable quantitative data on the existing situation. And the data show pretty definitively that however concerned the PTO and US patenters are about pendency time in the US, it is even longer in the EU (almost 50% longer in fact).”

    But it now appears the USPTO data on pendency may be fudged. See, the recent post by Dennis on this. RCEs are counted a new applications. Continuation are taken up quickly.

    If one compared apples to apples, original filings (e.g., no continuations) in both jurisdictions and exclude RCEs as new applications, I wonder what the data would actually show?

  5. 20

    Good for Germany and England. Swift justice is always good.

    I understand enforcement in China might even be swifter: one takes his patent to the local party boss and “works” things out. Most times, lawsuits are unnecessary if the boss in properly “convinced.”

    In fact, the Chinese procedure may be generally applicable anywhere there is no tradition of the rule of law.

  6. 19

    Nice try 6. If the patent system allows indefinite claims, and indefinte claims retard rather than promote progress in the useful arts as the US Supreme Court has so often said they do, then one might say that the higher the quality as measured by how indefinite and broad the claims are would be inappropriate as such claims do the most damage to the progress of the useful arts.

  7. 18

    Well Ned, I cannot discern any difference in patent enforcement in any one country, say, England or Germany, caused by the switch from pre-1978 national patent law to post-1978 EPC harmony.

    The same difference between enforcing patents in England, and enforcing them in Germany, exists now as then.

    But competition is a wonderful thing, that does promote the progress. Consider how the judges in England, and those in Germany, compete for the business. They compete by offering better decisions, reached more quickly and efficiently. Do please compare how slow and expensive patent litigation used to be, compared with how it is today. Do it in England or in Germany. You will find the same effect. But the national cultural differences remain, as striking as ever, whether in engineering skills, in scientific research, or in court procedure and practice. Vive la difference.

  8. 17

    Max, nice defense of Europe and FTF. But really, there is a cause and effect relationship between patent protection and progress in the useful arts. Why shouldn’t there further be a “fine tuning” relationship on top of that depending upon a number of factors, not just FTF. For example, I was thinking the English and the German patent systems clearly were better than most other European systems for a very long period of time. However, were they better during this same period than the US system?

    Now that both England and Germany have adopted the European patent system, has it helped or hurt their respective progress of the useful arts? Now, that would be an interesting study as well.

    But the bottom line is this: patent quality should be measured by the whether the patent is adding to or subtracting from the progress in the useful arts. If the Euro system underprotects patents, or costs way too much, there is a problem to the extent that it reduces the progress in the useful arts.

  9. 16

    Because patent quality and the maximization of the progress of the useful arts are practically 100% unrelated from one another?

  10. 15

    Oh Ned, you mean like speaking English is the cause of most of the world’s heart disease? That is what the data reveals.

    In the same way, “First to Invent” makes an industrial superpower while First to File turns countries into pygmies?

    We already know, don’t we, that the USA is different from Europe and why that is? Or are you saying that all those books comparing the USA and Europe to Mars and Venus have missed the obvious explanation, that those differences are caused by the USA’s unique First to Invent patent system.

    I commend to you a great book by a Michigan professor called Richard Nisbett called “The Geography of Thought”.

  11. 14

    Well, one thing, compare progress in various countries. See who does best. Then, figure out why.

    I understand the Japanese did just that back in the late 1800’s when they were trying to figure out why the West was so technically advanced.

    Thus a valid comparison of the Euro systems would be to compare it to the US, to the Japanese, to the Koreans, to the Chineses, etc.

  12. 13

    Ned asks us a riddle: Why don’t we define quality as that level that maximizes the progress of the useful arts.

    I answer: Good idea in theory. Lovely, if it can be done. But how does one set about measuring it?

  13. 12

    Good stuff from CC. Despite the absence of any EPO fee, despite the absence of any requirement that Applicant give any reason to “make special” and despite the absence of any constraint on Applicant how he prosecutes an application made special at the EPO, fewer than 5% of applications pending at the EPO are on its PACE programme of expedited handling.

    With so few cases on PACE, that means that each PACE case does indeed get handled expeditiously. Yet hardly anybody wants it.

    CC reminds us that the EPO is not an organ of the EU. We are all intelligent here. Can we try to distinguish between the problems of the EU (including how patents are litigated in Europe) and the problems of the EPO (how applications for patent are examined)? Those who want to assess what is in the EPO pipeline are free to do so, as soon as 18 months from the earliest declared priority date have elapsed. Knowing how strict Europe is with prosecution amendments, they need only to look at the WO (or EP-A publication) and the EPO search report. Then they can write a definitive clearance opinion.

    For F’s sake Dorothy, what more do you want from the EPO?

  14. 11

    While I understand and share Professor Mann’s misgivings about the small sample size, his discussion of timeliness betrays an impressive degree of ignorance about how the European patent system actually works. Of course, considering that he appears to be perfectly unaware of the difference between the EU and the European Patent Office (two completely separate entities with different sets of member states), that isn’t altogether surprising.

    When a European patent application is filed as a first filing (that is, not claiming priority from an earlier foreign or domestic application), a search report, with a first written opinion on patentability, is prepared well within the year. In my experience, in well over 90% of the cases, that search report not only cites all the most relevant prior art that is going to be cited throughout the examination procedure, but the written opinion also gives a clear contour of what may eventually be granted. Applications claiming an earlier priority are not given such a deadline, and in such cases the search report may indeed take some time in coming, but even then, after that search report, applicants are generally spared the merry-go-round of “non-final” and “final” decisions, RCEs, etc.

    Moreover, at the EPO the applicants always have the option of requesting accelerated examination without any extra charge (unlike what was recently mooted and then suspended at the USPTO). It is important to note that most applicants decline to do so (which is unsurprising, considering that a large portion of the cost of a European patent becomes due around the time of granting).

    Finally, as WIPO wisely points out, pendency times are not measured in the same way throughout patent offices. I wonder how each patent office measures pendency for divisionals, continuations, and continuations-in-part.

    So, when the professor asks the question of “But why should we care about a survey on timeliness, when actual data is available?”, the answer is simple: we don’t have any “actual data” on timeliness, understood as the compliance of the patent offices to the timescales wished by the applicants. We only have (highly disputable, and difficult to compare) data on pendency, which is a completely different thing. I am surprised that an academic from such a hallowed institution is not able to notice the difference between timeliness and pendency, and how applicants, at the end of the day, may care a lot less about the latter than about the former.

  15. 9

    The pertinent definition of “patent quality” is: the extent of complicance with the legal rules. Whether these rules are appropriate is something else, in particular as far as interpretation of statutory rules is concerned.

    Patent offices may raise this type of quality by aiming at predictable results. Which creates a false incentive. For instance, if *all* subject-matter is considered patentable, then there are no complaints about inconsistency. Perhaps about the substance, but that is outside the scope of this assessment.

  16. 8

    Yes, let’s divert into side tracks of interest – anything at all to avoid the conclusion of the study here (and the unending MaxDrei shill) of how great and wonderful the EP is.

    Much like the study – the drumbeat of EP this and EP that lacks substance.

    Pay no attention to that man behind the curtain…

  17. 7

    Thanks for the Link, Nits. Good point you make. For example, I found the graphs on WIPO’s page 43 interesting, comparing the historical rate of filing oppositions at the EPO with that at the USPTO.

  18. 6

    “And the data show pretty definitively that however concerned the PTO and US patenters are about pendency time in the US, it is even longer in the EU (almost 50% longer in fact).”

    The data also includes a note that it “may not be comparable between patent offices due to the procedural specificities of each office and methodological differences.” (link to wipo.int p.45)

  19. 5

    If you want to know about “quality” it is easy to ask the users, those whose daily work is to file, prosecute and litigate patents in Europe and the USA. But then all you get is opinions, and only from those who can be bothered to reply. In Europe, I suspect, very few small and medium sized EPO users would take the time to provide the EC with reliable, informative and useful replies. I suspect the reason why the EC does not state the sample size is that it is embarrassingly small.

    Better to use “the data” says the writer. But what data? When all seasoned users say that the available data paints a misleading picture, what then? Use worse than useless data to make policy decisions? I think not.

    Let’s compare Patent Offices with motor cars. Where is the data that reveal a BMW to be of higher “quality” than some other make? What methodology does a JD Power survey use?

    Is the EC looking at the right thing? Should it be looking at performance, rather than “quality”. Data on how BMW cars perform is abundant, and does enable comparison with other makes of car.

  20. 4

    Interesting point on the perception of timliness. My limited view was that the US has longer pendency times, especially if you figure in what seems to be the inevitable appeal. With the EPO sending out Rule 161 actions prior to Examination, things certainly start earlier than in the US (thank goodness they no longer have 1 month deadlines!).

    My perception on “quality” is that the EPO has it over the US-1)more art found that is actually relevant, 2) actual consideration of one’s arguments and thoughtful replies; and 3)much more of a “let’s cooperate” attitude than the USPTO.

  21. 1

    MaxDrei is needed back in Europe, rather than spending so much time on US Patent Law blogs.

Comments are closed.