Santa Clara – Duke Law Patent Quality Conference: Patent Quality – It’s Time

Guest Post from Professors Arti Rai and Colleen Chien.

“I know well the difficulty of drawing a line between the things which are worth to the public the embarrassment of an exclusive patent, and those which are not.“ – Thomas Jefferson, 1813

Santa Clara –Duke Law Patent Quality Conference: Patent Quality – It’s Time

At 1 pm today, USPTO Director Michelle Lee will be testifying before the House Judiciary Committee at a USPTO Oversight Hearing.  The hearing will focus on questions of patent quality raised by two recent GAO reports (here and here) as well as the issue of examiner reporting of time raised by a  report by the Department of Commerce’s Office of the Inspector General (here).

As we have discussed, the two of us are following closely the USPTO’s efforts to address issues of patent quality through its Enhanced Patent Quality Initiative (EPQI) – an urgent but also enduring challenge that one of our nation’s first patent examiners, Thomas Jefferson, struggled with.  Our institutions, the Duke Law Center for Innovation Policy and the Santa Clara High Tech Law Institute, are also co-sponsoring two conferences on EPQI and other levers for improving patent quality.

On Friday, September 9, we held the first of these conferences at Santa Clara Law School.  The conference brought together Deputy Commissioner for Patent Quality Valencia Martin-Wallace, Deputy Commissioner for Patents Andy Faile, former PTO Director David Kappos, EPO Director of Quality Support Alfred Spigarelli, GAO officials, industry representatives, and legal academics.  Here we provide a brief summary of the event.  Video and links to the presentations and related papers will be available here (a uncut version of the video is here). Several of the commentators will be publishing op-ed versions of their remarks with IPLaw360 from now until the December 13, 2016 conference, and the Berkeley Technology Law Journal has published short commentaries from a number of the speakers on their topics.

Valencia Martin-Wallace began by highlighting 6 aspects of the EPQI: case studies the USPTO is implementing in response to stakeholder requests; the Master Review Form (MRF), which is now being used as the exclusive review form by the Office of Patent Quality Assurance; other quality metrics; USPTO study of post-grant outcomes for purposes of improving ex ante patent examination, both in child applications of parents there are the subject of a post-grant review and more generally; a post-prosecution pilot; and a clarity of the record pilot.

Martin-Wallace’s comprehensive discussion of the MRF and quality metrics honed in on the difficult question of what quality means and how the PTO should measure it.  According to the USPTO, the MRF’s focus on clarity and correctness of the examination is a key aspect of how the agency measures product quality. The agency is also very interested in process quality, by which it means a focus on reducing rework and ensuring consistency.  The USPTO will also continue to monitor perceptions of quality through internal and external quality surveys. USPTO Case studies on 101 – the compliance of rejections with 35 U.S.C. 101 Official Guidance and the consistency of the application of 101 across art units/technology centers – will be issued on September 30 and December 9 respectively.  They will be followed by case studies on other types of rejections.

Martin-Wallace’s presentation was followed by a panel discussion of claim clarity and examination consistency.  Peter Menell discussed how claim clarity could be improved through a claim template [link] and the importance of putting all interactions with the examiner into the record.  Charles Duan also highlighted the importance of a clear prosecution record, particularly for purposes of allowing courts to find claim scope disavowal.  Jay Kesan focused on the lack of standardized claim language in software and suggested mechanisms for improving standardization.  Finally, Xavier Jaravel of Stanford presented his research with Josh Feng of Harvard showing that NPEs tend to purchase patents granted by lenient examiners that are incremental and vaguely worded; they estimate that a one-standard deviation change in the “examiner effect” could lower the rate of litigation and NPE purchased patents by 50%.

The conference then turned to issue of examiner time allocation, and in particular ensuring adequate time for search.  Deputy Commissioner for Patent Operations Andy Faile outlined the count system and the time that examiners are given to achieve each count.  On the first round of examination, examiners receive 1.25 counts for the first office action; 0.25 counts for the final rejection in that round; 0.75 counts for an allowance disposal: and 0.5 counts for termination of the first round of examination (“abandonment”).  The time allocated to achieve each count is a function of the area of technology – for example, while 16.6 hours are allocated per disposal in fishing lures, examiners are given almost double that, 27.7 hours per satellite communications disposal. It is also a function of GS-level of the examiner – as in the EPO, senior examiners at the USPTO are expected to produce more work than junior examiners.

Assistant GAO director Robert Marek next discussed highlights from the agency’s studies of patent quality and prior art search.  These studies relied heavily on a survey that produced 2669 USPTO examiner respondents.  The survey found that 70% of examiners reported having insufficient time to examine applications; the majority reported wanting more time for prior art search; and the majority also reported encountering vague and indefinite claims.  Examiners also reported conducting only limited searches for non-patent literature (NPL).  On the issue of search, the GAO recommends that the PTO develop guidance of what constitutes a good prior art search; identify key sources of NPL; monitor search quality; and assess the time examiners need for search.  On claim clarity, GAO recommends the PTO consider requiring patent applicants to include term glossaries or claim charts.

The morning concluded with a panel discussion of prior art search and time.  Colleen Chien discussed the findings of a recently released study that suggests that the EPO is much more likely to cite NPL in its search reports than PTO examiners are to rely upon NPL in their examination, despite that the PTO’s examiners are more likely to receive NPL, through IDS’. The paper argues that that the greater amount of time that EPO examiners spend on search (8-12 hours per app, vs. 4-5 hours at the PTO), contributes to this difference. However, the PTAB, which intensely reviews challenged patents, appears to cite NPL even more than EPO examiners do. Michael Frakes and Melissa Wasserman then summarized their empirical research on 1.4 million utility patent applications initiated after March 2001 and disposed of by July 2012.  Their research suggests that tightening of examiner time constraints as they move up GS-levels appears to lead to more grants, less examiner citation of prior art, and fewer time-intensive non-obviousness rejections.  Their analysis, which follows examiners as they rise up GS-levels within art units, capitalizes on the essentially random assignment of applications within an art unit and employs an examiner fixed-effect design. Moderator Karen Wong called the After Final Consideration Pilot 2.0 one of the PTO’s best inventions, and Steven Reid and others noted that the Post-Prosecution Pilot (P3) examination was also a valuable option for achieving resolution without the need for a full request for continued examination (RCE).

Jay Kesan and Colleen Chien kicked off the afternoon with a discussion of the differences between EPO and USPTO processes (the former bifurcates search and examination, doesn’t allow for continuations (but does tolerate divisionals), and charges higher fees, earlier), perceptions of the two institutions (based on a recent survey of ~650 practitioners), and outcomes. Drawing upon matched pair analysis, Kesan’s research suggests that on average EPO patents have fewer claims, and the claims are longer and have greater pendency. Chien’s paper suggests that the lower EPO allowance rate isn’t driven by a higher rejection rate but more applicant withdrawals.

EPO Patent Quality Director Alfred Spigarelli’s presentation was titled: “Patent Quality: Get it Right the First Time.” He emphasized the human resources component of the patent quality equation, from the hiring standards the EPO applies (e.g. examiners must be able to actively work in three languages), to the training each EPO examiner receives, to the low turnover rate. Search is viewed as the cornerstone of the patent examination process, with 60% of time dedicated to search.

On the panel that followed (“Once and Done” and Differentiating Between Patents), Laura Sheridan outlined a proposal to increase quality and predictability by introducing greater finality into the patent application process.  Under Sheridan’s proposal, examination would terminate after a predetermined maximum number of office actions, and the application would be then adjudicated by a panel.  Steven Yelderman discussed the dynamic effects on RCE practice of recent changes to fees as well as mechanisms by which post-grant review could feed information back into initial patent examination. Sandy Swain discussed steps that could be taken by applicants now, without any policy changes, stressing the importance of open communication and clear documentation. Alan Marco presented a study in progress of litigated patents, suggesting that small entities are more likely to litigate and finding that (with the exception of continuations) applicant characteristics are more likely to be predictive of litigation than examination characteristics.

Studies have shown that delays in patent examination are detrimental to startup firms. For these firms, long pendency could be considered a sign of poor examination quality.  Building on this work, Arti Rai discussed her ongoing research on accelerated examination through the Track 1 program.  She finds that the program is in fact disproportionately used by small and micro entities. However, the top filers in the program are large entities, and she is studying whether their applications show signs of poor quality. Josh Makower, an investor, inventor, and entrepreneur in the biotechnology and medical device industry, said that all his startups used TrackOne, underscoring the importance of patents to them. According to Oskar Liivak’s interpretation, 35 USC 115 makes it a felony for a patent applicant to claim more broadly than the actual invention. Brian Love noted that the EPQI had two important gaps – already granted patents and the flexing of the USPTO’s fee-setting authority. He recommended raising maintenance fees to cull low-value, low-quality patents and decreasing PTAB fees.  Love’s proposals generated vigorous discussion.

Dave Kappos provided the final keynote of the day, noting that patent quality has been a focus for as long as we’ve had a patent commissioner. He noted that many efforts he oversaw during his administration to increase quality – giving examiner 2.5 more hours of time, redesigning the IT system, creating the Edison Scholar program, ensuring Examiner received training from industry experts, and many more. What didn’t work, he noted, was working with the AIPLA to devise a set of quality indicia for applicant filings that all agreed upon and that could be applied to applications. What he would work on, were he still in office, would include Track 3 (some version of deferred examination) and allocation of more time, not across the board, but as appropriate.

How will we know if the EPQI has been successful in two years? When this question was posed to them, closing panelists Faile, Martin-Wallace, and PTO Silicon Valley Director John Cabeca mentioned a few milestones: greater transparency and engagement of all members of the patent community, creative rethinking of the count system, more time for certain applications, and the leveraging of state of the art examination processes and resources.

* * * * *

Arti Rai is the Elvin R. Latty Professor of Law at Duke Law School and co-Director, Duke Law Center for Innovation Policy.  Colleen Chien is Associate Professor of Law at Santa Clara University School of Law.

45 thoughts on “Santa Clara – Duke Law Patent Quality Conference: Patent Quality – It’s Time

  1. 15

    Applicants really wanting a high quality patent, and willing and able to pay for it, will have a quality prior art search done BEFORE preparing the patent application and its claims. [The PTO ought to encourage and reward that in some way, and so should investors.]
    No mere internal PTO application processing changes can make up for applications missing consideration of the most relevant prior art due to the very limited prior art searching time an examiner normally has available. Given available post-grant proceedings, serious prior art issues can no longer be avoided in patent litigation before settlement, trial or appeal as in the past.

    1. 15.1

      That is truly a “best practices” item of advice – and long one of my typical items discussed with clients.

      But the bottom line – of which I am sure that you are aware of (even if you no longer prosecute) – is that there is NO legal requirement for ANY searching, and the degree of searching (down to and including NO searching) is entirely up to the client.

      That being said, what type of “reward” system do you envision?

  2. 14

    “Michael Frakes and Melissa Wasserman…. …research suggests that tightening
    of examiner time constraints as they move up GS-levels appears to lead to
    more grants, less examiner citation of prior art, and fewer time-intensive
    non-obviousness rejections. ”

    Pretty sure you mean “obviousness rejections”.

  3. 13

    Here is the link to our research: link to

    The main idea is that within specific areas, patents eventually purchased by NPEs are initially granted by examiners that are less likely to issue certain blocking actions and make mistakes as measured by PTAB appeals and IPR. We also do a bunch of other tests of theories favorable to NPEs, using other data like EPO decisions.

    Very happy to hear constructive criticism and other feedback.


  4. 12

    “Their research suggests that tightening of examiner time constraints as they move up GS-levels appears to lead to more grants, less examiner citation of prior art, and fewer time-intensive non-obviousness rejections. ”

    Lol no kidding?

    “According to Oskar Liivak’s interpretation, 35 USC 115 makes it a felony for a patent applicant to claim more broadly than the actual invention. ”

    He’s probably right about that. Although you do have to consider mens rea and all that.

    1. 12.1

      What would the mens rea requirement of 35 USC 115 be?

      It would be a real shame if this “professor” was actually “teaching” this drivel to students. Cornell can’t be cheap. Imagine paying that tuition and having this guy pushing this nonsense.

      Demand a refund.

      1. 12.1.1

        “What would the mens rea requirement of 35 USC 115 be?”

        I would guess only what it states – the “willful” making of a “false statement”.


          …never mind that the oath signed with the application removes the willfully aspect – given as the oath attests that the inventor is swearing that he really does believe that he deserves the breadth claimed.

    2. 12.2

      LOL. I bet the DOJ is just itching to set aside money-laundering, racketeering, human trafficking and other similar cases so that it can prosecute an inventor for willfully claiming more than he was entitled to!

  5. 11

    A strategy to increase patent “quality” should focus on three things:

    (1) clarify the standards for patentability (i.e., aspects of 101, 112, and 103 are broken, as presently applied) – this will help ALL parties, including examiners, applicants, and the public
    (2) improve the Office correspondence platform and file wrapper – all amendments and arguments should be made via online text fields that enable ALL parties to easily search and review the record for arguments, amendments, etc.
    (3) improve the examiner searches – examiners should reduce citations to patent documents, and focus on broader web searches

  6. 10

    For what it’s worth, in my experience, patent examination quality in the software arts has gone up an order of magnitude since I started practicing ~2001. Even five years ago examination was often laughable and incompetent. The experience level and qualifications of the examiner corps has improved. To me, that’s the foundation of patent quality.

  7. 9

    McRo is out today and, as predicted from the oral arguments, the panel has laid a big dusty log in the punch bowl of contemporary 101 jurisprudence. Fixing this is going to waste a lot of time and money. Heckuva job, CAFC.

    1. 9.2

      The very long wait before the opinion suggested a tap dance. I haven’t read it yet, but I have notions of the expected choreography. I’m looking forward to digging in.

      Don’t go to sleep on the forces of counter-revolution.

      The geopolitical situation is the most fraught of my lifetime. Bond markets are shaky despite low cost energy, food, and goods. Deflation is fighting inflation- while the United States is near election of a truly radical regime, arriving in office with unprecedented support in statehouses and likely vert proof majorities in both houses of Congress and a Supreme Court pick in hand.

      If the wheels do come off the economy, we could see large, radical changes in the patent law- including a complete statutory and Supreme Court repudiation of Mayo/Alice, resulting in a kleptocracy that makes the first round look like a Sunday picnic. The effects will strike a tech industry reeling already, naturally, just to enhance everyone’e misery. Ritual sacrifice is what folks do in the face of fear and uncertainty.

      Warring with North Korea, Iran, Mexico, et al. during a sharp recession (or worse) with retrograde social legislation coming a mile-a- minute and the use of the courts as political weapons against dissent, nobody will know or care what becomes of patent policy for a good long time.

      Have a nice afternoon!

  8. 7

    According to Oskar Liivak’s interpretation, 35 USC 115 makes it a felony for a patent applicant to claim more broadly than the actual invention.

    This is great news for the prison industry!

  9. 6

    Jay Kesan focused on the lack of standardized claim language in software and suggested mechanisms for improving standardization.

    I’d like to hear more about these “mechanisms”, which — presumably — are more accurately characterized as “rules.”

    Oh, the irony.

  10. 5

    The problem here is that the normal incentives that drive quality are totally misaligned at the USPTO. Firms increase quality when forced to do so by competitive pressure or regulation, with the penalty for failing to increase quality being loss of market share, fines, etc.

    With the USPTO, there is no competitive pressure and zero cost to the office for failing to increase quality.

    At the very least, a patent held invalid in court should mean a refund from the office of all fees and costs- payable to the US Treasury or the prevailing party- as some minor measure of pain for doing the job badly. Naturally if a change in law or doctrine caused the invalidation, judges should have discretion not to zing the USPTO.

    Confusing compliance with quality as determined by outcome is the classic dodge of unaccountable organizations.

  11. 4

    “…Laura Sheridan outlined a proposal to increase quality and predictability by introducing greater finality into the patent application process. Under Sheridan’s proposal, examination would terminate after a predetermined maximum number of office actions, and the application would be then adjudicated by a panel.”

    I’ve stated numerous times over the years on this site that the USPTO should adopt something similar to the EPO’s oral proceedings. I guess you have to be an academic to get any notice from Dennis.

    1. 4.1

      BTW, the reason this proposal will never, ever be implemented is because the PTO’s position is that it is entitled to as many opportunities to get it wrong as they need. Or want. There’s no way the PTO is going to agree to any system that requires them to “adjudicate” an application after a certain number of rejections, especially if by “adjudicate” the good professor means “the patent will issue if the examiner’s rejections are reversed.”

  12. 3

    Talking about transparency, I recently reviewed a file wrapper where there were numbers of office actions, numbers of amendments and an equal if not more numbers of interviews. While the office actions spelled out the rejections, the amendments simply provided amended claims and a reference to an interview. The interview summaries were typical – very uninformative.

    So as a result of the review, I really had no clue as to what or was not important about the amendments in differentiating over the prior art. Needless to say, this kind of practice must stop. Unless interviews are recorded and transcribed, they must be ended.

    1. 3.1


      Take a step back and review your basic communication and negotiation theories.

      You go too far when what instead is needed is simply a more accurate summary of the interview and the inclusion of the “why” that you are looking for.

      That just does not need the drastic “record the entire exchange” line – which, would be counterproductive (again, if you understood communication and negotiations, would be quite obvious as to why).

      Hey, I have no per se problem with exacting “recordings,” it is simply that if you have that, you impede the very purpose of having such discussions to begin with, and you might as well simply not bother at all with any such avenue.

    2. 3.2

      How is this not sufficient? :

      It is the responsibility of the applicant or the attorney or agent to make the substance of an interview of record in the application file, except where the interview was initiated by the examiner and the examiner indicated on the “Examiner Initiated Interview Summary” form (PTOL-413B) that the examiner will provide a written summary. It is the examiner’s responsibility to see that such a record is made and to correct material inaccuracies which bear directly on the question of patentability.

      Examiners must complete an Interview Summary form PTOL-413 for each interview where a matter of substance has been discussed during the interview by checking the appropriate boxes and filling in the appropriate blanks. If applicant initiated the interview, a copy of the completed “Applicant Initiated Interview Request” form, PTOL-413A (if available), should be attached to the Interview Summary form, PTOL-413 and a copy be given to the applicant (or applicant’s attorney or agent), upon completion of the interview. If the examiner initiates an interview, the examiner should complete part I of the “Examiner Initiated Interview Summary” form, PTOL-413B, in advance of the interview identifying the rejections, claims and prior art documents to be discussed with applicant. The examiner should complete parts II and III of the “Examiner Initiated Interview Summary” form at the conclusion of the interview. The completed PTOL-413B form will be considered a proper interview summary record and it will not be necessary for the examiner to complete another PTOL-413 form. Discussions regarding only procedural matters, directed solely to restriction requirements for which interview recordation is otherwise provided for in MPEP § 812.01, or pointing out typographical errors in Office actions or the like, are excluded from the interview recordation procedures. Where a complete record of the interview has been incorporated in an examiner’s amendment, it will not be necessary for the examiner to complete an Interview Summary form.

      The Interview Summary form PTOL-413 shall include the date the interview was held and the substance of the interview shall be properly recorded. In a personal interview, the duplicate copy of the Interview Summary form along with any attachment(s) is given to the applicant (or attorney or agent) at the conclusion of the interview. In the case of a telephonic, electronic mail, electronic message system or video conference interview, the copy is mailed to the applicant’s correspondence address either with or prior to the next official communication. A copy of the form may be faxed or, if the Office has appropriate authorization to conduct communications via the Internet, a copy of the form may be e-mailed to applicant (or applicant’s attorney or agent) at the conclusion of the interview. If additional correspondence from the examiner is not likely before an allowance or if other circumstances dictate, the Interview Summary form should be mailed promptly after the telephonic, electronic mail, electronic message system or video conference interview rather than with the next official communication.

      The complete and proper recordation of the substance of any interview should include at least the following applicable items:

      (A) a brief description of the nature of any exhibit shown or any demonstration conducted;
      (B) identification of the claims discussed;
      (C) identification of specific prior art discussed;
      (D) identification of the principal proposed amendments of a substantive nature discussed, unless these are already described on the Interview Summary form completed by the examiner;
      (E) the general thrust of the principal arguments of the applicant and the examiner should also be identified, even where the interview is initiated by the examiner. The identification of arguments need not be lengthy or elaborate. A verbatim or highly detailed description of the arguments is not required. The identification of the arguments is sufficient if the general nature or thrust of the principal arguments can be understood in the context of the application file. Of course, the applicant may desire to emphasize and fully describe those arguments which he or she feels were or might be persuasive to the examiner;
      (F) a general indication of any other pertinent matters discussed;
      (G) if appropriate, the general results or outcome of the interview; and
      (H) in the case of an interview via electronic mail a paper copy of the contents exchanged over the internet MUST be made and placed in the patent application file as required by the Federal Records Act in the same manner as an Examiner Interview Summary Form, PTOL-413, is entered.

      Examiners are expected to carefully review the applicant’s record of the substance of an interview. If the record is not complete or accurate, the examiner may give the applicant a 2-month time period to complete the reply under 37 CFR 1.135(c) where the record of the substance of the interview is in a reply to a non-final Office action.

      link to

  13. 2

    Spigarelli’s Rule, to “Get it right first time” is one the EPO applies just as much to Applicants as to Examiners. Indeed, only be forcing Applicants to “Get it right first time” does the EPO have any chance to deliver its impressive quality numbers.

    But the EPO is master in its own house. Not so the poor old USPTO, which has to implement instead what the courts on appeal decide, guided by attorney at law advocacy rather than experience at the coal face of examination.

    So there is a limit to what can be gained, in making comparisons with the EPO.

    1. 2.1

      Max, how’s EPO doing in delivering high-quality, timely prosecuted and with low cost?

      Needless to say, if one is willing to pay enormous expenses over a great deal of time, one can almost perfect examination and always issue the highest quality work product.

      1. 2.1.1

        Ned, anon calls me a shill for the EPO. In answer to your question “How’s the EPO doing?” I’m at a loss how to reply. Why not ask those who continue to stump up its “enormous” fees? What do they think, of the comparative value for money offered by the USPTO and the EPO?

        Beware the stats though. For 80% of those who apply to the EPO for a patent, long pendency suits them (bearing in mind that, whenever they want, they can push the PACE button to get to grant asap). Long pendency puts off the expensive translation at grant, and the transition from a single EPO annuity to a national annuity paid to the Patent Office of as many of the 38 EPC Member States as the owner chooses. Besides, competitors don’t know i) what claims will go to issue and ii) what claims will be filed in the yet to be filed divisional applications. Most Applicants would choose to go to grant after, rather than before, the competitor’s product launch. Today, the EPO’s commercially-minded management is happy to oblige.


          Lots of “dancing” from you MaxDrei without any real answer from you for a decidedly straight forward question…

          (maybe this is part of why you get recognized as a shill…)


          The problem I see here is that one size does not fit all.

          Smaller inventors simply cannot afford super car prices. The just want to get to work.

  14. 1

    Is there a link to the “NPEs purchase lower quality patents” study?

    I ask because this is in direct conflict and has the opposite conclusion of a previous study (now somewhat dated) that Peter Zura presented.

    1. 1.1

      Anon, if you define low quality as equivalent to vague and indefinite claims, but not so indefinite as to be invalid, then I would think that I would think most unsophisticated NPE’s seeking to acquire patents for the purposes of asserting them for profit might be influenced by the apparent breath of the claims.

      1. 1.1.1

        Ned, Peter Zura’s old data dispelled that “notion,” and actually put forth that NPI’s gen erally had better vetting programs for what they purchased than what your typical NON-NPE evaluations.

        The reason why I ask is because this “academic” article smacks of more of the same Kool-Aid 0h-N0es “Tr011” beverages passed out in copious quantities by Big Corp.


            Most definitely not, Ned.

            That’s like saying that your opinion can substitute for facts.

            It can’t.

Comments are closed.