Guest Post by Dr. Shine Tu: Luck/Unluck of the Draw: An Emprical Study of Examiner Allowance Rates

Dr. Shine Tu is an Associate Professor of Law at the West Virginia University College of law.  For the past year, Dr. Tu, a former patent attorney with Foley & Lardner, has conducted a massive empirical examination of patent prosecution.  Massive, in the sense that Dr. Tu and his assistants examined the prosecution behavior of every patent issued in the last ten years.  In Luck/Unluck of the Draw: An Empricial Study of Examiner Allowance Rates, available at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1939508 and forthcoming in the Stanford Technology Law Review, Dr. Tu discusses some of the findings from his study. – Jason

Luck/Unluck of the Draw: An Emprical Study of Examiner Allowance Rates

Dr. Shine Tu

One of the greatest banes of a patent applicant is the examiner who will not allow an application that should be allowable.  During a trip to the PTO for an Examiner interview, I passed by the office of Examiner X with a sign that stated “0% allowance rate ☺ [ed.: smiley face].”  Evidently, the smiley face was inserted to add insult to injury.  I asked the examiner that I was with if Examiner X really had a 0% allowance rate, and she said “yes.”  (I later verified that this was correct)  I asked why this was the case, and she stated, “some examiners believe an allowance is an affront to their being.”  Of course, our patent system would be a complete failure if the ability to get a patent was solely determined by which examiner you were lucky/unlucky enough to draw.

In an attempt to determine the extent of this behavior, I set out to comprehensively study allowance rates of specific examiners by art unit, technology center and the PTO as a whole.  To be as comprehensive as possible, I coded over 1.5 million patents in an attempt to code every patent ever issued in the last 10 years (from January 2001-July 2011).

The results of this study were surprising.  I found two interesting populations of examiners.  First, I found a significant population of examiners who on average, issue a disproportionately small number of patents per year (low allowance rate examiners).  Second, I found a small but significant population of examiners, who on average, issue a disproportionately large number of patents per year (high allowance rate examiners).

The first population of low allowance rate examiners consist mainly of secondary examiners (junior examiner usually with less than 5 years of experience and no signatory authority).  These examiners, on average, issue a very small number of patents per year (less than 5 patents per year).   For example, between 2002-2011 in technology center 3700, approximately 17% (306 examiners) of the examiners issued less than 0.35% (823 patents) of the patents in this art unit.  Accordingly, these low allowance rate examiners issued approximately 2.7 patents per examiner.

In contrast, I found a second population of high allowance rate examiners that consist mainly of primary examiners (senior examiners usually with more than 5 years of experience and full signatory authority).  These examiners, on average, issue a high number of patents per year (more than 50 patents per year).  For example, between 2002-2011 in technology center 3700, approximately 12% (215 examiners) of the examiners issued approximately 51% (120,822 patents) of the patents in this art unit.  Accordingly, these high allowance rate examiners issued approximately 561.9 patents per examiner.

There are many good reasons why an application should not be allowed.  However, one reason that should not play a role is an uneven / inconsistent application of patentability rules.  Examiners should be consistent in the way they apply patentability rules within their own docket.  Additionally, examiners should be consistent in the way they apply patentability rules compared to examiners within their art unit.  This study suggests that examiners even within the same art unit may be applying the rules of patentability in an inconsistent manner. 

There are many limitations with this study.  First, I note that there is no ideal allowance rate.  Second, I note that this is only descriptive data.  Finally, I note that I focus only on issued patents.  I could not obtain data related to the total number of filed applications for each art unit, thus I cannot calculate the percentage allowance rate per art unit (because I do not have a denominator of “filed applications”).  This is a significant limitation because all of these results are limited by the “denominator” problem.

Additionally, it is possible both populations of examiners are doing an outstanding job.  That is, it is possible that the low volume examiners are simply getting harder cases (but I am skeptical of this explanation).   Additionally, it is possible that the low volume examiners are “doing a more thorough job” since they are mainly secondary examiners and thus have more time to spend per application for review and detailed analysis.  Furthermore, it is possible that high volume examiners are doing an adequate job because they are simply more experienced and not only know the field better, but know how to quickly make “good” rejections.  In order to obtain some substantive answers to overcome these limitations, I am currently reviewing a sample of prosecution histories to determine if these two populations of examiners truly are applying the patentability rules in an inconsistent manner. 

A full draft of “Luck/Unluck of the Draw: An Empirical Study of Examiner Allowance Rates” is available on SSRN at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1939508 and is forthcoming in the Stanford Technology Law Review.  Comments are appreciated.

195 thoughts on “Guest Post by Dr. Shine Tu: Luck/Unluck of the Draw: An Emprical Study of Examiner Allowance Rates

  1. 195

    Grandfathering under Article 163(1) EPC 1973 did not carry any duty to read and understand the new scriptures…

  2. 194

    American Cowboy,

    Your first error is in actually thinking that MaxDrei is correct in how he characterizes patent law in at the EPO.

  3. 193

    Max, when I, as a good American patent attorney, read your posts I constantly come away thinking how differently patent law is at the EPO vs USPTO, mostly procedure-wise, but often substance-wise.

    It makes me wonder how all of the efforts of the last 20 years to acheive “harmonization” have been worth it.

  4. 192

    Inviting Body Punches,

    I too have been a party to this posting mechanism of Paul Cole’s (over at ipwatchdog.com). I typically do not post here because the “intellectual dishonesty” is rampant here and Gene generally keeps a better control on his site. That being said, I would hope for better from Paul, as his views are ussually considered, albeit sometimes too one-sided and incomplete.

  5. 191

    Paul, this has become a pattern in your posts: you disappear after somebody asks a question about your basic premise or assumption.

    The rest of the pattern is that you re-emerge at a later date, and simply repeat your original premise, complete with original assumptions, unchanged.

    This is a time-honored strategy for dealing with criticism that you cannot, or will not, answer honestly. It relies both on you having greater stamina than your questioner, and on the relative unimportance of the issue upon which you are pontificating, the next time it comes around.

    I will take your silence on this matter as confirmation that your original premise is unsupportable, as it is unsupported when given the opportunity…and I await its inevitable resurrection, hopefully in this forum.

  6. 190

    Come on, Phelps, a guy who loves stats as much as Tu knows that a sample size of 1 is more than enough.

  7. 189

    Phelps–

    This was a better post.

    IMHO it points nowhere other than to a registration system.

    Your basic premise is that holding individual examiners accountable for unforgivable errors or malfeasance is too complicated and costly to be reasonably implemented, and that in any case, it is either unfair or unjust, as the examiners aren’t given a reasonable opportunity to achieve competence in the time allowed.

    Your arguments do reflect current realities, IMHO, that aren’t going to change any time soon. Meaningful management of public costs is no longer possible, and the only workable solution is to dismantle the public structure and shift the burden to the private sector to bear the costs.

    IMHO current examination before the PTO doesn’t get you squat. Examiners are expert in neither patent and administrative law and regulation, nor in technology–there are other parties in the private sector infinitely better-situated to carry out their functions.

    The PTO would still exist, but not to examine for patentability…there could still be some formal conditions of registrability, but even those could possibly be automated to some extent.

  8. 188

    Paul–IMO, they are already doing (b), and have been for a long time–it is only rarely that we actually see an Office Action that even mentions the dependent claims (aside from listing the claim numbers as rejected). Sometimes the Office Action will address a particular limitation of the dependent claims that is important, but usually all the claims are effectively lumped together.

  9. 187

    You have just shown that soup is dead wrong (it’s far easier to get counts per rejections).

  10. 186

    “It takes just as much “non-time” to indiscriminately reject as it does to allow.”

    Are you joking? I can reject indiscriminately within a couple of hours for case after case after case. There is literally no way that sht would fly for more than a few cases if I tried on allowances. Sure, I might could fake a case here or there, but that wouldn’t last long.

  11. 185

    “Where you run into trouble as an examiner is when you adequately search and examine the case, find the best art, apply it to the claims, and suggest ways to amend around the best art. Then you spend all of the time allotted (often more), the case fairly rapidly goes to allowance, and the count stream dries up.”

    PREACH IT BRO.

  12. 184

    Since your proposal for punishing examiners requires an appeal, it also doesn’t deal with the situation where the Examiner is in error, but the applicant chooses to file an RCE, rather than appeal. Likewise, your proposal doesn’t deal with valid rejections doled out over time. Appeals can only deal with invalid rejections. And your proposal is less harsh than the current system in that it would, presumably, require the Board to find that the examiner’s rejection was “one sided,” whereas the current system requires the Ex to work for free just because he or she lost, whether the rejection was one-sided or not.

    I agree that some examiners sometimes play the type of games you have mentioned. But neither the current system, which is harsher that the one you propose, nor your system can remedy this problem.

    It seems to me that the only remedy for the first kind of game you listed is to make appeal more attractive and accessible, so that applicants will feel more able to appeal “bad” actions. But this costs money and opens the door to frivolous appeals.

    And it seems to me that there is no good remedy for the second type of game, where the Examiner doles out valid rejections over time. There are times when examiners legitimately fail to find the best reference the first time around. And sometimes examiners fail to notice 112 issues the first time around. The Office’s system of requiring Ex’s to work for free should they raise new rejections not necessitated by an amendment is an attempt to encourage examiners to get it right the first time, but nobody is perfect, especially when they have 14 hours or so to learn new technology that took years to develop and compare that technology with ALL the technology that has come before.

    Thus, I think that room must be made for subsequent vaild actions based on new art. And if you are going to make such room, how are you going to distinguish between game playing and the legitimate use of new art and new rejections? Indeed, how can you be sure that the particular cases you think are cases of game playing really are cases of game playing?

    The POPA error is one that I had in mind. In addition, taking POPA membership as relevant is another. Examiner’s are protected, and bound, by the procedures enacted through collective bargaining with POPA whether they formally belong to POPA or not. Protection is tied to whether the exminer is a non-probationary employee or not. Non-probationary status occurs at 1 year. It used to occur at two years.

    As I recall, your paper also indicated that no counts were given for final actions. That’s not correct since .25 counts are given. Nothing in the paper seemed to hinge on this, so it’s minor.

    In addition, it seemed to me that your discussion on the different rates at which examiners do work appeared to fail to appreciate that the Office requires different work rates for different art units.

    And I think that the pay structure is outdated, for both examiners and attorneys. As I recall, the reference you cite dates to 2004.

    I think most of these were mentioned, or hinted to, above.

  13. 183

    It guess I’m sceptical that just because one ALJ believes he or she could easily determine which cases were one-sided that it actually is easy. The way these things usually go, standards of the usual kinds would have to be implemented. And a kind of case law would have to be developed. We don’t usually leave these kinds of decisions up to the pure discretion of the adjudicator.

  14. 182

    You wish to blame me for finding fault with your poor writing.

    Sorry, but that will not do.

  15. 181

    The situation that is not accounted for in your analysis is when examiners make a “bad” first non-action, and then make another “bad” second action that is final. This garners more counts by forcing the applicant to file an RCE. Either that or abandoning the application, which would also garner counts for the examiner.

    This is one of the main situations that I have encountered in practice. A 103 rejection based on reference 1 and reference 2. We would argue (successfully) that reference 2 does not apply based on a scientific argument. The examiner would then issue a FINAL office action based on reference 3, which would disclose basically the same information that reference 2 disclosed. Applicant is then forced to use the same argument that was made against reference 2. This looks like a valid rejection because it is based on new art.

    Second situation- an examiner has 4 different rejections (102, 103, 112 1st and 112 2nd). Instead of giving them all at once in one action, they get doled out one by one, non-final, then final, (RCE) then non-final then final. Each rejection is valid, however, the prosecution is prolonged.

    Both situations have happened to me countless times in private practice. Ideally, the actions should all be non-final (thus the second and subsequent actions would not garner counts), but this rarely happens. This is but a few of the games I believe an examiner plays to garner more counts.

    You also stated that my paper has “glaring” factual errors. The only one that I believe to be valid is the POPA one. That information was from an interview of two previous examiners. There are many limitations, with this study, but I believe I have pointed them out in my paper.

    Could you please elaborate on my “glaring” factual errors? I would like to correct them if they are valid.

  16. 180

    I note that I have talked with a well established PTO official. He was an ALJ for over 20 years. He had mentioned that it was easy to determine which cases were one sided (actually he said that most cases that come to the BPAI are fairly one sided). If this is correct, then it should be easy for the panel to determine if counts should be removed. He actually suggested that the applicant be refunded the cost of the appeal if the BPAI ruled in the applicant’s favor. I wouldn’t mind a system like that either.

  17. 179

    Oh I see, I’m to understand that I’m the one that told everybody up above that a requirement of the EPC is to identify to the EPO, and in your specification as filed, all that matter in the prior art universe of which you do not yet have any knowledge. You know, it is not me whose writings lack clarity but you who is over-eager to find fault with them and, in that over-eagerness, can’t be bothered to read them properly before leaping up to shout “muddled”.

  18. 178

    Did anybody except you ever suppose there was?

    Yes.

    You.

    See your posts on this thread (and the very point of my replies).

    You should choose your words more carefully. I am merely responding to your ill-worded post. If you wish to retract your statement and say what you mean in a more clear manner, please do so. But gathering from your response to a previous request for greater clarity, you do not seem so inclined.

  19. 177

    PDF, we need to distinguish what is required to be in the app as filed from what is required to go into the spec before the EPO will allow the case.

    As you know, it is the task of an EPO Examiner to object to every application that does not meet every single provision of the EPC and its Implementing Regulations. Typically, objections from PTO Examiners are overcome by amendment, right?

    Somebody like you who has been there and done that will surely know about the EPO case that was refused and went to the Board of Appeal because the Korean corporate applicant failed to disclose in the specification as filed its own relevant prior art. For the Examining Division, Applicant could not save the app from extinction by adding it in during prosecution.

    On that occasion, the Board of Appeal reversed, saving the app from failure. Given that the EPO doesn’t do Binding Precedent though, who knows what might happen next time up, in front of a differently constituted and less Applicant-friendly EPO Board of Appeal.

    But meanwhile, I do so much agree with you, that at the EPO you don’t have to include, in your initial drafted specification, those parts of the prior art universe of which you are still ignorant on your filing date, such as the art the EPO finds for you, after you file at the EPO. In your words:

    “…there is no requirement to include any such prior art that is not known to an applicant”

    Did anybody except you ever suppose there was?

    I’m enjoying this. Your turn again now.

  20. 175

    Been there, done that.

    The distinguishing word is “requires.”

    There, you fail. There is nothing you can provide that back up this “requirement” because the inclusion of prior art is not a hard and fast requirement. Analagous to the US law, there is no requirement for an applicant to perform a search of prior art and there is no requirement to include any such prior art that is not known to an applicant. An appllicant is not therefore required to state the difference and an applicant states what is new by the mere fact of submitting an application.

    42(1)(b): indicate the background art which, as far as is known to the applicant, can be regarded as useful to understand the invention

    “as far as is known… can be regarded as useful” = this quite simply is not a requirement that prior art MUST be included. There is no REQUIREMENT that an applicant MUST “set their invention in the context of the most relevant prior art.” Your naked assertion of this is just not an accurate statement of the law.

    As for your muddled interlocutor, your posts are often muddled and for that reason I typically ignore them. This post though caught my attention because like Paul Cole, you are making up law that twists what is actually required from what is otherwise helpful advice. I am amused when foreigners try to make up US law (or even foreign law). But dressing up helpful advice as law is just as incorrect as mistating law. By the way, your interlocutor is correct, as you do not make sense with your treatment of the word “obvious.” Perhaps you are bored or attempting to be provocative, but all that comes across is “muddle.”

  21. 174

    link to epo.org

    I’ll try. Get yourself to the EPC, for example on the EPO website, and discover there that the Implementing Regulations are deemed to be part of it. Go to Rule 42. Read. Then get back to me with any supplentary questions.

    Hope that is helpful for you (and also not too hard for my know-it-all muddled interlocutor above).

  22. 173

    The EPC requires inventors, in their specification, to set their invention in the context of the most relevant prior art.

    Cite please.

    Requires…?

  23. 172

    PhELPs: “as a member of a state bar, I am sympathetic to the weight of the responsibility born by attorneys.”

    Really? Unless you are in private practice and actually have to pay for insurance, or actually have to justify your actions to clients to either receive payment or retain them as clients, you have no actual understanding of the situation–only an abstract sense.

    Phelps: “unlike attorneys, examiners are already punished for bad work”

    That is laughable. Attorneys are not punished for bad work? Sorry, but that is a load of c.r.a.p. “Bad work” by an attorney is “punished” by malpractice suits, loss of clients, forfeiture of payment, and if there is a public interest involved, professional sanction.

    Phelps big issue appears to be with the fact that it is not the patent office that judges “bad work” by patent attorneys, but state boards and clients instead, thereby depriving the PTO of the power of which it is so desirous. I happen to agree that attorney conduct should be scrutinized more closely by the PTO–however, as a matter of efficiency, it is much easier for them to begin to look at examiner conduct.

    Phelps calls out Tu for “failing to explicate how the Office could implement a process for distinguishing between those judgments that are clear from those that aren’t”, yet proposes EXACTLY the same process for analyzing attorney conduct, using as his standard conduct that “anyone with any understanding of the subject matter would recognize as ineffectual in moving along prosecution”.

    What hypocrisy.

    Get real, Phelps.

  24. 171

    Such a statement is in effect a confession of pure ignorance of the state of the art of at least complex information processing, QA, QIP, and JIT concepts applied to methods of doing business.

  25. 170

    The EPC requires inventors, in their specification, to set their invention in the context of the most relevant prior art. That requirement makes sense, in any FtF jurisdiction, where patentability is judged as of the PTO filing date, AFTER the inventor has had a full opportunity to research the state of the art and AFTER she has completed the task of writing an enabling disclosure which can contrast what is perceived to be the invention against the background of the state of the art up to one day before the all-important PTO filing date.

    In contrast, in a First to Invent jurisdiction, the fiction has it that obviousness falls to be judged in relation to an act of conception, on some date long before the app is filed at the PTO.

    The concept of the “closest prior art” to an act of conception strikes me as an affront to logic. So I sympathise with those in a FtI jurisdiction who have difficulty coming to terms with the FtF notion of the “closest” prior art.

  26. 169

    I suppose that I need not respond since, by IBP’s own admission, he or she has not commented on any of the specifics of my post. However, as a member of a state bar, I am sympathetic to the weight of the responsibility born by attorneys. But, just to re-emphasize it, so the point is not lost, examiners are held accountable for “bad” Office actions. If an action is “bad,” the examiner must do the next action “for free,” without counts. Thus, examiners do have an incentive to produce good actions.

  27. 168

    no evidence in favour of unobviousness on the face of the documtnt

    There is no such fatal defect, because a patent application, in itself, is prima facia evidence of the inventor’s statement that he has an invention. It is all the positive evidence that is required.

    I think you are purposefully ignoring where the actual burden is. Remember, an applicant is deserving of a patent unless.

    In re Adams does not make the point you think it does. It did not create the legal need or switch the burden like your writing suggests.

    Mind, you, it may not be bad or imprudent to write as you suggest, but to make it sound like it is a legal requirement is a gross mistake in your position. You are not empowered to change the actual law to one that you wish were in place. It is your own wishful thinking that is being challenged.

  28. 166

    The question may be moot, but the answer to the larger question is not.

    😉

    Thanks Inviting Body Punches.

  29. 165

    Really,

    When you don’t realize that the “tard” is the examination, you will not, cannot, change.

    And it is precisely this lack of ability to change that makes registration so much more appealing.

    You keep on making my points for me 6. I ought to thank you, although I suspect that you have no clue that you are doing exactly that.

  30. 164

    Paul–

    How do you rationally justify your blanket prohibition on individual punishment?

  31. 163

    The answer is a registration system, where all those onerous tasks would only be undertaken where they were desired by an interested party, who would bear the vast majority of the associated financial burden.

    The gargantuan task then becomes parceled out into many fewer and much smaller manageable tasks.

    Does it then become a sport reserved for Kings?

    It already is, so the question is moot.

  32. 162

    Without commenting on any of the specifics in PhELPs’ post, I will comment on the general theme of attorney accountability.

    Hey PhELPs, we are already accountable to the PTO under the canons of ethics, not to mention to our state boards under similar canons and strict rules of practice, but also TO OUR CLIENTS WHICH IS WHY WE CARRY MALPRACTICE INSURANCE.

    You want to talk accountability? Let’s talk. Are any of the Secret Service public-sector hacks being held accountable? How many of the “punished” have taken early retirement?

    How many of those who have “been dismissed” from the Secret Service have ACTUALLY been re-assigned to different government units?

    Real accountability in the public sector? There is no such thing.

    How dearly I would love to be able hold civil servants, or public bodies, liable for damages arising from the same circumstances in which an attorney in private practice could be held liable by their clients.

    Don’t talk to me about accountability until you have some.

  33. 161

    There is no answer. If statutes that Congress draft are open to interpretation, and that’s a group of 535 possible initiators of a statute, try distilling what language means in 200k patents a year, and comparing it to prior language written by others. It’s a gargantuan task any way it’s done.

  34. 160

    “The CPC is going to be a huge cluster you know what. They’ve neglected the US classes in anticipation of the CPC, but it simply won’t work in many arts. For example software related classes are completely incompatible between the EU and US systems.”

    That’s because “software” isn’t an “art”. You’re just throwing together a cobbling of often functionally, but sometimes not, described artistic written works.

  35. 159

    “about the state art, especially software and business methods, and thus just rejects out of pure ignorance.”

    Considering that there is no “state of the art” regarding biz methods and softwarez then I suppose all examiners in those AU’s are always rejecting out of pure ignorance.

  36. 158

    “You are now worse off because you have wasted a cycle with nothing to show for it.”

    Not really. That only applies if you’re like yourself and draft like a tard.

  37. 157

    “I await something of substance, with a prescription for action, implementation, measurement, and accountability.”

    Which is what I thought about making a website specifically designed for the other day. But then I was like, meh. Let em wallow.

  38. 156

    I have said in an earlier posting: don’t punish individuals, change they system. That includes good training and realistic objectives.

    But I think that reasonableness is also called for. The decision in In re Klein when posted in this blog gained comments that the combination of references relied on could only have been put together without hindsight using the infinite improbability drive of the Heart of Gold spaceship and that you cannot combine a buffalo with a chicken to make buffalo wings. Objections cost applicants time and money quite disproportionate to the official fees within the USPTO, and good will and good sense on both sides are needed if the system is to work effectively.

  39. 155

    Others have already pointed out some of the glaring factual errors recited in the paper. But there there is one other that seems particularly relevant since the author seems to be working diligently to identify whether a problem exists in order to support instituting his proposed solution of punishing examiners.

    The author suggests that “another strategy to avoid bad rejections would be to punish examiners who engage in this behavior by removal of counts.” Specifically, the author suggests, “if an examiner rejection is reversed by the BPAI or is reviewed by quality control as issuing out a bad rejection, then counts for those actions would not be credited to the examiner.” The author concedes that “it is true that applicants may bring appeals based on uncertainty in the law,” but believes nevertheless that “those judgments that are clearly one sided for the applicant should result, at a minimum, in the loss of counts.” The author fails to explicate how the Office could implement a process for distinguishing between those judgments that are clear from those that aren’t. But the author asserts that “[a]pplication of punishments would dampen the effects of bad rejections by removing some of the incentives for issuing bad rejections.”

    Apparently, the author is unaware that, unlike attorneys, examiners are already punished for bad work. If an examiner has to re-open prosecution, either after a “bad” first action, or after losing an appeal, the Examiner must do the next action “for free,” without counts. The current process has roughtly the same net effect as the Author’s suggested loss of the previous counts. Except, that is, that examiners are, under the current system, also required to work for free even if they wrote a solid rejection that was undercut by factors outside their control, such as when an applicant swears behind a reference, or an applicant files data in an affidavit that overcomes a valid prima facie case.

    Thus, the author’s solution is already essentially in play. Perhaps, then, he need not be so concerned about whether his problem actually exists.

    On the other hand, since it takes two to tango, as they say, perhaps the author should look into whether the back log, and the low allowance rates are due, in substantial measure, to attorney fault (by the filing of amendments that anyone with any understanding of the subject matter would recognize as ineffectual in moving along prosection, for example). If so, perhaps attorneys ought to be punished by being required to repay their fees should they loose at the BPAI. Examiners are already on the hook this way. If attorneys are so sure of their position, maybe they should put their money, where their mouths are. After all, what’s good for the goose is good for the gander, no?

  40. 154

    For the importance of a description of new results in the specification or at least in the file wrapper see In re Adams (1966) where the Supremes refered to “the afterthought of an astute trial lawyer”. Unfortunately what many people think that they can get away with is based on wishful thinking.

    Not all results which are thought to be new will in fact turn out to be new. But failure to point out at least potentially new results in the specification is at least potentially a fatal defect because there is then no evidence in favour of unobviousness on the face of the documtnt. The lack of positive evidence is at least an indicator of lack of inventiveness. Based on this negative state of affairs, why should an examiner start with a presumption in favour of the patentee where the person drafting the specification had it in his/her power to provide such evidence and did not do so?

  41. 153

    It may be interesting because you are muddling obviousness.

    Obviousness is just not a different proposition as you seem to be implying. It is hard to tell what you think you are talking about, since you will not clarify. But if you insist on being confusing (and being confused), I will not stop you. Enjoy.

  42. 152

    Can I clarify? No, except to say that determining obviousness (theoretically at least) before the date of reduction to practice, before the writing of the enabling disclosure, as of the date of conception, is a rather different proposition than the exercise of determining obviousness as of a date posterior to that on which the spec was written. You tell me that nothing’s changed but i think something rather important has changed.

    I’m just drawing attention to how obviousness is handled objectively in a large First to File jurisdiction, Europe.

    The USA is now for the first time a FtF jurisdiction. How it will get to an objective treatment of obviousness under a FtF statute is interesting to me.

  43. 151

    Not following the rejection on obviousness here.

    Obviousness is a rejection extrinsic to the invention.

    You seem to have misplaced your inherency doctrine somewhere.

    Also, obviousness (in itself) has not changed under the move to AIA. All that has changed is what art can be considered or not (no swearing behind).

    Can you clarify?

  44. 150

    Paul,

    ALL such cases are presmed to have merit. That is the law.

    Further, it is not up to the applicant to spell out what is NEW over and above the state of the art. ALL that is required is for the applicant to describe what he believes his invention to be. You rely on a common but gross misconception that the applicant is well aware of the state of the art and writes his application in the form “this was old, and this is what I propose to be new.” Not only is this NOT the case, there is no legal basis for making it the case.

  45. 149

    So Paul do I see it right, that your TSR means Teaching/Suggestion/Result?

    My favourite is the blue squash ball example.

    Take as specimen claim: Squash ball, characterized in that it is blue. Is that subject matter obvious relative to prior art black and green balls? It might depend on whether the inventor told us in her spec that a human squash player can pick up the flight of a blue ball (against the white wall) faster than a black or gereen ball.

    With that statement in there, give her a patent.

    Without that statement, I think not.

    Subject matter same, prior art same, outcome different.

    Under the AIA, obviousness has to be thought through anew. For, under First to File, for the first time in the USA, the event of writing the spec is officially anterior to the date on which the validity of the claim has to be tested, not posterior any more. Isn’t that a relief?

  46. 148

    I entirely agree.

    But in the 19th centrry case of Loom Company v Higgins the Supreme Court where Justice Bradley said that if there is a new result not achieved before that is evidence of invention.

    So all the examiners need to to is to scan the description for new results/benefits/advantages/functions, identify the related claims and search those.

    There is no point in searching a claim for which no new result is stated because on the face of the document there is no evidence for the unobviousness of that claim.

    If the claim is supported by a new result, then an obviousness objection only arises if a convincing case can be made out under the TSR test. Some cases will fail that test but new result is a pretty good predictor. And if the case does end up in litigation, there will be identifiable and positive reasons why the patent was granted, and even if objections are made out on a more detailed record, the court will be aware that it is dealing with a case that, at least at the outset, appeared to have some merit.

  47. 147

    You may not be aware of this but the SCOTUS upheld business methods as statutory subject matter in a 5-4 decision in Bilski Vs Kappos.

    In addition to the Supreme Court expressly refusing to invalidate business methods, when specifically petitioned to do so, the United States Congress also chose not to eliminate business method patents or software as statutory subject matter in passing the new AIA.

    For any Examiner to withhold a patent to an applicant simply because the invention can be applied to processes for improving or conducting business or applied in a commercial manner is a gross violation of the law and an applicants rights. And IMHO is grounds for dismissal from the Patent Office.

  48. 146

    “With that said, do you think it is fair that an applicant will have to go through prolonged prosecution just because he is saddled with a junior examiner who can’t find a primary examiner willing to sign an allowance?”

    Of course it is not fair! And for many years I often attributed such acts to the incompetent, belligerent examiner that had simply has no clue about the state art, especially software and business methods, and thus just rejects out of pure ignorance.

    But your research has shed new light. And while I am certain there are a fair amount of low I.Q. and willfully obstinate examiners with a disregard for the law, I now understand that perhaps many are simply cogs caught up in a system that rewards them for such feckless behavior. After all, we are talking Government work here.

  49. 145

    In the yearly updates the USPTO publishes how many applications they receive as a whole during that year. I have the filing dates of the issued applications from the years 2001-2011 (July 15, 2011). I can then sort all of the patents that issued, that were filed from any specific year and then compare that against the PTO’s yearly report. However, this data is not available on a level where they break it down by art unit or even technology center.

  50. 144

    Actually, both are true.

    Actually it is not true. You need to look at a count per app, my friend, not a net count per any app.

    It takes just as much “non-time” to indiscriminately reject as it does to allow.

  51. 143

    Actually, both are true. You can get more counts by allowing after insufficient search and examination, simply because you can do more cases because you spent less time on them. You can also get more counts by sending out b.s. rejections for the exact same reason: you spent less time on the cases.

    Where you run into trouble as an examiner is when you adequately search and examine the case, find the best art, apply it to the claims, and suggest ways to amend around the best art. Then you spend all of the time allotted (often more), the case fairly rapidly goes to allowance, and the count stream dries up.

    You’re right about the Doll/Dudas hangover. It will take years for the reject bias to be extinguished (if ever).

  52. 142

    I think he was basing that off of remarks in earlier papers. Obviously the average salary has increased over time, but the point was that you can make more as an agent or attorney than you can as an examiner.

  53. 141

    Both of these are factually inaccurate. An allowance strategy is what can get you more counts than a rejection strategy

    Not true. While the RCE gravy train has been curtiled, you still can get more counts out of a single app by rejecting. Also, as far as I am aware, the “quality review” for allowances (and letting rejections slip by) is still in place.

    The point of the matter with probationary still applies, given the preceeding immense turnover rates as a secondary verification. The ability to join POPA is itself a bit of a red herring.

    Like it or not, the Examiner “X” is merely a symptom of those (cough Doll cough) who view quality as Reject-Reject-Reject (witness the exuberance over the rejection cliff graph and the equating of that graph with improved quality under the Duffas regime. While Kappos has come in and attempted to right the ship (“Quality does not equal rejct“), the subversive elements are still entrenched in some mid and upper levels of the Office.

  54. 140

    The CPC is going to be a huge cluster you know what. They’ve neglected the US classes in anticipation of the CPC, but it simply won’t work in many arts. For example software related classes are completely incompatible between the EU and US systems.

  55. 139

    Continued fact check of article:

    “Once examiners develop the skills necessary to become proficient examiners, many leave to enter the private sector making two to three times as much as the PTO pays. The average salary for an examiner is $60,000 in contrast to $150,000 for new law school graduates specializing in patent practice.”.

    The average examiner comes in as a GS-7, step 10, which pays 67,589. The average examiner with 2-3 years of experience is a GS-11, step 4 ($76,888), GS-12, step 2 ($82,836), or GS-13, step 1 ($95,326). Not sure where he’s getting his pay numbers, but it’s not possible for the average salary to be $60,000.

  56. 138

    Fact check on the actual article:

    “These examiners may be rejecting applications as a default because (1) a rejection strategy can artificially increase the measurement used to assess examiner productivity (“counts”) and (2) junior examiners may not wish to jeopardize their ability to join the Patent Office Professional Association (POPA)5 by issuing “low quality” patents within the first two years…Once an examiner becomes a member of POPA, it becomes more difficult to discharge the examiner.”

    Both of these are factually inaccurate. An allowance strategy is what can get you more counts than a rejection strategy, especially given the new count system with declining counts for each round of prosecution. Also, the POPA thing is wrong on several levels. The ability to join POPA is based on you paying the fee, not on rejection rate or tenure.

    Moreover, the difficulty in firing an examiner has to do with whether they’re a probationary employee, NOT on their POPA membership. The probationary period use to be 2 years, but is now 1 year after the President changed it.

    Regarding Examiner “X”, the 0% allowance sign and the mentality it represents are completely unprofessional and the examiner should be outed (if management hasn’t already read this blog and looked into it).

  57. 137

    I believe that someone wrote that Applicant is able to write valid claims from the outset, so apparently we should see few proper rejections of the independent claim (if you believe Applicant knows their field and their own invention’s proper place in the field). And then, Applicant won’t be picking a dependent claim at random to bring up into the independent claim after rejection, but will use their (now enhanced) knowledge of their own field to write a new, valid independent claim. Or so the theory goes…

    Hogwash on so many levels.

  58. 136

    I believe that someone wrote that Applicant is able to write valid claims from the outset, so apparently we should see few proper rejections of the independent claim (if you believe Applicant knows their field and their own invention’s proper place in the field). And then, Applicant won’t be picking a dependent claim at random to bring up into the independent claim after rejection, but will use their (now enhanced) knowledge of their own field to write a new, valid independent claim. Or so the theory goes…

  59. 135

    Even if the program is not still in place, the induced mindset is.

    Witness: 6.

    It just follows…

  60. 134

    I haven’t read all the comments, so I may be redundant, but …

    Under the previous Director, there was a “QC” progrom to reduce “false positive” allowances. If a junior Examiner were to be found to allow an unpatentable claim, there were severe consequences. The Examiner’s were afraid to present their allowable cases, so they rejected everything.

    Could this scenario still be in place?

  61. 133

    Based on my own detailed study of millions of patent applications, I believe it is misleading to conclude so casually that examiners are so clearly responsible for low allowance rates. Such a conclusion ignores the applicant’s role in the process. Let me explain. My data shows that the percentage of applications with an RCE in low allowance rate art units are most often remarkably LOW. The percentage of applications with two or more RCE’s is usually extremely low. Still further, the percentage of the low allowance cases with an appeal is also generally very low.

    If your primary explanation of uneven / inconsistent application of patentability rules” were accurate, you would certainly not expect the RCE and appeal rates to be very low. In other words, my data shows applicants in low allowance art units giving up sooner rather than later. It is hard to blame the examiner for a low allowance rate when a huge number of applicant’s are giving up before RCE.

    Here is my alternative theory on these low allowance art units…based on my own study of the data. I have looked carefully at the amount of time between filing and the first office action. I have also looked at the amount of time between RCE and the next substantive office action. I have also looked at the amount of time between the first office action and abandonment/allowance. The most shocking of all of these stats was the time from RCE to the next substantive office action. In some instances, this delay is averaging over TWO FULL YEARS.

    The numbers make it fairly clear to me that a low allowance rate often has more to do with an a frustrated applicant giving up because it has taken to long to get a patent…than with an examiner’s “uneven / inconsistent application of patentability rules.”

    The massive backlog of unexamined patent applications and unexamined RCE’s is NOT spread out evenly over the whole patent office. Instead, there is a massive clustering of unexamined applications/RCE’s in certain specific areas of the patent office (like art unit 3689). These areas are where the low allowance rates are most often found. Coincidence? I think not.

    But do not take my word for it. We created an online service that enables you to see all of our data. Pick any examiner or art unit and enjoy reviewing our corresponding statistics. We offer free trial access (just click free trial button on main page). The site is: http://www.patentcore.com

  62. 132

    “The information is readily available.”

    No information about what patents might actually be in effect and/or even a reasonable approximation of their valid scope. Which is what you seem uncapable of understanding. You want to provide 0 information about that key factor. That’s the big reason we did away with registration in the past and you’re rearing and ready for a comeback!

    “A patent that is not new disclosure gets overturned.”

    AFTER you pay to have it pop up? Id iot. Je sus Ch rist you are du m sir.

    “There is no such thing as disclosure”

    I make disclosures all the time you ta rd, most more valuable than any disclosure you’ll ever personally have the opportunity to work on in your life. And I literally cannot get a patent. Obviously my disclosures might not be patent worthy either, but under your system that would hardly matter, I could just mob them with apps and claim whatever the f I felt like.

    Listen I’m done talking to your du m a rse. If you wanna keep on, feel free.

    “The patent office is NOT in the inNOvation business.”

    Just fyi, it literally is when I say it is.

  63. 131

    One of the inevitable issues is the fuzziness of the current standard of obviousness. The Fed. Cir. tried to make it predictable or at least rationalizable with the TSM standard. But then the Supremes said “NO! It must be fuzzy!”

  64. 130

    Same here….Goes something like this:
    Here you go. Enjoy!!! Talk about Luck/Unluck of the draw…

  65. 129

    I don’t think that it will either be faster or that it would reduce the backlog.

    A perfect example is the mindset of 6 who is on the anti-patent inNOvation club. If you only examine the independent claim and that claim is rejected, you are back at square one for the dependent claim which rises up to take the independent claim’s place. You are now worse off because you have wasted a cycle with nothing to show for it. And that’s even taking for argument’s sake that the rejection was proper and could not be overcome.

    The value of examination is simply not worth the cost.

  66. 127

    Yeah, but having the unfortunate side effect of bringing manufacturing to a stand still.

    This is completely unfounded conjecture. Nothing is brought to a standstill. The information is readily available.

    No, I most certainly do not. More patents is a horrendous thing. More disclosure is a good thing. A happy medium is a so so thing.

    This is completely incorrect. A patent that is not new disclosure gets overturned. There is no such thing as disclosure without the quid pro quo (beyond what disclosure you already have, thus outside of the constitutional mandate and not a part of our discussion). The fact that you think patents are a horrendous thing I dictate a very unhealthy bias to any patent system. The patent office is NOT in the inNOvation business.

  67. 126

    Lulz that might be the case in some AU’s but I’ve seen docketing happen in my AU realtime. It goes something like this: Spe pulls up big ol spe docket with cases to hand out, grabs a big ol’ random group and away they go to the examiner’s docket.

  68. 124

    “Search fee is not the same as the fee for an RCE. It is just a search fee. Quite a bit less. ”

    So then you’d like a discount?

    “Further, it is not an RCE and is not intended to disrupt prosecution the way Herr Direktor has done with placing RCEs at the end of the line.”

    So then you’d like to not have to file an RCE to slip in your AF amend?

    I’m just not seeing when you’re proposing you’d pay this extra search fee and have the extra search done as opposed to when you’d do it now with an RCE. I’m just not seeing it and I’m going to need to see it before I can convince Davi on it.

  69. 123

    “Having a gargantuan “fleet” of applications at the ready will no doubt invigorate even more knowledge and sharing, thus fulfilling the constitutional mandate.”

    Yeah, but having the unfortunate side effect of bringing manufacturing to a stand still.

    “6, you do know that having MORE patents is a good thing, right?”

    No, I most certainly do not. More patents is a horrendous thing. More disclosure is a good thing. A happy medium is a so so thing.

  70. 122

    b) – yes get rid of the examination of the dependent claims. I bet applications would be examined faster and the backlog would be reduced.

  71. 121

    Isn’t Google supposed to provide file histories for most applications? Can you get the data this way? I did check on a few applications a few months ago, and although Google indicates they are continually crawling the PTO, several file histories had not been updated….they were about a year and a half old. Just thought it might be worth a check.

    Also, isn’t all of the data you require available for purchase from the PTO? I realize it might be expensive, to get file histories, but if you plan to offer this product commercially, it might be a good v2 update.

  72. 119

    Yeah…I’ve actually requested the patent application information via FOIA about 5 times now. Each time I get rejected. It is extremely frustrating, since this data is essential to clear up the “denominator” problems.

  73. 118

    Search fee is not the same as the fee for an RCE.  It is just a search fee.  Quite a bit less.  Further, it is not an RCE and is not intended to disrupt prosecution the way Herr Direktor has done with placing RCEs at the end of the line.

  74. 117

    what do you think it’ll cost?

    100k. TOPS (I wasn’t going to stress this, but since you are being a pr1ck, even that is way too high).

    You are doing it wrong otherwise.

    Or maybe your resistance comes from the clear vested interest you have in maintaining such ridiculous work.

  75. 116

    I wonder if you could get the PTO to give you the full data under the Freedom of Information Act? If you just got the number of abandonments for each examiner during the time period, you could use (allowances / [allowances + abandonments]) as a measure of where the examiner is on the allow/reject spectrum and it would inherently normalize for expectancy, grade level, and production/overtime.

    Thanks for writing this paper, it is very enlightening.

  76. 115

    Oh I hope you’re correct about that. I’ve never understood why, after turning back a 102 rejection without claim amendments on a first action, and then after turning back a 103 rejection without claim amendments on a second non-final action, I then have to face a third non-final action with the same 103 rejection just with a different secondary reference. If an examiner can’t, in two attempts, make a valid rejection, it seems a waste of resources to keep flogging it. I seem to prosecute a few cases per year that follow this pattern. Clients should be happy that their claims withstood a withering assault of irrelevant prior art, but they just see the cost of their patent filings double or triple for no apparent good reason.

  77. 114

    So you’ve never done a landscape, AND you can’t read?

    I said NOW they cost in the neighborhood of 100k. IF we also have to determine validity of 100’s or thousands of unexamined publications, what do you think it’ll cost?

    Are you still in school, or are you an techdirt engineer with no patent experience?

  78. 113

    1. You are right, it is not fair to lump all examiners into the same bin because examiners get different times to review applications. That is why my analysis is broken apart by TCs. I could have broken it down even further and compared examiners within each art unit, but that would have made the graphs overwhelming. The bottom line is that I had to make a comparison, and I chose to do it by TC. I don’t think that was an unfair comparison.

    2. Again, I agree with you that the grade level of the examiner makes a big difference. There were a lot of things that I would have loved to code for. Unfortunately, this information was not readily available. (not only would you have to track each examiner, but know what GS level they were during each specific year). I would love to get this information as well as if the examiner was in a “transitionary” period moving from secondary to primary.

    3. I have tried to control for this by removing examiners who allow only one patent per year. However, it is difficult again to determine if the examiner was canned, or simply not issuing patents.

  79. 112

    If you are charging at the level of 750K for a landscape project, you are doing something seriously wrong.

    No wonder you are so biased against the pure reasonableness of a registration system.

  80. 111

    Ability to access (and process) information at more than a factor of 100.

    That one alone crushes you.

  81. 110

    “Whosoever holds this hammer, if he be worthy, shall possess the power of Thor.”

    The problem, as alluded to above by IBP, is not who is running the show, but rather, how the show is being run.

    “Worthy,” as a focus, is a required attribute.

    Privatization alone, is meaningless. What is needed is to actually break down the process itself and re-engineer it. The “widget-as-one-size-fits-all” model is at the very core of the problem that the Office faces, both in regards to the object of examination and to the management of those performing the examination.

    What is needed is a concerted large scale effort to revitalize the process. Instead, the US has spent its capital on the AIA.

  82. 109

    sorry for the double commenting, because of the Internet problem, not responding fast enough that I think the first comment failed. Delete one of them, administrator, thanks!

  83. 108

    Some of my points(pls correct me if not appropriate):
    1: patent system should be tough enough to applicants. like some commenters have said, no amendments, eg, ’cause we should believe that players in this patent game have the ability to adapt to changes and the requirements!
    2: patent applicantions should only protect what the inventor/applicant REALLY invents! but look at patent claims now, what do they really mean to an ordinary person? so fuzzy! so un-understandable!
    3: Patent should be fostering society development, not hindering!

  84. 107

    Some of my points(pls correct me if not appropriate):
    1: patent system should be tough enough to applicants. like some commenters have said, no amendments, eg, ’cause we should believe that players in this patent game have the ability to adapt to changes and the requirements!
    2: patent applicantions should only protect what the inventor/applicant REALLY invents! but look at patent claims now, what do they really mean to an ordinary person? so fuzzy! so un-understandable!
    3: Patent should be fostering society development, not hindering!

  85. 105

    As a former examiner I can say that new examiners don’t tend to get the “harder” cases but rather the “easier” cases in that there exists prior art readily available to prevent patenting. These cases are normally docketed to a newer examiner by a supervisory examiner who conducts a limited review of the material. As examiners gain more skill in searching and analysis functions, as well as with the art, they will be given cases where the supervisors conduct less review of the material before hand. That will likely mean that patenting rates increase. Add to this the problem that as you increase in GS level your production requirement increases giving you less time to focus on a case. This creates an incentive for those that do not have to justify their allowance to supervisory people (e.g. primaries) an incentive to find more patentable subject matter.

  86. 104

    So you’ve never actually worked on a landscape project? And right now, we’re doing them with applications assumed valid. So when we’re having to study the validity of the applications we find, the cost jumps from 100k to 750k.

    Everybody has 750k laying around to spend before even starting on a new product? What kind of fantasy world are you living in?

  87. 102

    I understand the issues related to the differences between examiners and their allowance rates, but cant you associate this with lmost egal processes. Many legal cases are decided by the luck or unluck of the draw of a judge. Isnt that the basis for venue shopping. It appears that when opinion or reasonableness is at issue, two people will always differ and thus a different opinion and a different prosecution path.

    I believe this study sheds light on a glaring issue, but its clearly an indication of varying “reasonableness” that each individual examiner presents.

    While 3 examiners will provide a buffer, as stated above this will provide increased costs of examining. There is no perfect system that works for everyone … its all a balancing act. If you feel that the examiner is completely unreasonable, Wouldnt it be more efficient to Appeal to a 3 panel judge

    It would be interesting to see what examiner you get and their allowance rates … that would give you an indication if you want to RCE or Appeal.

  88. 100

    This is a long thread–I don’t know what’s been said. Perhaps the less experienced, low-allowance examiners are demonstrating a basic human tendency. Perhaps these examiners, like many of us, believe that they are perceived as being more intelligent if they find something wrong. I can recall, faintly, a study of college students reviewing papers. My recollection is that the students believed they were perceived as being more intelligent if they were more critical when reviewing the papers. As examiners mature in their careers, and are more secure, perhaps they are less prone to this tendency.

    Just a thought. Take care.

  89. 99

    Yes and sometimes those SPEs rise through the ranks and lower the allowance rate of the entire Patent Office (cough, John Doll, cough). As one who remembers him as just another SPE in my group, back in the 1980’s…

  90. 98

    No, it’s not fair, but if it’s any consolation, allowance rates should start soaring soon. There is an initiative coming from the top to limit actions per disposal to 2 or 3. If SPE bonuses are based on achieving this goal, they will become very allowance friendly.

  91. 97

    Hi Shine, I just read your paper and it is very interesting.

    A couple of points I want to bring up:
    1. I don’t think it’s fair to lump all art units, or even different TCs, into one basket with respect to a judgement of high/low allowance examiners. The different art units are allotted different amounts of time to examine each case corresponding to the complexity of the subject matter they deal with. If an examiner in TC 3700 gets 12 hours, and an examiner in TC 2400 gets 36, then it should be expected that even if the two examiners are applying the patent laws and rules in exactly the same way, that the TC 3700 examiner should have three times the number of allowances as the TC 2400 examiner (just because he has to do three times as many cases). If 50 allowances per year is “high” for one art unit, it may not be high at all for an art unit in a different TC. This is to say nothing of the disparity in subject matter that they’re dealing with and its relevance to the ease of getting an allowed claim. I know you said you don’t have data for abandonments by individual examiners, but if you did, that would be a good way to normalize for the widely different amounts of time different art units are given.

    2. Grade level of the examiner makes a big difference on the number of allowances. A primary examiner has to do about twice the production of a GS-5 junior. All else being equal, the primary should have twice the allowances as the GS-5 junior. For example, assume that a GS-5 examiner is a 20 allowance per year examiner at that level and gets a promotion every year. The expected yearly allowances are then 20, 23, 27, 30, 33, 38, 42, and 45 based solely on the increased production requirements. A comparison of individual examiners in the same art unit and at the same grade level would tend to prove inconsistency based on number of allowances.

    3. I think your “low allowance” examiner set is biased against the PTO. Lots of examiners get canned before the two year mark and they are likely to have allowed more than one case before then. New examiners won’t have been on production for the entire year, and even when they were they weren’t at full production. Neither of those cases is indicative of examiners being too rejection oriented.

  92. 95

    If you’re going to ask him about that, you might as well also ask him whether Escalades, Navigators, and Suburbans count as “compact” vehicles.

  93. 94

    Um, Fish – They are doing this. The USPC is being abolished on Jan. 1, 2013 and the CPC is being instituted worldwide. It’ll look an awful lot like ECLA.

    Teaching 7,000 examiners to use it will be a much tougher magic trick…

  94. 93

    On "laid open," the benefit of a public disclosure is like prior invention today. It eliminated intervening art disclosing the same subject matter.

    Sent from iPhone

  95. 92

    You could ask D-Kap if 5 U.S.C. 2635.809 extends to vehicle registration. Pretty sure people aren’t commuting between here and Florida, Texas, New York, Illinois, and Ohio every day. There are an excessive number of Pennsylvania and North Carolina plates in the garages as well, which I suppose are doable drives, but more likely those are local peeps enjoying some nice insurance fraud. Not like PTO staff are expected to respect the lawlz or anything…

  96. 90

    the fleet’s size will be gargantuan

    People keep onmaking my points for me.

    Having a gargantuan “fleet” of applications at the ready will no doubt invigorate even more knowledge and sharing, thus fulfilling the constitutional mandate.

    6, you do know that having MORE patents is a good thing, right? That’s why we have it in the constitution in the first place.

  97. 89

    With respect, to effect that goal would be to simply state the obvious.

    Unless the study is able to identify the effect of specific problems with sufficient particularity to recommend solutions that can be both effected and the results of which can be measured meaningfully, well then…

    I await something of substance, with a prescription for action, implementation, measurement, and accountability.

    Until then, this seems very much like pushing on a rope.

  98. 88

    My comments on the paper are as follows:
    1) The author is to be commended for the important and valuable work to characterize examiner-specific trends. It sheds important light on PTO’s work patterns and dynamics.

    2) The paper seems to conflate results of other authors calculating allowance rate and those calculating grant rates. The author also switches between these two terms as if they are equivalent. The author, on the other hand, is on the right track by attempting to estimate the grant rate. Allowance rates that the PTO publishes is not the probability that an application will issue as a patent – it is the fraction of all disposals in a given year that were allowances. This is a rather biased and manipulated metric, particularly by the way the PTO calculates RCE disposals. See the detailed explanation in Figures 1 and 4 in “Patent Reforms Must Focus on the U.S. Patent Office,” Medical Innovation & Business Journal, Vol. 2, No. 2, pp. 77-87 (Summer 2010). link to bitly.com.

    3) Importantly, if grant rates are properly calculated, there is no need to “correct” for RCEs because they do not have a separate serial number and tracing the application to issue permits estimating a lower bound of the grant rate, which becomes very tight for early filing dates. It is therefore unclear why the author steered away from an accurate determination of grant rate of earlier issued patents by singling out for analysis patents issued after a given date rather than selecting patent applications that were filed after a given date. The latter approach would have produced none of the downward bias in the early years shown in the Table on pages 14-15, or the artifact attenuation of grant rates in the short pendency part of the axis shown in Figures 2 and 3. The selection by application date can simply be done on the USPTO online patent search site by the Boolean text “APT/1 AND APD/1/1/1998->1/1/2011” and collecting all such issued patents. This would help produce more meaningful grant rate results.

    4) The author presents grant rate data that he characterizes as “uncorrected” for Continuations, CIPs and Divisions. While not making the “correction,” he nevertheless appears to accept a false notion propagated by other authors who through a fundamental misapprehension of the nature and types of these patent applications, believe these to be repeated attempts at the same claims; that a “correction” in their calculated grant rates is necesary. However, no such “correction” is necessary because unlike RCEs, the inventions claimed in such applications are essentially distinct from that in the parent cases. “Corrections” of this type resulted in calculated grant rates that exceed 100%. I discuss this in details in my paper “Bad Science in Search of ‘Bad’ Patents”, Federal Circuit Bar Journal, Vol. 17, pp. 1-30 (2007) available at link to bitly.com.

    5) By unsubstantiated inferences, the paper seems to propagate a fundamental misunderstanding of the purposes and uses of Continuations – holding that the purpose is merely “to keep applications alive to increase the likelihood of allowance.” The paper concludes: “Thus, it stands to reason that those in the software and computer industries focus on ‘weeding out bad patents’ while those who work in the biotech and chemical industries focus on “allowing good patents.” p. 17. No evidence to support this conclusion that appears to originate in the patent-bashing folklore is provided. The fact is that all industries legitimately use continuations to appropriate equivalent returns from their inventions. The higher Continuation usage in some industries are easily explained by basic product lifecycle differences as I elaborate with some empirical data in my paper on Continuations at link to bitly.com .

    6) The paper uses the term “Art Unit” incorrectly to mean Technology Group, or Technology Center. This is confusing and leads to outright errors. For example, the paper states that “art unit 2100 seems to have disbanded in 2007.” 2100 is a Technology Group, which is (fortunately) still operational, with many art units. See art units 21XX listed at link to uspto.gov .

    Sorry for the long note. I hope it helps the discussion here and the author.

    Ron

  99. 87

    Thus, no one looking at a published application would have any idea of the scope of patent claims which may eventually be allowed.

    How would right-to-use studies be conducted?

  100. 85

    A very low cost fleet of submarines! Yay! How could congress not have approved this already?

    Even ignoring the fact that if the cost plummets then the fleet’s size will be gargantuan compared to the small amount of apps we have today how could congress not know this is the way to go?

  101. 83

    “With that said, do you think it is fair that an applicant will have to go through prolonged prosecution just because he is saddled with a junior examiner who can’t find a primary examiner willing to sign an allowance? ”

    Was it “fair” to the hunter gatherer that got eaten by a lion that a lion was nearby when he went out hunting as opposed to when his brother went out hunting the day before? Of course it isn’t “fair” it is reality, it is the way things are.

    If the point of your paper is to encourage reform of this broken down old 18th century institution we call the PTO then by all means, encourage away.

  102. 81

    “examiners MIGHT be applying the patentability rules differently within art units.”

    Of course they will be. Any two examiners apply the patentability rules differently.

    As if you’re magically going to get all examiners to examine all alike in a given AU. A more absurd idea I’ve barely ever heard.

    Sure, they’ll all be similar in a given AU, but they’ll never be the same.

  103. 80

    Alright Ned, I’ll be happy to tell him that I’ve heard from at least a few attorneys saying they’d like the deletion of rule 56. However, I’m afraid I cannot support the summary allowance of all remaining 1995 cases.

    I will tell him about the suggestion to pay for a new search if you can tell me how that would differ from the current ability to submit an RCE.

    I could tell him about the last thing you’re proposing if I understood how that would help anyone but I’m afraid I do not see how that would help someone. Getting your stuff published earlier isn’t a “good” thing is it? I mean, if you want to publish your own prov then by all means, post it to PO.

  104. 78

    What TC are you in? In my art unit non-continuing new cases are assigned strictly based on the last two digits of the serial number. No cherry-picking here. In my experience it isn’t trivial to determine which cases are “easy” before actually digging in. I’ve had some ugly cases with only a handful claims, and other cases with dozens of claims that turned out to be unusually straightforward.

  105. 77

    How do you plan to account for the “KSR” effect in that divergent fields, as long as they touch upon a solution, are fair game?

    Innovation just doesn’t like to stay “in the lines.”

  106. 76

    The Patent Office needs to improve its classification system, especially in the software, business method and related arts. This will make searching easier and more accurate for the examiners (and for inventors).

    It also will help the public better locate relevant patents, for example when conducting right-to-use searches.

  107. 75

    I agree, the patentability rules (specifically 35 USC 103) are definitely being applied differently in different art units. But with something as subjective as what is or is not obvious, I don’t know how this can be helped. But I admit that the patent office is sorely in need of consistency.

  108. 74

    No. Better to do it 100 different times and obtain 100 different ways.

    You seem to want to ignore the fact that these thigns will still be there. Patent landscape projects also have the benefit of actually aligning with the constitutional directive of promoting (in its proper understanding of spreading the word).

    You are helping me make my points. Thank you.

  109. 73

    Burden x incidence = very low cost (and one that would actually be borne by the interested parties).

  110. 72

    Were you around when that was tried? I am sure that there are plenty of other factors that make your comparison more than just a little suspect.

  111. 71

    Get rid of amendments.

    Maybe people who care will actually become better writers.

    Counter-intuitive, but worth thinking about.

  112. 70

    Paul- the whole point of this study is to point out that there might be something wrong with the system. No one is going to bother to try to change anything if they believe that the current system is working fine.

    I think this study just shows that we might want to consider some of your reform suggestions since the system MIGHT not be working optimally.

    Additionally, I note that I have not criticized any particular individual examiner. I really have tried to be fair and objective in this analysis.

  113. 69

    Paul, reexaminations are effectively conducted by 3 examiners, all of whom have to agree. That should be the norm in all ex parte cases, IMHO. But it is expensive and would require that the examination fees be hiked to the 20k range, the same as reexaminations. This is why the original examinations should be deferred, as I suggested above, until the applicant/patentee really needs it. But even if he defers, he still should have a patent can be enforced provided that every claim in it is valid.

  114. 68

    By the way- if you wanted to avoid being included into this analysis the magic number would be 1 per year…not one per bi-week.

    If you allowed one case per bi-week, you would fall within the norms of many examiners.

  115. 67

    I agree with you completely. In fact, I think this behavior is indicative of the count system more than anything else. It is interesting to see that the trends are consistent through all technology groups. Accordingly, I think that you hit the nail on the head as far as the rationale for what we are seeing.

    With that said, do you think it is fair that an applicant will have to go through prolonged prosecution just because he is saddled with a junior examiner who can’t find a primary examiner willing to sign an allowance?

  116. 66

    I was very careful to say that there was no “magic number” and that it is even possible that things are working fine. This study is very limited in scope. That is, it only tells you that there are some examiners who are outliers. But even this may be fine.

    However, I believe that because the outliers are significantly effecting the allowance rates, examiners MIGHT be applying the patentability rules differently within art units.

    I will have to do a follow up (which I am currently doing) to (1) determine if the low allowance rate examiners are giving “bad rejection” and (2) if the high allowance rate examiners are allowing things too easily. Of course this follow up study cannot be definitive, because I am not an expert in all fields. Thus, I am focusing on the art units that I am most familiar with and trying to use control groups to get a better idea of what is going on.

  117. 65

    6, tell them you have been hearing from a lot of patent attorneys who would like

    1) the deletion of rule 56;
    2) the option to pay for another search if the examiner says that a new search is required; and
    3) the summary allowance of all remaining applications that have a filing date prior to June 8, 1995.

    He should also consider whether he can by rule effectively lay open (and thereby publish) provisional applications at the request of applicants (after they have been cleared for foreign filing) so that applicant my easily and effectively take advantage of the new grace period provisions regarding prior public disclosures.

  118. 64

    I accounted for this. I use the “working” examiner as the examiner who allowed the case. That is, if there is both a primary and a secondary working on the case, I only count the secondary as the “working examiner.” Thus, I don’t double count the patent with both the primary and secondary.

  119. 63

    Just a couple comments:

    1. Primary examiners have much higher production requirements than junior examiners. Since they have to do more cases, they tend to allow more cases. As such, a part of the difference in allowance rate can be explained by the higher production requirements of primary examiners.

    2. Junior examiners must get any allowance approved by a primary examiner. Sometimes it is faster to send out a rejection than to find someone that is to find a primary examiner willing to sign an allowance. Primary examiners tend to be more hesitant to sign off on a junior examiner’s case because they did not perform the search themselves.

  120. 62

    Also, at the low allowance side, it takes about a year for allowances to start to issue for an examiner. For the results to be most accurate, it seems like examiners with less than a year at the PTO should be excluded from the dataset (and probably less than two years if you calculate results based on calendar year).

  121. 59

    So this article is telling me that if I’m allowing two or more cases per bi-week, I’m allowing too much? I guess the magic number is 1 per bi-week?

  122. 58

    Regarding the high allowance examiners, I don’t know if your data accounts for this, but many primaries sign allowances for junior examiners. Thus their high allowance numbers are going to be in part due to other examiners’ cases that they signed. You’ll only get an accurate picture of those examiners if you exclude all of their alowances that also list a junior examiner.

  123. 57

    I think the USPTO should set up a commission to consider what should be expected during examination, and what the outcome should be. For example:

    (a) How realistic is it to expect the USPTO to issue valid patents, as some judges in some decisions have indicated? Or should substantial and credible arguments in favor of patentability suffice?

    (b) Is the current practice of an examiner issuing a rejection on each claim cost-effective, or would it be better to focus on the main independent claims as the EPO does?

    (c) Would a move to three-member examining divisions as in the EPO help iron out individual idiosyncrasies (as in the zero examiner)?

    (d) Should the Manual of Patent Examining procedure be reconsidered in its entirety with the objective of shortening and simplifying it? It is at present a complicated and difficult document, and while it identifies a patent application that should be rejected, does it yet sufficiently explain a patent application that should be allowed?

    The task of patent examination is not helped by the state of US case law, best described as ever-changing especially in the area of unobviousness, and by the paper blizzard of not very relevant references which prosecution attorneys are obliged to submit. It is easy to point a finger at a zero allowance examiner, but it is more constructive to reform the system than to criticize individuals working within it.

  124. 55

    instead we should spend 3x as much on patent landscape projects because 90% of the published patents are invalid?

    No. Better to do it right once, the first time than do it differently 100 different times.

  125. 54

    or maybe the small sample size is indicative that we spend WAY WAY too much on examination.

  126. 53

    That sample set seems too small to be helpful. I know in our litigation, the issue tends to be art not cited in prosecution that leads to invalidity. So that doesn’t have a lot to do with the Examiner allowing something they shouldn’t have other than impugning the quality of ONE of hundreds of searches they do each year.

    You’re probably better off with BPAI statistics due to the substantive resolution and larger sample size.

  127. 51

    So we are kind of doing this in a slightly different way. I have coded for all patent litigations that ended in 2010. This was about 2000 litigations that corresponded to about 5000 patents. We then paired as many patents as we could to the examiner database. From this, we had about 2100 patents that were litigated. Then we culled it down to approximately 200 patents that went to final judgement with an opinion (most litigations settled). Of those 200 patents, 84 were found invalid. We are now doing a more detailed analysis of those 84 patents to determine how/why they were held invalid, and if it could have been prevented by “better” examination. (Additionally we are going to add to this to include all patent litigations that ended in 2009 so as to hopefully get our numbers up)….but again- stay tuned…that will be the next paper.

  128. 50

    This is clearly the biggest limitation with our dataset. I have tried to get this data from the PTO, with fairly high resistance. If you are able to get it from them, I would be happy to include the data.

  129. 48

    I think this is very useful, but you might consider looking at BPAI appeal statistics. An Examiner’s reversal rate on appeal is more of an “objective” measure, and would include both issued and unissued patents. Your methodology would not identify 0% approval Examiners, but would include 100% approval Examiners, so it could be skewed in that respect. However, if a low allowance Examiner always wins on appeal, a low allowance rate might make sense.

    Applications that are appealed may not be a representative sample set, but the data could be useful.

  130. 47

    The solution is that the examining corp should be eventually replaced with semantic artificial intelligence. Many of the examiners are technically illiterate and their searches are basically just word matching anyway (fyi: I am both a former US examiner and have gone through the examination process as an inventor so this is an informed opinion). At least with an A.I. solution the examination would be uniform and have some hope of improving as the technology improves.

  131. 46

    or you can apply the famous Ned Heller Business School of Logic and take it that medical methods are all now patent ineligible. That should be quite a few patents that the government won’t have to take eminent domain over in order to help the health care system.

  132. 45

    Yes deferred, perhaps permanently. There are a lot of patents that do not need to be examined as they just go on top of an already large pile of patents that are licensed as a pile. Why should these folks be required to pay for an examination that is otherwise of no use to them and is a big waste of time and money. What they want and need is a patent that describes and claims the invention, and that provides a basis of prior art against others. If and when they actually want to enforce the patent they can. They can request examination. The public can request examination. But they can still enforce if they first disclaim any overbroad claim so as to reduce the risk that any claim might be held invalid.

    On the other hand, if one knows the patent is important from the get go because it protects a vital business or product, for example, the opportunity to request examination should be present. However, I would expect and demand that the PTO spend a lot more time and effort on the examination process than they currently do, and end forever the whole concept of compact prosecution. Multiple searches might be required, for example, as the applicant introduces new claim limitations that were previously ingored. The examination process must be continuous and not be disrupted by political appointees who polish their resumes by deciding that RCEs to go to the end of the queue.

    We need people who think outside the box. The acedemics who conduct such research as exemplified here demonstrate just how grotesguely flawed and, I might add, wasteful the current system actually is. We need real reform. It is time for a change.

  133. 43

    “If anyone has concerns about the potential submarine, they can file the request to examine. ”

    So the burden falls to the public rather than the patent owner to make the submarine pop up! Even better! I don’t understand how congress hasn’t implemented this already!

  134. 42

    It would be useful to give the ratio of issued patents to published applications for each examiner and a frequency distribution of the number of examiners in each art unit at each ratio. Though admittedly less useful than knowing the exact number of applications examined by and patents granted by each examiner, it would still be useful. It would also be useful to compare the ratios of individual examiners to the overall ratio in the examiners’ art units. These statistics would tend to support or fail to support what many practitioners believe based on their experience: the examiner’s personality and inclination to allow or reject applications have a substantial effect on the outcome in many applications.

  135. 41

    Nope. They would be published for all to see. If anyone has concerns about the potential submarine, they can file the request to examine.

    Likely a large number of apps will never be examined allowing the PTO to focus on applications that someone, anyone care about.

  136. 40

    Dr Shine Tu said: “all of these results are limited by the “denominator” problem.”

    There clearly is some value in this research. But candidly, if you are trying to compare relative allowance rates among different examiners, I don’t see how you can get past the “denominator problem”.

    It’s like trying to determine the rate at which passing meteors actually strike the earth by counting strikes alone. Without knowing how many meteors didn’t strike, how is it possible to know anything about the rate? And if you categorized the strikes by type of meteor, for example, by chondrites, irons, stony etc., and noticed a difference in the raw numbers of each, without knowing how many of each type didn’t strike, how is it possible to draw any conclusions about the relative rates of strikes for each different type? Or to state that a passing chondrite (for example) has a higher rate of strinking than a passing iron?

  137. 39

    Yes. The data presented in the paper is actually stacked in favor of the PTO. I removed from the dataset any examiner who only issued one (and only one) patent in a year. Thus, I believe, the most junior examiners (who needed time to ramp up, and thus only issued one patent that year) are excluded. However, they are counted later on if they issued more than one patent the next year.

    I believe this solves a lot of other issues-(1) those examiners who were only briefly at the PTO, but left before issuing more than one patent, (2) those examiners who are primary examiners who mainly review the work of secondary examiners but issued one patent by themselves, (3) those examiners who have issued one patent, but have not issued any since, (4) those examiners hired in December who may have issued only one patent because of the ramp up time, and (5) examiners who came back (or just recently started) to the PTO and needed time to ramp up during their return year, and thus only issued one patent that year. In each of these cases, I have excluded the examiner from the dataset.

  138. 38

    Did you consider that junior examiners, especially examiners with less than 1-2 years experience, tend to spend the majority of their time on first office actions, and once in the office for a longer period of time have a more diverse docket of FOAs, responses, etc.?

  139. 37

    Thanks for the response! Any idea how much you’ll be selling the information for? Keep in mind, price it too high and someone will just write a script that does it for free 😀

  140. 35

    On a rough cut basis, I agree with you. But as many re-opens and overturning the examiners as I’ve seen, I think it would be a great v2 of your data to account for such information.

  141. 34

    nah, hopefully they’re in teh voodoo chemical arts where no one really enables or suppports anything because they don’t know how it works. As in prometheus, these are all just unpatentable laws of nature anyway.

  142. 33

    lol, and then by default, like now, clients go ahead with the exam fee at filing so that they gain an enforceable patent.

    So you’re really just advocating to add a deferred examination track to what we already have.

  143. 32

    He’s rather generous on the whole lunch break thing. You know it isn’t like you see me on here all day. As impossible as it might be for you to tell given your own personal schedule I actually do work a long day. And I mean “work” for the time which I’m referring to.

  144. 31

    I’ve got one. I’d like to hear how Mr. Kappos is dealing with examiners who spend all their on the Interwebz, rather than examining.

  145. 30

    LOL

    The US tried that for over 40 years. It was worse than having no patent system at all.

  146. 29

    Great comment. We are already doing this, and it should be out in a followup paper. Additionally, we are looking at the type of rejections issued by primary versus secondary examiners. We have already seen some very interesting trends. More to come 🙂

  147. 28

    You are correct, it is kind of unfair to attribute the allowance to an examiner who dragged his feet and lost at the BPAI.

    Unfortunately, it would be very difficult to do this type of filtering with the data that I currently have coded. I believe that this subset is a fairly small subset, and shouldn’t effect the overall conclusions.

  148. 27

    The PTO should do a study of examiners in the top and bottom 5% of allowance rates to see what can be learned and what changes can be made.

  149. 26

    If the prosecution was reopened, and a patent issued from that reopened case, then it would have been included in this dataset.

  150. 25

    OK. Yes, the patents that issued as a result of a BPAI decision was actually classified as an allowance by the examiner.

    I did not filter out to categorize those decisions as allowed by the BPAI.

  151. 24

    BTW guys, I’m having a supar secrit meeting with one Dividious Kappozinous in a week or so. If there are any super important matters you would like to have presented to his excellency then you may air them here.

  152. 23

    Also, I would kind of like for them to out this examiner, just so that we can verify that they in fact work in softwarelol or biz methods and we can all nod our heads in agreement that they’re doing a super good job.

  153. 18

    One rather large reason for this discrepancy is docketing practice. The primary examiner with over a decade of experience has much more say in what cases are added to their own docket. The junior examiner has almost no input, and often gets the cases that were misclassified, poorly worded, or with difficult subject matter or overly broad claims. The primary examiner may be expected to turn in twice as much work as a junior examiner in the same art unit, so sometimes it behooves them to take more “easy” cases.

    Additionally, the primary examiners may get cases that are extremely focused on one area of the art. Their prior art search may bring up mostly patents that they examined personally. Junior examiners rarely get to be so “dialed in,” and so may not be able to find the allowable subject matter as quickly as a primary. The comparison is more valid if you compare the same classes and subclasses of patents, not just the examiners in one tech center.

  154. 17

    Also, Dr. Tu – I forgot to mention. Thanks for putting this information together. We need better data and metrics throughout the gov’t (and a lot of private industry).

  155. 16

    Seems like it would be interesting to do a followup study to see how the allowance rates of individual examiners (generally at the margins – very high/low allowance rates) change as a matter of the individual examiner’s experience. Also, whether the tenure of the examiner at the PTO affects the allowance rate (e.g., do examiners employed only 2-3 years at the PTO have higher or lower rates in general than those employed for a longer term [of course at the time that the latter set were employed for the same length of time as the former set])

  156. 15

    Right. And if so, this wouldn’t be an accurate classification since the Examiner not only rejected the claims but dragged his feet all the way to the BPAI and lost.

  157. 14

    Duh, I tend to agree. However to answer the next post point, I would make patents all but unenforceable until examined. May I suggest that if a patent has not been examined that it cannot be enforced if any claim in it is invalid.

  158. 10

    Except for the fact that with going to Registration, you have just saved the innovation community BILLIONS that go to fund a federal agency that is incompetent at best or malfeasant as worst.

    It’s time for Registration.

    It’s a ridiculous suggestion to ignore the economic trade-off of a Registration system.

  159. 9

    I note that we are actually doing the analysis to determine if secondary examiner who work with low volume SPEs also have low volumes. Similarly, we are currently analyzing to determine if high volume SPEs also have high volume secondaries who work under them. It stands to reason to think that the behavior of the supervisor would affect the behavior of the junior examiner. We just have to see if the data bears out the theory.

  160. 8

    The patents that were issued as a result of a BPAI decision was included in the dataset. I did not flag these patents as special or put them into a special category.

    The only patents that were not included were, plant patents and design patents. I also did not remove patents that had claims that were canceled in reexamination.

  161. 7

    We are actually working to commercialize this product, and it should be available within the next year or so. The data is actually too big to post online. The summary of most of the data is present in my paper, which you can download from SSRN.

  162. 6

    And then you’re right back to square one of the problem. Except that no one knows which of the issued patents have any validity.

    What a ridiculous suggestion for the largest IP market in the world.

  163. 3

    The results are not surprising. There are a lot of great patent examiners, but the PTO still needs to implement better accountability and reliability. Still too many false positives and false negatives.

    I favor the proposal of allowing the PTO to not examine any application until someone, anyone (including anonymous requests) files a request with a fee to examine. If no one cares about the application, why waste limited resources? Otherwise, the application sits idle and will expire after 20 years. Applicant gets a filing date and nothing else until the next step is taken.

  164. 2

    In a case like this, I think it is entirely appropriate to out “Examiner X.” That Examiner is not doing their job, and practitioners need to know if their efforts (and their clients’ money) are going to be wasted by attempting to use reason.

    Failing that, access to your data (or at least your methods for generating it) would be a boon to the profession. Given that this information is public, there should be no ethical or professional misgivings in making that data open.

    Another interesting question I would be interested to hear the answer to is whether there are particular *supervisors* who have low allowance rates among their subordinate Examiners. Some of those low-allowance Examiners may simply be working under supervisors who themselves have an ax to grind.

  165. 1

    Exactly why we need a patent registration system rather the 6,000+ individual patent offices at the PTO. Let the practitioner determine where patentability lies. Then, anyone that wants to object can file a request for examination.

Comments are closed.