Guest Post by Professor Camilla Hrdy (Rutgers Law)
Can generative AI models like ChatGPT be “reverse engineered” in order to develop competing models? If so, will this activity be deemed legal reverse engineering or illegal trade secret misappropriation?
I have now written a few articles exploring this question, including Trade Secrecy Meets Generative AI and Keeping ChatGPT a Trade Secret While Selling It Too. But when I first asked this question a year and a half ago, I was getting responses purely in the negative. I asked a panel at a trade secret conference at Georgetown in 2023, “Can ChatGPT be reverse engineered?” Several members of the panel laughed. I would talk to AI experts, and the answer I got was along the lines of: “it’s not going to happen.”
But one of my students, Devin Owens at Akron Law, who has both a patent law and a computer science background, insisted to me that reverse engineering was possible using “model extraction attacks.” A model extraction attack entails imputing a massive number of queries into a “target AI model” and using the target’s responses to train a new model that mimics the original’s behavior and functionality. Devin wrote a student note about this, arguing that AI models are so vulnerable to extraction attacks that they can’t really be “owned” by anyone.
Now it seems clear that at least partial reverse engineering of generative AI models is indeed possible, and of increasing concern to AI developers.
A few weeks ago, OpenAI momentously alleged that DeepSeek (a competing Chinese AI model) was “inappropriately” developed from ChatGPT using “knowledge distillation.” In comparison to “model extraction attacks,” “knowledge distillation” is less frowned-upon and is typically motivated by efficiency and cost reduction, rather than exact replication. The difference between these two is nuanced. Knowledge distillation typically involves training a smaller ‘student’ model to mimic a larger ‘teacher’ model’s outputs on specific tasks for efficiency, while model extraction attacks are more aggressive attempts to replicate a model’s entire functionality through systematic querying designed to extract the underlying architecture and parameters. Some sources depict knowledge distillation as a form of “model extraction” or “model stealing.” Apparently OpenAI sees neither as an “appropriate” means of copying.
Assume it’s true that DeepSeek used the outputs of ChatGPT to develop a competing model. A major question is whether this is legal “reverse engineering,” or instead a violation of trade secret law.
This legal question is about to be tested. A new trade secret lawsuit was filed on Wednesday, Feb. 26, 2025, alleging that extracting data from a generative AI is misappropriation of trade secrets and breach of contract, among other things.
In OpenEvidence, Inc. v. Pathway Medical, Inc., OpenEvidence alleges that a Canadian company, Pathway Medical, used a so-called “prompt injection attack” to extract “trade secrets” from OpenEvidence’s generative AI model with the goal of developing a competing system.
OpenEvidence’s claims include: (1) acquisition of trade secrets in violation of the Defend Trade Secrets Act, (2) breach of contract due to violation a terms of use, (3) unauthorized access to a computer system in violation of the Computer Fraud and Abuse Act, and (4) circumvention of access control measures to obtain copyrighted content in violation of the Digital Millennium Copyright Act.
OpenEvidence (which is not affiliated with OpenAI) distributes a popular generative AI tool for use by medical professionals and patients. OpenEvidence’s large language model, similar to ChatGPT, appears to users as a chatbot which can be used to ask natural language questions about medical issues, like diagnoses, treatments, and medication side effects. (Complaint, 14-15). OpenEvidence is open to the general public for free, but general public users only get two questions per week. (Complaint, 16). Meanwhile, licensed medical professionals can get unlimited access, upon proving their license number and attesting, through a terms of use, to be a licensed medical professional. (Complaint, 16-17).
OpenEvidence’s trade secret misappropriation case is, at least initially, going to look very strong to the U.S. district court in Massachusetts where it was filed. Judge Myong J. Joun has been assigned to case. There will be two major issues. This court, and courts in future cases, will have to grapple with them.
- Are There Trade Secrets?
The first issue is what exactly a generative AI model’s trade secrets are? Trade secrets must be, inter alia, not “generally known” or “readily ascertainable,” derive “independent economic value” from secrecy, and be subject to “reasonable” secrecy precautions.
As I discussed in Keeping ChatGPT a Trade Secret While Selling Too, the main types of trade secrets that generative AI companies might be able to protect include algorithms, code, training data, and aspects of the models’ overall system architecture, including how it was trained, developed, implemented, and “fine-tuned.”
The primary trade secret identified in OpenEvidence’s Complaint is the system prompt code. This refers to the instructions given to a generative AI model in the order to guide its responses to users, customize the model, and enhance its performance. OpenEvidence alleges the system prompt code is OpenEvidence’s “crown jewel.” (Complaint, 10-11, 7).
Case law precedents from traditional software products are likely to apply to generative AIs. In software cases, courts have permitted distributors of software to assert trade secrecy in their source code, even after widespread public distribution, because the code is typically compiled into “object code,” making it difficult to “decompile,” and so still legally secret.
Still, a bunch of questions jumped out at me:
- How hard is it to figure out system prompt code, how many strategic prompts were required to learn this information, and is it perhaps “readily” ascertainable using “proper” means?
- Does system prompt code really derive economic value from remaining a secret? Is it so important to the functioning of the model that it impacts an economic advantage, actual or potential, due to staying secret?
- Can OpenEvidence prove it took “reasonable” secrecy measures, given that it released the model not only to the entire community of medical professionals, but also to the general public?
Contractual precautions will be important, both for establishing that the putative secret is “not readily ascertainable” and for establishing that plaintiff took “reasonable” secrecy precautions.
OpenEvidence does have a “Terms of Use.” It states, among other things, that users agree the “software” “contains proprietary and confidential information that is protected by applicable intellectual property and other laws.” OpenEvidence implies this means “Pathway owed and continues to owe confidentiality obligations to OpenEvidence…” (Complaint, 25).
But I don’t think this Terms of Use is sufficient to generate an express duty of confidentiality. It’s a mass-market, non-negotiated “contract of adhesion.” I’m not suggesting the Terms of Use are unenforceable for contract law purposes. But courts are unlikely to view this as placing end users under a duty of confidentiality for purposes of trade secret law. Instead, OpenEvidence will have to argue under an “improper means” theory.
- Did Defendant Use “Improper Means”?
The second issue is whether the actions defendant took to acquire the putative trade secrets qualify as “improper means.” As just noted, OpenEvidence makes loose allusions to a duty to maintain secrecy, but this is, in my opinion, not an “insider” case. The question will be whether defendant used “improper means” to access OpenEvidence trade secrets.
A few facts or fact questions jump out at me regarding “improper means”:
- The defendant allegedly accessed the unlimited version of the model, available only to medical professionals, by using a third party’s medical identification number to “impersonate” a licensed practitioner. (Complaint, 2, 18).
- The defendant allegedly used a “prompt injection attack” in order to mislead the OpenEvidence AI model into revealing sensitive data, including the OpenEvidence system prompt code. (Complaint, 20). Prompt injection attacks are portrayed as a type of “hacking.” The Eleventh Circuit’s recent holding in Compulife v. Newman, holding using a “bot” to “scrape” insurance quotes off a website, suggests this method might be viewed as inherently “improper.”
- The defendant, at least according to the Complaint’s allegations, pretty clearly violated the Terms of Use, and courts have indicated violating a terms of use makes it more likely acquisition of trade secrets was improper.
- Some key provisions include:
- “The Services are intended for physicians and other healthcare professionals. By using the Services, you represent and warrant that you have the right, authority, and capacity to agree to and abide by these Terms and that you are not prohibited from using the Services or any portion thereof.”
- “No part of the OpenEvidence Content may be reverse engineered or included in other software.”
- “[You agree you will not] attempt to access … any content contained therein through the use of any engine, software, tool, agent, device or mechanism (including scripts, bots, spiders, scraper, crawlers, data mining tools or the like) other than through software generally available through web browsers[.]”
This case raises some big picture questions about protecting generative AI models as trade secrets.
First, how hard is it, in fact, to reverse engineer generative AI’s? A few years ago, people apparently thought reverse engineering generative AIs would be hard if not impossible. But now it’s not so clear.
Second, how will courts view data extraction through strategic prompting in order to learn about how a particular model was developed? Will they see this as akin to buying a product on the open market and picking it apart, i.e., traditional legal reverse engineering? Or will they view this as acquisition by improper means, like hacking into a computer or flying a plane over an unfinished plant to see what’s inside?
Third, how much deference will courts give to contracts? Can attaching a terms of use that prohibits reverse engineering turn otherwise-lawful reverse engineering into acquisition by improper means? When construing the scope of liability under the Computer Fraud and Abuse Act (CFAA), the Supreme Court has indicated merely violating a contract isn’t necessarily a CFAA violation. But courts in some trade secret law cases have held that accessing information in knowing violation of a terms of use is by definition “improper.”