Defending Human Agency in the Age of Agentic AI

This article is adapted from a talk I gave to the Ethereum Foundation on March 31, 2026.

Love AI or hate it, people are definitely talking about it and—more importantly—to it. The artificial intelligence governance discussion largely concerns the provenance of its inputs (training data, scraping and explainability) and concerns about its outputs (bias and hallucinations). This focus has become dangerously incomplete.

In 2026, the primary vector of risk is no longer what an AI produces, but the data footprint generated as it acts: geolocation, API calls, payment events, timing correlations, search queries, execution paths and the actual prompts. All this data combines to form unprecedentedly clear and durable records of private human thought. Right now, those personal thought records are sitting in indexed files at AI companies. It is inevitable that those thoughts will be subpoenaed or sold.

The Invisible Surface Area of AI

Large language model prompts are not communications with others; they are conversations with ourselves. Unlike email, phone calls or texts, they lack social friction, audience awareness or self‑censorship. Prompt logs therefore approximate unguarded cognition.

Modern AI deployments increasingly rely on multistep agents that orchestrate external systems. Even locally hosted models depend on third‑party infrastructure for search, retrieval, inference, transactions and billing. Each interaction leaves a persistent log.

For litigators, this exhaust is evidentiary gold. For government, it enables narrative reconstruction of intent. For citizens and firms, it is a silent surveillance backdoor.

The Evidentiary Danger to the Sanctity of Our Thoughts

Although a new technology, the law will likely do little to protect these prompts and identities from compelled protection. U.S. law holds that the Fifth Amendment protects against compelled testimonial evidence but not the contents of voluntarily created documents. Fisher v. United States, 425 U.S. 391 (1976); United States v. Hubbell, 530 U.S. 27 (2000). The European Court of Human Rights derives a similar right against self‑incrimination from Article 6 ECHR but has a similar exception. See Saunders v. United Kingdom, App. No. 19187/91 (ECHR 1996); Jalloh v. Germany [GC], App. No. 54810/00 (ECHR 2006). 

LLM prompts expose a gap: the process of an LLM chat produces a transcript of our real-time thoughts, yet its capture is treated as a third‑party business record.

The fact that these chats can be seized by governments, courts or even sold to third parties causes a real threat to the idea of intellectual independence. Our LLM chats constitute unguarded thoughts, not just outside conversations, and they are fundamentally tied to our digital IDs and geolocations. For the first time in history, third parties can know exactly what we were thinking, when we were thinking it and where we were at the time. 

(RIght now, I am focusing on the prompts’ content. However, there is also a deep and abiding danger to leaving the behavioral patterns of AI users’ unencrypted. That cumulative pattern-weaving privacy danger will need to be discussed some other time.) 

Without structural protections, AI creates an repository of our deepest, unguarded human intent. It doesn’t have to be this way. AI can function without trampling our privacy by decoupling the chats from our identity. I propose the Zero‑Knowledge Firewall: a layered technical‑legal architecture designed to make such records technically infeasible.

From Trust‑Me Compliance to Technical Infeasibility

By their nature, AI services log, correlate and profile your chats in persistent records. In a broad privacy sense, this data will generally not be “sold.” However, history demonstrates that these types of assurances fail under subpoenas, breaches or regulatory compulsion. If your AI record, connecting your private chats to your identity, is kept at rest by the AI provider, it will get out. The government may acquire your prompts via warrant; they may even show up in divorce proceedings. And that does not even take into account the chance of a data breach. All of those possibilities are statistical certainty. Indeed, the warrants and subpoenas are already happening. 

Recently, in United States v. Heppner, Judge Rakoff of the Southern District of New York— addressing “a question of first impression nationwide”—ruled that written exchanges between a criminal defendant and generative AI platform Claude were not protected by attorney-client privilege or the work product doctrine even though the chats were intended to inform the defendant’s conversations with his attorney. Judge Rakoff’s decision indicated that these chats could never be privileged.

The scope of the chats and prompts can be extremely broad and not confined to one user. In The New York Times Company v. Microsoft Corporation, 23-cv-11195 (S.D.N.Y.), a federal court ordered the company to preserve all chat-logs, including deleted chats, on the theory that they might contain evidence of copyright infringement by the AI model. The court directed the company to “preserve and segregate all output log data that would otherwise be deleted moving forward.” These logs include chats deleted by the user, or flagged “temporary,” meaning that even if you are deleting your own chats and history, and have no knowledge of the litigation, third parties may still get your chats.

AI prompts can even be used to back into your identity by searching for people who discuss certain topics with the LLMs. In the last year, the United States Department of Homeland Security, obtained a search warrant compelling an AI service provider to locate prompts discussing certain child abuse topics and then to unmask and affirmatively identify any user. In that case, the government did not even have the prompts, merely the fact that an unidentified person had discussed with an undercover agent that the unidentified man was using an AI chatbot to discuss certain topics.  Although the context was overtly criminal, that outcome should raise a concern for anyone who is having conversations with an LLM that may be unpopular with a government.

The concept of these prompts and chat logs is still very new but the legal trends are clear: if the chats and identity are attributable to each other by the AI companies, they are subject to forced disclosure. The only durable privacy standard is technical infeasibility. Therefore, we should strive to use systems designed without sensitive correlations between prompts and identity. In these systems, no disclosure will happen because the data never exists.

The Zero‑Knowledge Firewall

For most of legal history, there has been a stable and morally intuitive line:

Thinking is not evidence. Doing is. 

Courts have built doctrines—mens rea, work product and the right to silence—on the premise that internal cognition is fundamentally different from externalized action. Similarly, legal doctrines such as the hearsay rule are built upon the premise that words said to another person have distinct and thorny reliability problems. The current use of AI models threatens to upend those distinctions by purporting to provide access to unguarded inner thoughts. At a recent Ethereum Foundation conference, I presented a talk about how a zero‑knowledge firewall could reestablish and re-center our protections where the courts and law enforcement are misclassifying thinking as behavior merely because it passes through a machine.

The proposed zer-knowledge firewall could accomplish this through three mutually reinforcing layers. 

1. Zero Knowledge Verification: Identity Without Attribution

Classic identity (ID) verification and zero-knowledge (ZK) verification represent two different paradigms in security. Classic verification methods (such as showing a driver's license or a bank's maintenance of “know your client” records) prioritize trust in a central authority to verify and hold sensitive data. Under this classic verification regime, a user provides a full document (e.g., ID card, passport) or sensitive data (e.g., SSN, birthday) to a bank or website. The service provider inspects, verifies and stores this personal ID data. However, all of that attribution and identity verification is not always necessary. For example, when we go to our house or car, all that we really need is the key to get in, not any proof of ownership. Therefore, although access to a home is obviously important, we can prove ownership via possession of the key. 

ZK verification allows a user (prover) to prove to another party (verifier) that a statement is true—such as “I have remaining access credits to the LLM,” “I am over 18,” or “I am a citizen”—without disclosing any specific, underlying information. Under this process, the user generates a cryptographic proof (a digital assertion) that their ID documents meet certain criteria. The verifier only receives a “true” or “false” result, not the document itself. While backed with robust mathematical proofs, this process is analogous to a circumstance with which many of us have experience. 

Imagine that you are trying to get into a bar or nightclub. There is a trusted doorman there who is diligently inspecting IDs. That doorman does not keep an image of the IDs or pass any of the details from the IDs to the bartender. Both of those options would be insecure, wasteful, embarrassing and silly. Rather, the doorman simply announces that this person is able to enter. The bartender trusts the verification process based upon a trusted third party’s ephemeral inspection of personal ID. At the risk of oversimplifying things, we can think of the ZK verification regime in similar terms. 

Under a ZK verification regime, only the necessary “predicate” (e.g., above 18, yes/no) is shared.  A digital wallet generates a ZK proof for a website that you are over 18. The website confirms the cryptographic proof, never knowing your name or exact birth date.

This abstraction of identity can be critical to preserving privacy and cognitive sovereignty. Even if an investigator obtains chat logs from an LLM, if they are using ZK verification, those logs can no longer answer: “Who was this user?”  Rather, they are narrowly designed to simply answer, “was this the same user?” The LLM does not care who we are, simply that the identity is the same. Governments and courts need specific attribution. This is because evidence requires attribution but thought does not.

2. Local Proxy: Keeping Cognition Where It Belongs

Currently, an AI user’s prompts are transmitted in their unvarnished form to the LLM, where they are paired with your identity, geolocation and other data. The local proxy ensures that prompts, drafts, speculative questions and exploratory reasoning: are generated locally, processed ephemerally and never leave the user’s device in attributable form.

In practical terms, this means that prompt construction, chain‑of‑thought generation, “what if” reasoning and hypothetical legal or moral analysis all occur on the user’s  device, not the host servers of the LLM. Cloud systems may still be used but only after cognition has been reduced to task‑bounded queries, not raw thought streams. This technical distinction matters, historically, because the law has treated internal reasoning as non‑discoverable, non‑testimonial and non‑culpable until translated into action.

By keeping prompts local and disposable, the local proxy restores the temporal fragility of thinking—the fact that thoughts normally disappear unless deliberately recorded.

In other words, it makes cognition forgettable again.

3. Unlinkable Inference: Preventing Behavioral Reconstruction

Even if the first two layers enable us to keep AI-associated cognition local, agentic AI still requires external services. Each prompt requires logged records of what is computed, what is searched on the internet, which databases are accessed and certain other execution systems. That digital footprint can create a robust picture of the user and their prompts. However, if the first two layers are implemented, those patterns cannot easily be correlated across time, cannot be tied to a persistent self-defining identity and therefore cannot be clustered into a behavioral profile.

So, how can we create this ZK verification regime and local prompts?

Another Use For The Blockchain

One possibility might be a novel use for an existing technology: blockchain. This technology is usually associated with cryptocurrencies, decentralized finance and NFTs. However, the technology is not confined to those financial and collectible use cases. 

Vitalik Buterin, the cofounder of the Ethereum Network, and Davide Crapis, the head of AI for the Ethereum Foundation, in February proposed a new way to deal with this problem using the Ethereum concept known as Zero Knowledge Proof of Identity. Perhaps typically, from the perspective of people known best for the monetary and contractual uses of their technology, they focus on a monetary aspect of identity: ZK API Usage Credits: LLMs and Beyond. Still, whatever its analytical starting point, the proposal replaces the LLM user’s identity with anonymized confirmation under their technology. Under this Ethereum proposal, users deposit funds once and generate ZK proofs for each request, proving authorization without revealing identity or enabling linkability. It is too early to test the efficacy of this proposal, but it is an interesting approach.

Companies have already been building blockchain-based technologies on this front. For example, there are companies utilizing zero-knowledge proofs (ZKPs), specifically zk-SNARKs and the Semaphore protocol, to enable anonymous verification of identity. It allows users to prove they are a unique human (member of a group) without revealing their identity, biometric data or linking different actions. While the details do not yet matter, all of this activity demonstrates that ZK identity is attainable.

Conclusion—From Scaling to Sovereignty

The danger is not what AI thinks. It does not. Rather, the danger is that the AI remembers for us. The zero‑knowledge firewall safeguards the thought–action boundary that underwrites criminal law, due process, and human agency itself.