Storage in Regulated Industries: The Deletion Paradox and Crypto-Shredding

Written by Skye Tran

You have to keep data forever. You also have to delete it on command. Crypto-shredding is how you do both.

The Situation: Storage Used to Be Simple

Once upon a time, storage was straightforward - or at least, that’s how it felt back then.

Buy a suitable storage service type such as object, block, or file system. If you ran out of space, you could either delete old stuff to free up space or pay to expand your storage.

The old normal is long gone.

These days, storage isn’t just about capacity. Integrity, retention, and sovereignty become the trendy topics whenever storage is being discussed.

And when it comes to regulatory requirements, especially across different jurisdictions, you may experience conflicting ones that make you scratch your head.

The deletion paradox is one big headache you may encounter.

The Problem: The Deletion Paradox

The deletion paradox is the tension between two conflicting regulatory requirements.

On one hand, compliance frameworks demand retention. On the other hand, privacy frameworks demand the right to erasure (a.k.a. the right to be forgotten).

So how do you retain data while allowing people to request on-demand deletion? How do you prove you’ve deleted something that was supposed to be immutable?

The deletion paradox isn’t a theoretical problem.

It’s a common challenge for organizations operating across multiple jurisdictions in regulated industries (e.g., Australian banks serving EU customers).

The Solution: Crypto-Shredding

What is Crypto-Shredding?

Imagine you run a bakery and as part of your normal operations, you keep detailed records of every sales transaction. This could include what each customer bought and how much the order was.

If any customer opts to scan their digital loyalty card, you will also capture the customer’s name and email address associated with the loyalty program.

The tax authorities expect you to keep all sales records for 7 years for audit purposes. Meanwhile, under privacy law, a customer has the right to request deletion of their personal information.

If you keep their record, you’re violating privacy law. If you delete it, you will get into trouble with tax matters.

To satisfy both requirements, you keep two things separate:

  1. A permanent, tamper-proof ledger of every sales transaction - what was sold, when, and for how much. If a customer scans their loyalty card, their sales entry is then tagged against a unique Customer ID.

  2. A locked box for each customer, labeled with the same Customer ID, containing their name and contact details. Each box has its own key, and only you have the key.

When tax authorities request to audit your sales, you show them the ledger, which is complete and unaltered.

When a customer wants his personal data deleted, you locate the box with his Customer ID and destroy the key, thus making his data instantly and permanently unreadable.

That’s exactly how crypto-shredding (a.k.a. cryptographic erasure) works. The data remains exactly where it is, but the personal information it contains becomes cryptographically unrecoverable.

The regulators see the records. The customer’s data is effectively deleted. You’ve satisfied both requirements without violating the integrity of your audit trail.

Prerequisites for Crypto-Shredding

If you look more closely at the analogy above, it implies that being a bakery owner means understanding the need to maintain a master list of customers and keep them separate from transactional data.

This also highlights three critical prerequisites for implementing crypto-shredding.

1. Data Classification

You need to know which is personally identifiable information (PII) and which is regular transactional data.

Without classification, you wouldn’t know what to keep in the ledger for tax authorities and what to lock away and potentially delete upon request.

Always keep PII and transactional data separate. This will make privacy handling much more manageable.

2. Linking

You must have a common key, such as a unique Customer ID, that links personal data to transactional data.

Without that link, you can’t determine which key to destroy. And you also can’t prove that you’ve destroyed the right one.

3. Archiving

Before you can delete anything, you need to know where the data is currently.

Data has a lifecycle, and it follows what I call “data temperature”.

As data moves through its lifecycle, it’s important to proactively move cold data out of the fast-access operational systems and into long-term storage.

Without archiving, you could end up trying to delete data that’s scattered across operational systems, data warehouse, backups and cold storage. Data deletion would then become inefficient, risky, and more difficult to verify.

Classification tells you what to encrypt. Linking tells you which key to destroy. Archiving tells you where the data lives. Without all three, crypto-shredding is just a very risky operation based on guesswork.

Real-World Examples & Lessons Learned

Crypto-shredding is not just a theoretical solution taken from a research paper. It is recognized by the National Institute of Standards and Technology (NIST) as a valid data sanitization technique in its Special Publication 800-88.

Here are 2 real-world examples of crypto-shredding, each with a different lesson that may interest you.

Media Sanitization at Google

In October 2025, Google announced the transition from the “brute force disk erase” to cryptographic erasure for customer data whenever a deletion request is initiated.

Instead of overwriting the entire drive, Google securely deletes the cryptographic keys that are used to encrypt the data. Once the keys are gone, the data is rendered unreadable and unrecoverable.

Google's previous brute-force erase method was environmentally costly. Any drive that failed verification was physically destroyed—destroying of thousands of devices each year.

With crypto-shredding, Google effectively eliminates the need to destroy drives, thus reusing more hardware and recovering valuable rare earth materials.

Lesson learned: Companies are opting for crypto-shredding for compliance, business efficiency, and sustainability.

Conduktor Gateway (Kafka)

Apache Kafka is a distributed event streaming platform for building real-life data pipelines and streaming applications.

In Apache Kafka, data records are only ever added to the very end of a partition file and cannot be modified after they’re written.

When a customer requests deletion, you can’t simply delete data records from an immutable log in Apache Kafka.

Conduktor Gateway solves this challenge with a cost-efficient approach. Instead of expensive per-user cloud keys, they derive user-specific keys from a central master key (stored in an external KMS such as AWS KMS) and stores them securely within Kafka itself.

Upon deploying Conduktor, every message from Apache Kafka essentially flows through the Conduktor Gateway and is encrypted with a unique per-user key.

When a deletion request arrives, Conduktor permanently removes the key for that UserID, thus instantly making all associated data unreadable.

Lesson learned: The hardest part of crypto-shredding isn’t encryption but key management strategy.

What You Need To Do

If you’re facing the deletion paradox, here’s where to start.

If You Only Have 30 Minutes

Audit one single storage resource.

Seriously, pick an S3 bucket, a Blob container, or an NFS share, and ask yourself:

  • What data are we storing? Why do we need it?

  • Does the data contain any PII?

  • Is the encryption enabled?

  • Do you have a process for securely destroying keys?

  • Can you prove deletion if a GDPR request arrives in your mailbox tomorrow?

Any question you can’t answer is a hint on where to dig deeper.

If You Have More Time

1. Classify your data

Protecting data without knowing what you have isn’t compliance. It’s pure optimism.

A simple way to start is to create a data inventory. List your storage resources, the types of data they hold, and the regulations that apply (GDPR, HIPAA, APRA, etc.).

2. Enable per-subject encryption

Crypto-shredding only works if each individual (customer, patient, employee) has their own unique encryption key. Without this, you can't destroy a single key to render a single person’s data unrecoverable.

Consider implementing per-subject encryption (like Conduktor Gateway) or a key management service that supports per-user keys.

Don’t forget to store the keys separately from the data they protect.

3. Implement a key management strategy

Define who can access the encryption keys and how keys are destroyed.

Store keys separately from the data they protect (zero-trust architecture), implement strict access controls and audit logging for key operations, and ensure key destruction is cryptographically verifiable.

Also don’t forget to document everything - regulators need to see your proof of compliance.

Reference




Skye Tran
Previous

Manual Segmentation Is Death by a Thousand Cuts
Next

Compute in Regulated Industries: The Translation Gap