This is some text inside of a div block.
Home
/
TRM Tech Blog
/
Incremental Processing When Old Data Keeps Changing: Spark and Iceberg at TRM
Engineering
November 13, 2026

5 min

Incremental Processing When Old Data Keeps Changing: Spark and Iceberg at TRM

‍

We recently moved both from daily full rebuilds to incremental processing, spanning more than 70 blockchains. In the initial 14-chain rollout, runtime for the shared transfer dataset fell by 39%

‍

XX
[
Dileep Vasireddy,
 ]

At TRM Labs, we enrich blockchain transfers with time-aware attribution so investigators can see which entity controlled an address at the time of each transfer. New transfers usually arrive in recent partitions, while attribution updates can reach years into the past. Daily full rebuilds captured both. For two of our core datasets, which together hold roughly 90 TB, that meant repeatedly processing large amounts of unchanged data.

We recently moved both from daily full rebuilds to incremental processing. The rollout spanned more than 70 blockchains. In the initial 14-chain rollout, runtime for the shared transfer dataset fell by 39%. For the downstream transfer summary dataset, migrating from BigQuery full rebuilds to incremental Spark processing cut processing time by 84%.

We achieved these savings by matching the update strategy to the type of change and reusing previously computed results for unchanged data. This kept incremental processing efficient even when changes spread across most historical partitions.

Key takeaways

  • Reuse previously computed results. Reusing prior work reduced large joins and shuffles, keeping incremental runs faster even when changes touched most historical partitions.
  • Match the update strategy to the type of change. Rebuild partitions when transfers change, and recompute affected fields when only context changes.
  • Use fine-grained updates for sparse changes. Iceberg deletion vectors let us replace affected rows for sparse attribution changes without rewriting entire partitions.

We started with the shared transfer dataset, which holds the enriched blockchain transfers used by downstream pipelines. When transfer data changed, we rebuilt complete partitions. When only attribution or other context changed, we recomputed affected rows and reused the results for unchanged rows. We then extended incremental processing to the downstream transfer summary dataset, using Iceberg deletion vectors to reduce data rewrites. In the rest of this post, we explain why a tiny update can reach so far back in history, how we detect and process only what changed, and how we verified and operate the pipeline in production.

Why small changes affect years of data

New transfers usually arrive in recent date partitions. A blockchain reorganization or decoding correction may revise existing transfers, but those changes are usually concentrated in a small number of recent partitions.

Attribution changes can reach much farther back. An address may have transacted for years before the entity attributed to it, or its category, is updated. For example, a transfer may be sent to an address we can't yet attribute. If new intelligence later links that address to an exchange, we update the attribution on its historical transfers. Changing attribution for a few addresses can therefore affect a large share of the dataset’s historical partitions.

Attribution also has a valid time range. A label valid from May 12 until the start of May 20 applies to a May 15 transfer, but not one on May 10 or May 20. Changing the label or its dates requires updating the affected historical transfers; deleting it requires removing the old label.

A sanctions designation works the same way. If an exchange becomes sanctioned from a particular date, transfers from that date onward carry the sanctions label, while earlier transfers keep the classification that applied at the time.

This is a variation of the slowly changing dimension (SCD) problem: correcting attribution resembles a Type 1 update, while preserving attribution across effective time intervals resembles Type 2 history. Because we store attribution in transfer rows, changing one address’s attribution can require updating years of historical transfers.

In one daily production run for Ethereum in October 2026, attribution changed for just 0.09% of the dataset’s addresses. Those changes touched 86% of its daily partitions. By partition count, the update looked close to a full rebuild. That count, however, did not tell us how many rows needed new attribution or how much computation we could reuse.

Two paths: Rebuild partitions or recompute rows

A partition may need updated attribution while its underlying transfer data remains valid. We use that distinction to choose between rebuilding the partition from its sources and recomputing affected fields in existing rows.

Change Processing path Work we reuse
Transfers changed within a partition Partition Rebuild: rebuild the partition from source transfers and context Unchanged partitions
Attribution or other context changed, transfers unchanged Row Recompute: read existing rows and recompute affected fields Existing transfer facts and unaffected values

Partition Rebuild reads source transfers for the affected partitions, joins current attribution and other context, and overwrites those partitions. Its scope follows the changed transfer data, so a historical backfill can select old partitions as well as recent ones.

Row Recompute starts from the existing dataset. It finds transfers affected by changes to sender or receiver attribution and other context, recalculates the affected fields and dependent values, and preserves unaffected values. This avoids repeating transfer normalization and unrelated source joins.

With our partition-level copy-on-write approach, every affected partition still needs to be rewritten. The savings come from reduced computation rather than fewer bytes written.

Each partition follows only one incremental path per run. Partitions selected for Partition Rebuild are excluded from Row Recompute to prevent competing writes.

Before: Daily full rebuild

Each run reads the full transfer history and context, recomputes joins and derived fields, and rebuilds the complete table before quality checks.

After: Incremental processing

Transfer changes route to Partition Rebuild, context changes to Row Recompute, and oversized workloads to a full refresh. Each table write commits independently. The checkpoint advances only after all writes and quality checks succeed and the new change-detection baseline is saved. A failed run leaves the previous checkpoint in place.

Detecting changes between runs

A fixed “last 24 hours” window can miss changes when runs are delayed or historical data is backfilled. We instead track changes since the last successful run, using a saved comparison baseline and a checkpoint that records the run’s snapshot time.

For attribution, we create a deterministic fingerprint of each address’s attribution intervals and values. Comparing these fingerprints with the saved baseline identifies addresses whose attribution was added, changed, or removed. Those addresses become the lookup set for recomputation.

For transfers, we use the source table’s partition metadata to identify partitions modified between the last checkpoint and the current run’s snapshot time.

Every incremental source read uses the same snapshot timestamp, captured when the run starts. A retry reuses that timestamp, so it sees the same inputs as the first attempt. These timestamps also give us known boundaries for comparing runs.

After the output checks pass, we save the new comparison baseline and advance the checkpoint. If any required step fails, the checkpoint stays unchanged so the changes are not marked complete.

When to fall back to a full refresh

We kept the full-refresh path for workloads that exceed our incremental limits and for runs where an operator needs to rebuild the baseline.

Large transfer backfills or oversized attribution and context change sets can trigger a full refresh. For example, a major sanctions reclassification on October 8, 2026, required attribution updates for millions of existing addresses across multiple blockchains. The broader set of attribution changes that day affected billions of historical transfers involving those addresses. The run logs which limit was exceeded, and the output must pass the same quality checks before the checkpoint advances.

Initially, we also used partition coverage as a fallback signal. If changes touched too many partitions, the pipeline switched to a full refresh. For some large blockchains, attribution changes reached nearly every historical partition almost every day, repeatedly triggering that fallback.

Testing showed that Row Recompute was still faster, even when it rewrote most of the table. It reused existing transfer data and avoided repeating temporal joins for unchanged rows. We adjusted the fallback logic to allow Row Recompute even when changes affected most partitions, while keeping the other fallback checks in place.

Optimizing joins and query planning

Reusing the changed-address set

To find transfers affected by attribution changes, we join existing transfer rows with the changed-address set twice: once on the sender address and once on the receiver address. Referencing the same Spark DataFrame in both joins does not guarantee that Spark computes the set only once. We write it to disk so both joins can read the prepared data without recomputing it.

Handling join skew

Some blockchain addresses appear in far more transfers than others. During attribution joins, these addresses can concentrate work in a few Spark tasks that take much longer to finish than the rest. Salting spreads that work across multiple tasks, but applying it to every address unnecessarily duplicates attribution records and increases data movement and memory use.

For full refreshes, including fallback runs, we added an optional selective-salting approach. A preprocessing step identifies heavily used addresses, and only their attribution records expand across salt buckets. Records for other addresses remain unexpanded.

Keeping query planning efficient

When changes affected thousands of historical partitions, listing each date separately made the filter expression itself expensive to plan. We group consecutive dates into timestamp ranges and combine them into a balanced filter expression. This reduces planning overhead while allowing Iceberg to skip unrelated partitions.

Verifying parity with the full refresh

Before enabling incremental processing for each chain, we ran it alongside the full-refresh pipeline and compared their outputs. We checked row counts in every partition and used row hashes to compare record contents in a sample of partitions. We also ran the pipeline’s existing checks for required fields and unique keys.

Reducing data rewrites with Iceberg V3 deletion vectors

Incremental processing reduced repeated computation, but our partition-overwrite approach still rewrote affected partitions even when only a few rows changed. We addressed that cost in the downstream transfer summary dataset using Apache Iceberg V3 merge-on-read (MoR) with deletion vectors.

Dense transfer changes continue to use partition-level copy-on-write. Sparse attribution changes use merge-on-read: deletion vectors mark old row versions as deleted, and replacement rows are written to new data files. Readers apply the deletion vectors when reading the table, so each sparse update can leave existing data files in place.

Moving this workload from BigQuery to Spark also gave us the write support needed for Iceberg V3 deletion vectors. BigQuery can query these tables, but its current Iceberg V3 support does not include writes, so Spark performs the incremental updates.

Results: Less processing time, fewer tasks

For the shared transfer dataset, the initial 14-chain rollout reduced combined daily Spark job runtime from approximately 13.6 hours to 8.3 hours, a 39% reduction. That saved about 5.4 hours of Spark job runtime per day.

We measured these results on August 13, 2026, comparing each chain’s daily runs before and after its cutover. The figures sum job runtimes across chains and exclude quality checks and cluster setup.

For the downstream transfer summary dataset, we replaced daily BigQuery full rebuilds with incremental Spark processing using Iceberg merge-on-read and deletion vectors. The migration reduced aggregate processing time by approximately 84% and the number of Airflow tasks by 63%. Retiring the legacy BigQuery workload also reduced average slot usage by approximately 480 slots.

What’s next

We are exploring whether joining some attribution later could reduce historical rewrites by keeping stable transfer records separate from changing attribution. We need to do that while keeping customer queries fast and predictable.

Across both datasets, the savings came from reusing valid computation and choosing how much data to rewrite for each change. Our next step is to extend incremental processing to final data products.

If you enjoy building data systems at scale and making them faster and more efficient, join our engineering team at TRM.

‍

XX
[
Dileep Vasireddy,
 ]
No items found.
Subscribe to our latest insights
You can unsubscribe at any time. Read our Privacy Policy.