Completed Inline Table Maintenance

[zhangyue19921010]

Completed Inline Table Maintenance

BackGround

Currently, Lance supports various kind of table maintenance, including cleanup, optimize_indices(create delta index or merge index), and Compaction. To maintain a healthy Lance table, users typically need to:

1. Configure a regular write task
2. Configure a scheduled Clean task
3. Configure regular index maintenance task
4. Configure regular Compaction task

Overall, this imposes a high burden on users, and the maintenance difficulty and dependencies of multiple tasks are complex.

Currently, for Clean, Lance supports providing a Clean up Hook after Commit. We can extend this idea by encapsulating numerous table maintenance operations into TableMaintainer. After the Commit is completed, corresponding operations will be executed according to the strategy, so as to reduce the user's burden and lower the threshold for use.

Summary

Introduce a unified TableMaintainer abstraction to run table maintenance actions inline after a successful
user commit, including:

• cleanup
• optimize_indices
  • create a new delta index
  • merge several delta index
• compaction
• etc

Provide a certain degree of intelligent inference to determine which table maintenance operations should be triggered and which should be ignored

Goals

• Unify TableMaintainer behavior behind a single trait.
• Automatically execute configured table maintenance operations inline after a successful write commit.
• Provide configurable failure handling per operation: BestEffort or Strict.
• Prevent recursive operation triggering when operations perform internal commits.

Covered Table maintenance operations

1. CleanupOperation
   • Deletes old versions / unreferenced files using an explicit CleanupPolicy.
   • Does not create a dataset version.
2. OptimizeIndexOperation
   • For existing non-system indices, append delta indices to cover currently unindexed fragments.
   • Implemented via Dataset::optimize_indices(OptimizeOptions::append()...).
   • May create new versions.
3. OptimizeIndexOperation
   • Merge indices.
   • Implemented via Dataset::optimize_indices(OptimizeOptions::merge(k)).
   • May create new versions.
4. CompactionOperation
   • Compacts files using compact_files(&mut Dataset, CompactionOptions, ...).
   • May create new versions (no-op if nothing to compact).

1. Configuration

Add rust/lance/src/dataset/table_tablemaintainer.rs (or similar):

  #[derive(Debug, Clone, Default)]
  pub struct TableMaintenanceContext {
      pub optimize_index: Option<OptimizeIndexOperation>,
      pub compaction: Option<CompactionOperation>,
      pub cleanup: Option<CleanupOperation>,
  }

  #[derive(Debug, Clone)]
  pub enum FailurePolicy {
      BestEffort,
      Strict,
  }

  #[derive(Debug, Clone, Default)]
  pub struct OptimizeIndexOperation {
      pub options: lance_index::optimize::OptimizeOptions, // merge/retrain/index_names
      pub failure_policy: FailurePolicy,
  }

  #[derive(Debug, Clone, Default)]
  pub struct CompactionOperation {
      pub options: crate::dataset::optimize::CompactionOptions,
      pub failure_policy: FailurePolicy,
  }

  #[derive(Debug, Clone, Default)]
  pub struct CleanupOperation {
      pub policy: crate::dataset::cleanup::CleanupPolicy,
      pub failure_policy: FailurePolicy,
  }

2. TableMaintainer

  #[async_trait::async_trait]
  pub trait TableMaintainer: Send + Sync {
      fn id(&self) -> &'static str;

      async fn check_and_schedule(&self, ctx: &TableMaintenanceContext<'_>) -> crate::Result<<type,bool>>;
      async fn execute(&self, ctx: &mut TableMaintenanceContext<'_>) -> crate::Result<()>;
  }

3. Runner: TableMaintenanceRunner

Responsibilities

• Construct table maintenance operations.
• Execute sequentially in a fixed order.
• Apply per-operation FailurePolicy.
• Enforce recursion guard (operation-internal commits do not trigger operation again).

Execution Order (Decision Complete)

1. OptimizeIndexOperation
2. CompactionOperation
3. CleanupOperation

Failure Policy Semantics

• BestEffort: log error, continue to next operation.
• Strict: return Error::PostCommitOperationFailed { operation_id, source }.

4. Hook Point & Recursion Guard

4.1 Centralize post-commit behavior in Dataset::apply_commit

Refactor so low-level commit code only commits manifests and returns (no auto-cleanup in io/commit.rs).

Dataset::apply_commit:

1. commits the user transaction
2. updates self.manifest / self.manifest_location / self.fragment_bitmap
3. if run_table_maintenance && table_maintainer.is_some(): runs the runner with ctx.dataset = self

4.2 Recursion Guard

Introduce internal options:

  pub struct ApplyCommitOptions {
      pub table_maintenance: Option<TableMaintenanceContext>,
      pub run_table_maintenance: bool,
  }

• User commit path: run_table_maintenance = true
• Maintenance-internal commits (index optimize, compaction commit): run_table_maintenance = false

[majin1102]

5.1 Centralize post-commit behavior in Dataset::apply_commit

Agree that we should have a commit hook in lance. Wondering if we should build this ability by providing a commitTable spec in lance-namespace

I think the term "table service" mentioned in this topic might be slightly misleading. IMO, a table maintenance service represents a distinct layer in the data ecosystem—at the very least, it should sit above the metadata service.

As I understand it, the main idea is to complete the commit hook and provide the ability to perform compaction and optimizing_indices after commit. However, I'm a bit concerned about the workload implications in a production environment: for cleanup, the workload is primarily metadata scanning and garbage collection, which is relatively lightweight. But index optimization and fragment compaction involve substantial data processing—I don't think it's a good idea to execute these directly within a commit operator.

I think an ideal solution for inline optimization might be to provide dedicated optimizing_indices and compaction operators that trigger asynchronously after commit in each engine. This would likely be a separate effort.

What do you think?

[zhangyue19921010]

Thank you for your suggestion @majin1102 :) The relevant expressions have been revised. The current discussion mainly aims to construct a complete Table Maintenance abstraction to unify all Lance table maintenance operations.

One of the usage is that after the completion of a regular write commit (similar to the current Clean up Hook after Commit), this Table Maintenance can be used to perform synchronous or asynchronous Lance table maintenance operations according to user-specified requirements (providing a certain ability to intelligently infer the triggering timing), so as to reduce the user's operation and maintenance costs and usage thresholds.

Of course, this can also serve as an implementation foundation for the Long running table service, if necessary.

Whether these table maintenance operations are triggered synchronously or asynchronously, and regardless of the amount of data to be processed, they will not block the regular write process, including commits, meaning they will not affect the timeliness of data visibility. Of course, users can also choose the previous method and organize their own scheduling logic as needed.

[Xuanwo]

Hi, thank you for sharing this! Before discussing the tech details, can you demonstrate how will our users use the new API and what's the difference?

[zhangyue19921010]

Hi @Xuanwo Thanks for your attention.

For ordinary users, they only need to configure some parameters in regular write tasks to control behaviors such as cleanup, optimize_index, and compaction, instead of configuring a whole set of separate operation and maintenance tasks to trigger and maintain them regularly as before. This type of usage is similar to the current clean-up hook after writing is completed.

What's more. we will provide a certain degree of intelligent inference to determine which table maintenance operations should be triggered and which should be ignored, for example, automatically inferring when to create incremental indexes, when to merge multiple increments; implementing incremental Compaction awareness, and so on

For advanced users or online services, they can directly interface with this TableMaintainer abstraction to achieve unified and flexible control over table operation and maintenance behaviors.