Skip to content

Version-based time travel should go no further back than the log retention and deleted file retention duration #1000

Open
@OussamaSaoudi

Description

@OussamaSaoudi
Collaborator

Describe the bug

Today, a snapshot can be constructed for any version of the table. However, a snapshot scan is only valid if all the data files are present in the table. We must block snapshots of the table that existed before the delta.deletedFileRetentionDuration and delta.logRetentionDuration.

Depends on #1001

Initially reported by @scovich

To Reproduce

No response

Expected behavior

No response

Additional context

This requires that the timestamp of the version at the snapshot be no longer ago than the retention durations. If in-commit timestamps is enabled, then the ICT must be respected.

To speed up ICT reads, the CRC can be used to fetch the ICT at the snapshot's version. See here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @OussamaSaoudi

        Issue actions

          Version-based time travel should go no further back than the log retention and deleted file retention duration · Issue #1000 · delta-io/delta-kernel-rs