RowId for Daft-Lance Integration

[Jay-ju]

Technical Proposal for Daft-Lance Integration

Current Implementation

Daft effectively utilizes URL-based lazy processing to perform joins, aggregations, and complex operations (e.g., window functions) in multi-modal workflows. This paradigm delays downloading content until its actual use, which offers significant flexibility.

Identified Storage-Side Challenges

1. Excessive small files: High S3 QPS due to fragmented requests
2. Limited encoding capabilities: Cannot leverage columnar compression or advanced encoding techniques
3. Underutilized Lance features: Point queries and dynamic column addition (add_column) remain inaccessible

Proposed Integration Approach

To harness the complementary strengths of Daft and Lance, we suggest:
Replace URL-based download interfaces with row_id-based point queries via lance_take(),
Substitute upload interfaces with Lance's atomic update() operations.

Anticipated Advantages

• Unified execution leveraging Daft's distributed processing and Lance's storage optimizations
• Elimination of small-file overhead through batched columnar access
• Full utilization of Lance features: compression, point queries, and schema evolution

@jaychia @kevinzwang @universalmind303 WDYT？

[kevinzwang]

HI @Jay-ju , thanks for writing this up. I think this approach makes sense.

As a first step, could you share some sample workloads that would benefit from this? I'd like to set up some baseline benchmarks that we can work towards improving.