feat: Add offset support to Spark Connect

[plotor]

Changes Made

Related Issues

#3581
#4547

Checklist

• Documented in API Docs (if applicable)
• Documented in User Guide (if applicable)
• If adding a new documentation page, doc is added to docs/mkdocs.yml navigation
• Documentation builds and is formatted properly (tag @/ccmao1130 for docs review)

[codecov]

Codecov Report

❌ Patch coverage is 69.23077% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.86%. Comparing base (072858e) to head (4fffe6d).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
src/daft-connect/src/spark_analyzer.rs	69.23%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4962      +/-   ##
==========================================
+ Coverage   76.29%   76.86%   +0.56%     
==========================================
  Files         918      918              
  Lines      128703   126910    -1793     
==========================================
- Hits        98195    97545     -650     
+ Misses      30508    29365    -1143     

Files with missing lines	Coverage Δ
src/daft-connect/src/spark_analyzer.rs	80.17% <69.23%> (-0.22%)	⬇️

... and 56 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[greptile-apps]

Greptile Summary

This PR adds comprehensive OFFSET support to Daft's Spark Connect implementation, enabling pagination functionality that was previously missing. The implementation consists of three key components:

1. Core Rust Implementation: Added offset functionality to spark_analyzer.rs by implementing an offset method that validates input parameters (ensuring non-negative values), converts to u64, and integrates with the logical plan builder. The implementation follows the same pattern as the existing limit method for consistency.

2. SQL Query Support: Added test coverage in test_spark_sql.py that validates both OFFSET ... LIMIT and LIMIT ... OFFSET syntax variations work correctly with SQL queries. The tests use a large dataset (1024 rows) with shuffled data to ensure robust validation under realistic conditions.

3. DataFrame API Support: Added programmatic offset testing in test_basics.py through the test_range_limit_offset function, which validates that df.offset() works correctly both independently and in combination with limit() operations.

The implementation addresses GitHub issues #3581 and #4547, providing essential pagination capabilities for Spark Connect users. The changes integrate seamlessly with Daft's existing logical plan architecture and maintain consistency with Spark's DataFrame API. All tests use shuffled data followed by ordering to ensure the offset mechanism works correctly regardless of initial data arrangement.

Confidence score: 4/5

• This PR is safe to merge with minimal risk as it adds well-tested functionality without modifying existing behavior
• Score reflects comprehensive test coverage and implementation that follows established patterns in the codebase
• Pay close attention to the Rust implementation in spark_analyzer.rs to ensure proper error handling for edge cases

_{3 files reviewed, no comments}

_{Edit Code Review Bot Settings | Greptile}

[plotor]

Hi @universalmind303, I see you created the EPIC #3581, and this PR is to add offset support to Spark Connect. Please take your time to review it when you're free.

[plotor]

Hi @srilman, Since I don't have merge permission, could you help me merge this PR?