do not merge -- Mg/try lance schema changes

[malcolmgreaves]

DO NOT MERGE

[greptile-apps]

Greptile Summary

This PR refactors the Daft codebase to improve Lance integration by making schema handling more robust. The main changes include:

1. DaftExtension Class Refactoring: The DaftExtension class in daft/datatype.py is moved from being defined inside the _ensure_registered_super_ext_type() function to module level. This improves code organization and makes the PyArrow extension type accessible throughout the module without requiring the registration function to be called first.

2. Enhanced Schema Preservation in Lance Data Sink: The Lance data sink (daft/io/lance/lance_data_sink.py) is modified to explicitly reconstruct Arrow tables using the stored PyArrow schema. Instead of directly using micropartition.to_arrow(), it now uses pa.Table.from_batches() with the stored schema to ensure metadata preservation, particularly important for Lance-specific encodings like compression and blob storage.

3. Blob Storage Test Coverage: A new test test_lancedb_write_blob validates the blob encoding functionality, ensuring that binary data with Lance-specific metadata can be written and retrieved correctly using Lance's specialized blob API.

These changes collectively improve the integration between Daft and Lance by ensuring schema consistency and supporting advanced Lance features like blob encoding for efficient binary data storage.

Confidence score: 4/5

• This PR is generally safe to merge with some attention to the schema handling changes
• Score reflects solid refactoring and test coverage, but lowered due to schema reconstruction logic that could impact data fidelity
• Pay close attention to daft/io/lance/lance_data_sink.py for potential schema compatibility issues

_{3 files reviewed, 1 comment}

_{Edit Code Review Bot Settings | Greptile}

[greptile-apps]

style: Missing mode parameter - should specify mode='create' for consistency with other tests

[codspeed-hq]

CodSpeed Performance Report

Merging #4975 will degrade performances by 53.86%

_{Comparing mg/try_lance_schema_changes (fe9db2e) with main (0129008)}

Summary

❌ 1 regressions
✅ 23 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
❌	test_iter_rows_first_row[1 Small File]	124.7 ms	270.2 ms	-53.86%

[codecov]

Codecov Report

❌ Patch coverage is 86.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.15%. Comparing base (0129008) to head (fe9db2e).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
daft/datatype.py	92.30%	1 Missing ⚠️
daft/io/lance/lance_data_sink.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4975      +/-   ##
==========================================
- Coverage   79.59%   79.15%   -0.44%     
==========================================
  Files         917      917              
  Lines      127014   127016       +2     
==========================================
- Hits       101094   100544     -550     
- Misses      25920    26472     +552     

Files with missing lines	Coverage Δ
daft/datatype.py	94.39% <92.30%> (-0.19%)	⬇️
daft/io/lance/lance_data_sink.py	82.81% <50.00%> (ø)

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.