chore(dev): use pyproject.toml to manage the dev dependencies

[xy-xin]

Changes Made

Currently daft uses requirememnts-dev.txt and requirements-doc.txt to manage the dev and doc dependencies but use uv manage the release dependences. But uv has the same capabilities. We could remove requirements.txt and leverage uv to manage all the packages.

Related Issues

Close #4682
Close #4754

[greptile-apps]

Greptile Summary

This PR consolidates dependency management by migrating from separate requirements-dev.txt and requirements-doc.txt files to using uv's dependency groups feature in pyproject.toml. The change standardizes the project on a single dependency management tool (uv) instead of the previous hybrid approach where release dependencies used uv/pyproject.toml while development and documentation dependencies used separate requirements files.

The migration introduces dependency groups in pyproject.toml (dev, test, tracing, docs, lint, storage, sql, ai) that organize dependencies by purpose. All build workflows, CI configurations, and documentation processes are updated to use uv sync --all-extras --all-groups instead of pip install -r requirements-*.txt. The Makefile, GitHub Actions workflows, Read the Docs configuration, and contributor documentation are all updated to reflect this unified approach.

This change aligns with modern Python packaging best practices by leveraging PEP 735 dependency groups and uv's superior dependency resolution capabilities. It eliminates the maintenance overhead of keeping multiple requirements files in sync and provides a single source of truth for all project dependencies.

Confidence score: 2/5

• This PR has significant missing dependencies that will likely cause immediate build failures
• Critical analysis reveals many pinned dependencies from requirements-dev.txt are missing from pyproject.toml, including pyarrow==20.0.0, ray[data,client]==2.34.0, and boto3==1.36.20 which had specific compatibility constraints
• Files requiring urgent attention: pyproject.toml (missing critical dependencies), .github/workflows/daft-profiling.yml (will fail due to missing deps)

_{9 files reviewed, 3 comments}

_{Edit Code Review Bot Settings | Greptile}

[greptile-apps]

logic: uv needs to be installed first. Add pip install uv or use a setup action that includes uv before running uv sync

Suggested change

	uv sync --all-extras --all-groups
	pip install uv
	uv sync --all-extras --all-groups

[greptile-apps]

logic: Naming conflict: sql dependency group has same name as the optional dependency in line 55. This may cause confusion or conflicts in uv.

Suggested change

	sql = [
	sql-dev = [

[greptile-apps]

style: installing 'latest' uv version could introduce build instability if uv releases breaking changes. Consider pinning to a specific version

[codspeed-hq]

CodSpeed Performance Report

Merging #4849 will improve performances by 40.6%

_{Comparing xy-xin:uv (b8d98a0) with main (9902072)}

Summary

⚡ 1 improvements
✅ 23 untouched benchmarks

Benchmarks breakdown

	Benchmark	BASE	HEAD	Change
⚡	test_iter_rows_first_row[1 Small File]	125.3 ms	89.2 ms	+40.6%

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.34%. Comparing base (af3fdd9) to head (c3eaf3c).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4849      +/-   ##
==========================================
- Coverage   79.78%   79.34%   -0.44%     
==========================================
  Files         896      896              
  Lines      125817   125819       +2     
==========================================
- Hits       100377    99828     -549     
- Misses      25440    25991     +551     

see 15 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[xy-xin]

@desmondcheongzx @jaychia ，could you pls review this pr?

[desmondcheongzx]

Thanks for the PR @xy-xin! I noticed that requirements-dev.txt wasn't deleted so I went ahead and did it to see what happens in CI. Seems to be failing for now but I'll take a closer look.