chore: Extensions architecture POC

[michael-s-molina]

SUMMARY

This PR introduces new packages and functionality for introducing extensions as proposed in SIP #34162. Example extensions that were used for validating the system have been moved to the following draft PR: #34726

This PR introduces the following packages:

superset-frontend/packages/superset-core
Our package with UI components, utility functions, types, and APIs used by extensions.

superset-core
The host APIs used by backend extensions.

superset-cli
CLI commands to build, bundle and hot-rebuild extensions.

superset-extensions build     Builds extension assets.

superset-extensions bundle    Packages the extension into a .supx file.

superset-extensions dev       Provides hot-rebuilding of assets.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

The Dataset references extension displays relevant information for SQL authors in SQL Lab:

• Datasets referenced in a SQL Query: This helps users discover the origin of the data, especially for long SQL queries.
• Owners of each referenced dataset: This is crucial for dealing with access permissions or quickly finding out who to contact in case of problems.
• Latest available partition: This is important for understanding data completeness issues.
• Estimated row count for each dataset: This helps users grasp the implications of joins.

Screen.Recording.2025-06-04.at.10.48.32.mov

TESTING INSTRUCTIONS

1 - Set the ENABLE_EXTENSIONS feature flag to True.

2 - Run npm ci on superset-frontend.

3 - Install the development dependencies by running pip install -r requirements/development.txt. This will pull in the new cli package that contains the superset-extensions command.

4 - Bundle an extension by going into its root folder and executing superset-extensions bundle. This will build the extension and generate a .supx bundle of the extension.

5 - Upload the .supx file to Superset using Swagger and the extensions endpoint.

6 - Go to SQL -> SQL Lab and check the extension.

For development mode:

1 - Execute steps 1, 2 and 3 above.

2 - Configure Superset's LOCAL_EXTENSIONS variable like:

LOCAL_EXTENSIONS = [
    "./extensions/dataset_references",
]

3 - Run in development mode by going into the extension root folder and executing superset-extensions dev. This will trigger hot-rebuilding of your frontend assets. Backend assets are currently reloaded by the default reloading mechanisms of Superset using superset run --reload.

4 - Go to SQL -> SQL Lab and check the extension.

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[korbit-ai]

Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment /korbit-review.

Your admin can change your review schedule in the Korbit Console

[codecov]

Codecov Report

❌ Patch coverage is 13.01483% with 528 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.91%. Comparing base (fbcdf69) to head (0d9c10b).
⚠️ Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/extensions/utils.py	0.00%	129 Missing ⚠️
superset/extensions/api.py	0.00%	83 Missing ⚠️
superset/extensions/startup_upsert.py	0.00%	61 Missing ⚠️
superset/initialization/__init__.py	16.98%	42 Missing and 2 partials ⚠️
superset/extensions/local_extensions_watcher.py	26.31%	42 Missing ⚠️
superset/daos/extension.py	0.00%	39 Missing ⚠️
superset/extensions/models.py	0.00%	33 Missing ⚠️
superset/extensions/discovery.py	0.00%	29 Missing ⚠️
superset/extensions/types.py	0.00%	18 Missing ⚠️
superset/core/api/types/models.py	51.72%	14 Missing ⚠️
... and 7 more

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #31934       +/-   ##
===========================================
+ Coverage        0   71.91%   +71.91%     
===========================================
  Files           0      588      +588     
  Lines           0    42307    +42307     
  Branches        0     4458     +4458     
===========================================
+ Hits            0    30425    +30425     
- Misses          0    10712    +10712     
- Partials        0     1170     +1170     

Flag	Coverage Δ
hive	46.53% <13.01%> (?)
mysql	70.92% <13.01%> (?)
postgres	70.99% <13.01%> (?)
presto	50.18% <13.01%> (?)
python	71.87% <13.01%> (?)
sqlite	70.54% <13.01%> (?)
unit	100.00% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[michael-s-molina]

@villebro I'm creating a task list so we can track what we need to do to merge this PR. Feel free to add/edit items:

• Update the PR description referencing the draft PR that will contain the extensions folder
• Adjust the upload logic based on our security discussions
• Add tests to the new upload logic
• Remove any UI component from @apache-superset/core as v0.0.1 will only contain APIs
• Add docstrings to Typescript APIs
• Add docstrings to Python APIs
• Remove the extensions table from the metastore
• Update PR [WIP] feat: example extensions #34726 with the most up to the date code from the extensions folder and remove the folder

[villebro]

@michael-s-molina I'm wondering if we should actually close this PR and open up new ones with the subsets of changes, cross referencing to this one for context?

[michael-s-molina]

@michael-s-molina I'm wondering if we should actually close this PR and open up new ones with the subsets of changes, cross referencing to this one for context?

@villebro I prefer the feature branch approach as we can test everything together before merging things to master. Check this comment where I share my thoughts about PR organization.