Skip to content

Introduce Apache OpenDAL (Open Data Access Layer) Provider #50728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 10 commits into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions airflow-core/src/airflow/utils/db.py
Original file line number Diff line number Diff line change
@@ -732,6 +732,14 @@ def create_default_connections(session: Session = NEW_SESSION):
),
session,
)
merge_conn(
Connection(
conn_id="opendal_default",
conn_type="opendal",
extra='{"source_config":{"path":"", "conn_id":"opendal_default", "operator_args": {}}}',
),
session,
)


def _create_db_from_orm(session):
1 change: 1 addition & 0 deletions dev/breeze/src/airflow_breeze/global_constants.py
Original file line number Diff line number Diff line change
@@ -656,6 +656,7 @@ def generate_provider_dependencies_if_needed():
"cncf-kubernetes",
"common-io",
"common-messaging",
"common-opendal",
"docker",
"elasticsearch",
"fab",
60 changes: 60 additions & 0 deletions providers/common/opendal/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@

.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
.. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
.. IF YOU WANT TO MODIFY TEMPLATE FOR THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
``PROVIDER_README_TEMPLATE.rst.jinja2`` IN the ``dev/breeze/src/airflow_breeze/templates`` DIRECTORY
Package ``apache-airflow-providers-common-opendal``

Release: ``1.0.0``


``Common OpenDAL Provider``


Provider package
----------------

This is a provider package for ``common.opendal`` provider. All classes for this provider package
are in ``airflow.providers.common.opendal`` python package.

You can find package information and changelog for the provider
in the `documentation <https://airflow.apache.org/docs/apache-airflow-providers-common-opendal/1.0.0/>`_.

Installation
------------

You can install this package on top of an existing Airflow 2 installation (see ``Requirements`` below
for the minimum Airflow version supported) via
``pip install apache-airflow-providers-common-opendal``

The package supports the following python versions: 3.10,3.11,3.12

Requirements
------------

================== ==================
PIP package Version required
================== ==================
``apache-airflow`` ``>=3.0.1``
================== ==================

The changelog for the provider package can be found in the
`changelog <https://airflow.apache.org/docs/apache-airflow-providers-common-opendal/1.0.0/changelog.html>`_.
22 changes: 22 additions & 0 deletions providers/common/opendal/docs/changelog.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
``apache-airflow-providers-common-opendal``


1.0.0
.....
36 changes: 36 additions & 0 deletions providers/common/opendal/docs/commits.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@

.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
.. NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
.. IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
`PROVIDER_COMMITS_TEMPLATE.rst.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
.. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENERATED. IT WILL BE OVERWRITTEN!
Package apache-airflow-providers-common-opendal
------------------------------------------------------

`Apache OpenDAL <https://opendal.apache.org/>`__.
Apache OpenDAL is a Open Data Access Layer and theme is ``One Layer, All Storage.``


This is detailed commit list of changes for versions provider package: ``common-opendal``.
For high-level changelog, see :doc:`package information including changelog <index>`.

.. airflow-providers-commits::
27 changes: 27 additions & 0 deletions providers/common/opendal/docs/conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Disable Flake8 because of all the sphinx imports
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Configuration of Providers docs building."""

from __future__ import annotations

import os

os.environ["AIRFLOW_PACKAGE_NAME"] = "apache-airflow-providers-common-opendal"

from docs.provider_conf import * # noqa: F403
16 changes: 16 additions & 0 deletions providers/common/opendal/docs/connections/opendal.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
105 changes: 105 additions & 0 deletions providers/common/opendal/docs/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@

.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
``apache-airflow-providers-common-opendal``
=====================================================

.. toctree::
:hidden:
:maxdepth: 1
:caption: Basics

Home <self>
Changelog <changelog>
Security <security>

.. toctree::
:hidden:
:maxdepth: 1
:caption: Guides

Connection Types <connections/opendal>
Operators <operators/opendal>

.. toctree::
:hidden:
:maxdepth: 1
:caption: References

Python API <_api/airflow/providers/common/opendal/index>

.. toctree::
:hidden:
:maxdepth: 1
:caption: System tests

System Tests <_api/tests/system/common/opendal/index>

.. toctree::
:hidden:
:maxdepth: 1
:caption: Resources

Example DAGs <https://github.com/apache/airflow/tree/providers-common-opendal/|version|/providers/common/opendal/tests/system/opendal>
PyPI Repository <https://pypi.org/project/apache-airflow-providers-common-opendal/>
Installing from sources <installing-providers-from-sources>

.. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENERATED. IT WILL BE OVERWRITTEN AT RELEASE TIME!
.. toctree::
:hidden:
:maxdepth: 1
:caption: Commits

Detailed list of commits <commits>


apache-airflow-providers-common-opendal package
------------------------------------------------------

`Apache OpenDAL <https://opendal.apache.org/>`_ is a Open Data Access Layer. it provides a abstraction on various
object storage systems, and key value stores to access data and manipulate data.


Release: 1.0.0

Provider package
----------------

This package is for the ``common.opendal`` provider.
All classes for this package are included in the ``airflow.providers.common.opendal`` python package.

Installation
------------

You can install this package on top of an existing Airflow 2 installation via
``pip install apache-airflow-providers-common-opendal``
For the minimum Airflow version supported, see ``Requirements`` below.

Requirements
------------

The minimum Apache Airflow version supported by this provider distribution is ``2.10.0``.

================== ==================
PIP package Version required
================== ==================
``apache-airflow`` ``>=2.10.0``
``opendal`` ``>=0.45.18``
================== ==================
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
.. include:: /../../../../devel-common/src/sphinx_exts/includes/installing-providers-from-sources.rst
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
65 changes: 65 additions & 0 deletions providers/common/opendal/docs/operators/opendal.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
Apache OpenDAL Operators
==========================

.. _howto/operator:`OpenDALTaskOperator`:

OpenDALTaskOperator
---------------

Executes OpenDALTaskOperator

For parameter definition take a look at :class:`~airflow.providers.common.opendal.operators.opendal.OpenDALTaskOperator`.

Using the operator
""""""""""""""""""

An example usage of the OpenDALTaskOperator with fs backend:

.. exampleinclude:: /../tests/system/common/opendal/example_opendal_operator.py
:language: python
:dedent: 4
:start-after: [START howto_read_operator_opendal]
:end-before: [END howto_read_operator_opendal]

.. exampleinclude:: /../tests/system/common/opendal/example_opendal_operator.py
:language: python
:dedent: 4
:start-after: [START howto_write_operator_opendal]
:end-before: [END howto_write_operator_opendal]

.. exampleinclude:: /../tests/system/common/opendal/example_opendal_operator.py
:language: python
:dedent: 4
:start-after: [START howto_copy_operator_opendal]
:end-before: [END howto_copy_operator_opendal]

.. exampleinclude:: /../tests/system/common/opendal/example_opendal_operator.py
:language: python
:dedent: 4
:start-after: [START howto_copy_from_aws_to_gcs]
:end-before: [END howto_copy_from_aws_to_gcs]



Reference
"""""""""

For further information, look at: https://opendal.apache.org/
Loading