Use GitHub API to fetch Github artefacts in our scripts

[amoghrajesh]

Body

We've been frequently hitting GitHub rate limits in CI over the past few days. Upon investigation, it appears that while we do pass down the GITHUB_TOKEN to the breeze commands and related CI scripts, we don’t actually use the GitHub API. Instead, we rely on fetching files via raw.githubusercontent.com, which doesn't utilize the token and thus bypasses the mechanisms that would help avoid rate limiting.

Reference:
https://stackoverflow.com/questions/77054807/how-do-you-use-raw-githubusercontent-com-in-a-github-action#comment135840009_77054807

---

Shell Scripts referencing raw.githubusercontent.com

• get_devel_deps.sh - not needed
• [common.sh - we have no GitHub token during building images
• [entrypoint_ci.sh](https://github.com/apache/airflow/blob/main/scripts/docker/[entrypoint_ci.sh](https://github.com/apache/airflow/blob/main/scripts/docker/entrypoint_ci.sh)
• [restricted_environments.sh](https://github.com/apache/airflow/blob/main/docker-stack-docs/docker-examples/restricted/[restricted_environments.sh](https://github.com/apache/airflow/blob/main/docker-stack-docs/docker-examples/restricted/restricted_environments.sh) - this is just example we do not run it regularly

---

Python Files referencing raw.githubusercontent.com

• [vendor_k8s_json_schema.py](https://github.com/apache/airflow/blob/main/scripts/ci/pre_commit/[vendor_k8s_json_schema.py](https://github.com/apache/airflow/blob/main/scripts/ci/pre_commit/vendor_k8s_json_schema.py)
• [airflow_version_check.py](https://github.com/apache/airflow/blob/main/scripts/ci/[airflow_version_check.py](https://github.com/apache/airflow/blob/main/scripts/ci/airflow_version_check.py)
• [add_back_references.py](https://github.com/apache/airflow/blob/main/dev/breeze/src/airflow_breeze/utils/[add_back_references.py](https://github.com/apache/airflow/blob/main/dev/breeze/src/airflow_breeze/utils/add_back_references.py)
• [validate_version_added_fields_in_config.py](https://github.com/apache/airflow/blob/main/dev/[validate_version_added_fields_in_config.py](https://github.com/apache/airflow/blob/main/dev/validate_version_added_fields_in_config.py)
• [facets.py](https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/openlineage/facets.py)
• [discord.py](https://github.com/apache/airflow/blob/main/providers/discord/src/airflow/providers/discord/notifications/discord.py)
• [exampleinclude.py](https://github.com/apache/airflow/blob/main/devel-common/src/sphinx_exts/exampleinclude.py)
• [run_generate_constraints.py](https://github.com/apache/airflow/blob/main/scripts/in_container/[run_generate_constraints.py](https://github.com/apache/airflow/blob/main/scripts/in_container/run_generate_constraints.py)
• [build_prod_params.py](https://github.com/apache/airflow/blob/main/dev/breeze/src/airflow_breeze/params/[build_prod_params.py](https://github.com/apache/airflow/blob/main/dev/breeze/src/airflow_breeze/params/build_prod_params.py)
• [cdxgen.py](https://github.com/apache/airflow/blob/main/dev/breeze/src/airflow_breeze/utils/cdxgen.py)
• [common_build_params.py](https://github.com/apache/airflow/blob/main/dev/breeze/src/airflow_breeze/params/[common_build_params.py](https://github.com/apache/airflow/blob/main/dev/breeze/src/airflow_breeze/params/common_build_params.py)
• [install_airflow_and_providers.py](https://github.com/apache/airflow/blob/main/scripts/in_container/[install_airflow_and_providers.py](https://github.com/apache/airflow/blob/main/scripts/in_container/install_airflow_and_providers.py)
• [slack.py (Operator)](https://github.com/apache/airflow/blob/main/providers/slack/src/airflow/providers/slack/operators/slack.py)
• [release_candidate_command.py](https://github.com/apache/airflow/blob/main/dev/breeze/src/airflow_breeze/commands/[release_candidate_command.py](https://github.com/apache/airflow/blob/main/dev/breeze/src/airflow_breeze/commands/release_candidate_command.py)
• [get_provider_info.py](https://github.com/apache/airflow/blob/main/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/[get_provider_info.py](https://github.com/apache/airflow/blob/main/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/get_provider_info.py)
• [slack.py (Notification)](https://github.com/apache/airflow/blob/main/providers/slack/src/airflow/providers/slack/notifications/slack.py)
• helm_template_generator.py (unit)
• test_slack.py (notification unit)
• [example_slack_webhook.py](https://github.com/apache/airflow/blob/main/providers/slack/tests/system/slack/[example_slack_webhook.py](https://github.com/apache/airflow/blob/main/providers/slack/tests/system/slack/example_slack_webhook.py)
• [example_slack.py](https://github.com/apache/airflow/blob/main/providers/slack/tests/system/slack/[example_slack.py](https://github.com/apache/airflow/blob/main/providers/slack/tests/system/slack/example_slack.py)
• test_slack.py (operator unit)
• helm_template_generator.py (helm tests)

---

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[aritra24]

@amoghrajesh if you need a couple more hands helping move these I can pitch in.

[potiuk]

I don't think we need to "fix" all of those- I already reviewed and marked a few that should stay as they are.

[amoghrajesh]

Thank you @potiuk, it was a bulk fetch and I didn't triage them yet, thanks!