use of debugpy to run airflow components via breeze

[rawwar]

Using debugpy to run airflow components so that we can use IDE's as Vscode to attach to the component to debug.

Ports for components are as follows:

BREEZE_DEBUG_SCHEDULER_PORT = "50231"
BREEZE_DEBUG_DAG_PROCESSOR_PORT = "50232"
BREEZE_DEBUG_API_SERVER_PORT = "50233"
BREEZE_DEBUG_CELERY_WORKER_PORT = "50234"
BREEZE_DEBUG_EDGE_PORT = "50235"
BREEZE_DEBUG_FLOWER_PORT = "50236"

Env variables to use debugpy to run a component are below

BREEZE_DEBUG_SCHEDULER
BREEZE_DEBUG_APISERVER or BREEZE_DEBUG_WEBSERVER
BREEZE_DEBUG_TRIGGERER_PORT
BREEZE_DEBUG_CELERY_WORKER_PORT
BREEZE_DEBUG_FLOWER
BREEZE_DEBUG_EDGE
BREEZE_DEBUG_DAG_PROCESSOR

When using breeze, we can add above env to files/airflow-breeze-config/environment_variables.env.

For example, to run the scheduler via debugpy:

BREEZE_DEBUG_SCHEDULER=true

debugpy will be listening at port 50231. I use VS Code. So, I can add the following to the configuration in "launch.json" and start debugging:

{
            "name": "Scheduler Debug(Breeze)",
            "type": "debugpy",
            "request": "attach",
            "justMyCode": false,
            "connect": {
                "host": "localhost",
                "port": 50231,
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "/opt/airflow"
                }
            ]
        },

Once I start breeze, the Scheduler pane will be waiting as shown in the screenshot:

In VSCode, I can place an arbitrary debug point and start debugging and it will pause as shown below:

I am sure this can be done via PyCharm as well, but I haven't tested it yet.

[jscheffl]

Pretty cool!

Can you add something about this into the Contribution docs as well?
...and would it make sense adding this to breeze CLI as command line instead of ENV?

[rawwar]

Pretty cool!

Can you add something about this into the Contribution docs as well? ...and would it make sense adding this to breeze CLI as command line instead of ENV?

Yes. I'm adding CLI support as well. And, I'll update docs.

[potiuk]

Pretty cool!

Can you add something about this into the Contribution docs as well? ...and would it make sense adding this to breeze CLI as command line instead of ENV?

Very much agree with Jens here .... and that's nice. I usually use intelliJ's debug server, there, it works a bit differently (you need to add debug code to connect to server running in IntelliJ) - but maybe we could do something similar.

[rawwar]

Very much agree with Jens here .... and that's nice. I usually use intelliJ's debug server, there, it works a bit differently (you need to add debug code to connect to server running in IntelliJ) - but maybe we could do something similar.

I just looked into how to do this in IntelliJ and yeah, its different. No option to directly connect to debugpy. But, I'll look into it and add something similar