Detected blocking call inside the event loop

[chemelli74]

Describe the bug
Home Assitant recently added the ability to detect I/O function that block event loop:

2024-06-12 09:40:41.237 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to listdir inside the event loop by integration 'aws'
at homeassistant/components/aws/notify.py, line 38: return await session.get_available_regions(service) (offender: /usr/local/lib/python3.12/site-pa
ckages/botocore/loaders.py, line 311: api_versions = os.listdir(full_dirname)), please create a bug report at https://github.com/home-assistant/core
/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+aws%22

After trying a first fix in #1120, I was redirected to botocore.
I opened a issue there: boto/botocore#3222, but was closed as botocore doesn't use event loop at all.
After some more discussions in the original PR, a new idea was raised and I applied it to a new fix in #1196.
Lately I was asked to open a issue to discuss the approach.

Checklist

• I have reproduced in environment where pip check passes without errors
• I have provided pip freeze results
• I have provided sample code or detailed way to reproduce
• I have tried the same code in botocore to ensure this is an aiobotocore specific issue
• I have tried similar code in aiohttp to ensure this is is an aiobotocore specific issue
• I have checked the latest and older versions of aiobotocore/aiohttp/python to see if this is a regression / injection

pip freeze results
pip_freeze.txt

Environment:

• Python Version: 3.12.4
• OS name and version: alpine-release 3.20.1

Additional context
Add any other context about the problem here.

[jakob-keller]

Thank you for opening this issue as well as your patience!

Let me try to summarize my current understanding based upon the history referenced and my own investigation into the botocore loaders module. I also have a few clarifying questions for you.

Is there an issue?

I understand that there is a warning by Home Assistant about blocking calls in its aws component. The warning appears to be correct in the sense that botocore, and by extension aiobotocore, uses file I/O to lazy load the data models needed to do its job.

What practical issues arise besides the warning in home assistant. Are there any negative effects?

What can be done?

If there are even any actual issues, mitigations or fixes may potentially be applied at Home Assistant, botocore and/or aiobotocore.

Home Assistant

@thehesiod has provided suggestions on pre-warming the model cache that could be implemented by Home Assistant.

Has this been attempted? What other steps has Home Assistant taken to mitigate negative effects, if any? Does Home Assistant use long-living sessions or are they frequently recreated?

botocore

As @thehesiod stated, issues arising from botocore loaders are caused by their design, i.e. their reliance on file I/O and a large number of JSON files to be processed. The botocore teams seems to be aware of the situation, attempted optimisations in the past, but have not shared any plans on further redesigning that aspect of botocore. We can assume that botocore loaders will stay as they are for the foreseeable future.

aiobotocore

We already touched upon a few things that could be done by aiobotocore, including:

1. Async loaders
   The purpose of aiobotocore is to allow for async calls to AWS services. We currently do not provide async implementations of botocore's loaders. This might be feasible and I have started to work in that direction. If successful, that would provide a path to non-blocking loaders. However, I do not want the raise any expectations and cannot make any promises yet.

2. Surgical patches
   There are also more surgical patches that might work around particular blocking codepaths, such as the proposals in Fix detected blocking call inside the event loop #1120 and Load botocore i/o methods in executor #1196. As you have noted, @thehesiod is sceptical about resorting to threads / the default executor, and I have voiced concern with regards to drift they introduce and the added complexity we need to deal with going forward. Also, these patches likely conflict with 1. while not fixing blocking file I/O in the vast majority of cases (paginators, waiters, ...). I would like to exhaust other options, before considering this path.

I hope this is a fair summary of the discussion so far and look forward to your responses.

[jakob-keller]

1. Async loaders
   The purpose of aiobotocore is to allow for async calls to AWS services. We currently do not provide async implementations of botocore's loaders. This might be feasible and I have started to work in that direction. If successful, that would provide a path to non-blocking loaders. However, I do not want the raise any expectations and cannot make any promises yet.

FYI, I am no longer working on an asynchronous loaders patch, since it turned out to be too complicated and not worth the effort IMO.