Specify API for `StreamingBody`

[jakkdl]

botocore.StreamingBody is a wrapper class for urllib3.response.HTTPResponse, which explicitly defines an API that exposes some of the underlying methods and adds additional ones on top. It does not otherwise forward attribute calls to the underlying HTTPResponse afaict.

aiobotocore.StreamingBody uses wrapt.ObjectProxy to wrap aiohttp.ClientResponse, which means it forwards all underlying methods, and additionally specifies most of the methods that botocore.StreamingBody does. This means it has a larger API, but one that is tightly tied to aiohttp.ClientResponse.

The soon-to-be-added aiobotocore.HttpxStreamingbody is in a tough position. It currently uses wrapt.ObjectProxy to wrap httpx.Response, but httpx.Response differs against both of the former in that it requires you to specify the chunk size up-front, so unless we write a complicated helper method we cannot expose a read method that takes a parameter to specify the number of bytes to read.

There's also differences in e.g. aiohttp.ClientResponse exposing a sync close() method, which httpx.Response doesn't.

It's all kind of a mess at the moment, and if we want the aiobotocore API of [Httpx]StreamingBody to be library-agnostic between aiohttp & httpx we'll likely need to stop using wrap.ObjectProxy and explicitly define the API.
We then have to decide whether to remove functionality, write complicated functions to translate between them, petition httpx to implement more functionality, make it possible to directly access the underlying stream when users do want to opt into library-specific functionality, and/or raise helpful error messages when something breaks because the backend changed.

See #1085 (comment) and #1085 (comment) for previous discussion.

[jakob-keller]

My initial thought is that we should implement aiobotocore.AioStreamingBody as a subclass of botocore.StreamingBody and patch any methods that are blocking and/or rely on urllib3 functionality. The basic API would be determined by upstream, but with asynchronous implementations where required, which is our standard approach.

That basic API could be extended by:

1. internal adapter interfaces that abstract away aiohttp and httpx specifics. This simplifies calling codepaths in aiobotocore.
2. legacy methods which have been part of the old aiobotocore.StreamingBody, but would otherwise be removed. This improves backwards compatibility for users.
3. a public property (.raw_stream?) that gives access to the wrapped response. For advanced users or libraries building on top of aiobotocore.

Does that make any sense?

[jakkdl]

yeah I think that's the sensible way of going about it.

The biggest problem in practice is going to be writing the adapter class to make httpx.Response support everything that the aiohttp interface has allowed doing. #1085 (comment) is ~100 lines of fairly complex code (and I have not tried understanding it, nor trust the current LLM's to implement that well), because httpx wants you to specify the chunk size up front. Wholly banning read(amt=X) is perhaps bad, but with some state handling we can maybe add some limitations that makes it play nice with httpx.Response

for close vs aclose we'll either have to fully deprecate StreamingBody.close, or make it error on httpx.

All the StreamingBody tests don't currently run against httpx, and I haven't checked the full API of aiohttp.ClientResponse so there might be more nasty surprises.

[jakkdl]

I don't think we gain much, if anything, from implementing it as a subclass of botocore.StreamingBody. There's a bunch of methods (mostly dunders) that aiobotocore.StreamingBody shouldn't have, and there's a ton of methods that go from sync to async, leaving only readable, tell and _verify_content_length as having the same signature - and for all of them the implementation will differ.

[jakkdl]

As to verifying content length, the url in the comment is broken, but should be psf/requests#1855

But as opposed to requests, both aiohttp and httpx appear to verify content length. Although these are semi-unrelated/old issues and I haven't dug into the source code to confirm:
aio-libs/aiohttp#7208 (comment)
encode/httpx#643

so I think those parts of StreamReader can be scrapped as well.