My application stopped working after upgrading aiobotocore from version 2.17.0 to 2.21.0. If someone could help ?

[EtceterisCDev]

Introduction

It's my first post here.
So please bear with me; I'll do my best to be clear.

Context

We use (and we appreciate a lot) aiobotocore on our application for more than 2 years.

The main components of our solution are :

• python 3.12.6
• FastAPI
• Pydantic

We have done an update of our requirements.txt from aiobotocore 2.17.0 to 2.21.1.

In fact, the break changes appear from aiobotocore 2.17.0 to 2.18.0.

With this update, our put instruction fails.
If we just change the requirements.txt and then go back in 2.17.0, everything is working fine again.

The part where it break is here :

                case BaseModel() | dict() | list():
                    if isinstance(object,dict) or isinstance(object,list):
                        print("Traitement demandé : dict ou list")
                        out_datas = json.dumps(object)
                    else:
                        print("Traitement demandé : BaseModel")
                        out_datas = object.model_dump_json()

                    buffer_io = BytesIO(out_datas.encode('utf-8'))
                    buffer_io.seek(0)

                    try:
                        res = await s3_client.put_object(
                                                    Bucket=object_definition.object_base, 
                                                    Key=s3_object_global_name, 
                                                    Body=buffer_io.getvalue(),
                                                    ContentType=s3_content_type,
                                                )
                        print(f"Retour sur l'appel S3 put_object {res}")

                        # Vérifiez si l'objet existe
                        res_head = await s3_client.head_object(
                            Bucket=object_definition.object_base, 
                            Key=s3_object_global_name
                        )
                        print("Informations sur l'objet après upload :", res_head)
                    except Exception as e:
                        print("DÉBUT DEBUG put_object ---")
                        print("Bucket S3 :", object_definition.object_base)
                        print("Clé S3    :", s3_object_global_name)
                        print("ContentType:", s3_content_type)
                        print("Données envoyées (JSON):", out_datas)
                        print("Longueur données envoyées (bytes):", len(buffer_io.getvalue()))
                        print("FIN DEBUG ---")

                        texte_erreur = f"Erreur driver S3 - Opération put_object - Bucket={object_definition.object_base} - Key={s3_object_global_name} - ContentType={s3_content_type}"
                        raise ValueError(f"{texte_erreur}\n{e}")

The error message is:

An error occurred (InvalidArgument) when calling the PutObject operation: Invalid Argument.

I read the release notes without finding nothing.

Is there anybody able to help me ?
Have i miss something ?

Thanks a lot for your help in advance
Christian

[thehesiod]

of critical importance is if your botocore matches our requirements in each version

[EtceterisCDev]

Thanks for the advice!
In my requirements.txt, there is only mention on aiobotocore.
I use virtual env.

And i check it with pip freeze.

WHen i change from aiobotocore version, it automatically brings the appropriate botocore version

[jakob-keller]

Salut and thank you for reporting the issue!

aiobotocore 1.18.0 included support for botocore 1.36.x, which introduced several changes related to S3, in particular with regards to checksums:
https://github.com/boto/botocore/blob/66445e9364aa6e42e5b69733d3952d1107a4bc51/CHANGELOG.rst#L15-L27

Are you working with third-party S3-compatible object storage services, i.e. not AWS S3 buckets? Many users of such services have reported issues with their applications recently.

Another possibility is that your application somehow sets custom configuration that needs to be adjusted to be compatible with botocore >=1.36.0 and thus aiobotocore >= 1.18.0.

[EtceterisCDev]

Thanks for your quick reply!

Yes, I'm working with OVH's S3 service based on OpenStack.

I'm specifically using these parameters in my configuration:

• region
• endpoint
• aws_access_key
• aws_secret_key

I'll further investigate the two points ⚠️ mentioned in your suggestions, related to "feature".

1.36.0
======

* api-change:``apigateway``: Documentation updates for Amazon API Gateway
* api-change:``bedrock-agent-runtime``: Now supports streaming for inline agents.
* api-change:``cognito-identity``: corrects the dual-stack endpoint configuration
* api-change:``partnercentral-selling``: Add Tagging support for ResourceSnapshotJob resources
* api-change:``s3``: This change enhances integrity protections for new SDK requests to S3. S3 SDKs now support the CRC64NVME checksum algorithm, full object checksums for multipart S3 objects, and new default integrity protections for S3 requests.
* api-change:``security-ir``: Increase minimum length of Threat Actor IP 'userAgent' to 1.
* api-change:``sesv2``: This release introduces a new recommendation in Virtual Deliverability Manager Advisor, which detects elevated complaint rates for customer sending identities.
* api-change:``workspaces``: Added GeneralPurpose.4xlarge & GeneralPurpose.8xlarge ComputeTypes.
* api-change:``workspaces-thin-client``: Mark type in MaintenanceWindow as required.
* enhancement:AWSCRT: Update awscrt version to 0.23.4
* ⚠️feature:``s3``: The S3 client attempts to validate response checksums for all S3 API operations that support checksums. However, if the SDK has not implemented the specified checksum algorithm then this validation is skipped. Checksum validation behavior can be configured using the ``when_supported`` and ``when_required`` options - in code using the ``response_checksum_validation`` parameter for ``botocore.config.Config``, in the shared AWS config file using ``response_checksum_validation``, or as an env variable using ``AWS_RESPONSE_CHECKSUM_VALIDATION``.
* feature:``s3``: Added support for the CRC64NVME checksum algorithm in the S3 client through the optional AWS CRT (``awscrt``) dependency.
* ⚠️feature:``s3``: S3 client behavior is updated to always calculate a CRC32 checksum by default for operations that support it (such as PutObject or UploadPart), or require it (such as DeleteObjects). Checksum behavior can be configured using ``when_supported`` and ``when_required`` options - in code using the ``request_checksum_calculation`` parameter for ``botocore.config.Config``, in the shared AWS config file using ``request_checksum_calculation``, or as an env variable using ``AWS_REQUEST_CHECKSUM_CALCULATION``. Note: Botocore will no longer automatically compute and populate the Content-MD5 header.

[jakob-keller]

In that case, please take a look at boto/boto3#4398. This is not aiobotocore related.

[EtceterisCDev]

Thanks a lot

I have done these updates in my code:

• New object to import:

  from aiobotocore.config import AioConfig

• New constants

  S3_compatible_config = AioConfig(
      request_checksum_calculation="when_required",
      response_checksum_validation="when_required"
  )

• Adding a parameter to my create_client call:

          async with self.s3_session.create_client(
                                                  's3',
                                              region_name=self.DRIVER_SETTINGS.S3_REGION,
                                              endpoint_url=self.DRIVER_SETTINGS.S3_ENDPOINT_URL,
                                              aws_access_key_id=self.DRIVER_SETTINGS.S3_AWS_ACCESS_KEY_ID,
                                              aws_secret_access_key=self.DRIVER_SETTINGS.S3_AWS_SECRET_ACCESS_KEY,
                                              use_ssl=self.DRIVER_SETTINGS.S3_USE_SSL,
                                              config=S3_compatible_config,
                                          ) as s3_client:

And now it's working with the last aiobotocore version !!!