Checksum is getting included in the file content while using the self hosted flash storage "Flash Blade S200 NVMe 4.5.7"

[snowsagarshirsath]

Describe the bug

Hello,

We have self hosted s3 storage with pure flash blades with Version "Flash Blade S200 NVMe 4.5.7".

There is a Java Program which uses the software.amazon.awssdk.s3
https://mvnrepository.com/artifact/software.amazon.awssdk/s3

While uploading any file through this client s3, the checksum or payload of file is getting included in the file content, currently we have used 2.29.52 version and excluded the checksum which works fine, but later versions does not work, looks like adding checksum is enforced somewhere, this leads to file corruption issue.

Sample code:

        putObj = PutObjectRequest.builder()
                .bucket(S3ClientSingleton.bucket)

// .checksumAlgorithm(ChecksumAlgorithm.SHA256)
.key(String.valueOf(destinationS3Path))
.metadata(metadata)
.build();

        return s3client.putObject(putObj, AsyncRequestBody.fromFile(sourceFilePath)).whenComplete((response, error) -> {
            stopWatch.stop();
            long timeTaken = stopWatch.getTime();

            if (error != null) {
                fLogger.severe(() -> String.format("Failed to upload file to S3 %s, error: %s",
                        destinationS3Path, error.getMessage()));
                throw new CompletionException(error);
            } else {
                fLogger.info(() -> String.format(
                        "Uploaded Artifact to S3 Finished, s3_upload_time_taken=%dms, file: %s etag: %s",
                        timeTaken,
                        sourceFilePath,
                        response.eTag()
                ));
            }
        });

Please help us to get it fixed or find the root cause.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

With the latest aws s3 sdk the checksum should not get included in the file content. and file should not get corrupted

Current Behavior

File getting corrupted

Reproduction Steps

Deploy S3 server storage with Pure storage "Flash Blade S200 NVMe 4.5.7"
Use client https://mvnrepository.com/artifact/software.amazon.awssdk/s3/2.31.64

Possible Solution

Support to be included for flash storage in client s3

Additional Information/Context

No response

AWS Java SDK version used

2.31.64

JDK version used

17.0

Operating System and version

Mac os 15.5 (24F74)

[debora-ito]

In SDK version 2.30.0 the default checksum behavior changed, for more info check the AWS SDK Reference Guide about the checksum configuration, the announcement made at the time and the blog post that S3 released with Python SDK examples.

To disable the checksum calculation you need to set the config AWS_REQUEST_CHECKSUM_CALCULATION to "WHEN_REQUIRED", see code examples here: #5802 (comment)

QUESTION: When you say "File getting corrupted", what do you mean exactly? The file gets uploaded with no errors, but the file is corrupted?

[snowsagarshirsath]

Thanks, Debora
Yes files are getting upload fine but gets corrupted, i have attached the files for your reference.

Book1.xlsx - Original file, md5 checksum d6ffa46104a847dd378b9d95c3dcef24

Book1-Corrupted.xlsx - Uploaded to S3, md5 checksum b7e673ae8c5218d20a870c9f4b16d198

[debora-ito]

@snowsagarshirsath

How are you checking the md5 after the upload?
Can you share a self-contained code sample we can use to reproduce? Ideally a code sample we can simply copy+paste and run on our side.

[snowsagarshirsath]

After upload i am getting etag in the response.
Here is my code,

`import org.apache.commons.lang3.time.StopWatch;
import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectResponse;

import java.nio.file.Path;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.*;
import java.util.logging.Logger;

public class FileUploadTask{
private static final Logger fLogger = Logger.getLogger(FileUploadTask.class.getName());

private final Path sourceFilePath;
private final Path destinationS3Path;

public FileUploadTask(Path sourceFilePath, Path destinationS3Path) {
    this.sourceFilePath = sourceFilePath;
    this.destinationS3Path = destinationS3Path;
}

public CompletableFuture<PutObjectResponse> upload() throws Exception {
        StopWatch stopWatch = new StopWatch();
        stopWatch.start();
        S3AsyncClient s3client = S3ClientSingleton.getAsyncInstance();
        Map<String, String> metadata = new HashMap<>();
        metadata.put("author", "MLPredictor");
        PutObjectRequest putObj;

        putObj = PutObjectRequest.builder()
                .bucket(S3ClientSingleton.BUCKET)

// .checksumAlgorithm(ChecksumAlgorithm.SHA256)
.key(String.valueOf(destinationS3Path))
.metadata(metadata)
.build();

        return s3client.putObject(putObj, AsyncRequestBody.fromFile(sourceFilePath)).whenComplete((response, error) -> {
            stopWatch.stop();
            long timeTaken = stopWatch.getTime();

            if (error != null) {
                fLogger.severe(() -> String.format("Uploaded Artifact to S3 Failed time_take=%d ms, file: %s, error: %s",
                        timeTaken, destinationS3Path, error.getMessage()));
                throw new CompletionException(error);
            } else {
                fLogger.info(() -> String.format(
                        "Uploaded Artifact to S3 Finished, s3_upload_time_taken=%d ms, file: %s etag: %s",
                        timeTaken,
                        sourceFilePath,
                        response.eTag()
                ));
            }
        });
}
public static void main(String[] args) throws Exception {
    FileUploadTask task = new FileUploadTask(Path.of("<LocalFIlePath>"), Path.of("<S3Path>"));
    task.upload().join();
}

}`