S3Client.deleteObjects doesn't work in 2.30.0

[wendigo]

Describe the bug

deleteObjects now fails with Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Missing required header for this request: Content-Md5. (Service: S3, Status Code: 400, Request ID: 181B4664E1D4B169, Extended Request ID: 39697cb5cac07bf2a227e0cd44c7b2e9184e7598e2abe41c55eb5028b44676ca) on Minio and other S3-compatible storages, while deleteObject with a single location works just fine.

This fails regardless of the:

.responseChecksumValidation()
.requestChecksumCalculation()

set to WHEN_REQUIRED. WHEN_REQUIRED seems to work fine for other API calls but not for deleteObjects.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I'd expect deleteObjects to work the same as in the 2.29.x.

Current Behavior

For deleteObjects:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Missing required header for this request: Content-Md5. (Service: S3, Status Code: 400, Request ID: 181B46929A850D93, Extended Request ID: 39697cb5cac07bf2a227e0cd44c7b2e9184e7598e2abe41c55eb5028b44676ca)

Since I'm passing list of locations, I'm not sure why the Content-Md5 would be required.

For deleteObject: passes.

Reproduction Steps

Batch delete files stored on Minio. Single delete works fine.

Possible Solution

Downgrade to 2.29.x

Additional Information/Context

No response

AWS Java SDK version used

2.30.0

JDK version used

23

Operating System and version

Linux

[zoewangg]

Taking a look

[debora-ito]

Hi @wendigo, we tested deleteObjects with the aws s3 endpoint and it succeeds with the WHEN_REQUIRED setting.

My guess is that the issue is on the Minio/third-party side, they have to accommodate the changes released in 2.30.0. If the issue is indeed on the third-party side, I can't really explain why deleteObject works fine.

One workaround is to pin the Java SDK version to 2.29.52 until Minio adds support for the change.

[wendigo]

@debora-ito why the deleteObject works while deleteObjects doesn't? This doesn't make much sense

[debora-ito]

The exception is being returned by Minio, the SDK is only unmarshalling the response. There's probably a difference in how Minio implemented the s3 compatibility internally.

[lucix-aws]

DeleteObjects has always required a checksum, before today SDKs would compute an MD5 one for these requests. After today's object integrity change, SDKs now send a CRC32 checksum instead. Minio's implementation still appears to expect MD5.

DeleteObject singular optionally used checksums before, all the SDKs changed in that regard is sending one by default now.

[henricook]

To save other readers the trip over to the Minio issues board, they have not yet decided whether they're going to adapt to this change or not. So there's no timeline on resolution.

My personal opinion is that obviously they'll be forced to do it eventually if they want to stay current and keep calling themselves S3-compatible, unless the AWS team roll this change back.

Until they decide on a timeline I'm forced to not upgrade to 2.30.x.

[henricook]

Update: Minio have said they'll make the change (thanks Minio team!). No timeline yet but it does sound like it's mostly a cherry-pick.

minio/minio#20845 (comment)

[debora-ito]

Thank you for keeping us posted @henricook.

Minio released support for the S3 checksum changes, so I'll go ahead and close this.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.