Upgrade AWS SDK to 2.30; update tests to disable default integrity protection

[yashmayya]

• Recent dependabot AWS SDK upgrades have been failing in CI due to some failed assertions in S3PinotFSTest - Bump software.amazon.awssdk:bom from 2.29.52 to 2.30.4 #14907, Bump software.amazon.awssdk:bom from 2.29.52 to 2.30.0 #14825.
• The reason for this is the default integrity protection change - S3 default integrity protection change - version `2.30.0` aws/aws-sdk-java-v2#5802. This isn't supported by the S3 mock testcontainer library that we're using for testing.
• The solution is to disable the integrity protection in the test client (see S3Client#getObject response input streams contains more than it should aws/aws-sdk-java-v2#5819 (comment)).

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 63.72%. Comparing base (59551e4) to head (8836115).
Report is 1617 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #14909      +/-   ##
============================================
+ Coverage     61.75%   63.72%   +1.97%     
- Complexity      207     1469    +1262     
============================================
  Files          2436     2708     +272     
  Lines        133233   151423   +18190     
  Branches      20636    23376    +2740     
============================================
+ Hits          82274    96498   +14224     
- Misses        44911    47672    +2761     
- Partials       6048     7253    +1205     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (+99.99%)	⬆️
integration	100.00% <ø> (+99.99%)	⬆️
integration1	100.00% <ø> (+99.99%)	⬆️
integration2	0.00% <ø> (ø)
java-11	63.70% <ø> (+1.99%)	⬆️
java-21	63.61% <ø> (+1.99%)	⬆️
skip-bytebuffers-false	63.72% <ø> (+1.97%)	⬆️
skip-bytebuffers-true	63.59% <ø> (+35.87%)	⬆️
temurin	63.72% <ø> (+1.97%)	⬆️
unittests	63.72% <ø> (+1.97%)	⬆️
unittests1	56.28% <ø> (+9.38%)	⬆️
unittests2	34.03% <ø> (+6.30%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Jackie-Jiang]

Thanks for fixing this! Can you help also double check if the usage of S3Client in S3PinotFS won't be affected?

[yashmayya]

This is only an issue when the new client is used with an S3 server that doesn't support this data integrity protection feature (https://aws.amazon.com/blogs/aws/introducing-default-data-integrity-protections-for-new-objects-in-amazon-s3/). So this won't be a problem for users using it with Amazon S3 itself. Do we recommend / support using this same S3PinotFS class with other S3 compatible object stores (that may or may not support these new default data integrity protections)?

[Jackie-Jiang]

This is only an issue when the new client is used with an S3 server that doesn't support this data integrity protection feature (https://aws.amazon.com/blogs/aws/introducing-default-data-integrity-protections-for-new-objects-in-amazon-s3/). So this won't be a problem for users using it with Amazon S3 itself. Do we recommend / support using this same S3PinotFS class with other S3 compatible object stores (that may or may not support these new default data integrity protections)?

I don't think so. It is fine as long as it works with Amazon S3.