Fatal error condition in s3_checksum_stream aws_base64_encode

[ScepticMatt]

Mountpoint for Amazon S3 version

1.10.0

AWS Region

uk.io.cloud.ovh.net

Describe the running environment

We observed a crash while using the mount-s3 tool with the OVH S3-compatible endpoint (https://s3.uk.io.cloud.ovh.net). While OVH endpoints are not officially supported, the behavior might highlight a broader issue affecting other configurations as well. Below are the details:

1. Multiple Rename Operations:

   • Several rename operations were attempted over a span of time, all failing with the following warning:

     rename failed: operation not supported by Mountpoint
     

   • These failures occurred consistently but did not immediately cause the service to terminate.

2. Checksum-Related Fatal Crash:

   • Following the last rename operation, the service crashed with a checksum-related error:

     Fatal error condition occurred in /workplace/mountpoint-s3/mountpoint-s3-crt-sys/crt/aws-c-s3/source/s3_checksum_stream.c:38:
     aws_base64_encode(&checksum_result_cursor, impl->encoded_checksum_output) == AWS_OP_SUCCESS
     

   • It is unclear if the crash is directly related to the repeated rename operations or an independent checksum issue triggered afterward.

3. Checksum Uploads Disabled:

   • The --upload-checksums=off flag was used because checksum uploads are not compatible with OVH endpoints. We did not test the behavior with checksum uploads enabled.

This sequence suggests the possibility of an underlying issue in how the tool handles unsupported operations or subsequent checksum processing. We hope this information helps in evaluating the behavior and identifying any potential improvements for robustness. Thank you!

Mountpoint options

ExecStart=/usr/bin/sudo /usr/bin/mount-s3 <removed> /mnt/o0 \
    --endpoint-url=https://s3.uk.io.cloud.ovh.net \
    --force-path-style \
    --upload-checksums=off \
    --storage-class=STANDARD \
    --bucket-type=general-purpose \
    --allow-delete \
    --allow-overwrite \
    --allow-other \
    --uid=1000 \
    --gid=1000 \
    --file-mode=0666 \
    --dir-mode=0777 \
    --maximum-throughput-gbps=1 \
    --write-part-size 100000000 \
    --foreground

What happened?

Fatal error condition occurred in /workplace/mountpoint-s3/mountpoint-s3-crt-sys/crt/aws-c-s3/source/s3_checksum_stream.c:38

Relevant log output

Nov 20 17:54:20 rsch-0 sudo[1520250]: Fatal error condition occurred in /workplace/mountpoint-s3/mountpoint-s3-crt-sys/crt/aws-c-s3/source/s3_checksum_stream.c:38: aws_base64_encode(&checksum_result_cursor, impl->encoded_checksum_output) == AWS_OP_SUCCESS
Nov 20 17:54:20 rsch-0 sudo[1520250]: Exiting Application
Nov 20 17:54:20 rsch-0 sudo[1520250]: ################################################################################
Nov 20 17:54:20 rsch-0 sudo[1520250]: Stack trace:
Nov 20 17:54:20 rsch-0 sudo[1520250]: ################################################################################
Nov 20 17:54:20 rsch-0 sudo[1520250]: /usr/bin/mount-s3(+0x4e573f) [0x5c48080f873f]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /usr/bin/mount-s3(+0x4e26d4) [0x5c48080f56d4]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /usr/bin/mount-s3(+0x59edff) [0x5c48081b1dff]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /usr/bin/mount-s3(+0x59ee90) [0x5c48081b1e90]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /usr/bin/mount-s3(+0x4f809f) [0x5c480810b09f]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /usr/bin/mount-s3(+0x57dad8) [0x5c4808190ad8]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /usr/bin/mount-s3(+0x57ddae) [0x5c4808190dae]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /usr/bin/mount-s3(+0x57eda5) [0x5c4808191da5]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /usr/bin/mount-s3(+0x5796a4) [0x5c480818c6a4]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /usr/bin/mount-s3(+0x4e6aa2) [0x5c48080f9aa2]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /usr/bin/mount-s3(+0x4eeb44) [0x5c4808101b44]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /usr/bin/mount-s3(+0x56ac72) [0x5c480817dc72]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /lib/x86_64-linux-gnu/libc.so.6(+0x9ca94) [0x7e11d509ca94]
Nov 20 17:54:20 rsch-0 sudo[1520250]: /lib/x86_64-linux-gnu/libc.so.6(+0x129c3c) [0x7e11d5129c3c]
Nov 20 17:55:16 rsch-0 sudo[1520248]: pam_unix(sudo:session): session closed for user root

[dannycjones]

Hey @ScepticMatt, thanks for providing so much detail here.

We've seen this issue once before failing a CI run but it appears very difficult to reproduce. That being said, there does appear to be an issue here that we should find the root cause of and address it.

We'll let you know when we have more to share here.

[dannycjones]

Given the other case we saw didn't include rename requests, I'll update the issue title.