BlockCountExceedsLimit on azblob when uploading large files

[BaurzhanSakhariev]

Hi, I have a question about uploading large files (>5GB)

We use OpenDAL java binding 0.46.3 and getting “BlockCountExceedsLimit”, message: “The uncommitted block count cannot exceed the maximum limit of 100,000 blocks. (crate/crate#17136)

I couldn't create an isolated small test. Tried modifying BlockingWriteTest to repeat write 100K times and also played around with the size of the generated data but tests passes.

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-storage/blobs/connectivity/invalid-blob-or-block-content-or-invalid-block-list suggested to mitigate that by committing blocks (put block API)

Looks like it has been implemented already in #4181 and released in 0.45.1 so version we use (0.46.3) includes it.

Is it a bug or I'm missing some configuration parameter for azblob?

[Xuanwo]

Hi, it looks like we didn't expose the chunk API to java side:

opendal/bindings/java/src/operator_output_stream.rs

Lines 44 to 52 in e9df3eb

	fn intern_construct_write(
	env: &mut JNIEnv,
	op: &mut BlockingOperator,
	path: JString,
	) -> crate::Result<jlong> {
	let path = jstring_to_string(env, &path)?;
	let writer = op.writer(&path)?;
	Ok(Box::into_raw(Box::new(writer)) as jlong)
	}

Instead, we expose a very small chunk 16KiB:

opendal/bindings/java/src/main/java/org/apache/opendal/OperatorOutputStream.java

Lines 38 to 41 in e9df3eb

	private static final int MAX_BYTES = 16384;
	private final Writer writer;
	private final byte[] bytes = new byte[MAX_BYTES];

We will need to expose https://docs.rs/opendal/0.50.2/opendal/operator_functions/struct.FunctionWriter.html#method.chunk to java side so that you can configure it to 8MiB or 16MiB instead.

Also cc @tisonkun @G-XD for ideas.

[BaurzhanSakhariev]

Hi @Xuanwo, thanks for the pointers, looks like I can patch OperatorOutputStream locally as an interim solution.

I will give it a try and report back.

[BaurzhanSakhariev]

Opened a feature request: #5421