Faster rendering

[akre54]

Hubble depends on the WebMWriter library to record webm video within the WebMEncoder class.

WebMWriter uses canvas.toDataURL under the hood to serialize the canvas state as a base64-encoded string, which it later turns into a blob with the atob function. In my exploration this is the major cause of latency in recording with Hubble.

I did some testing with the more recent canvas.toBlob API with massive speedup improvements, but it seems like the webm-wasm library is even better suited to this task. See this example for how it could work in practice. I also looked into the MediaRecorder API (which Hubble exposes as StreamEncoder) but it seems to only support realtime stream capture (even though the stream API supports manual frame additions).

Further gains might be made with an OffscreenCanvas (which deck appears to support) to keep everything off the main thread, though my theory is that simply replacing the renderer will be sufficient.

[chrisgervang]

Nice find. Did some reading up on the spec, since toBlob wasn't an option I knew about when first implementing the encoders. toBlob is probably a better option for all the other encoders currently using toDataUrl, and webm-wasm best for webm.

I think the loaders.gl video module would be the best place we have to implement webm-wasm, since it facilitates packaging everything related to workers and standardizes the api about blobs.

Tangentially related, Ib Green and I previously worked together on the "builder"-style interface used by hubble encoders for this kind of frame-by-frame writer, and demonstrated it in GIFBuilder within loaders.gl. Hubble still uses its own gif builder due to a minor issue, but it's appealing to me to add more builder-style writers into loader.gl.

[akre54]

Awesome. I've only really used loaders.gl for decoding but if that's the right place to add it let's do it. The Hubble encoder would wrap loaders.gl's video encoder then?

Until the browser's MediaRecorder API supports adding frames async, it seems like this is something that needs to be handled in userland. And webm-wasm + getImageData seems to be the best and fastest for our use cases.

If it's good with you and Ib, I'm happy to start an rfc or strawman pull request against Loaders

[chrisgervang]

It's good with me, and I'm sure Ib as well. Yeah, I imagine the Hubble encoder would wrap a loaders builder.

We never covered the builder design with an RFC yet - just a small mention of them in the encoding RFC, so some design documentation would certainly add value.

[akre54]

Okay I started a branch as a proof-of-concept but running into some issues with webpack and loading wasm. I might want your eyes sometime next week if I can't power through it.