Proposal for a new architecture of image rendering pipeline

[marty-wang]

(Note: This has already been discussed with Microsoft team. Post here for public awareness and feedback.)

Context

We are facing the following issues:

• Cached images ares still rendered asynchronously, which makes them appear on the screen a few frames later than adjacent UI.
• High cpu usage for animated images, such as gifs
• Lack of extensibility of decoding other image types, e.g. webp

The current implementation uses BitmapImage as the image source, which internally takes care of the loading, decoding and rendering of the images, and the caller of BitmapImage has no control over any of these steps.

Proposal

Similarly like what is done on the macOS side, we propose to build a pipeline to handle the loading, decoding and rendering all by ourselves, like the following diagram. The loading step manages the queue of loading images from different sources, and then hand the raw image data over to the decoding step, which takes care of the decoding of common image types, but can also allow app developers to inject their own decoders for any image types, including the common ones, and then cache the decoded image data (SoftwareBitmap) into a memory cache, with sensible eviction policy, and at last render the decoded image data synchronously.

For animated images, such as gifs, we can use keyframe animations to render them on UI.Composition layer, which I believe is hardware accelerated and effectively brings down the CPU usage to near 0. (This is how we did for the macOS side and it works really well).