Skia useImage fails without error

[AdamGerthel]

I'm currently replacing my old 9-slice button solution with Skia, but I'm having issue with useImage. Quite often, one or several of the images fail to load, but they do so without any error message. It happens on both iOS and Android, but more frequently on iOS.

Note: I should probably add that I'm primarily looking for the most performant way to create these buttons (in various sizes and colors). Perhaps there are better ways? Path effects? Atlas? Creating snapshots of pre-rendered button backgrounds and reusing them?

Skia.image.load.fail.mp4

The left sides of the buttons do not seem to fail, or at least not as often. I suspect it could have to do with the load that is caused. The left side image is the first one in the button component.

Another example.

Example.mp4

As you can see, whenever a button renders, there seems to be chance that it completely fails. This happens both in simulators and real devices and in both debug and release builds. Images are bundled assets, they're not loaded remotely. My previous implementation (using expo-image and regular views) was constructed in a similar fashion but never failed in image loading.

Button component

import type {
  StyleProp,
  ViewStyle,
  TextStyle,
  LayoutChangeEvent,
  TouchableOpacityProps,
  PressableProps,
  GestureResponderEvent,
  ColorValue,
  LayoutRectangle
} from 'react-native'
import React, { useState, useRef, useEffect, memo, useCallback } from 'react'
import { View, Pressable } from 'react-native'
import Animated, {
  useSharedValue,
  useAnimatedStyle,
  withTiming,
  Easing
} from 'react-native-reanimated'
import { Icon, type TIconProps, type TIconName } from '../icon/icon'
import { Text } from '../text/text'
import style from './button.style'
import ButtonBackground from './button-background/button-background'

const AnimatedPressable = Animated.createAnimatedComponent(Pressable)

const LONGPRESS_DELAY = 250
const ONPRESS_ANIMATION_DURATION = 100
const DISABLED_ANIMATION_DURATION = 0
const DEFAULT_ONLOAD_ANIMATION_DURATION = 0
const DEFAULT_OPACITY = 1
const ONPRESS_OPACITY = 0.3
const DISABLED_OPACITY = ONPRESS_OPACITY

type TButtonProps = TouchableOpacityProps & {
  label?: string
  size?: 'small' | 'mini' | 'regular'
  align?: 'right' | 'left'
  disabled?: boolean
  used?: boolean
  wrapperStyle?: StyleProp<ViewStyle>
  labelStyle?: StyleProp<TextStyle>
  icon?: TIconName
  iconProps?: Omit<TIconProps, 'name'>
  tint?: ColorValue
  onLoadAnimationDuration?: number
} & PressableProps

export const Button = memo(function Button({
  label,
  size = 'regular',
  align,
  disabled = false,
  used,
  wrapperStyle = {},
  labelStyle = {},
  icon,
  iconProps = {
    width: style.icon.width,
    height: style.icon.height
  },
  hitSlop,
  tint,
  onLoadAnimationDuration = DEFAULT_ONLOAD_ANIMATION_DURATION,
  onPressIn,
  onPressOut,
  ...rest
}: TButtonProps) {
  const withLabel = label !== undefined
  const [outerDimensions, setOuterDimensions] = useState<LayoutRectangle>()
  const onLayout = (event: LayoutChangeEvent) => {
    if (
      !outerDimensions ||
      outerDimensions.width !== event.nativeEvent.layout.width ||
      outerDimensions.height !== event.nativeEvent.layout.height
    ) {
      setOuterDimensions(event.nativeEvent.layout)
    }
  }
  const sideWidth = outerDimensions ? outerDimensions?.height * 1 : 0 // N is the ratio between the width and height of the side image
  const isReadyToDisplay = useRef(false)
  const animatedOpacity = useSharedValue<number>(0)
  const animatedStyles = useAnimatedStyle(() => ({ opacity: animatedOpacity.value }))
  const disabledValue = useRef(disabled)

  const fade = (toValue: number, duration: number) => {
    if (duration > 0) {
      animatedOpacity.value = withTiming(toValue, {
        duration,
        easing: Easing.inOut(Easing.ease)
      })
    } else {
      animatedOpacity.value = toValue
    }
  }

  useEffect(() => {
    disabledValue.current = disabled

    if (!isReadyToDisplay.current) {
      return
    }

    if (disabled) {
      fade(DISABLED_OPACITY, DISABLED_ANIMATION_DURATION)
    } else if (!disabled) {
      fade(DEFAULT_OPACITY, DISABLED_ANIMATION_DURATION)
    }
  }, [disabled])

  const onReadyToRender = useCallback(() => {
    isReadyToDisplay.current = true
    fade(disabledValue.current ? DISABLED_OPACITY : DEFAULT_OPACITY, onLoadAnimationDuration)
  }, [])

  const onPressInWrapper = (event: GestureResponderEvent) => {
    fade(ONPRESS_OPACITY, ONPRESS_ANIMATION_DURATION)

    if (onPressIn) {
      onPressIn(event)
    }
  }

  const onPressOutWrapper = (event: GestureResponderEvent) => {
    fade(disabledValue.current ? DISABLED_OPACITY : DEFAULT_OPACITY, ONPRESS_ANIMATION_DURATION)

    if (onPressOut) {
      onPressOut(event)
    }
  }

  const wrapperStyles = [
    style.outerWrapper,
    { minWidth: sideWidth * 2 },
    wrapperStyle,
    disabled && style.wrapperDisabled,
    !withLabel && icon ? style.wrapperIconOnly : undefined
  ]

  const labelStyles = [
    style.label,
    labelStyle,
    disabled && style.labelDisabled,
    used && style.labelUsed
  ]

  switch (align) {
    case 'right':
      wrapperStyles.push(style.wrapperRight)
      break
    case 'left':
      wrapperStyles.push(style.wrapperLeft)
      break
    default:
      break
  }

  let contentWrapperStyle = style.wrapper

  if (size === 'small') {
    labelStyles.push(style.labelSmall)
    iconProps.width = style.iconSmall.width
    iconProps.height = style.iconSmall.height
    contentWrapperStyle = style.wrapperSmall

    if (!withLabel && icon) {
      wrapperStyles.push(style.wrapperSmallIconOnly)
    }
  }

  if (size === 'mini') {
    labelStyles.push(style.labelMini)
    iconProps.width = style.iconMini.width
    iconProps.height = style.iconMini.height
    contentWrapperStyle = style.wrapperMini

    if (!withLabel && icon) {
      wrapperStyles.push(style.wrapperMiniIconOnly)
    }
  }

  return (
    <AnimatedPressable
      style={[wrapperStyles, animatedStyles]}
      onPressIn={onPressInWrapper}
      onPressOut={onPressOutWrapper}
      accessible
      accessibilityLabel={used ? `${label} (used)` : label}
      accessibilityState={{ disabled }}
      accessibilityRole="button"
      touchSoundDisabled={true}
      disabled={disabled}
      delayLongPress={LONGPRESS_DELAY}
      delayPressIn={0}
      onLayout={onLayout}
      {...rest}
    >
      <View style={contentWrapperStyle}>
        {withLabel && <Text style={labelStyles}>{label}</Text>}

        {icon && (
          <View style={[style.iconWrapper, !withLabel && style.iconWrapperIconOnly]}>
            <Icon {...iconProps} name={icon} />
          </View>
        )}
      </View>
      {outerDimensions && (
        <ButtonBackground
          height={outerDimensions.height}
          width={outerDimensions.width}
          sideWidth={sideWidth}
          tint={tint as string}
          onLoad={onReadyToRender}
        />
      )}
    </AnimatedPressable>
  )
})

Button Background component

import React, { useRef, memo } from 'react'
import { PixelRatio, StyleSheet } from 'react-native'
import { Canvas, Mask, Rect } from '@shopify/react-native-skia'
import { SkiaImage } from '../../skia-image/skia-image'

const lineBottom = require('~eldrum/assets/graphics/line-bottom.png')
const lineTop = require('~eldrum/assets/graphics/line-top.png')
const LINE_WIDTH = 1000

const LEFT_SIDES = [
  require('~eldrum/assets/graphics/button-left.png'),
  require('~eldrum/assets/graphics/button-left-2.png'),
  require('~eldrum/assets/graphics/button-left-3.png')
]

const RIGHT_SIDES = [
  require('~eldrum/assets/graphics/button-right.png'),
  require('~eldrum/assets/graphics/button-right-2.png'),
  require('~eldrum/assets/graphics/button-right-3.png')
]

function getSideVariant(number: number) {
  if (number < 0.3333) {
    return 0
  } else if (number < 0.6666) {
    return 1
  }

  return 2
}

const ButtonBackground = memo(function ButtonBackground({
  width,
  height,
  sideWidth,
  tint = '#000',
  onLoad
}: {
  width: number
  height: number
  sideWidth: number
  tint?: string | undefined
  onLoad?: () => void
}) {
  const lineThickness = PixelRatio.roundToNearestPixel(5)

  const randomNumberA = useRef(Math.random()).current
  const randomNumberB = useRef(Math.random()).current
  const topOffset = (randomNumberA * LINE_WIDTH) / 6
  const bottomOffset = (randomNumberB * LINE_WIDTH) / 6
  const sideLeftVariant = LEFT_SIDES[getSideVariant(randomNumberA)]
  const sideRightVariant = RIGHT_SIDES[getSideVariant(randomNumberB)]
  const roundedSideWidth = PixelRatio.roundToNearestPixel(sideWidth)
  const lineWidth = width - roundedSideWidth * 2
  const imagesLoaded = useRef([false, false, false, false]).current

  const handleLoad = (index: number) => () => {
    imagesLoaded[index] = true

    if (imagesLoaded.every(state => state)) {
      onLoad?.()
    }
  }

  return (
    <Canvas
      style={[
        style.buttonWrapper,
        {
          height: PixelRatio.roundToNearestPixel(height),
          width: PixelRatio.roundToNearestPixel(width)
        }
      ]}
    >
      <Rect
        x={sideWidth}
        y={lineThickness}
        height={height - lineThickness * 2}
        width={width - sideWidth * 2}
        color={tint}
      />

      <SkiaImage
        source={sideLeftVariant}
        x={0}
        y={0}
        tint={tint}
        height={height}
        width={roundedSideWidth}
        fit="fitWidth"
        onLoad={handleLoad(0)}
      />

      <Mask
        mode="alpha"
        mask={
          <Rect color={tint} x={roundedSideWidth} y={0} width={lineWidth} height={lineThickness} />
        }
      >
        <SkiaImage
          source={lineTop}
          x={roundedSideWidth}
          y={0}
          tint={tint}
          height={lineThickness}
          width={lineWidth + topOffset}
          fit="fitHeight"
          onLoad={handleLoad(1)}
        />
      </Mask>

      <Mask
        mode="alpha"
        mask={
          <Rect
            color={tint}
            x={roundedSideWidth}
            y={height - lineThickness}
            width={lineWidth}
            height={lineThickness}
          />
        }
      >
        <SkiaImage
          source={lineBottom}
          x={roundedSideWidth}
          y={height - lineThickness}
          tint={tint}
          height={lineThickness}
          width={lineWidth + bottomOffset}
          fit="fitHeight"
          onLoad={handleLoad(2)}
        />
      </Mask>

      <SkiaImage
        source={sideRightVariant}
        x={width - roundedSideWidth}
        y={0}
        tint={tint}
        height={height}
        width={roundedSideWidth}
        fit="fitWidth"
        onLoad={handleLoad(3)}
      />
    </Canvas>
  )
})

const style = StyleSheet.create({
  buttonWrapper: {
    flexDirection: 'row',
    position: 'absolute',
    zIndex: -1,
    top: 0,
    left: 0
  }
})

export default ButtonBackground

Skia image component

import React, { useEffect } from 'react'
import {
  Image,
  useImage,
  type DataSourceParam,
  type ImageProps,
  type SkiaProps,
  type RectCtor,
  Mask,
  Rect,
  rect
} from '@shopify/react-native-skia'
import { logger } from '~eldrum/helpers/logger'

type TSkiaImageProps = {
  source: DataSourceParam
  tint?: string // Must be Hex color
  onLoad?: () => void
} & Omit<SkiaProps<ImageProps>, 'image'> &
  Required<RectCtor>

export const SkiaImage = ({ source, tint, onLoad, ...restProps }: TSkiaImageProps) => {
  const image = useImage(source, error => {
    logger.error(error)
  })

  useEffect(() => {
    if (image) {
      onLoad?.()
    }
  }, [image])

  return (
    <Mask mode="alpha" mask={<Image image={image} {...restProps} />}>
      <Rect rect={rect(restProps.x, restProps.y, restProps.width, restProps.height)} color={tint} />
    </Mask>
  )
}

[AdamGerthel]

Update:

I decided to try to preload and cache the images instead of relying on useImage:

const imgFactory = Skia.Image.MakeImageFromEncoded.bind(Skia.Image)

const lineBottom = require('~eldrum/assets/graphics/line-bottom.png') as string
const lineTop = require('~eldrum/assets/graphics/line-top.png') as string
const LINE_WIDTH = 1000

const LEFT_SIDES = [
  require('~eldrum/assets/graphics/button-left.png'),
  require('~eldrum/assets/graphics/button-left-2.png'),
  require('~eldrum/assets/graphics/button-left-3.png')
] as string[]

const RIGHT_SIDES = [
  require('~eldrum/assets/graphics/button-right.png'),
  require('~eldrum/assets/graphics/button-right-2.png'),
  require('~eldrum/assets/graphics/button-right-3.png')
] as string[]

class ButtonBackgroundImageCache {
  cachedImages?: {
    lineTop: SkImage
    lineBottom: SkImage
    left: SkImage[]
    right: SkImage[]
  }

  async cacheImage(source: string): Promise<SkImage> {
    const uri = typeof source === 'string' ? source : Image.resolveAssetSource(source).uri
    const data = await Skia.Data.fromURI(uri)
    const image = imgFactory(data)

    if (!image) {
      throw new Error('Failed to load image')
    }

    return image
  }

  async generateCache() {
    this.cachedImages = {
      lineTop: await this.cacheImage(lineTop),
      lineBottom: await this.cacheImage(lineBottom),
      left: await Promise.all(LEFT_SIDES.map(this.cacheImage.bind(this))),
      right: await Promise.all(RIGHT_SIDES.map(this.cacheImage.bind(this)))
    }
  }
}

export const buttonBackgroundCache = new ButtonBackgroundImageCache()

// I call buttonBackgroundCache.generateCache() on app startup, and I'll then use buttonBackgroundCache.cachedImages in my button background component.

This kind of solved the issue and the button backgrounds are generated much faster. BUT now I'm noticing that some backgrounds fail to render altogether (i.e. all parts).

[AdamGerthel]

Ok, so it renders fine sometimes, and sometimes not. I've never seen another RN library show such inconsistent errors. I'm out.

ssieu.mp4

[AdamGerthel]

Tried upgrading to 1.7.5 from 1.5.0 (which Expo SDK 52 is pinned to) and it's actually gotten worse. See my loom recording below:
https://www.loom.com/embed/83ca960e1ce24657ac1f27c37794ad86?sid=9e679ab1-7347-449a-9f1d-481f70a38442

I should add that it does seem like the loading isn't the issue. I'm saying that because I'm now at a point where I'm preloading the images for the buttons but not for the bigger "card" background - that one still uses useImage. So it seems like it's the rendering that fails intermittently. The buttons will either completely fail (none of the Image parts appear), and for the card, it seems like any individual part can fail.

UPDATE: I should also add that the failure to render does not happen when I use createPicture and draw it using Skia directly instead of using the components, i.e:

return createPicture(
      canvas => {
        const paint = Skia.Paint()
        const topOffset = (randomNumberA * LINE_WIDTH) / 6
        const bottomOffset = (randomNumberB * LINE_WIDTH) / 6
        const lineWidth = width - roundedSideWidth * 2

        // Draw background rect
        paint.setColor(Skia.Color(tint))
        canvas.drawRect(
          {
            x: sideWidth,
            y: lineThickness,
            height: height - lineThickness * 4,
            width: width - sideWidth * 2
          },
          paint
        )

        // Draw left side
        canvas.drawImageRect(
          leftSideImage,
          { x: 0, y: 0, width: 160, height: 160 },
          { x: 0, y: 0, width: roundedSideWidth, height },
          paint
        )

        // Draw top line with mask
        canvas.save()
        canvas.clipRect(
          { x: roundedSideWidth, y: 0, width: lineWidth, height: LINE_HEIGHT },
          ClipOp.Intersect,
          true
        )
        canvas.drawImageRect(
          lineTopImage,
          { x: 0, y: 0, width: LINE_WIDTH, height: LINE_HEIGHT },
          { x: roundedSideWidth, y: 0, width: lineWidth + topOffset, height: LINE_HEIGHT },
          paint
        )
        canvas.restore()

        // Draw bottom line with mask
        canvas.save()
        canvas.clipRect(
          {
            x: roundedSideWidth,
            y: height - lineThickness,
            width: lineWidth,
            height: lineThickness
          },
          ClipOp.Intersect,
          true
        )
        canvas.drawImageRect(
          lineBottomImage,
          { x: 0, y: 0, width: LINE_WIDTH, height: LINE_HEIGHT },
          {
            x: roundedSideWidth,
            y: height - lineThickness,
            width: lineWidth + bottomOffset,
            height: lineThickness
          },
          paint
        )
        canvas.restore()

        // Draw right side
        canvas.drawImageRect(
          rightSideImage,
          { x: 0, y: 0, width: 160, height: 160 },
          {
            x: width - roundedSideWidth,
            y: 0,
            width: roundedSideWidth,
            height
          },
          paint
        )
      }

[AdamGerthel]

I ended up rewriting both components using createPicture which never failed to render/load properly, which the declarative (i.e. Reactive way) did a lot.

I did noticed thought that there were many more issues in 1.7.5 so I probably won't be upgrading anytime soon.

[wcandillon]

Hello Adam,

Thank you for posting this. Your intuition of building your own loaders is correct. We offer useImage as a convenience tool but if it doesn't fit for you, we definitely recommend you build your own logic there.

I just had a look at the screen recording. is there any chance you could reproduce the issue in a small standalone example? is it possible that you are dealing with texture cache busting somehow? https://shopify.github.io/react-native-skia/docs/animations/textures/ maybe useImage as texture is what you need?

[AdamGerthel]

Yes! I'm actually planning on making a reproducible repository as soon as I can. Turns out, the imperative API is failing from time to time as well from version 1.7.2 and above.

Will take a look at the textures link and get back to you on that. One thing though: These specific components that are failing are not using animations, and textures seem to live under the "Animations" section of the docs - does it still apply even when not using animations?

Update: Since I'm using pictures (with images in them), usePictureAsTexture might be a good solution.

[AdamGerthel]

See #2831

[wcandillon]

in the example you provided, no image rendering seems to be failing

[AdamGerthel]

In the sample repository? If you look at the video you'll see that some of the images will sometimes not render/paint (not sure if this is a terminology issue 😅). The "picture" draws 8 images, one for each side and one for each corner. In the video you can see that sometimes several pieces seem to be missing (i.e. not rendering). Anyway, let's keep the discussion in the issue, I'll try to make a smaller repro sample.

[wcandillon]

Correct, I am genuinely by your usecase and see if there is a potential error but please beware that just to run the example and investigating it, it a lot of time/context-switching. I felt a bit overwhelmed by the complexity of the implementation and simply render the images side by side without any additional logic, and they will render as expected on very re-render.

I'm curious to see how small you can make the repo.