Server-side video stream encryption for client-side decrypt and playback

[jcalfee]

Has the question been asked before?

• I have searched the existing issues

Question

I'm trying to encrypt files in Nodejs (on the server) for play back in the browser. If your curious, this has to do will role based security where some servers have access to certain videos and backup and encrypt with a public key then distribute that to less trusted servers that stream the files. The client gets authenticated and use the decryption key in the browser without the less trusted servers ever having access. This is pretty portable to a plugin or an electron app.

So, on the web side of things, there is a great example from webrts whereby insertable-streams/endtoend-encryption/js/worker.js is using a Streams API TransformStream to setup the streams for some week XOR encryption / decryption (not shown here yet, this is just how it sets it up):

function handleTransform(operation, readable, writable) {
  if (operation === 'encode') {
    const transformStream = new TransformStream({
      transform: encodeFunction,
    });
    readable
        .pipeThrough(transformStream)
        .pipeTo(writable);
  } else if (operation === 'decode') {
    const transformStream = new TransformStream({
      transform: decodeFunction,
    });
    readable
        .pipeThrough(transformStream)
        .pipeTo(writable);
  }
}

Something very helpful, the encode function logs the first 30 frames:

function encodeFunction(encodedFrame, controller) {
  if (scount++ < 30) { // dump the first 30 packets.
    dump(encodedFrame, 'send');
  }
 // ... then some weak XOR encryption
}

It gets complex though. I see this is not simply straight data. It is the audio and video streams in separate frames with metadata. I'm not even sure if the size of these frames will be predictable and consistent (something I need to know for strong AES CTR encryption). I would rather encrypt and decrypt on the entire stream and not break it down like this, however, there does not seem be a way to do this and still feed it into <video srcObject={stream}.../>. Keep in mind, the <video ../> element will interact with the stream to get Content-Range blocks of data so the user can fast-forward and re-wind in what is typically a very large stream. So, I need to work with this element and give it what it needs. This is also probably the main way to get decent GPU and hardware support for rendering. Do you think I really have to break the stream up like this on server and get inside and encrypt each frame? If so, it looks like I'll probably need to do this exactly like the browser is going to do this: read and dissemble the stream and encrypt / decrypt each frame. This is just what their example does, except it is browser to browser instead of node to browser. So, in node, I probably should be able to create a "dump" just like the one I'm seeing in the web browser or at least some thing like it so I can debug this and figure out what is going on.

Here is the node example I have so far:

The dump is the same (so far):

function dump(encodedFrame, direction, max = 16) {
  const data = new Uint8Array(encodedFrame.data);
  let bytes = '';
  for (let j = 0; j < data.length && j < max; j++) {
    bytes += (data[j] < 16 ? '0' : '') + data[j].toString(16) + ' ';
  }
  const metadata = encodedFrame.getMetadata();
  console.log(performance.now().toFixed(2), direction, bytes.trim(),
      'len=' + encodedFrame.data.byteLength,
      'type=' + (encodedFrame.type || 'audio'),
      'ts=' + (metadata.rtpTimestamp || encodedFrame.timestamp),
      'ssrc=' + metadata.synchronizationSource,
      'pt=' + (metadata.payloadType || '(unknown)'),
      'mimeType=' + (metadata.mimeType || '(unknown)'),
  );
}

I want this output from node, but this is only from the browser (so far):

12030114.40 send b1 0a 00 e4 2a 1f b8 af c3 38 80 7a cb 66 fb d5 len=439 type=delta ts=892793352 ssrc=1448182488 pt=96 mimeType=video/VP8
worker.js:44 12030114.60 send b1 0a 00 e4 2a 1f b8 af c3 38 80 7a cb 66 fb d5 len=439 type=delta ts=924677304 ssrc=3772434219 pt=96 mimeType=video/VP8
worker.js:44 12030118.60 recv b1 0a 00 e4 2a 1f b8 af c3 38 80 7a cb 66 fb d5 len=439 type=delta ts=892793352 ssrc=1448182488 pt=96 mimeType=video/VP8
worker.js:44 12030134.60 send 68 36 ba e0 3c 0e 0b 21 8d 17 25 d5 c8 71 88 07 len=72 type=audio ts=1404123116 ssrc=1144326229 pt=111 mimeType=audio/opus
worker.js:44 12030135.10 send 68 37 35 e5 a7 85 d1 90 1a 89 95 e0 fa eb fd a2 len=74 type=audio ts=406336411 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030135.80 recv 68 37 35 e5 a7 85 d1 90 1a 89 95 e0 fa eb fd a2 len=74 type=audio ts=406336411 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030155.00 send d1 0a 00 e4 0a 5f b8 ab f7 a7 47 a7 d8 85 17 f0 len=680 type=delta ts=892797042 ssrc=1448182488 pt=96 mimeType=video/VP8
worker.js:44 12030155.30 send d1 0a 00 e4 0a 5f b8 ab f7 a7 47 a7 d8 85 17 f0 len=680 type=delta ts=924680994 ssrc=3772434219 pt=96 mimeType=video/VP8
worker.js:44 12030157.40 send 68 36 88 b9 9c 76 4f fe 6b 4a 41 2b 2b 89 28 2a len=73 type=audio ts=406337371 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030157.50 send 68 37 27 67 7e 2e 7d e4 67 17 d0 f4 a2 09 8a b1 len=67 type=audio ts=1404124076 ssrc=1144326229 pt=111 mimeType=audio/opus
worker.js:44 12030162.00 recv 68 36 88 b9 9c 76 4f fe 6b 4a 41 2b 2b 89 28 2a len=73 type=audio ts=406337371 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030165.60 recv d1 0a 00 e4 0a 5f b8 ab f7 a7 47 a7 d8 85 17 f0 len=680 type=delta ts=892797042 ssrc=1448182488 pt=96 mimeType=video/VP8
worker.js:44 12030165.70 recv d1 0a 00 e4 0a 5f b8 ab f7 a7 47 a7 d8 85 17 f0 len=680 type=delta ts=892797042 ssrc=1448182488 pt=96 mimeType=video/VP8
worker.js:44 12030181.70 recv 68 37 43 55 60 a4 94 73 30 60 31 c5 72 db e1 84 len=72 type=audio ts=406338331 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030204.50 recv 68 37 43 55 e5 54 56 cd 21 ac 0b e1 16 d4 89 bb len=64 type=audio ts=406339291 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030204.60 recv 68 36 99 ec 1c 4e bb 8c 44 92 d1 7b 1b de ee 28 len=70 type=audio ts=406340251 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030212.40 recv 31 0a 00 e3 e2 af b8 a7 ff a7 47 a7 d8 85 17 f0 len=957 type=delta ts=892800822 ssrc=1448182488 pt=96 mimeType=video/VP8
worker.js:44 12030228.10 recv 68 37 43 55 d3 2a b3 9d 7d 52 f2 f0 30 b8 09 ff len=69 type=audio ts=406341211 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030243.90 recv f1 0d 00 e3 c2 ef b8 80 4f a7 47 a7 d8 85 17 f0 len=1122 type=delta ts=892804602 ssrc=1448182488 pt=96 mimeType=video/VP8
worker.js:44 12030251.60 recv 68 37 41 6e 88 5f cd 9b 90 0d 2e a2 95 04 4c c7 len=71 type=audio ts=406342171 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030274.10 recv 68 9d 4f f5 70 b6 c7 ce a0 25 63 c8 43 8d 29 78 len=82 type=audio ts=406343131 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030290.60 recv 31 11 00 e3 b3 0f b8 84 19 4c 60 7a bc 0c 84 a5 len=957 type=delta ts=892808292 ssrc=1448182488 pt=96 mimeType=video/VP8
worker.js:44 12030297.50 recv 68 36 72 e5 e3 91 e1 e2 f8 0c 9b 52 e8 ba 22 db len=75 type=audio ts=406344091 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030321.00 recv 68 36 99 eb 7e 26 41 79 62 36 99 bd 95 00 a4 14 len=74 type=audio ts=406345051 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030337.40 recv f1 12 00 e3 9b 3f c8 6c 08 fd 00 64 c8 a0 ac 2e len=1081 type=delta ts=892812072 ssrc=1448182488 pt=96 mimeType=video/VP8
worker.js:44 12030343.60 recv 68 37 09 9b 74 f8 bc ae 80 27 94 f8 7c 6a 20 c3 len=73 type=audio ts=406346011 ssrc=253231432 pt=111 mimeType=audio/opus
worker.js:44 12030343.80 recv 68 36 88 ba 3a 65 2d c4 e6 0e 32 fb e8 70 2c 38 len=75 type=audio ts=406346971 ssrc=253231432 pt=111 mimeType=audio/opus

So, I start down this path using 'node:stream/web' but I'm finding it does not provide the getMetadata() function:

import { TransformStream } from 'node:stream/web'

const readStream = fs.createReadStream(input, { highWaterMark: 64 * 1024 })
const readable = new ReadableStream({
  async start(controller) {
    await pipeline(readStream, chunk => {
      controller.enqueue(chunk)
    })
    controller.close()
  },
})
const writable = new WritableStream({
  // ...
})
const transformStream = new TransformStream({
  transform: (encodedFrame, controller)=> {
    dump(encodedFrame, 'decode') // TypeError: encodedFrame.getMetadata is not a function
  },
})
readable
  .pipeThrough(transformStream)
  .pipeTo(writable)
handleTransform('encode', readable, writable)

Do you know, did I miss something or is this just not available in node? Maybe new ReadableStream() the wrong thing to use? Or, perhaps I need some sort of shim so node acts more like the Web Stream API. Or, better yet, I just need to know more about these streams so I can handle them correctly. Any help is appreciated. Thank you!

[Borewit]

Interesting and challenging things you are working on @jcalfee, but to me it appears this has nothing to do with this music-metadata project.

But streams you usually use to flow or transform data chunks (bunch of bytes). That .getMetadata() does not directly work on a raw data chunk sounds logic. I would expect you need to wrap that bunch of bytes in something able to decode that bunch of bytes, and knows how to extract the metadata from it. Unless you stream objects rather then chunks, which maybe be possible, I never done that so far.

In principal Node.js has a Web Stream API, maybe minor differences in implementation, but it should bot be necessary to shim. The whole point of Web Stream API as that this part of the ECMAScript, which should result in code which can work both in browser and in Node.js.

I can confirm that working with the Web Stream API can be very tricky.

[jcalfee]

but to me it appears this has nothing to do with this music-metadata project.

Thanks for the clarification. I saw MP4 format support here and was not sure and, of course, streams. I had only 2 days to put this together so I really was not sure.

Looks like I need to take a few more days to explore other possible high-level encoding methods then dig in to the data in the stream itself, like you say, with different APIs. There are other players that I can explore, even native wasm compiled ones which may have fall-back optimizations so video is still fast. That is probably going to be more modern and simple. I was considering do the processing in a chrome engine like browserless so the API runs both sides in the browser (like the p2p examples out there), then intercept and save the encrypted stream using something like node-datachannel. However, I really don't want to support such a heavy weight stack though esp if this is the only reason it is needed. Also, I don't want to be locked in to playing the video only on the browser. But your suggestion is the most straight forward, it is going to take me a while to figure out where to start unless I can get on the right path up-front (so I reached out). Yes, higher level streams (objects instead of chunks for example). Maybe some of the popular video players have clues in them.

Fantastic, thank you for the feedback. If anyone else has experience please let me know or let me know if you want me to post some answers when I have them.

[jcalfee]

And, what project is on topic if you know? ;)

[Borewit]

And, what project is on topic if you know? ;)

You raised an issue on the music-metadata project, and that happen also to be the appropriate project to discuss. For most users that is obvious.

For generic help for implementation challenges I can advise you raise your questions at https://codereview.stackexchange.com/.