Please provide a short (approximately 100 word) summary of the following web Content, written in the voice of the original author. If there is anything controversial please highlight the controversy. If there is something surprising, unique, or clever, please highlight that as well. Content:
Title: Real-Time Video Processing with WebCodecs and Streams
Site: webrtchacks.com
WebRTC used to be about capturing some media and sending it from Point A to Point B. Machine Learning has changed this. Now it is common to use ML to analyze and manipulate media in real time for things like virtual backgrounds, augmented reality, noise suppression, intelligent cropping, and much more. To better accommodate this growing trend, the web platform has been exposing its underlying platform to give developers more access. The result is not only more control within existing APIs, but also a bunch of new APIs like Insertable Streams, WebCodecs, Streams, WebGPU, and WebNN. 
 So how do all these new APIs work together? That is exactly what W3C specialists,  François Daoust  and  Dominique Hazaël-Massieux  (Dom) decided to find out. In case you forgot, W3C is the World Wide Web Consortium that standardizes the Web. François and Dom are long-time standards guys with a deep history of helping to make the web what it is today. 
 This is the first of a two-part series of articles that explores the future of real-time video processing with WebCodecs and Streams. This first section provides a review of the steps and pitfalls in a multi-step video processing pipeline using existing and the newest web APIs. Part two will explore the actual processing of video frames. 
 I am thrilled about the depth and insights these guides provide on these cutting-edge approaches – enjoy! 
 {“editor”, “ chad hart “} 
 
 
 About Processing Pipelines 
 In simple WebRTC video conferencing scenarios, audio and video streams captured on one device are sent to another device, possibly going through some intermediary server. The capture of raw audio and video streams from microphones and cameras relies on  getUserMedia . Raw media streams then need to be encoded for transport and sent over to the receiving side. Received streams must be decoded before they can be rendered. The resulting video pipeline is illustrated below. Web applications do not see these separate encode/send and receive/decode steps in practice – they are entangled in the core WebRTC API and under the control of the browser. 
 
 If you want to add the ability to do something like remove users’ backgrounds, the most scalable and privacy respective option is to do it client-side before the video stream is sent to the network. This operation needs access to the raw pixels of the video stream. Said differently, it needs to take place between the capture step and encode steps. Similarly, on the receiving side, you may want to give users options like adjusting colors and contrast, which also require raw pixel access between the decode and render steps. As illustrated below, this adds an extra process steps to the resulting video pipeline. 
 
 This made  Dominique Hazaël-Massieux  and me wonder how web applications can build such media processing pipelines. 
 The main problem is raw frames from a video stream cannot casually be exposed to web applications. Raw frames are: 
 
 large – several MB per frame, 
 plentiful – 25 frames per second or more, 
 not easily exposable – GPU to CPU read-back often needed, and 
 browsers need to deal with a variety of pixel formats (RGBA, YUV, etc.) and color spaces under the hoods. 
 
 As such, whenever possible, web technologies that manipulate video streams on the web ( HTMLMediaElement ,  WebRTC ,  getUserMedia ,  Media Source Extensions ) treat them as opaque objects and hide the underlying pixels from applications. This makes it difficult for web applications to create a media processing pipeline in practice. 
 Fortunately, the  VideoFrame  interface in  WebCodecs  may help, especially if you couple this with the  MediaStreamTrackProcessor  object defined in  MediaStreamTrack Insertable Media Processing using Streams  that creates a bridge between WebRTC and WebCodecs. WebCodecs lets you access and process raw pixels of media frames. Actual processing can use one of many technologies, starting with good ol’ JavaScript and including  WebAssembly ,  WebGPU , or the  Web Neural Network API  (WebNN). 
 After processing, you could get back to WebRTC land through the same bridge. That said, WebCodecs can also put you in control of the encode/decode steps in the pipeline through its  VideoEncoder  and  VideoDecoder  interfaces. These can give you full control over all individual steps in the pipeline: 
 
 For transporting the processed image somewhere while keeping latency low, you could consider  WebTransport  or WebRTC’s  RTCDataChannel . 
 For rendering, you could render directly to a canvas through  drawImage , using WebGPU, or via an  <video>  element through  VideoTrackGenerator  (also defined in  MediaStreamTrack Insertable Media Processing using Streams ). 
 
 Inspired by  sample code  created by  Bernard Aboba  – co-editor of the WebCodecs and WebTransport specifications and co-chair of the WebRTC Working Group in W3C – Dominique and I decided to spend a bit of time exploring the creation of processing media pipelines. First, we wanted to better grasp media concepts such as video pixel formats and color spaces – we probably qualify as web experts, but we are not media experts and we tend to view media streams as opaque beasts as well. Second, we wanted to assess whether technical gaps remain. Finally, we wanted to understand where and when copies get made and gather some performance metrics along the way. 
 This article describes our approach, provides highlights of our  resulting demo code , and shares our learnings. The code should not be seen as authoritative or even correct (though we hope it is), it is just the result of a short journey in the world of media processing. Also, note the technologies under discussion are still nascent and do not yet support interoperability across browsers. Hopefully, this will change soon! 
 Note: We did not touch on audio for lack of time. Audio frames take less memory, but there are many more of them per second and they are more sensitive to timing hiccups. Audio frames are processed with the Web Audio API. It would be very interesting to add audio to the mix, be it only to explore audio/video synchronization needs. 
 The demo 
 Our  demo  explores the creation of video processing pipelines, captures performance metrics, evaluates the impacts of choosing a specific technology to process frames, and provides insights about where operations get done and when copies are made. The processing operations loop through all pixels in the frame and “do something with them” (what they actually do is of little interest here). Different processing technologies are used for testing purposes, not because they would necessarily be a good choice for the problem at hand. 
 The demo lets the user: 
 
 Choose a source of input to create an initial stream of VideoFrame: either a  Nyan-cat -like animation created from scratch using  OffscreenCanvas , or a live stream generated from a camera. The user may also choose the resolution and framerate of the video stream. 
 Process video frames to replace green with blue using WebAssembly. 
 Process video frames to turn them into black and white using pure JavaScript. 
 Add an H.264 encoding/decoding transformation stage using WebCodecs. 
 Introduce slight delays in the stream using regular JavaScript. 
 Add an overlay to the bottom right part of the video that encodes the frame’s timestamp. The overlay is added using WebGPU and WGSL. 
 Add intermediary steps to force copies of the frame to CPU memory or GPU memory, to evaluate the impact of the frame’s location in memory on transformations. 
 
 
 Once you hit the “Start” button, the pipeline runs and the resulting stream is displayed on the screen in a  <video>  element. And… that’s it, really! What mattered to us was the code needed to achieve that and the insights we gained from gathering performance metrics and playing with parameters. Let’s dive into that! 
   
 Note: these APIs are new and may not work in your browser 
 Technologies discussed in this article and used in the demo are still “emerging” (at least as of March 2023). The demo currently only runs in  Google Chrome Canary  with WebGPU enabled (“Unsafe WebGPU” flag set in  chrome://flags/ ). Hopefully, the demo can soon run in other browsers too. Video processing with WebCodecs is available in the  technical preview of Safari  (16.4) and is under development in  Firefox . WebGPU is also under development in  Safari  and  Firefox . A greater unknown is support for  MediaStreamTrack Insertable Media Processing using Streams  in other browsers. For example, see this  tracking bug in Firefox . 
 Timing Stats 
 Timing statistics are reported to the end of the page at the end and as objects to the console (this requires opening the dev tools panel). Provided the overlay was present, display times for each frame are reported too. 
 
 We’ll discuss more on this in the  Measuring Performance  section. 
 The role of WebCodecs in client-side processing 
 WebCodecs is the core of the demo and the key technology we’re using to build a media pipeline. Before we dive more into this, it may be useful to reflect on the value of using WebCodecs in this context. Other approaches could work just as well. 
 What about the Canvas? Do we need WebCodecs? 
 In fact, client-side processing of raw video frames has been possible on the web ever since the  <video>  and  <canvas>  elements were added to HTML, with the following recipe: 
 
 Render the video onto a  <video>  element. 
 Draw the contents of the  <video>  element onto a  <canvas>  with  drawImage  on a recurring basis, e.g. using  requestAnimationFrame  or the more recent  requestVideoFrameCallback  that notifies applications when a video frame has been presented for composition and provides them with metadata about the frame. 
 Process the contents of the <canvas> whenever it gets updated. 
 
 We did not integrate this approach in our demo. Among other things, the performance here would depend on having the processing happen out of the main thread. We would need to use an  OffscreenCanvas  to process contents in a worker, possibly coupled with a call to  grabFrame  to send the video frame to the worker. 
 WebCodecs advantages 
 One drawback to the Canvas approach is that there is no guarantee that all video frames get processed. Applications can tell how many frames they missed if they hook onto  requestVideoFrameCallback  by looking at the  presentedFrames  counter, but missed frames were, by definition, missed. Another drawback is that some of the code ( drawImage  or  grabFrame ) needs to run on the main thread to access the  <video>  element. 
 WebGL and WebGPU also provide mechanisms to import video frames as textures directly from a  <video>  element, e.g. through the  importExternalTexture method  in WebGPU. This approach works well if the processing logic can fully run on the GPU. 
 WebCodecs gives applications a direct handle to a video frame and mechanisms to encode/decode them. This allows applications to create frames from scratch, or from an incoming stream, provided that the stream is in non-containerized form. 
 Note on containerized media 
 One important note – media streams are usually encapsulated in a media container. The container may include other streams along with timing and other metadata. While media streams in WebRTC scenarios do not use containers, most stored media files and media streamed on the web use adaptive streaming technologies (e.g. DASH, HLS) that are in a containerized form (e.g. MP4, ISOBMFF). WebCodecs can only be used on non-containerized streams. Applications that want to use WebCodecs with containerized media need to ship additional logic on their own to extract the media streams from their container (and/or to add streams to a container). For more information about media container formats, we recommend  The Definitive Guide to Container File Formats  by  Armin Trattnig . 
 Processing streams using… Streams 
 So, having a direct handle on a video frame seems useful to create a media processing pipeline. It gives a handle to th<truncated>