Saturday 4 May 2013
Web Audio Progress
Our Web Audio implementation is making great progress. This is mainly due to the efforts of Ehsan Akhgari, who is, astoundingly, cranking out one or two features per day. Paul Adenot and I are spending hours every day just reviewing his code. I think this is partly due to Ehsan and I laying down some pretty good infrastructure at the outset.
Our current goal is to have a basically complete implementation for Firefox 24, which branches from trunk in about eight weeks. There are a few things we need to do to get there:
- Complete the feature set. At this point that mainly means adding all the node types that aren't implemented yet: MediaStreamAudioDestinationNode, MediaStreamAudioSourceNode, MediaElementAudioSourceNode, ConvolverNode, OscillatorNode, and WaveShaperNode. The first three are all related and shouldn't be too hard since we designed Web Audio from the start to share infrastructure with MediaStreams (which are already integrated into media elements) --- internally, a Web Audio node is just a special kind of MediaStream. We still need to implement HRTF and soundfield panning modes for PannerNode. We need to implement OfflineAudioContext. For some of the audio algorithms that aren't very well specified, we're borrowing code from Blink. This is suboptimal but there's ongoing discussion about what level of detail we should specify the audio algorithms at.
- Work on latency. Right now audio output has pretty bad latency, especially on Android, FirefoxOS and Windows; the biggest problems are intrinsic issues with the platform APIs we're using. On older versions of Android and Windows XP it may be a lost cause, because good APIs simply aren't available. For Windows Vista and up we're writing a new audio output backend using WASAPI. On FirefoxOS we may rip out the Android code we're using and replace it with PulseAudio. There is additional work to do to better integrate the MediaStreamGraph (that drives MediaStream and AudioNode processing) with our libcubeb audio backends for lower latency and better tracking of the audio clock. This latency work is desperately needed for WebRTC as well as Web Audio.
- Work on throughput. Right now we're focusing on having a good clean design and functional correctness. For example, all communication and synchronization with the MediaStreamGraph real-time processing thread is asynchronous, using message passing. Updates to Web Audio and MediaStream graphs are batched so all changes performed by a script happen atomically on the real-time thread. But we haven't done any profiling, tuning, or optimization of the actual processing code. In particular we'll clearly need SIMD implementations of basic audio primitives such as mixing to get near-maximum performance, especially on mobile.
- Test and fix bugs, needless to say.
Contributors welcome! The Web Audio bug (779297) has a lot of dependencies to choose from :-).
One interesting issue that Jean-Marc Valin brought up recently is the prospect of a loudness war on the Web. Some areas, such as Sony's Playstation products and some broadcast TV regions, are trying to mitigate the loudness war by standardizing acceptable dB levels for all content. It might be a good idea to do this on the Web too, and have browsers automatically limit the volume of content that exceeds those levels. We're still thinking about whether and how this should be done, and talking about it on public-audio.
Comments