- Native multimedia: why, what, and how?
- Codecs-the horror, the horror
- Rolling custom controls
- Multimedia accessibility
- Synchronising media tracks
- Summary
This chapter is from the book
This chapter is from the book
This chapter is from the book
Synchronising media tracks
HTML5 will allow for alternative media tracks to be included and synchronised in a single <audio> or <video> element.
You might, for example, have several videos of a sporting event, each from different camera angles, and if the user moves to a different point in one video (or changes the playback rate for slow motion), she expects all the other videos to play in sync. Therefore, different media files need to be grouped together.
This could be a boon for accessibility, allowing for sign-language tracks, audio description tracks, dubbed audio tracks, and similar additional or alternative tracks to the main audio/video tracks.
This can be accomplished with JavaScript, or declaratively with a mediagroup attribute on the <audio> or <video> element:
<div> <video src="movie.webm" autoplay controls mediagroup=movie></video> <video src="signing.webm" autoplay mediagroup=movie></video> </div>
This is very exciting, and very new, so we won’t look further: the spec is constantly changing and there are no implementations.
Video conferencing, augmented reality
As we mentioned earlier, accessing a device’s camera and microphone was once available only to web pages via plugins. HTML5 gives us a way to access these devices straight from JavaScript, using an API called getUserMedia. (You might find it referred to as the <device> element on older resources. The element itself has been spec’d away, but the concept has been moved to a pure API.)
An experimental build of Opera Mobile on Android gives us a glimpse of what will be possible once this feature is widely available. It connects the camera to a <video> element using JavaScript by detecting whether getUserMedia is supported and, if so, setting the stream coming from the camera as the src of the <video> element:
<!DOCTYPE html>
<h1>Simple web camera display demo</h1>
<video autoplay></video>
<script type="text/javascript">
var video = document.getElementsByTagName('video')[0],
heading = document.getElementsByTagName('h1')[0];
if(navigator.getUserMedia) {
navigator.getUserMedia('video', successCallback, errorCallback);
function successCallback( stream ) {
video.src = stream;
}
function errorCallback( error ) {
heading.textContent =
"An error occurred: [CODE " + error.code + "]";
}
} else {
heading.textContent =
"Native web camera streaming is not supported in this browser!";
}
</script>
Once you’ve done that, you can manipulate the video as you please. Rich Tibbett wrote a demo that copies the video into canvas (thereby giving you access to the pixel data), looks at those pixels to perform facial recognition, and draws a moustache on the face, all in JavaScript (see Figure 4.7).
)
Figure 4.7 Remy Sharp, with a magical HTML5 moustache. (Photo by Julia Gosling)
Norwegian developer Trygve Lie has made demos of getUserMedia that use Web Sockets (see Chapter 10) to send images from an Android phone running the experimental Opera Mobile build to a desktop computer. See https://github.com/trygve-lie/demos-html5-realtime for the source code and a video demonstrating it.
Obviously, giving websites access to your webcam could create significant privacy problems, so users will have to opt-in, much as they have to do with geolocation. But that’s a UI concern rather than a technical problem.
Taking the concept even further, there is also a Peer-to-Peer API being developed for HTML, which will allow you to hook up your camera and microphone to the <video> and <audio> elements of someone else’s browser, making it possible to do video conferencing.
In May 2011, Google announced WebRTC, an open technology for voice and video on the Web, based on the HTML5 specifications. WebRTC uses VP8 (the video codec in WebM) and two audio codecs optimised for speech with noise and echo cancellation, called iLBC, a narrowband voice codec, and iSAC, a bandwidth-adaptive wideband codec (see http://sites.google.com/site/webrtc/).
As the project website says, “We expect to see WebRTC support in Firefox, Opera, and Chrome soon!”