- Native multimedia: why, what, and how?
- Codecs-the horror, the horror
- Rolling custom controls
- Multimedia accessibility
- Synchronising media tracks
- Summary
This chapter is from the book
This chapter is from the book
This chapter is from the book
Codecs—the horror, the horror
Early drafts of the HTML5 specification mandated that all browsers should have built-in support for multimedia in at least two codecs: Ogg Vorbis for audio and Ogg Theora for movies. Vorbis is a codec used by services like Spotify, among others, and for audio samples in games like Microsoft Halo.
However, these requirements for default format support were dropped from the HTML5 spec after Apple and Nokia objected, so the spec makes no recommendations about codecs at all. This leaves us with a fragmented situation, with different browsers opting for different formats, based on their ideological and commercial convictions.
Currently, there are two main container/codec combinations that developers need to be aware of: the new WebM format (www.webmproject.org) which is built around the VP8 codec that Google bought for $104 million and open licensed, and the ubiquitous MP4 format that contains the royalty-encumbered H.264 codec. H.264 is royalty-encumbered because, in some circumstances, you must pay its owners if you post videos that use that codec. We’re not lawyers so can’t give you guidance on which circumstances apply to you. Go to www.mpegla.com and have your people talk to their people’s people.
In our handy cut-out-and-lose chart (Table 4.1), we also include the Ogg Theora codec for historical reasons—but it’s really only useful if you want to include support for older versions of browsers with initial <video> element support like Firefox 3.x and Opera 10.x.
Table 4.1 Video codec support in modern browsers.
|
WEBM (VPS CODEC) |
MP4 (H.264 CODEC) |
OGV (OGG THEORA CODEC) |
|
|
Opera |
Yes |
No |
Yes |
|
Firefox |
Yes |
No |
Yes |
|
Chrome |
Yes |
Yes—See Note, support will be discountinued |
Yes |
|
IE9+ |
Yes (but codec must be installed manually) |
yes |
No |
|
Safari |
No |
Yes |
No |
Marvel at the amazing coincidence that the only two browsers that support H.264 are members of the organization that collects royalties for using the codec (www.mpegla.com/main/programs/AVC/Pages/Licensors.aspx).
A similarly fragmented situation exists with audio codecs, for similar royalty-related reasons (see Table 4.2).
Table 4.2 Audio codec support in modern browsers.
|
.OGG/.OGV (VORBIS CODEC) |
Mp3 |
MP4/M4A (AAC CODEC) |
WAV |
|
|
Opera |
Yes |
No |
No |
Yes |
|
Firefox |
Yes |
No |
No |
Yes |
|
Chrome |
Yes |
Yes |
Yes |
Yes |
|
IE9+ |
No |
Yes |
Yes |
No |
|
Safari |
No |
Yes |
Yes |
Yes |
The rule is: provide both a royalty-free WebM and an H.264 video, and both a Vorbis and an MP3 version of your audio, so that nobody gets locked out of your content. Let’s not repeat the mistakes of the old “Best viewed in Netscape Navigator” badges on sites, or we’ll come round and pin a “n00b” badge to your coat next time you’re polishing your FrontPage CD.
Multiple <source> elements
To do this, you need to encode your multimedia twice: once as WebM and once as H.264 in the case of video, and in both Vorbis and MP3 for audio. Then, you tie these separate versions of the file to the media element.
Previously, we’ve used the <video src="..."> syntax to specify the source for our video. This works fine for a single file, but how do we tell the browser that there are multiple versions (using different encoding) available? Instead of using the single src attribute, you nest separate <source> elements for each encoding with appropriate type attributes inside the <audio> or <video> element and let the browser download the format that it can display. Faced with multiple <source> elements, the browser will look through them (in source order) and choose the first one it finds that it thinks it can play (based on the type attribute—which gives explicit information about the container MIME type and the codec used—or, missing that, heuristic based on file extension). Note that in this case we do not provide a src attribute in the media element itself:
- <video controls>
- <source src=leverage-a-synergy.mp4 type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'>
- <source src=leverage-a-synergy.webm type='video/webm; codecs="vp8, vorbis"'>
- <p>Your browser doesn't support video.
- Please download the video in <a href=leverage-a-synergy.webm>webM</a> or <a href=leverage-a-synergy.mp4>MP4</a> format.</p>
- </video>
Line 1 tells the browser that a video is to be inserted and gives it default controls. Line 2 offers an MP4 version of the video. We’ve put the mp4 first, because some old versions of Mobile Safari on the iPad have a bug whereby they only look at the first <source> element, so that if it isn’t first, it won’t be played. We’re using the type attribute to tell the browser what kind of container format is used (by giving the file’s MIME type) and what codec was used for the encoding of the video and the audio stream. If you miss out on the type attribute, the browser downloads a small bit of each file before it figures out that it is unsupported, which wastes bandwidth and could delay the media playing.
Notice that we used quotation marks around these parameters—the spec uses 'video/mp4; codecs="avc..."' (single around the outside, double around the codec). Some browsers stumble when it’s the other way around. Line 3 offers the WebM equivalent. The codec strings for H.264 and AAC are more complicated than those for WebM because there are several profiles for H.264 and AAC, to cater for different categories of devices and connections. Higher profiles require more CPU to decode, but they are better compressed and take less bandwidth.
We could also offer an Ogg video here for older versions of Firefox and Opera, after the WebM version, so those that can use the higher-quality WebM version pick that up first, and the older (yet still HTML5 <video> element capable) browsers fall back to this.
Inside the <video> element is our fallback message, including links to both formats for browsers that can natively deal with neither video type but which is probably on top of an operating system that can deal with one of the formats, so the user can download the file and watch it in a media player outside the browser.
OK, so that’s native HTML5 video for users of modern browsers. What about users of legacy browsers—including Internet Explorer 8 and older?
Video for legacy browsers
Older browsers can’t play native video or audio, bless them. But if you’re prepared to rely on plugins, you can ensure that users of older browsers can still experience your content in a way that is no worse than they currently get.
Remember that the contents of the <video> element can contain markup, like the text and links in the previous example? Here, we’ll place an entire Flash video player movie into the fallback content instead (and of course, we’ll also provide fallback for those poor users who don’t even have that installed). Luckily, we don’t need to encode our video in yet another format like FLV (Flash’s own legacy video container); because Flash (since version 9) can load MP4 files as external resources, you can simply point your custom Flash video player movie to the MP4 file. This combination should give you a solid workaround for Internet Explorer 8 and older versions of other browsers. You won’t be able to do all the crazy video manipulation stuff we’ll see later in this chapter, but at least your users will still get to see your video.
The code for this is as hideous as you’d expect for a transitional hack, but it works anywhere that Flash Player is installed—which is almost everywhere. You can see this nifty technique in an article called “Video for Everybody!” by its inventor, Kroc Camen (http://camendesign.com/code/video_for_everybody).
Alternatively, you could host the fallback content on a video hosting site and embed a link to that between the tags of a video element:
<video controls>
<source src=leverage-a-synergy.mp4 type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'>
<source src=leverage-a-synergy.webm type='video/webm; codecs="vp8, vorbis"'>
<embed src="https://www.youtube.com/v/cmtcc94Tv3A&hl=
en_GB&fs=1&rel=0" type="application/x-shockwave-flash"
allowscriptaccess="always" allowfullscreen="true"
width="425" height="344">
</video>
You can use the HTML5 Media Library (http://html5media.info) to hijack the <video> element and automatically add necessary fallback by adding one line of JavaScript in the page header.
Sending differently compressed videos to handheld devices
Video files tend to be large, and sending very high-quality video can be wasteful if sent to handheld devices where the small screen sizes make high quality unnecessary. There’s no point in sending high-definition video meant for a widescreen monitor to a handheld device screen, and most users of smartphones and tablets will gladly compromise a little bit on encoding quality if it means that the video will actually load over a mobile connection. Compressing a video down to a size appropriate for a small screen can save a lot of bandwidth, making your server and—most importantly—your mobile users happy.
HTML5 allows you to use the media attribute on the <source> element, which queries the browser to find out screen size (or number of colours, aspect ratio, and so on) and to send different files that are optimised for different screen sizes.
This functionality and syntax is borrowed from the CSS Media Queries specification www.w3.org/TR/css3-mediaqueries but is part of the markup, as we’re switching source files depending on device characteristics. In the following example, the browser is “asked” if it has a min-device-width of 800 px—that is, does it have a wide screen. If it does, it receives hi-res.webm; if not, it is sent lo-res.webm:
<video controls>
<source src=hi-res.webm ... media="(min-device-width: 800px)">
<source src=lo-res.webm>
...
</video>
Also note that you should still use the type attribute with codecs parameters and fallback content previously discussed. We’ve just omitted those for clarity.