You've seen how HTML5 gives you the first credible alternative to third-party plugins. The incompatible codec support currently makes it harder than using plugins to simply embed video in a page and have it work cross-browser.
There are already a number of ready-made scripts available that allow you to easily leverage the HTML5 synergies in your own pages, without having to do all the coding yourself. The Kaltura player (http://www.html5video.org/) is an open source video player that works in all browsers. jPlayer (http://www.happyworm.com/jquery/jplayer/) is a very liberally-licensed jQuery audio player that degrades to Flash in legacy browsers, can be styled with CSS and can be extended to allow playlists.