# Friday, December 19, 2008

Silverlight BabySmash Audio Files

Overview

In this post I will provide an overview of the changes from the original wav media files, to wma files required by the Silverlight version.

The main sections are:

  • Introduction – Discusses the background behind the use of audio files in the original WPF version and an overview of the changes required.
  • Code Changes – Discusses the changes required to move from win32 api implementation to the MediaElement.
  • Audio File Changes – Discusses uses Microsoft Expression Media Encoder 2 to convert the wav files to wma files.
  • Summary – Overview of the changes made and the advantages of the change.

Introduction

As Scott indicated in his first post Introducing BabySmash, he started using techniques that he already knew. As a result the laughs and giggles played when a key is pressed where included as embedded wav files. When moving to the Silverlight environment this approach require some modifications to optimise the download experience and cater for the differences of running cross platform within a browser.

Code Changes

The original audio files used by the WPF application where wav files as in made use of class called Win32Audio, which wrapped calls to the underlying Windows unmanaged win32 api’s.

    public class Win32Audio
    {
        #region WIN32 Const Definitions
        public const UInt32 SND_ASYNC = 0x0001;
        public const UInt32 SND_LOOP = 0x0008;
        public const UInt32 SND_MEMORY = 0x0004;
        public const UInt32 SND_NOSTOP = 0x0010;
        #endregion

        // this is the overload we want to play embedded resource...
        public static Dictionary<string, string> cachedWavs = new Dictionary<string, string>();
        public static object cachedWavsLock = new object();

        [DllImport("winmm.dll")]
        public static extern bool PlaySound(byte[] data, IntPtr hMod, UInt32 dwFlags);

        [DllImport("winmm.dll", SetLastError = true)]
        private static extern bool PlaySound(string pszSound, IntPtr hmod, UInt32 fdwSound);

        public void PlayWavResource(string wav)
        {
            string s = GetWavResource(wav);
            PlaySound(s, IntPtr.Zero, SND_ASYNC);
        }

For the Silverlight version pinvokes to the underlying operating system are not allowed from  both a security point and cross platform point of view. The answer was to implement a new class XamlAudio that makes use of the MediaElement class present in both WPF and Silverlight to play the audio files.

    public class XamlAudio
    {
        private MediaElement player;
        readonly Queue<string> soundsToPlay = new Queue<string>();
        private bool inPlay;

        private Dictionary<string, Stream> media = new Dictionary<string, Stream>();

        //TODO Add lock and config value for sound queue limit
        public void PlayWavResourceYield(string s)
        {
            if (player == null)
                return;

            if (inPlay && soundsToPlay.Count < 5)
            {
                soundsToPlay.Enqueue(s);
                return;
            }

            PlaySound(s);
        }

The second major difference in the code is the shift from  the use of embedded audio files, as indicated in my earlier post Silverlight BabySmash Performance – The Asynchronous Story, the Silverlight version asynchronously downloads the audio files on demand. As result the initial download size for the deployed application is smaller.

Audio File Changes

The encoding format was also a major difference, changing from the use of wav files to Window Media  Audio files (*.wma). To convert the files I used Microsoft Expression Encoder.

The steps I used to convert the files where as follows:

  1. Download and install Microsoft Expression Encoder 2.
  2. Open Expression Encoder.
  3. Import the existing wmv files. Either using File Import menu or by licking on the Import button at the bottom of the window.
  4. Select the Encode tab.
  5. In the profile audio group expand the advanced properties.
  6. Change the properties to a smaller size to optimise the smaller size.
    1. Codec WMA
    2. Mode CBR
    3. Bitrate 20kps
    4. Sample Rate 44.1kHz
    5. Bits Per Sample 16
    6. Channels Mono
  7. Start the encoding process by click the encode button at the bottom of the window or the Encode File menu item.
  8. Once the encoding is complete select Output tab in the top right.
  9. Open the output folder by clicking o the Open Job button in the Job Output Section.

From the the output folder I copied the converted wma files to the ClientBin folder of the web site so that they where easily accessible  to the Silverlight application when using a WebClient asynchronous download request.

File WAV Encoding (kbps) WAV Files Size (KB) WMA Encoding (kbps) WMA File Size (KB)
babygigl2 20 6 20 15
babylaugh 88 14 20 11
ccgiggle 88 9 20 10
EditedJackPlaysBabySmash 352 227 20 25
falling 128 29 20 14
giggle 88 15 20 13
laugingmice 89 17 20 14
rising 128 26 20 13
scooby2 20 5 20 13
smallbumblebee 705 862 20 31

One interesting point from the comparison of the conversion is that for the small files with the same bit rate encoding the WMA files where larger. If further optimisation in terms of files size is required, then this is an area that could be further investigated to determine the exact encoding settings to trade-off audio quality against file size.

Summary

  • Changes to the C# code to make use of the built in MediaElement to play the audio files.
  • Cross platform audio support using wma files instead of wmv files.
  • Use of Expression Encoder to convert the source wmv files to a web optimised encoding.
# Sunday, November 16, 2008

Silverlight BabySmash Performance – The Asynchronous Story

Overview

In this blog entry I will cover some the factors effecting perceived performance and the asynchronous code used to optimise the user experience.

Introduction

When looking at the performance of the BabySmash Silverlight application it was important to look at a number of factors to help improve the perceived performance of the application. These included:

  • Initial download size.
  • Use of a secondary assembly.
  • Use of media files.

So let's start of by profiling the current application see what has been done.

Http Download Profile

Initial DownloadInitial DownloadSecondary DownloadSecondary DownloadLoad On DemandLoad On Demand

Looking at the HTTP profile of the application there are three distinct phases:

  1. Synchronous download of the initial html code and the Silverlight application.
  2. Asynchronous download of secondary components.
  3. Asynchronous download of media files on demand as the user presses keys.

Initial Download

The default.htm file together with the initial BabySmashWeb.xap combined are around 60kb. This xap file contains all the code necessary to interact with the user, request the additional resources and log information to the central ADO.Net Data Service.

Using this approach the loading phase of the the Silverlight application is very short and the user immediately shown the “Smash Canvas” as quickly a possible.

    <div id="silverlightControlHost">
        <object data="data:application/x-silverlight-2," type="application/x-silverlight-2" width="100%" height="100%">
            <param name="source" value="ClientBin/BabySmashWeb.xap"/>
            <param name="onerror" value="onSilverlightError" />
            <param name="background" value="white" />
            <param name="minRuntimeVersion" value="2.0.31005.0" />
            <param name="autoUpgrade" value="true" />
            <param name="initParams" value="logKeys=true" />
            <a href="http://go.microsoft.com/fwlink/?LinkID=124807" style="text-decoration: none;">
	            <img src="http://go.microsoft.com/fwlink/?LinkId=108181" alt="Get Microsoft Silverlight" style="border-style: none"/>
            </a>
        </object>
        <iframe style='visibility:hidden;height:0;width:0;border:0px'></iframe>
    </div>

Secondary Download

Once the application is loaded, the constructor of the main code behind Page class is called

        public Page()
        {
            InitializeComponent();

            var controller = new Controller();
            controller.Launch();
            mainPage = controller.Windows[0];

            KeyUp += new KeyEventHandler(Page_KeyUp);

            LayoutRoot.Children.Add(mainPage);
        }

Which calls starts a Controller class, which in turn starts the asynchronous requests for the letters and the initial media file.

       public void Launch()
       {
           CheckForUpdatesAsync();

           SetupInitialWindowState();

           //here to pre-cache letter
           CoolLetter.InitLetterStateAsync();

           //Startup sound
           audio.PlayWavResourceYield(".Resources.Sounds." + "EditedJackPlaysBabySmash.wav");
       }

The rest of this section will discuss how the letters stored in the secondary BabySmashEnglishLetters.xap are downloaded and stored for use when the user presses a key. The initial technique for loading the audio files is discussed in the implementation of PlayWavResourceYeild in the Load On Demand section below.

   public partial class CoolLetter 
	{
      public CoolLetter()
      {
         this.InitializeComponent();
      }

	    static readonly Dictionary<string, string> _letters = InitLetters();

        public static void InitLetterStateAsync()
        {
            // No work to do as the private _letters will be initialised above
        }

The InitLetters method is called by the .Net Framework when the static class is initialised. The method utilises a WebClient class instance to asynchronously download the required secondary assembly

       private static Dictionary<string, string> InitLetters()
       {
           var addressUri = new Uri("BabySmashEnglishLetters.xap", UriKind.Relative);
           var letterLoader = new WebClient();
           letterLoader.OpenReadCompleted += letterAssemblyLoaded;
           letterLoader.OpenReadAsync(addressUri);
           return new Dictionary<string, string>();
       }

Once the asynchronous call is completed, reflection is used to dynamically call the GenerateAlphanumericCharacters method of the AlphaNumericLetterGenerator class.

	    private static void letterAssemblyLoaded(object sender, OpenReadCompletedEventArgs e)
	    {
	        if ((e.Error != null) || e.Cancelled) return;

	        // Convert the downloaded stream into an assembly
	        var a = LoadAssemblyFromXap("BabySmashLetters.dll", e.Result);

	        var generator = a.CreateInstance("BabySmashLetters.AlphaNumericLetterGenerator");

	        if (generator == null) return;


	        var generated = generator.GetType().InvokeMember("GenerateAlpanumericCharacters", BindingFlags.Public | BindingFlags.InvokeMethod | BindingFlags.Instance, null, generator,null) as Dictionary<string, string>;
            if ( generated == null ) return;
	        foreach (var pair in generated)
	        {
	            _letters.Add(pair.Key, pair.Value);
	        }
	    }
Loading the assembly is achieved by extracting the requested assembly from the xap stream as follows.

	    public static Assembly LoadAssemblyFromXap(string relativeUriString, Stream xapPackageStream)
        {
            var uri = new Uri(relativeUriString, UriKind.Relative);
            var xapPackageSri = new StreamResourceInfo(xapPackageStream, null);
            var assemblySri = Application.GetResourceStream(xapPackageSri, uri);

            var assemblyPart = new AssemblyPart();
            var a = assemblyPart.Load(assemblySri.Stream);
            return a;
        }

Load On Demand

By the time that the third stage of load on demand media files is reached the main application is loaded, the secondary assembly is loading or loaded and the introduction sound file has played. This has resulted in the download of around 160kb of files and kept the time wait time for the user to a minimum.

Loading the external dependencies has also given us the following advantages:

  • Allow for the possibility of localising the application to other non-english languages.
  • Allow media files to be updated independently of the compiled code.
  • Allow the media files to to be sourced from alternative locations e.g. File stored locally on the client computer.
  • Reduced bandwidth from 406kb for users that visit the page but do not start using the application.

The process for loading the additional media files is similar to the process to load the assembly secondary xap file but has the additional complexity that the user can be pressing keys very quickly. As a result the application needs to respond very quickly by queuing the key requests then asynchronously downloading and playing the associated sound file.

        public void PlayWavResourceYield(string s)
        {
            if (player == null)
                return;

            if (inPlay && soundsToPlay.Count < 5)
            {
                soundsToPlay.Enqueue(s);
                return;
            }

            PlaySound(s);
        }

If the application is currently playing a media file then the request is placed on queue for later playback otherwise the sound will be played. If the media file is requested for the first time ten it will be asynchronously downloaded, added to the available media list then played.

        private void PlaySound(string resourceName)
        {
            var mediaFile = GetMediaFileFromResourceName(resourceName);

            lock (media)
            {
                if (media.ContainsKey(mediaFile) == false)
                {
                    var wc = new WebClient();
                    wc.OpenReadCompleted += delegate(object sender, OpenReadCompletedEventArgs e)
                                                {
                                                    lock (media)
                                                    {
                                                        if (media.ContainsKey(mediaFile) != false) return;
                                                        media.Add(mediaFile, e.Result);
                                                        player.SetSource(e.Result);
                                                    }
                                                };
                    wc.OpenReadAsync(new Uri(mediaFile, UriKind.Relative));
                    return;
                }
            }

            var sound = media[mediaFile];
            if (sound == null) return;
            if (sound.CanSeek) sound.Position = 0;
            player.SetSource(sound);
        }

If the media file is requested again then the version available in the media list will be reused and played to the user.

Summary

In summary when looking Silverlight applications one aim is to improve the perceived user performance by:

  • Quick Feedback – Provide quick feedback to the user so that the can interact with application. Small initial xap file allowed the “Smash Canvas” and initial introduction message where played without the overhead of other resources.
  • Thinking Asynchronously - Performing actions asynchronously so that the user action is not blocked while necessary processing is occurring.  Requesting media on demand allowed content to played as needed.
  • Separating The Application – Look at architecting the application so that it can be split into different parts that can be managed independently if required. Using separated code/media files allows for further expansion points and hosting options.
# Tuesday, October 28, 2008

BabySmash At PDC

Well Scott Hanselman has delivered his session at PDC NET Framework: Overview and Applications for Babies. And from the tweet streams it looks like he had a full room and fantastic feedback.

As Scott indicated in his earlier blog post he covers a lot of content in a short space of time so I would suggest watching the recorded video of his session TL49 (once it becomes available at http://channel9.msdn.com in the next 24 hours) that is now online from Channel9 or download the WMV file directly for offline viewing.

As an overview Scott’s presentation included different variants of his original WPF full framework  BabySmash application expanded to use different platforms & technologies i.e.

  • BabySmash Surface PC Demo.
  • BabySmash WPF application using MEF.
  • ADO.Net Data Service.
  • Windows Mobile Running Silverlight 2 BabySmash.
  • ASP.Net Charting.
  • ASP.NET MVC Charting.
  • BabySmash running under Silverlight 2 in the Browser.
  • Silverlight 2 Charting.

Scott ended the presentation by sending the address to the Silverlight version out to twitter users. As users hit the keys, the results were sent to the ADO.Net Data Service which then displayed the live results using a Silverlight Chart and Microsoft Virtual earth.

This is where my involvement started. Scott decided to use initial port of BabySmash to Silverlight that I created in his PDC presentation, to deliver on the MMO (Massively Multiplayer Online) request for BabySmash.

So the BabySmash Silverlight MMO PDC version was born so to speak. Now that Scott has publically demonstrated the Silverlight version, I will continue adding blog posts to my existing BabySmash series. In the new posts I will discuss further details of the Silverlight version of BabySmash and the Silverlight Reporting that Scott and I collaborated upon to create the final result for the presentation.

# Friday, August 08, 2008

BabySmash Silverlight Refactorings - Part 2 - Routed Events

One of the original problems with the initial port of BabySmash to Silverlight was loading the XAML storyboard animations for UserControls. In this post I will cover the changes required from a Silverlight point of view to replace the RoutedEvent in the UserControl.Triggers section with either equivalent XAML or C# code depending on your need.
# Monday, August 04, 2008

BabySmash Silverlight Refactorings - Part 1 - Adding Letters

Discussion of one alternative to using the FormattedText object that is not currently available in Silverlight.
# Saturday, August 02, 2008

BabySmash on Silverlight

Introduction to "BabySmash Web Edition". A Silverlight port of BabySmash, a Full Framework WPF application created by Scott Hanselman and other community contributors.