Overview

In this post I will provide an overview of the changes from the original wav media files, to wma files required by the Silverlight version.

The main sections are:

• Introduction – Discusses the background behind the use of audio files in the original WPF version and an overview of the changes required.
• Code Changes – Discusses the changes required to move from win32 api implementation to the MediaElement.
• Audio File Changes – Discusses uses Microsoft Expression Media Encoder 2 to convert the wav files to wma files.
• Summary – Overview of the changes made and the advantages of the change.
  

Introduction

As Scott indicated in his first post Introducing BabySmash, he started using techniques that he already knew. As a result the laughs and giggles played when a key is pressed where included as embedded wav files. When moving to the Silverlight environment this approach require some modifications to optimise the download experience and cater for the differences of running cross platform within a browser.

Code Changes

The original audio files used by the WPF application where wav files as in made use of class called Win32Audio, which wrapped calls to the underlying Windows unmanaged win32 api’s.

    public class Win32Audio
    {
        #region WIN32 Const Definitions
        public const UInt32 SND_ASYNC = 0x0001;
        public const UInt32 SND_LOOP = 0x0008;
        public const UInt32 SND_MEMORY = 0x0004;
        public const UInt32 SND_NOSTOP = 0x0010;
        #endregion

        // this is the overload we want to play embedded resource...
        public static Dictionary<string, string> cachedWavs = new Dictionary<string, string>();
        public static object cachedWavsLock = new object();

        [DllImport("winmm.dll")]
        public static extern bool PlaySound(byte[] data, IntPtr hMod, UInt32 dwFlags);

        [DllImport("winmm.dll", SetLastError = true)]
        private static extern bool PlaySound(string pszSound, IntPtr hmod, UInt32 fdwSound);

        public void PlayWavResource(string wav)
        {
            string s = GetWavResource(wav);
            PlaySound(s, IntPtr.Zero, SND_ASYNC);
        }

For the Silverlight version pinvokes to the underlying operating system are not allowed from  both a security point and cross platform point of view. The answer was to implement a new class XamlAudio that makes use of the MediaElement class present in both WPF and Silverlight to play the audio files.

    public class XamlAudio
    {
        private MediaElement player;
        readonly Queue<string> soundsToPlay = new Queue<string>();
        private bool inPlay;

        private Dictionary<string, Stream> media = new Dictionary<string, Stream>();

        //TODO Add lock and config value for sound queue limit
        public void PlayWavResourceYield(string s)
        {
            if (player == null)
                return;

            if (inPlay && soundsToPlay.Count < 5)
            {
                soundsToPlay.Enqueue(s);
                return;
            }

            PlaySound(s);
        }

The second major difference in the code is the shift from  the use of embedded audio files, as indicated in my earlier post Silverlight BabySmash Performance – The Asynchronous Story, the Silverlight version asynchronously downloads the audio files on demand. As result the initial download size for the deployed application is smaller.

Audio File Changes

The encoding format was also a major difference, changing from the use of wav files to Window Media  Audio files (*.wma). To convert the files I used Microsoft Expression Encoder.

The steps I used to convert the files where as follows:

1. Download and install Microsoft Expression Encoder 2.
2. Open Expression Encoder.
3. Import the existing wmv files. Either using File Import menu or by licking on the Import button at the bottom of the window.
4. Select the Encode tab.
5. In the profile audio group expand the advanced properties.
6. Change the properties to a smaller size to optimise the smaller size.
   1. Codec WMA
   2. Mode CBR
   3. Bitrate 20kps
   4. Sample Rate 44.1kHz
   5. Bits Per Sample 16
   6. Channels Mono
7. Start the encoding process by click the encode button at the bottom of the window or the Encode File menu item.
8. Once the encoding is complete select Output tab in the top right.
9. Open the output folder by clicking o the Open Job button in the Job Output Section.

From the the output folder I copied the converted wma files to the ClientBin folder of the web site so that they where easily accessible  to the Silverlight application when using a WebClient asynchronous download request.

One interesting point from the comparison of the conversion is that for the small files with the same bit rate encoding the WMA files where larger. If further optimisation in terms of files size is required, then this is an area that could be further investigated to determine the exact encoding settings to trade-off audio quality against file size.

Summary

• Changes to the C# code to make use of the built in MediaElement to play the audio files.
• Cross platform audio support using wma files instead of wmv files.
• Use of Expression Encoder to convert the source wmv files to a web optimised encoding.

Overview

In this blog entry I will cover some the factors effecting perceived performance and the asynchronous code used to optimise the user experience.

Introduction

When looking at the performance of the BabySmash Silverlight application it was important to look at a number of factors to help improve the perceived performance of the application. These included:

• Initial download size.
• Use of a secondary assembly.
• Use of media files.

So let's start of by profiling the current application see what has been done.

Looking at the HTTP profile of the application there are three distinct phases:

1. Synchronous download of the initial html code and the Silverlight application.
2. Asynchronous download of secondary components.
3. Asynchronous download of media files on demand as the user presses keys.

Initial Download

The default.htm file together with the initial BabySmashWeb.xap combined are around 60kb. This xap file contains all the code necessary to interact with the user, request the additional resources and log information to the central ADO.Net Data Service.

Using this approach the loading phase of the the Silverlight application is very short and the user immediately shown the “Smash Canvas” as quickly a possible.

    <div id="silverlightControlHost">
        <object data="data:application/x-silverlight-2," type="application/x-silverlight-2" width="100%" height="100%">
            <param name="source" value="ClientBin/BabySmashWeb.xap"/>
            <param name="onerror" value="onSilverlightError" />
            <param name="background" value="white" />
            <param name="minRuntimeVersion" value="2.0.31005.0" />
            <param name="autoUpgrade" value="true" />
            <param name="initParams" value="logKeys=true" />
            <a href="http://go.microsoft.com/fwlink/?LinkID=124807" style="text-decoration: none;">
	            <img src="http://go.microsoft.com/fwlink/?LinkId=108181" alt="Get Microsoft Silverlight" style="border-style: none"/>
            </a>
        </object>
        <iframe style='visibility:hidden;height:0;width:0;border:0px'></iframe>
    </div>

Secondary Download

Once the application is loaded, the constructor of the main code behind Page class is called

        public Page()
        {
            InitializeComponent();

            var controller = new Controller();
            controller.Launch();
            mainPage = controller.Windows[0];

            KeyUp += new KeyEventHandler(Page_KeyUp);

            LayoutRoot.Children.Add(mainPage);
        }

Which calls starts a Controller class, which in turn starts the asynchronous requests for the letters and the initial media file.

       public void Launch()
       {
           CheckForUpdatesAsync();

           SetupInitialWindowState();

           //here to pre-cache letter
           CoolLetter.InitLetterStateAsync();

           //Startup sound
           audio.PlayWavResourceYield(".Resources.Sounds." + "EditedJackPlaysBabySmash.wav");
       }

The rest of this section will discuss how the letters stored in the secondary BabySmashEnglishLetters.xap are downloaded and stored for use when the user presses a key. The initial technique for loading the audio files is discussed in the implementation of PlayWavResourceYeild in the Load On Demand section below.

   public partial class CoolLetter 
	{
      public CoolLetter()
      {
         this.InitializeComponent();
      }

	    static readonly Dictionary<string, string> _letters = InitLetters();

        public static void InitLetterStateAsync()
        {
            // No work to do as the private _letters will be initialised above
        }

The InitLetters method is called by the .Net Framework when the static class is initialised. The method utilises a WebClient class instance to asynchronously download the required secondary assembly

       private static Dictionary<string, string> InitLetters()
       {
           var addressUri = new Uri("BabySmashEnglishLetters.xap", UriKind.Relative);
           var letterLoader = new WebClient();
           letterLoader.OpenReadCompleted += letterAssemblyLoaded;
           letterLoader.OpenReadAsync(addressUri);
           return new Dictionary<string, string>();
       }

Once the asynchronous call is completed, reflection is used to dynamically call the GenerateAlphanumericCharacters method of the AlphaNumericLetterGenerator class.

	    private static void letterAssemblyLoaded(object sender, OpenReadCompletedEventArgs e)
	    {
	        if ((e.Error != null) || e.Cancelled) return;

	        // Convert the downloaded stream into an assembly
	        var a = LoadAssemblyFromXap("BabySmashLetters.dll", e.Result);

	        var generator = a.CreateInstance("BabySmashLetters.AlphaNumericLetterGenerator");

	        if (generator == null) return;


	        var generated = generator.GetType().InvokeMember("GenerateAlpanumericCharacters", BindingFlags.Public | BindingFlags.InvokeMethod | BindingFlags.Instance, null, generator,null) as Dictionary<string, string>;
            if ( generated == null ) return;
	        foreach (var pair in generated)
	        {
	            _letters.Add(pair.Key, pair.Value);
	        }
	    }

	    public static Assembly LoadAssemblyFromXap(string relativeUriString, Stream xapPackageStream)
        {
            var uri = new Uri(relativeUriString, UriKind.Relative);
            var xapPackageSri = new StreamResourceInfo(xapPackageStream, null);
            var assemblySri = Application.GetResourceStream(xapPackageSri, uri);

            var assemblyPart = new AssemblyPart();
            var a = assemblyPart.Load(assemblySri.Stream);
            return a;
        }

Load On Demand

By the time that the third stage of load on demand media files is reached the main application is loaded, the secondary assembly is loading or loaded and the introduction sound file has played. This has resulted in the download of around 160kb of files and kept the time wait time for the user to a minimum.

Loading the external dependencies has also given us the following advantages:

• Allow for the possibility of localising the application to other non-english languages.
• Allow media files to be updated independently of the compiled code.
• Allow the media files to to be sourced from alternative locations e.g. File stored locally on the client computer.
• Reduced bandwidth from 406kb for users that visit the page but do not start using the application.

The process for loading the additional media files is similar to the process to load the assembly secondary xap file but has the additional complexity that the user can be pressing keys very quickly. As a result the application needs to respond very quickly by queuing the key requests then asynchronously downloading and playing the associated sound file.

        public void PlayWavResourceYield(string s)
        {
            if (player == null)
                return;

            if (inPlay && soundsToPlay.Count < 5)
            {
                soundsToPlay.Enqueue(s);
                return;
            }

            PlaySound(s);
        }

If the application is currently playing a media file then the request is placed on queue for later playback otherwise the sound will be played. If the media file is requested for the first time ten it will be asynchronously downloaded, added to the available media list then played.

        private void PlaySound(string resourceName)
        {
            var mediaFile = GetMediaFileFromResourceName(resourceName);

            lock (media)
            {
                if (media.ContainsKey(mediaFile) == false)
                {
                    var wc = new WebClient();
                    wc.OpenReadCompleted += delegate(object sender, OpenReadCompletedEventArgs e)
                                                {
                                                    lock (media)
                                                    {
                                                        if (media.ContainsKey(mediaFile) != false) return;
                                                        media.Add(mediaFile, e.Result);
                                                        player.SetSource(e.Result);
                                                    }
                                                };
                    wc.OpenReadAsync(new Uri(mediaFile, UriKind.Relative));
                    return;
                }
            }

            var sound = media[mediaFile];
            if (sound == null) return;
            if (sound.CanSeek) sound.Position = 0;
            player.SetSource(sound);
        }

If the media file is requested again then the version available in the media list will be reused and played to the user.

Summary

In summary when looking Silverlight applications one aim is to improve the perceived user performance by:

• Quick Feedback – Provide quick feedback to the user so that the can interact with application. Small initial xap file allowed the “Smash Canvas” and initial introduction message where played without the overhead of other resources.
• Thinking Asynchronously - Performing actions asynchronously so that the user action is not blocked while necessary processing is occurring.  Requesting media on demand allowed content to played as needed.
• Separating The Application – Look at architecting the application so that it can be split into different parts that can be managed independently if required. Using separated code/media files allows for further expansion points and hosting options.

Well Scott Hanselman has delivered his session at PDC NET Framework: Overview and Applications for Babies. And from the tweet streams it looks like he had a full room and fantastic feedback.

As Scott indicated in his earlier blog post he covers a lot of content in a short space of time so I would suggest watching the recorded video of his session TL49 (once it becomes available at http://channel9.msdn.com in the next 24 hours) that is now online from Channel9 or download the WMV file directly for offline viewing.

As an overview Scott’s presentation included different variants of his original WPF full framework  BabySmash application expanded to use different platforms & technologies i.e.

• BabySmash Surface PC Demo.
• BabySmash WPF application using MEF.
• ADO.Net Data Service.
• Windows Mobile Running Silverlight 2 BabySmash.
• ASP.Net Charting.
• ASP.NET MVC Charting.
• BabySmash running under Silverlight 2 in the Browser.
• Silverlight 2 Charting.

Scott ended the presentation by sending the address to the Silverlight version out to twitter users. As users hit the keys, the results were sent to the ADO.Net Data Service which then displayed the live results using a Silverlight Chart and Microsoft Virtual earth.

This is where my involvement started. Scott decided to use initial port of BabySmash to Silverlight that I created in his PDC presentation, to deliver on the MMO (Massively Multiplayer Online) request for BabySmash.

So the BabySmash Silverlight MMO PDC version was born so to speak. Now that Scott has publically demonstrated the Silverlight version, I will continue adding blog posts to my existing BabySmash series. In the new posts I will discuss further details of the Silverlight version of BabySmash and the Silverlight Reporting that Scott and I collaborated upon to create the final result for the presentation.

One of the original problems with the initial port of BabySmash to Silverlight was loading the XAML storyboard animations for UserControls.

In this post I will cover the changes required from a Silverlight point of view to replace the RoutedEvent in the UserControl.Triggers section with either equivalent XAML or C# code depending on your need.

Refactoring Routed Event Changes

XAML in WPF allows tools such as Expression Blend to define triggered events without the use of code. In the original BabySmash XAML similar the following exists, which starts a storyboard when the user control is loaded e.g.

            <
            UserControl.Triggers
            >
          

            <
            EventTrigger
            RoutedEvent
            ="FrameworkElement.Loaded"
            >
          

            <
            BeginStoryboard
            Storyboard
            ="{StaticResource
EyesSB}"
            />
          

            </
            EventTrigger
            >
          

            </
            UserControl.Triggers
            >
          

As Shawn Wildermuth indicated on the Silverlight forums Routed Events are not available in Silverlight 2.0 Beta 2 as a result the event trigger either needs to be deleted or commented out in the XAML and alternative approach taken to achieve the same result.

Looking Richard Griffen's Silverlight Tweening Adventures with Baby Smash! MyTweenerTest it shows that the XAML can easily be refactored by moving the animations inside a BeginStoryBoard element. Hence not require the use of the Loaded event e.g.

            <
            UserControl.Resources
            >
          

            <
            BeginStoryboard
            x:Name
            ="Begin"
            >
          

            <
            Storyboard
            x:Name
            ="EyesSB"
            RepeatBehavior
            ="Forever"
            >
          

            <
            DoubleAnimationUsingKeyFrames
            BeginTime
            ="00:00:00"
            Storyboard.TargetName="CircleEye1"Storyboard.TargetProperty="(UIElement.Opacity)"RepeatBehavior="Forever">

            <
            SplineDoubleKeyFrame
            KeyTime
            ="00:00:02.1000000"
            Value
            ="1"
            />
          

            <
            SplineDoubleKeyFrame
            KeyTime
            ="00:00:02.1000000"
            Value
            ="0"
            />
          

            <
            SplineDoubleKeyFrame
            KeyTime
            ="00:00:02.300000"
            Value
            ="0"
            />
          

            <
            SplineDoubleKeyFrame
            KeyTime
            ="00:00:02.300000"
            Value
            ="1"
            />
          

            <
            SplineDoubleKeyFrame
            KeyTime
            ="00:00:7.300000"
            Value
            ="1"
            />
          

            </
            DoubleAnimationUsingKeyFrames
            >
          

            <
            DoubleAnimationUsingKeyFrames
            BeginTime
            ="00:00:00"
            Storyboard.TargetName="CircleEye2"Storyboard.TargetProperty="(UIElement.Opacity)"RepeatBehavior="Forever">

            <
            SplineDoubleKeyFrame
            KeyTime
            ="00:00:02.1000000"
            Value
            ="1"
            />
          

            <
            SplineDoubleKeyFrame
            KeyTime
            ="00:00:02.1000000"
            Value
            ="0"
            />
          

            <
            SplineDoubleKeyFrame
            KeyTime
            ="00:00:02.300000"
            Value
            ="0"
            />
          

            <
            SplineDoubleKeyFrame
            KeyTime
            ="00:00:02.300000"
            Value
            ="1"
            />
          

            <
            SplineDoubleKeyFrame
            KeyTime
            ="00:00:7.300000"
            Value
            ="1"
            />
          

            </
            DoubleAnimationUsingKeyFrames
            >
          

            </
            Storyboard
            >
          

            </
            BeginStoryboard
            >
          

            </
            UserControl.Resources
            >
          

However I wanted to look at a way of using the code to apply dynamic changes to the XAML and walkthough the process of refactoring.

Routed Event  - Replacement Using Code

In the next few steps I will introduce a series of refactorings that create the code behind and them reduce the amount of code that needs to be repeated.

1. The first refactoring defines a Loaded event in each of the shapes (e.g. Circle, Heart, Hexagon, Rectangle, Square, Star, Trapezoid and Triangle)
2. The next refactoring reduces the code by using a extension method to remove the need for the duplicated event handler implementation.
3. The third refactoring introduces a convention that the eyes animation storyboard should be called EyesSB and applied to UserControls.

Overall the aim of the refactorings are to:

1. Reduce the amount of code that has to be implemented by developers or designers.
2. Provides documented implementation of how the storyboard is applied.
3. Move the code required into a separate class that can be unit tested.
4. Allows new shapes to be created that do not require any additional code to be written.

Overall the effect is that the Routed event section is no longer requires in each of the XAML files.

Refactoring 1 - Inline Event

Change the code behind for each of the shapes and add a ShapedLoaded event e.g.

            public CoolHeart()

{

            this.InitializeComponent();

            this.Loaded
+= ShapeLoaded;

}

            private
            void ShapeLoaded(object sender,
RoutedEventArgs e)

{

  var eyes = FindName("EyesSB") as Storyboard;

            if (eyes
!= null)

      eyes.Begin();

}

Refactoring 2 - Extension Method

To aid with readability and reuse across all the shapes the refactoring above can further be improved. By moving to an extension method in a separate class. As a result we can remove ShapeLoaded from each shape e.g.

            public CoolHeart()

{

            this.InitializeComponent();

            this.Loaded
+= this.BeginStoryBoardAnimation("EyesSB");

}

With the definition of the extension method as follows

            /// <summary>
          

            ///
Helper class for XAML that performs common tasks
          

            ///
</summary>
          

            public
            static
            class XamlHelper

{

            ///
<summary>
          

            /// Begins the named storyboard
animation.
          

            /// </summary>
          

            ///
<remarks>No exception if thrown if the storyboard is not found</remarks>
          

            ///
<param name="control">The control that contains the storyboard</param>
          

            ///
<param name="storyboardName">Name of the storyboard to be started</param>
          

            ///
<returns>An event handler instance </returns>
          

            public
            static RoutedEventHandler
BeginStoryBoardAnimation(this UserControl control, string storyboardName)

    {

            return (source,
args) =>

                   {

                       var eyes = control.FindName(storyboardName) as Storyboard;

            if (eyes
!= null)

                           eyes.Begin();

                   };

    }

}

Refactoring 3 - Naming Convention

Remove the initialisation of the Loaded event from the constructor of each shape and update the FigureGenerator so that the EyesSB is started if it exists.

            public
            static UserControl
NewUserControlFrom(FigureTemplate template)

{

    UserControl retVal = null;

            //We'll
wait for Hardware Accelerated Shader Effects in SP1
          

            if (template.Letter.Length
== 1 && Char.IsLetterOrDigit(template.Letter[0]))

    {

          retVal = new CoolLetter(template.Fill,
template.Letter);

    }

            else
          

    {

          retVal = template.GeneratorFunc(template.Fill);

          retVal.StartStoryboardAnimation("EyesSB");

    }

With the updated definition of the extension method

            /// <summary>
          

            /// Helper class for XAML that performs common
tasks
          

            /// </summary>
          

            public
            static
            class XamlHelper

{

            /// <summary>
          

            /// Begins the named story board animation.
          

            /// </summary>
          

            /// <remarks>No exception if thrown if the
story board is not found</remarks>
          

            /// <param name="control">The control that
contains the story board</param>
          

            /// <param name="storyboardName">Name of
the storyboard to be started</param>
          

            /// <returns>An event handler instance </returns>
          

            public
            static RoutedEventHandler
BeginStoryBoardAnimation(this UserControl control, string storyboardName)

{

            return (source, args) => StartStoryboardAnimation(control,
storyboardName);

}

            /// <summary>
          

            /// Starts the named story board animation.
          

            /// </summary>
          

            /// <remarks>No exception if thrown if the
story board is not found</remarks>
          

            /// <param name="control">The control that
contains the story board</param>
          

            /// <param name="storyboardName">Name of
the storyboard to be started</param>
          

            public
            static
            void StartStoryboardAnimation(this UserControl
control, string storyboardName)

{

    var eyes = control.FindName(storyboardName) as Storyboard;

            if (eyes != null)

        eyes.Begin();

}

• Call a web service to dynamically convert the letters to the associated XAML path for a defines font.
• Create a small WPF application that generates the XAML path geometry and embed this in a C# dictionary object within the code.

The second of the two options was the quickest (code snippet below). In the future another enhancement could look at creating web service or using a xap file with different fonts.

The next piece to the process was to load the path data into a Geometry instance. Using Alex Golesh's PathFigureCollectionConverter the shapes are now loaded and the updated version now supports generation of letters. One problem I have now if that the closed in shapes are not rendering properly i.e. the O is a solid oval.

var code = new StringBuilder(); 

code.Append("var arial = new Dictionary<string,
string>();\r\n");

var lettersToCreate = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";

for (var i = 0;i < lettersToCreate.Length; i++)

{

    var letter = lettersToCreate.Substring(i, 1);

    var fText = new FormattedText(

    letter,

    CultureInfo.CurrentCulture,

    FlowDirection.LeftToRight,

new Typeface(

new FontFamily("Arial"),

        FontStyles.Normal,

        FontWeights.Heavy,

        FontStretches.Normal),

    300,

    Brushes.Black

    );

    var g = fText.BuildGeometry(new Point(0,
0)).GetAsFrozen() as Geometry;

// Get the Path info from the geometry

    var p = g.GetFlattenedPathGeometry();

    code.Append("arial.Add(\"" + letter
+ "\", \"" + p.ToString().Replace("F1M", "M
") + "\");\r\n");

}

After listening to Scott Hanselman's podcast on BabySmash I wanted to have a go at creating a proof on concept Silverlight "BabySmash Web Edition". I have embedded the current version at the bottom of the page if you want to skip the technical discussion.

The web edition has the following differences from the original BabySmash

• Uses Silverlight 2.0 Beta 2 instead Windows Presentation Foundation application for Windows XP/Vista.
  • Silverlight 2.0 download size much smaller (Around 4 MB)
  • No need to install the .Net Framework 3.5 (Around 33MB).
• Smaller download size for the application (Currently around 450KB).
• Hosted inside the web browser.
• Can be run on all platforms supported by Silverlight 2.0 Beta 2 (i.e. Window XP/Vista and Intel Mac's).

What are the limitations?

• Capturing system keys. Obviously limited to keys that the plugin  can capture.
  • For my son I use an external usb numeric keyboard similar to one made by Targus.
• Mouse click limited to left click only. Right click brings up Silverlight Configuration

What is working?

• Keyboard and basic mouse integration.
• Ability to play audio.
• Basic options dialog UI.

What is missing?

• Animations (Character blinks & fades).
• Click animations.
• ClickOnce deployment/updates - Not needed for web version.
• Letters.
• Loading and Saving options.
• Text to speech.

Work yet to do:

• Suggest refactoring's to allow common WPF/Silverlight codebase.
• Install experience for users that do not have Silverlight 2.0 installed.
• Complete the options dialog.
• Open the options dialog by clicking in each of the corners of the control.
• Animations (Character Blinks & fades).
• Add support to display letters.
• Add full screen mode.
• Mouse scroll wheel support.
• Test with experimental 2.0 Moonlight plugin.
• Include sound files for letters and numbers.
• Separate out sound files into another downloadable component.

So here is the current version. Any suggestions/feedback appreciated.

By using the application please note the BabySmash Licensing Terms