Optimizing IIS Smooth Streaming for CPU-constrained devices
February 2, 2012 2 Comments
Today’s media delivery landscape is highly heterogenous – modern software solutions can expect to serve users over wildly different network connections and on multiple software and hardware platforms. In such a situation it is very cost-effective to use a single media delivery technology to serve all of these viewers.
When we combine the use of standard codecs such as H.264 and AAC with an adaptive streaming technology and a sufficiently clever player implementation, this can be the reality – the choice of widely supported codecs means that the media stream will play on every modern platform and the use of adaptive streaming means that viewers will be served content that matches their available bandwidth and computational resources.
One of the more widespread adaptive streaming technologies is IIS Smooth Streaming, developed by Microsoft. Together with the server-side technology, Microsoft also provides a player implementation for the Silverlight platform. This article gives you an overview of how to configure the Silverlight player for optimal behavior on CPU-constrained devices.
Standard behavior of the Silverlight Smooth Streaming client
The heuristics engine that controls the active quality level takes most of its guidance from the available bandwidth, measured by looking at how long fragments of the media presentation take to arrive at the player. This information is very accurate and the player can respond to changing network conditions within seconds. In all but the most extreme situations, the user will never notice any pause in playback caused by changing network conditions.
Another important aspect of the playback experience is to ensure that there is enough computational power – CPU and GPU resources – available to play back the media without dropping frames. The main indicator for this is the number of video frames that could not be displayed because they were decoded too late – the count of dropped frames.
Unfortunately, the default heuristics are highly effective at bandwidth-focused quality level selection but hopelessly optimistic when it comes to matching the quality level to available computational power. A lower quality level is selected only when the playback experience is truly horrible and even then, the player will soon try again to play the quality level that it could not handle. The end result for the viewer is that video playback is always jittery and every minute or two, becomes even worse as an even higher quality level is momentarily and unsuccessfully tried.
Thus, the default heuristics of the Silverlight Smooth Streaming client are not capable of ensuring an adequate playback experience on devices where CPU constraints are the limiting factor.
For a visual demonstration of the differences between optimized and unoptimized Smooth Streaming playback on Silverlight, please see this annotated image of various playback quality level indicators:
Adjusting the heuristics
The parameters used by the Silverlight Smooth Streaming client when performing its heuristics calculations are easily configurable. However, public documentation on this does not exist. At best, you can find a forum post from 2010 with guesses about the effects various settings. I will describe here the most relevant settings; there are many more but they do not affect the topics covered by this article. If you would like to know more about the other settings, as well, please leave a comment.
The settings file is a simple XML file that should be included in your Silverlight media player (with Build Action = Content) and its path specified in the ConfigPath property of the Smooth Streaming media element. If your media player is based on Microsoft Media Platform: Player Framework, then all you need to do it place the file at the root of your project and name it Config.xml – the file will be loaded automatically, without you having to set its path.
Inside the XML file, we have the root element LiveSmoothStreamingSettings and more elements under this, one for each configurable component of the Smooth Streaming Client. The one of interest for us is named HeuristicsEngineSettings.
To make sure that the playback experience is acceptable on CPU-constrained devices, we need to demand a near-perfect rendering quality – if any video frames are dropped, we have to consider it a sign that the quality level is too high. Additionally, we need to make sure that the built-in optimism of the heuristics engine does not too eagerly try to play higher quality levels.
To make the playback engine paranoid of any dropped frames, we add the following attributes to the HeuristicsEngineSettings element:
Note that the names of the attributes are inaccurate – these are not percentages but ratios (i.e. 1 is the maximum value, not 100). These values were derived experimentally and seem to work fine for lowering the quality level on devices such as low powered netbooks, while simultaneously preserving high quality level playback on high-end devices.
The framerate heuristics are fairly inaccurate, so it helps to increase the size of the history it uses to perform its calculations, by adding the following attributes:
Even with these settings, the player will try to constantly use a higher bitrate thanks to its built-in optimism. This can cause problems where the end-user will experience periodic stuttering of the video every few minutes. However, we can significantly decrease this optimism by also adding the following attributes:
Finally, we have arrived at a configuration that forces the player to be very careful about trying to play high quality levels. There is still one thing left to do – the media itself must be designed to support these optimizations.
Optimizing the media streams
If the differences in processing cost between quality levels are very large (e.g. a jump from 500 kbps to 1500 kbps), the playback experience can end up being undesirable even with optimized heuristics – simply because the player is unable to slowly ramp up the quality until it hits an obstacle.
My recommendation is to ensure that the maximum gap between quality levels is around 300 kbps. For example, the low-end streams should have bitrates of 700, 1000, 1300 and 1600 kbps, perhaps with even smaller differences between them. Low-end devices tend to hover around 1 Mbps quality levels, so further away from this point, the spacing can be increased.
Minimizing undesirable side-effects
One problematic side-effect of these settings is that it might now take longer for the media player to climb to high quality levels. This is intentional but might give a bad first impression for users on high-end devices. My recommended fix for this is to manually override the startup quality level – by default, the Silverlight Smooth Streaming client starts playing the lowest quality level and quickly climbs up from that. In my experience, 1 Mbps is the ideal quality level to start with when using the optimized settings.
It is relatively simple to implement this – when the ManifestReady event is raised, adjust the set of allowed allowed quality levels to only include the level closest to 1 Mbps. Once the player has actually started playing the video, reset the set of allowed quality levels to include all of them – the player heuristics will take over from this point.
Hardware-accelerated video decoding… is useless
Silverlight 5 includes support for decoding the video using the GPU. This greatly improves playback on all the supported devices and may make the optimizations listed in this article unnecessary, since even CPU-constrained devices will be able to play HD video at very high quality levels.
However, there exists a major obstacle here - hardware-accelerated video decoding does not work for DRM-protected videos! Since almost all commercial content will require the use of DRM technologies such as SilverHD DRM, this feature of Silverlight 5 is effectively useless for most large-scale scenarios.
Thus, for the foreseeable future, these optimizations will remain of vital importance to ensure the best playback experience.
Here is a Silverlight Smooth Streaming Client configuration file that includes all of the abovementioned settings.
<?xml version="1.0" encoding="utf-8"?> <LiveSmoothStreamingSettings> <HeuristicsEngineSettings FrameRateHeuristicsSuspendTrackIfMoreThanXPercentDroppedFrames="0.05" FrameRateHeuristicsFlushBufferIfMoreThanXPercentDroppedFrames="0.05" FrameRateHeuristicsSuspendNextLowestTrackIfMoreThanXPercentDroppedFrames="0.25" FrameRateHeuristicsOverallHistorySize="16" FrameRateHeuristicsTrackHistorySize="12" FrameRateHeuristicsResetTrackHistoryAfterXSecondsOfInactivity="20" SwitchingFrequencyChunksBeforeUpgrade="15" SwitchingFrequencyStepSizeForUpgrade="1" FrameRateHeuristicsMaxSuspensions="2" /> </LiveSmoothStreamingSettings>