Vista Multimedia Playback and Network Throughput


A few weeks ago a poster with the handle dloneranger reported in the 2CPU forums that he experienced reduced network throughput on his Vista system when he played audio or video. Other posters chimed in with similar results, and in the last week attention has been drawn to the behavior by other sites, including Slashdot and Zdnet blogger Adrian Kingsley-Hughes.


Many people have correctly surmised that the degradation in network performance during multimedia playback is directly connected with mechanisms employed by the Multimedia Class Scheduler Service (MMCSS), a feature new to Windows Vista that I covered in my three-part TechNet Magazine article series on Windows Vista kernel changes. Multimedia playback requires a constant rate of media streaming, and playback will glitch or sputter if its requirements aren’t met. The MMCSS service runs in the generic service hosting process Svchost.exe, where it automatically prioritizes the playback of video and audio in order to prevent other tasks from interfering with the CPU usage of the playback software:




When a multimedia application begins playback, the multimedia APIs it uses call the MMCSS service to boost the priority of the playback thread into the realtime range, which covers priorities 16-31, for up to 8ms of every 10ms interval of the time, depending on how much CPU the playback thread requires. Because other threads run at priorities in the dynamic priority range below 15, even very CPU intensive applications won’t interfere with the playback.


You can see the boost by playing an audio or video clip in Windows Media Player (WMP), running the Reliability and Performance Monitor (Start->Run->Perfmon), selecting the Performance Monitor item, and adding the Priority Current value for all the Wmplayer threads in the Thread object. Set the graph scale to 31 (the highest priority value on Windows) and you’ll easily spot the boosted thread, shown here running at priority 21:




Besides activity by other threads, media playback can also be affected by network activity. When a network packet arrives at system, it triggers a CPU interrupt, which causes the device driver for the device at which the packet arrived to execute an Interrupt Service Routine (ISR). Other device interrupts are blocked while ISRs run, so ISRs typically do some device book-keeping and then perform the more lengthy transfer of data to or from their device in a Deferred Procedure Call (DPC) that runs with device interrupts enabled. While DPCs execute with interrupts enabled, they take precedence over all thread execution, regardless of priority, on the processor on which they run, and can therefore impede media playback threads.


Network DPC receive processing is among the most expensive, because it includes handing packets to the TCP/IP driver, which can result in lengthy computation. The TCP/IP driver verifies each packet, determines the packet’s protocol, updates the connection state, finds the receiving application, and copies the received data into the application’s buffers. This Process Explorer screenshot shows how CPU usage for DPCs rose dramatically when I copied a large file from another system:




Tests of MMCSS during Vista development showed that, even with thread-priority boosting, heavy network traffic can cause enough long-running DPCs to prevent playback threads from keeping up with their media streaming requirements, resulting in glitching. MMCSS’ glitch-resistant mechanisms were therefore extended to include throttling of network activity. It does so by issuing a command to the NDIS device driver, which is the driver that gives packets received by network adapter drivers to the TCP/IP driver, that causes NDIS to “indicate”, or pass along, at most 10 packets per millisecond (10,000 packets per second).



Because the standard Ethernet frame size is about 1500 bytes, a limit of 10,000 packets per second equals a maximum throughput of roughly 15MB/s. 100Mb networks can handle at most 12MB/s, so if your system is on a 100Mb network, you typically won’t see any slowdown. However, if you have a 1Gb network infrastructure and both the sending system and your Vista receiving system have 1Gb network adapters, you’ll see throughput drop to roughly 15%.


Further, there’s an unfortunate bug in the NDIS throttling code that magnifies throttling if you have multiple NICs. If you have a system with both wireless and wired adapters, for instance, NDIS will process at most 8000 packets per second, and with three adapters it will process a maximum of 6000 packets per second. 6000 packets per second equals 9MB/s, a limit that’s visible even on 100Mb networks.


I caused throttling to be visible on my laptop, which has three adapters, by copying a large file to it from another system and then starting WMP and playing a song. The Task Manager screenshot below shows how the copy achieves a throughput of about 20%, but drops to around 6% on my 1Gb network after I start playing a song:




You can monitor the number of receive packets NDIS processes by adding the “packets received per second” counter in the Network object to the Performance Monitor view. Below, you can see the packet receive rate change as I ran the experiment. The number of packets NDIS processed didn’t realize the theoretical throttling maximum of 6,000, probably due to handshaking with the remote system.




Despite even this level of throttling, Internet traffic, even on the best broadband connection, won’t be affected. That’s because the multiplicity of intermediate connections between your system and another one on the Internet fragments packets and slows down packet travel, and therefore reduces the rate at which systems transfer data.


The throttling rate Vista uses was derived from experiments that reliably achieved glitch-resistant playback on systems with one CPU on 100Mb networks with high packet receive rates. The hard-coded limit was short-sighted with respect to today’s systems that have faster CPUs, multiple cores and Gigabit networks, and in addition to fixing the bug that affects throttling on multi-adapter systems, the networking team is actively working with the MMCSS team on a fix that allows for not so dramatically penalizing network traffic, while still delivering a glitch-resistant experience.


Stay tuned to my blog for more information.


Comments (156)

  1. Anonymous says:

    "God help us all if anybody tries to use Vista for important stuff like medical equipment monitoring."

    If someone was so eager to play mp3s and videos on medical equipment monitors, they probably don’t deserve to live anyways.

    I speculate the presumption of MSFT is that people don’t do Gigabit networking and HD multimedia at the same time. Given that HD videos usually compressed with processing expensive algorithm, it was reasonable to give the MM process a boost if someone was trying to stream it across the network and it would still well under the limit of 100Mb network. Consider the fact that Vista is a desktop OS I doubt the networking performance got the same treatment and priority that the Windows Server can get.

    They probably will fix the network scheduling algorithm by the time of releasing Server 2008 and Vista SP1. (I hope)

  2. Anonymous says:

    These links may provide some additional info on "Gigabit network suffering slow (relatively speaking) transfer rates"-

    http://web2.minasi.com/forum/topic.asp?whichpage=1&TOPIC_ID=25099#116458

    which will point you to –

    http://blog.tmcnet.com/blog/tom-keating/microsoft/remote-desktop-slow-problem-solved.asp

    Also look into (with an elevated command prompt)-

    netsh interface tcp set global autotuninglevel=disabled

    This will set the TCP window size to what is normally used.  In Vista and Longhorn they use a larger window size which works great with  Vista and Longhorn.

    To reset it you would enter –

    netsh interface tcp set global autotuninglevel=normal

  3. Anonymous says:

    Peter – yes, I’ve seen heart monitor goes beep. I’ve even seen a real expensive machine that goes "Bing!" in action 🙂

    And no, you don’t need HD audio for beeps and more importantly those life-critical equipment usually runs on RTOS instead of Windows. AFAIK, windows has never been certified to run on life-supporting system.

    Non life-support system such as visualization systems(those hi-res video you’re talking about) and management console, however, often times they DO deploy windows. Just so you know, if you do loss your life over machines, chances are it isn’t windows.

  4. Anonymous says:

    It seems to me that Vista has things backwards here. Rather than ensuring that music playback can be performed within its realtime constraints, and having programmable facilities within the kernel to ask for that, it’s arbitrarily degrading other bits of the system "just in case" they might interfere. The dependency is incorrect. It ought to be the scheduler making these kinds of decisions – not the sound subsystem.

    It smells more like a nasty hack – as if last-minute testing showed that there were glitches in playback with the latest kernel, so quick fixes needed to be bludgeoned in.

  5. Anonymous says:

    Ignoring multi-core issues for the moment, realize that DPCs are implemented with a queue — new DPCs are entered either at the front (High Importance DPC) or at the rear (regular DPCs) by interrupt handlers.  The irpt handler is most concerned with getting DPC queued and then re-enabling the hardware interrupts.  As a DPC completes its processing, the next DPC (if any) is removed from the queue and processed. The important take-away is that NO scheduled threads can be run while pending DPCs are present.

    If the audio RT threads MUST run every 10ms, then it only takes some DPCs queued just before or during the RT thread’s execution to delay the final mixing and output enough to hear a glitch.

    So why not give the Audio a larger output buffer so DPC delays don’t matter?  Remember that you must fill the buffer before starting the audio to play, and the larger the buffer, the longer the "delay" until the sound is heard from the speakers.  A good musician can detect audio delays in the order of 4ms or so, and most people will notice 20ms delay, particularly in cases like watching video showing the source of an impulse sound such as a drum.  So, keeping a 15ms buffer from under-run requires 10ms audio thread scheduling AND keeping DPCs from delaying that thread by more than say 4ms.  (Actually, multiple threads must run every 10ms because audio data is "pulled" from upstream sources, and these sources must execute within the "pull" to generate new audio data.)  I hope this gives a little insight into how difficult it is to keep audio from glitching.

    So that leaves "optimizing" the DPC stuff to prevent it from delaying the RT audio. (I don’t know the network stack like I do audio, so I’ll only attempt to discuss the issue.)  For a high-end datapoint, a 1gbit LAN interface with a bunch of minimum-sized packets (say 64 bytes each) would queue a DPC about every 2 microseconds.

    Because DPCs "preemptively interrupt" the executing thread, sharing any type of data between the DPC and a thread is complicated.  Suppose the DPC needs to look at something like the protocol handler table, and suppose that network threads must frequently "update" this table.  What if a DPC interrupts the thread doing a table update operation?  The DPC can’t look at the table because the data may be inconsistent, but the DPC can’t go to sleep to let the thread finish updating the table. So, as threads update the table, they must be able to acquire write ownership on it and this might require disabling interrupts during the update.  But what if that thread is swapped with a higher priority thread while IRPTs are disabled?  It’s a really complicated problem to solve correctly.

    In practice, this serialized-data-access problem means that a DPC must share as little data as possible with the associated threads.  But, for example, if the DPC can’t even determine which protocol queue to drop the packet into, networking breaks down.  I suspect that the current DPC-based network handling is forced to maintain DPC-specific state and thus have to process more of the packet in the DPC to avoid the access serialization problems.  Maybe things like firewall packet filtering must be done in the DPC, as well.  At some point, even the packet’s data must be mapped or copied into the appropriate application’s buffer, so maybe the DPC does this as well.

    Bottom line is that I don’t know exactly why the networking DPCs can’t do minimal processing, but solving the problem suggests that Microsoft will have to get this stuff out of the DPC environment and into RT thead scheduling, like they did with multimedia.

    Eventually, I’m sure Mark will write an excellent article on the "new network RT scheduling algorithm" being released in an upcoming OS. For now, the "fix" will probably be some improved feedback from the audio RT system to let the network throttling vary in response to the unused time in the 8ms RealTime audio window.

    Hope this spew of mine is at least a little helpful in comprehending the difficult design tradeoffs I see were made in Vista’s audio and network stacks.

  6. Anonymous says:

    The more I read about this issue the stranger it becomes. There are so many problems it’s even hard to know where to start.

    1. Why does the audio system require the following:

    "For instance, in Vista, the audio engine runs with a periodicity of 10 milliseconds. That means that every 10 milliseconds, it MUST wake up and process the next set of audio samples, or the user will hear a "pop" or “stutter” in their audio playback. It doesn’t matter how fast your processor is, or how many CPU cores it has, the engine MUST wake up every 10 milliseconds, or you get a “glitch”."

    Audio is not a hard real time system, it should be a soft real time. You process enough ahead of time and send it to the audio card’s buffer, that if you miss a period by say +/-5 ms it does not cause a problem. Unless the OS doesn’t trust the audio card.

    The part with, "it doesn’t matter how many CPUs you have" is also quite wrong. If I have more CPUs, then there’s a better chance one of them should be free, or not have enough load to be able to process every 10ms.

    "So it doesn’t matter how much horsepower your machine has, it’s about how many interrupts have to be processed."

    But won’t a machine with more horsepower handle interrupts faster? And this gets us into problem 2.

    2. "Network DPC receive processing is among the most expensive, because it includes handing packets to the TCP/IP driver, which can result in lengthy computation. The TCP/IP driver verifies each packet, determines the packet’s protocol, updates the connection state, finds the receiving application, and copies the received data into the application’s buffers."

    Why does it need to verify each packet, one would think that can be done on the network card. Packet protocol, update of connection and receiving application should be figured quite fast for today’s computers, and I do hope that it is not copying the whole data, and just updating pointers (I assume the data has already been copied from the network drive since it does checksum on it). And check this link out (from 2003 none the less):

    http://www.microsoft.com/whdc/device/network/NetAdapters-Drvs.mspx

    "TCP and IP Checksum Offload: For most common network traffic, offloading checksum calculation to the network adapter hardware offers a significant performance advantage by reducing the number of CPU cycles required per byte. Checksum calculation is the most expensive function in the networking stack"

    "In the Windows Performance Lab, we have measured TCP throughput improvements of 19% when checksum was offloaded during network-intensive workloads."

    And then there’s 3, which is quite funny in itself – hard codding a throttling value in code. This is from the same article from 2003 from MS (different application, but same principle):

    "One reason to load the table from the registry is to avoid hard-coded values in the driver. This also allows for experimenting with the table entries to find the optimal values for several representative workloads."

  7. tom says:

    Great article Mark, thank you for giving a clear explanation of the problem.

  8. Serge Munger says:

    is it the reason that explain why my games are A LOT less performant with windows Vista? does playing audio in games and having Teamspeak (or ventrilo at the same time) reduce the graphic and procesor output for my games because audio is at higher priority (especially in MMORPG where the network is important)?

    is there a way to put this value back to normal?

    thanks

  9. Richard McBeef says:

    Running XP I just tried playing an MP3 file, a video file and downloading several files from the internet all at the same time.  The audio and video files played perfectly and there was no slowdown in my network speed.

    "mechanisms employed by the Multimedia Class Scheduler Service (MMCSS), a feature new to Windows Vista"

    Seems to me Microsoft tried to "fix" something that wasn’t broken.

  10. Doug says:

    Very useful to know. I’ve spent a good bit of time trying to figure out why my file transfers were so slow. During all of that debugging, I probably had a music file running in the background…

    Serge: Games performance is probably more related to video driver issues. My understanding is that the new Vista video driver model is theoretically just as fast as the XP video driver model, but it requires a lot of stuff to be re-written by the driver developers for best performance. Until it all gets re-written, they’re using pieces from the XP driver (changing Vista’s input to what the XP driver expects, then changing the XP driver’s output to what Vista expects), and the result is that some tasks are done less efficiently.

    On my laptop, I could hardly get any video files to play without glitching. CPU usage would go up to 100% and stay there, even though the media file played just fine with only 20% CPU usage on XP. Then by accident I was running another program that was incompatible with Aero and forced Vista to switch into the XP-style video mode. Suddenly my video started playing back smoothly at 20% video usage, just like XP.

  11. Matt says:

    Serge,

    I don’t own Vista yet, but the first thing I thought of when I started reading this post was "games".  Good question…

  12. Anonymous says:

    Mark,

    A sincere thank you for explaining the issue in detail and not “suger coating” it—you might renew my faith in Microsoft.

  13. Dr.Butt says:

    Running XP I just tried playing an MP3 file, a video file and downloading several files from the internet all at the same time.  The audio and video files played perfectly and there was no slowdown in my network speed.

    Monday, August 27, 2007 5:28 PM by Richard McBeef

    *****

    It won’t effect your internet speed; just your LAN speed.

  14. Courtney says:

    my first thought was to disable the MMCS service, but the Windows Audio service is dependent on it.

    So I ran regedit, and changed the key

    HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesAudiosrvDependOnService

    Just remove MMCS from that key in the list, and set MMCS to disabled in services, then reboot.

    As soon as I rebooted I was able to copy files at 40mb/s+ while listening to audio

    -Courtney

  15. Andrew says:

    Why is XP not having this problem then? This seems like unnecessary feature of Vista.

    I’ve spent countless hours for couple months trying to figure out why copying files over network was going at 5MB/s

  16. zzz says:

    "While DPCs execute with interrupts enabled, they take precedence over all thread execution, regardless of priority, on the processor on which they run"

    Am I reading this correct: The DPCs can all run on core #1 and multimedia can run on the other cores? -> no interruption of multimedia?

  17. Dan_IT says:

    Serge: No, the MCSS service is only used by multimedia applications like WMP (actually that’s probably all that uses it ATM).

    You can revert to old behavior by right clicking My Computer, clicking Manage, finding the Services entry on the left pane and finding the MCSS service mentioned and stopping it.  To make this setting permanent, you have to go into properties and set the startup type to Disabled.  However it will not make your games run any faster. (Hmm, Courtney notes it’s dependent on the Windows Audio user-mode stack and offers a workaround… I didn’t know that.  Thanks Courtney!)

    Games are slower overall because of all the added functionality and features in Vista… even with many services, eyecandy, and programs disabled I still find programs run better in XP (and even better in Linux!).  For example, BioShock runs horribly on my computer in Vista with input and audio lag making it unplayable.  This didn’t surprise me too much as my computer didn’t meet the minimum specs in the CPU department.  But, in XP, it ran quite acceptably.  I was even able to up the details settings without performance degradation.

    I would recommend gamers to stick to Windows XP for the time being, unless you have a DirectX 10 video card and plan on playing DirectX 10 games (and you really, REALLY need DirectX 10 for some reason).  Also note that DirectX 10 cards will not be able to take advantage of new DirectX 10.1 features in Vista SP!.  You’d need a new card for those.

    Richard: As noted in the article, you have to have quite high bandwidth (over 100mbits/s) to be able to notice any slowdown.  Also there was something broken they fixed… audio stuttering or jittering in WMP and other media applications.  Now if they only did the same thing for games I might switch over to Vista permanently.

    Doug: Unless you’re speaking of LAN transfers on a 1gbps network, I doubt that was your problem.

    I have noticed problems when running on an IPv4 network.  IPv6 drivers are enabled by default for every network adapter, meaning every time you try to connect to a remote computer/website, Vista tries using IPv6 first.  If your network doesn’t support it, this only results in wasted time.  You can disable this protocol by finding the moved "Network Connections" folder (go to Network Control Panel and click on whatever side entry relates to showing network adapters) and then right clicking Properties on desired adapters and unchecking the IPv6 protocol entries.

    Andrew: XP can have this problem if enough apps are suching up the CPU.  Some multimedia apps, like Winamp, solve the problem themselves by boosting their own priority (Winamp runs at High by default) but this can be dangerous if done to High or above because the application is not stable because then it may freeze the entire system (fortunately Winamp is stable.  I’ve never had problems in that area).  

    Whenever I have an application which is running a tad pokey, especially games, I boost it to Above Normal using Process Explorer or Task Manager (cmon Mark, fix Procexp so it can replace Taskmgr when UAC is enabled).  This works with most programs (a few games go wonky though).

    barrkel: I think all programmers do this,  It’s called implementing the required and requested features with the least amount of work.  We’re a lazy people. 😉

    zzz: I’m guessing you could minimize DPCs and interrupts on one core by putting only threads essential to music playback there, but most likely this would make the core underutilized.  And just like with memory, you don’t want to leave it unused because you just end up wasting time which you could be saving by filling it with work.

  18. adam says:

    An excellent explanation Mark.

    Maybe I am oversimplifying things, but you mention it limits to 10,000 packets. Wouldn’t the obvious solution be to increase this number (or make it configurable). Obviously if you make it too high, your songs can skip. If you have a multi-core CPU (as most will be in the next few years), then it would seem to me to be unnecessary. Perhaps Windows could detect whether this optimisation is really helpful.

  19. Vinicius Canto [MVP] says:

    Great post Mark!

    Vinicius Canto

    MVP Windows Server – Admin Frameworks

    Brazil

  20. Brie Bruns says:

    "I have noticed problems when running on an IPv4 network.  IPv6 drivers are enabled by default for every network adapter, meaning every time you try to connect to a remote computer/website, Vista tries using IPv6 first.  If your network doesn’t support it, this only results in wasted time."

    Actually, the time wasted should be very little.  Unless the hostname has an AAAA record associated with it, the system shouldn’t use IPv6 to try and communicate with the host.

    The only time it becomes a major delay, is when the remote host supports IPv6, and either the local or remote host isn’t properly configured for IPv6.  Then, it has to time out, then switches to IPv4.

    IIRC, Teredo is on by default in Vista, which means (provided the teredo relay isn’t down and firewall isn’t blocking) you have functioning IPv6 on your end enough to be useful.

  21. Tanveer Badar says:

    Windows Vista does allow a fine grained control of priority boost, you just need to find where to look for it instead of disabling the service (MMCSS).

    http://msdn2.microsoft.com/en-us/library/ms684247.aspx

  22. F16PilotJumper says:

    Please elaborate on why this kludge is required in Vista, when XP degrades gracefully in both the Single CPU+High Network Load+Multimedia case, and the insufficient resources to process multimedia properly case.

    See:
    http://episteme.arstechnica.com/eve/forums/a/tpc/f/99609816/m/910004196831?r=628001007831#628001007831

  23. Bill2 says:

    Why do Vista need seuch a system ?

    Ever an old Win2K, on un PIII 1Ghz, with 512Mb can play an audio file without glitchs during network transfert.

    Today, with have "big" CPU, and Vista cannot play audio file smoothly during network transfert ? Such a shame …

  24. Courtney Malone says:

    I wrote up the details on my fix on my blog

    http://courtneymalone.com/2007/08/28/a-note-on-vista-network-speed/

  25. Julian W says:

    Regarding multiple network cards. If I disable the other network cards will the performance go up (you said 3 cards 1/3 network performance?)

    also whats up with  this key?

    HKLMSoftwareMicrosoftWindows NTCurrentversionMultimediaSystemProfileSystemResponsiveness

    seems this is an easy registry edit to allocate less cpu time for MMCSS

  26. Jerry Schneider says:

    I developed lots of audio drivers for the older Microsoft OSes, and interestingly, had to use the DPC to process audio mixing and such so that it wouldn’t glitch on systems with lots of thread activity.  The trick to avoid using all the CPU cycles for audio was to keep track of how much time was being spent in the DPC vs. the scheduled threads.

    In Vista, trying to "guess" how much of the CPU is being used for MM operations is unrealistic, since it spans the domain from 128kb MP3 playback to DRM’ed HD audio/video playback.  

    The MMCSS algorithm defaults to 8ms for MM threads and 2ms for the other threads, but the flaw in the algorithm is that the IRPT and DPC times are not subtracted from the wall-clock times.

    Suppose that a really busy network transfer used up 50% of the cpu in the DPC routines.  MMCSS should then see that only 4ms is available for mm out of the 8ms maximum.  The audio stack must do it’s part, then, by letting the MMCSS know if that 4ms was enough to process 10ms of output.  For simple audio, it may need only 1ms to generate 10ms of audio output.  OTOH, it may need more than 4ms to insure no glitching, and THAT is the point that the network should be throttled back.

    I think Vista gives most of the info needed to do this right — the new kernel thread time accounting now subtracts DPC (and probably IRPT) from the thread’s scheduling quantum.  The audio stack "pulls" output from the hardware back through the various drivers, which should allow determination of how well the computational load is being handled.

    Looks like SP1 is needed now more than ever!  Tell Balmer to forget the market-timing issues and just get a Vista SP out in the field.  I’m even thinking seriously about ripping Vista off my new laptop and desktop if SP1 is not going to appear in the near future.

    Thanks for your as usual well-stated description of the problem!

  27. Steve says:

    Mark,

    I’ve been following your insights for years. Once again, thank you for the excellent explanation. We have 2 users on the network who use Vista and have complained w/network issues.

    Once again Mark, thanks for the years and all the great utilities.

    PS: I use XP but have tried Vista. After spending over 15 hours of learning and tweaking, I finally gave up and uninstalled it. After 3 weeks of hard disk thrashing (I continually fought w/Vista services which "magically" reset to default thrash mode), it became futile. You may want to pass up the chain, it is for these reasons Vista will not be quickly adopted.

  28. Gigel says:

    All good and nice, but :

    1.  It seems that people have this problem even AFTER they disabled MMCSS.

    2.  The performance impact is way bigger (from 500..1000 mbps to sub-100mpbs or even lower. This is by no means a minor impact.

    3. As other pointed out,it seems like a classic problem of "Shoot yourself in the foot"  because previous versions of windows do not have this issue nor audio playback smoothness problems.

    4. If you really want to help maybe you should take a look on the original forum page (http://forums.2cpu.com/showthread.php?t=83112) and see that the impact reported by other users and what you explain here does not quite add up.

    5. As a personal note : with today’s computing power  this kind of problem on dual/quad core machines is ….well I don’t know if I should say "funny" or "pathetic" 🙂

  29. c3 says:

    "100Mb networks can handle at most 12MB/s, so if your system is on a 100Mb network, you typically won’t see any slowdown".

    I think you need to learn a bit about networking. Not every frame is full. At half full, the limited number of frames is reached at 7.5MB/s, well within a 100Mb networks ability.

  30. Jimbo says:

    This whole thing reeks of shoddy hack.

    Seriously, who’s idea was this? Have they been taken out and shot yet?

  31. tomthebomb says:

    sell out. you where all about discovering and fixing bugs/glitches, or even malware. now you are part of the MS propaganda. this kind of problem shows how flawed windows vista is. davinci would have torn the whole thing apart and started again.

  32. SteveO says:

    Does this take into account jumbo frames if you’re using a gigabit network?  If your frame size is, say, 9000 bytes, at 10000 packets/sec and 8bits/packet, that’s 720Mbit (theoretical) throughput.  That’s still less than 100% but it’s not horrible…

  33. Mr. Obvious says:

    This doesnt make any sense design wise.

    Why does every OS so far, even WinXP and every *NIX OS doesnt have any trouble handling multimedia + network ?

    Furthermore pc hardware get faster over time not slower – the thinking behind this design implies that hardware gets slower because previously in other OS it worked fine.

    Once again, braindead developers at M$ which doesnt surprise me at all

  34. Dave says:

    So a non-privileged userland application is able to modify global kernel parameters?

    Sounds like a good idea to me.

    Any other bits of the kernel I can mess with from a non-priv account?

  35. Vajk says:

    Why have to guess a good value for max packets/sec while playing audio? Why not have some intelligent monitoring to find if audio playback thread is starving for cpu? Like check for empty buffers? Profile normal playback cpu needs, and make sure it receives that?

    vajk

  36. Rob says:

    What I find most discouraging about this isn’t the hack to workaround what was probably a non-issue, but the fact that <i>copying a file</i> takes 41% of the CPU.  What kind of networking stack has that kind of processor overhead?

  37. Chris says:

    To me, this seems like overly optimistic resource allocation…

    I agree 100% that it needed to happen, (and I’m greatful for Mark’s detailed response/ explanation) however, it seems to be at so much overhead that it degrades other services…

    Given the multi-core CPU scenario we live in now, is this optimization even so necessary?

    I remember trying to set up MS ISA 2 years ago on a SMP server (in a hurry) and got very poor performance due to "CPU’s fighting over controlling the NIC’s" – (I realize that now that you can pin a NIC to a particular CPU)… Is this similar?

    Is there any way to pin or isolate these competing processes such that we are not stuck with such low limits to utilization?

    thanks again for a great explanation Mark.

  38. Zachary Pruckowski says:

    What I think a few posters are missing is the fact that this has nothing to do with overall CPU speed.  Overall, CPUs are fast and getting faster.  This has to do with priority over the course of milliseconds.  

    Both networking and media playback require instant results – because TCP/IP gets processed when it’s received, and because most audio is sampled at over 40 kHZ.  On a 2 GHz machine, that means that it’s playing something every 22 microseconds (about once every 45,350 processor cycles, and it takes a few cycles to play something).  If it delays too long, or drops a few samples, you hear skipping.

    As a result, the concern is that if both network functions and multimedia need the CPU *right now* then you have a collision.  That’s why MS limited the networking so it only comes in on average once every 100 microseconds, to prevent that.

  39. K says:

    One addition: Microsoft completely rewrote the TCP/IP stack for Vista, and doing so they surely made some… hm, mistakes. This might be another reason of strange behavior.

    See http://www.microsoft.com/technet/community/columns/cableguy/cg0905.mspx and lots of other pages.

  40. Jay Levitt says:

    Actually, multimedia glitching is a pretty well-known problem to heavy audio and video users (i.e. multitrack work, editing, not just playback).  A whole alternate universe of drivers has sprung up to deal with this.  It’s much less of a problem than it used to be, but under heavy loads, it’s still an issue.

    Sounds like Microsoft tried to be over-proactive about it, and (as others have said) shot themselves in the foot.  I wonder if this isn’t something that went into Vista back in 2001 as an obvious necessity, and then wasn’t looked at toward the end of the release cycle…

  41. Zachary Pruckowski says:

    Responding specifically to Chris, my understanding is that Vista can’t do the kind of things you talk about (segregate functions by processor) even if it would be an effective solution, because they simply can’t target dual-core machines yet.

    There’s a significant adoption lag that Microsoft has to adapt to.  Let’s briefly look at the gaming world for an example, and then come back to this.  Currently, game designers/programmers for many games have to make sure the game can run on computers are far back as pre-HT P4s.  Therefore, they can’t fully optimize for a two-core world across the line yet.  

    The same situation occurs with Vista.  Given the average age of the PC install base, and Vista’s minimum CPU requirements (800 MHz single core), they can’t optimize for dual-core in that sort of a manner.  Also, remember that Vista was designed in a period from 2001-2006.  Dual-cores only became available in mid-2005, and weren’t really prevalent in most selling models until mid-2006 (release of Conroe in July, and subsequent price war).  The Vista RTM was shortly thereafter.

  42. s t says:

    Something as essential as music playback and file copying should not require days of investigation by users. Someone write better docs!

    PS – I have a *phone* running a 200 MHz ARM with a piddly little OS and am able to play streaming MP3’s on a broadband wireless network with no hassles. And a dual-core, 2 GHz desktop uses 41% CPU to simply copy files???

  43. s t says:

    Something as essential as music playback and file copying should not require days of investigation by users. Someone write better docs!

    PS – I have a *phone* running a 200 MHz ARM with a piddly little OS and am able to play streaming MP3’s on a broadband wireless network with no hassles. And a dual-core, 2 GHz desktop uses 41% CPU to simply copy files???

  44. guy says:

    I did some playing on my XP system and got similar results as Richard.  I transfered a copy of Win2KSP4 while I listened to a podcast I have downloaded on my local system.  I did not see any decay in the file transfer speed over the LAN.   I do not consider Mark a sellout because he reported  on this issue.  He identified it, he did not condone it.  IMHO, this seems like a misplaced performance tweak on MS part.  I hope it goes away in SP1.  We still have not moved to Vista, I have intentionally ordered new systems with XP.  It works and I am not willing to turn my shop into a test bed for a first release of an OS.

  45. Sean Siler [MSFT] says:

    Hi, I am the IPv6 Program Manager at Microsoft.  

    Regarding the comment "Unless the hostname has an AAAA record associated with it, the system shouldn’t use IPv6 to try and communicate with the host."

    Even if the destination has an A and an AAAA record, Vista will prefer IPv4 over Teredo.  The order of precedence is IPv6, IPv4, THEN Teredo.  So a destination host with an A and AAAA record will always be reached using IPv4, NOT TEREDO.

    The only time Teredo would be used is if the destination host ONLY had a AAAA record, and there are darn few of those out there.  In other words, leaving IPv6 (and Teredo) enabled on your home PC have absolutely no impact on your networking performance.

  46. Lothar says:

    Rob: … <i>copying a file</i> takes 41% of the CPU.  What kind of networking stack has that kind of processor overhead?

    I regarded this kind of behavior when copying files using SMB. The same file copied from the same server using FTP was many times faster.

    Regards, Lothar

  47. Erik says:

    @Chris   re: (I realize that now that you can pin a NIC to a particular CPU)

    can you elaborate?  I’m looking around and can’t find any info – this could be helpful on one of my servers.

    Thanks

  48. Mike E says:

    I have had a ton of issues with media playback in Vista on both single core and dual core platforms, with and without Aero even when it could easily support it.  My solution has been to shutdown as many ancillary services as possible, and there are a lot.  Most, if not all of the security services are gone, and that was strictly for my sanity.  Many of the disk services and indexing are shutdown. I do actually know where my content is and don’t need any help finding it.  And then I shutdown some more things that just seemed to be hanging out and not really providing any immediately useful service.  The result is a fairly smooth running system that runs aero, the sidebar, other applications, and my i-tunes videos full screen without glitches.  Prior to shutting all of these things down, i-tunes videos were unwatchable, streaming internet video (ie simple slideshows) were unwatchable, and the whole media experience was enough to make me beg for XP back, or even 2000 pro.  My disk access has gone from essentially constant to only when I’m actively doing something.  Now I have my LED for power that stays on, and my disk activity LED is no longer solid 24/7.  Things that ran fine on XP systems with less than half the performance of the current system now run like they should.  Moving large files across the Gigabit network also move like they should.  Vista still won’t play wmv files that run fine on my other XP boxes.  Winamp will play them just fine on VISTA, but Media Player says there is a problem with my WHQL video card.  

    Why all of this?  There are many other issues with Vista playing media than the one network issue mentioned here and eliminating many of the ancillary services will go a long way, but not nearly all the way, to solving them.  There error codes that Windows Media Player provide are of course useless because there is no description for them.  I would have thought that all of the effort going into the driver certification process and application certification process would have resulted in a more stable and better performing system, but alas this has not been the case.

  49. Redesign It says:

    > Despite even this level of throttling, Internet

    > traffic, even on the best broadband connection, won’t

    > be affected.

    This is only a valid analogy for someone downloading a single file off the Internet from a single source.

    Throw in P2P-anything and you get huge number of small packets which Vista will happily throttle for you [sigh], regardless of your actual bandwidth.

    One problem is you use packets/sec regardless of how ‘full’ those packets are.  Another is that Vista apparently has a horribly inefficient TCP stack… 40% CPU usage for a file copy?

  50. Ken Davis says:

    "Games are slower overall because of all the added functionality and features in Vista… even with many services, eyecandy, and programs disabled I still find programs run better in XP (and even better in Linux!).  For example, BioShock runs horribly on my computer in Vista with input and audio lag making it unplayable.  This didn’t surprise me too much as my computer didn’t meet the minimum specs in the CPU department.  But, in XP, it ran quite acceptably."

    – Not to mention all the DRM throughout the audio and video stack.  Benefitting who exactly?  Not really the consumer, they’re better off with XP arguably.  DX10 runs a fair bit quicker on XP and Linux (with the "backported" un-official release), that surely means something is very wrong.

  51. lee says:

    So this explains the problems with MCE’s having stutter problems? As they push data over the network while doing ‘media playback’ at the same time?

    The Transcode360 forums are full of this problem too.

  52. gvb says:

    Correction: "the standard Ethernet frame size is about 1500 bytes" should be "the *MAXIMUM* standard Ethernet frame size is about 1500 bytes."

    This is significant because it is a "well known fact" that most ethernet frames are a lot less than the maximum length.  It is only when pumping lots of data (e.g. file transfers) that maximum size frames are used heavily, and even then the acknowledgment frames going back to the data source are typically 64 bytes (minimum size).

    The minimum frame is 64 bytes, which works out to about 164,000 frames/second so throttling at 10,000 frames/second will throttle the network a *whole* lot worse than the calculation based on 1500 byte packets.

    Note that there is the same amount of overhead in a 1500 byte frame and a 64 byte frame, meaning the amount of useful data that gets passed in shorter frames is even worse.

  53. LinuxGuy says:

    Robert Love (Linux Kernel Developer and author of Linux Kernel Development) has a good article comparing Linux’s implementation vs. Vista’s – he states that even on Gigabit Ethernet Linux equivalent of Vista’s DPC is unable to generate more than minuscule amount of CPU Usage.

    So can it be concluded that Vista’s Network performance is abysmal and to plug that Microsoft had to play around with prioritizing Multimedia and penalizing the network performance? Why would this ugly hack be required if Vista network implementation did not consume this much CPU.

    Mark – Answer is really appreciated. Thanks.

    Here is the URL for Robert Love’s blog post – http://blog.rlove.org/2007/08/those-dang-dpcs-clogging-mmcss.html

  54. Neil Prestemon says:

    The real questions, I suppose, have to do with the various virtualization scenarios – XP under Vista, or Vista under XP, and whether this resource allocation can be "fooled" under these circumstances.

  55. foobarbaz says:

    I just tried killing MMCSS along with the registery hack for the audio service, and it works like a charm. I was stuck at 10% transfers, and now I hit 80%. Fantastic! The interesting thing is why I was experiencing this before. Well, the majority of my network transfers are with media files, which means that I open a folder with lots of media files (i.e., AVIs) and copy them to another folder with lots of media files. No music playing, no video playing, and yet Vista slows down. Why? Because explorer is busy creating previews of all those media files, hence kicking in MMCSS and it’s silly slowdown! Well there you go, problem solved, another useless piece of software goes down.

  56. Anon says:

    And, yes, the same hardware running Linux can play back a glitch free mp3 while keeping the network link fully loaded with send and receive data.

    The answer to this particular problem – switch to Linux.

  57. SteveO says:

    "The minimum frame is 64 bytes, which works out to about 164,000 frames/second so throttling at 10,000 frames/second will throttle the network a *whole* lot worse than the calculation based on 1500 byte packets."

    True, but how often does one send out more than 10000 little frames per second?  Usually when you’re transferring more than 10000 frames it’s because they’re part of, for example, a large TCP stream, and those frames are likely the maximum size, i.e. that of the MTU.

  58. Pete R says:

    Mark,

    Thanks for an honest discussion of the issue. It’s sad that MS can spend billions developing Vista and an obvious magic-number hack (and a buggy one too!) like this is acceptable for production release to the world.

  59. Leo Davidson says:

    "PS – I have a *phone* running a 200 MHz ARM with a piddly little OS and am able to play streaming MP3’s on a broadband wireless network with no hassles. And a dual-core, 2 GHz desktop uses 41% CPU to simply copy files???"

    Does your phone transfer files at gigabit network speeds? I presume it does not so it appears you didn’t read the article and are making an invalid comparison.

  60. Leo Davidson says:

    "Why would this ugly hack be required if Vista network implementation did not consume this much CPU."

    It seems to be a scheduling/throttling issue, rather than one caused by something using too much CPU. The problem is that fixed magic numbers were chosen, perhaps based on the worst-case hardware, and obviously without thinking about the requirements of gigabit networks.

    I don’t think Vista is actually using 100% CPU in this situation; the problem is that it’s still throttling the network "just in case" of something that won’t happen.

    With gigabit networks so common now, and for years, it’s surprising that this slipped through but it looks like it’s being addressed and it’s great to have it explained openly, and in detail, so soon after the issue was diagnosed.

  61. LinuxGuy says:

    Leo Davidson said –

    "It seems to be a scheduling/throttling issue, rather than one caused by something using too much CPU. The problem is that fixed magic numbers were chosen, perhaps based on the worst-case hardware, and obviously without thinking about the requirements of gigabit networks."

    If I could do network I/O without taxing the CPU I would not land into trouble with scheduling and neither would I need to do any throttling. The problem is not magic numbers themselves – the problem lies int the fact that they were needed.

    There is no excuse to use more than tiny amount of CPU to do network I/O even at Gigabit speeds if the OS and networking stack are designed sanely. Read the blog post I linked to.

  62. Stewart says:

    I’m not so sure that the issue as described here scopes the symptoms broadly enough.  I have a 100Mb network, 3GHz PC and all the latest Realtek HD audio drivers. I generally use iTunes connected my file server where my audio files are stored, and a number of other players for DVD and movie files both locally and on the server. The PC was previously happily running Windows 2000 and 2003 Server (it’s my development box), so I know it’s not the hardware.  Under Vista, sound output is dire- sound playback hops, skips and squeaks to the extent that I now do my editing on another PC running Win2k.  I have the same problem on a Sony AR31S which also has HD Audio playback.  

    Using the same perfmon settings as above, I’m not seeing the same profile.  My my network access is very ‘spikey’- zero to 400kps spikes while iTunes plays MP3.  

    All these drivers are relatively frequently updated and I’ve noticed that the performance changes somewhat between versions (up and down), but is never resolved. Even setting iTunes to use it’s max buffer size (which would hopefully overcome a variable speed network problem) does not help.

    I feel that the problem is perhaps deeper in the HD Audio system that Vista introduces.  The early Realtek HD drivers were rubbish and if I recall MS pushed a HD Audio fix early one- perhaps there’s more to learn and resolve?  Unfortunately I’ve not the network diagnostics and audio driver skills to provide better diagnosis.  I cannot identify any perfmon counters to measure HD Audio drivers, but iTunes CPU usage runs around 5-15%.

    I’m just hanging out for a proper fix (doh!)  and wait for a fix and currently play music on a regular CD player.  Bummer.

  63. FusionGuy says:

    It’s refreshing to hear a valid, lucid explanation of what the issue really is instead of the background "noise" created by the FUD mongers at Slashdot.  Thank you once again for leading the way to understanding core Windows technologies.

  64. Lucio3 says:

    "It’s refreshing to hear a valid, lucid explanation of what the issue really is instead of the background "noise" created by the FUD mongers at Slashdot.  Thank you once again for leading the way to understanding core Windows technologies."

    This is a "lucid explanation" of a really BAD DESIGN.

    This is not FUD, its purely bad design… Do you know another OS in the world that needs more than 40% of the processor to handle a GigE traffic?

  65. Dupe says:

    Do you even know how to measure kernel time as a percentage of total processing time in any OS?  I’ll give you a hint — it doesn’t show up in a typical CPU utilization chart.

  66. CableGuy says:

    This is not only a Windows problem. You can find the same on many other OS.

  67. Leo Davidson says:

    Lucio, nobody is claiming their isn’t a problem here. MS (or at least two MS employees in their blogs) are being quite open about the fact there is a problem that needs fixing.

    The FUD that I believe FusionGuy was referring to was people jumping to baseless conclusions that this issue was caused by the evil multimedia DRM in Vista and other such nonsense.

    It’s great that Mark and Larry (http://blogs.msdn.com/larryosterman/default.aspx) from Microsoft have been able to shed light on this new issue in such detail and I hope it puts an end to the FUD. The issue itself still needs to be solved but it’s clear that it’s being taken seriously, being worked on by the people that need to fix it, and not being hidden or excused.

  68. mike says:

    Mark – sorry, I know you’re one of the most credible living authorities on Windows, but you have a lot to learn about IP networks.

    A "standard Ethernet frame" isn’t 1500 bytes at all – as has been pointed out, the minimum is just 64 bytes, and a realistic average for a TCP stream would be somewhere between 600 and 900 bytes. If you’re using (the increasingly popular) UDP or some other connectionless protocol, the average tends to get smaller.

  69. I like sausages says:

    "It’s great that Mark and Larry (http://blogs.msdn.com/larryosterman/default.aspx) from Microsoft have been able to shed light on this new issue in such detail and I hope it puts an end to the FUD. The issue itself still needs to be solved but it’s clear that it’s being taken seriously, being worked on by the people that need to fix it, and not being hidden or excused."

    No.  This is where you’re downright WRONG.

    There should be no "issue" in the first place.  Have you read any of the notes in this thread?  Re-read everything, re-read all of them.  Everyone is saying the same thing: how exactly did a FLAW like this get 1) created, 2) pass QA/testing, 3) shipped, and 4) not noticed after the release?

    What needs to happen is that Microsoft needs to fire the FTE (or terminate the contract of the dashtrash individual) who idealised/designed the methodology used, and also needs to terminate whoever was involved in the writing/production of the related code.

    Yes, someone needs to get fired over this.  I will repeat myself: FIRED.  Terminated.  No more job.  No more free Starbucks coffee, no more little ball-rubbing managerial meetings with PMs, no more time wasting.  Whoever wrote the code needs to be axed, NOW.  Oh, but I’m sure if they’re a FTE, they won’t be terminated, because Microsoft never fires FTEs — they just relocate them into other depts. and let them make the same mistakes over and over.  Or maybe they’ll become a PM.  😉

    I’m well aware of Microsoft’s business practises — I mean, everyone who works there has taken Microsoft’s SBC 2007, right?  SBC talks about *taking responsibility* and making good decisions, and that’s what Microsoft supposedly prides itself on doing.  Was this engineering flaw even remotely a good decision?  No.  Was this engineering methodology discussed with other programmers?  Probably.  And no one chimed in with "Uh, this probably isn’t a good idea, Joe…"?  Fire them all.

    So who’s going to pay?  Whose ass is on the line?  Oh right, I forgot: the consumer’s.  Every Vista customer will suffer because one or two half-ass engineers at Microsoft decided "this way is AWESOME".

    This entire engineering flaw proves that Microsoft is intentionally hiring programmers who need to "think opposite of UNIX" (and not "think outside the box"), does *absolutely NO form of decent QA* on their products, and simply put, doesn’t give a rats ass about a better end-user/consumer experience — all they care about is more services, more abstraction (WINDOWS MEDIA LIBRARY ENUMERATION DEVICE MANAGEMENT SERVICE!!!! YEAH!!!), and more CRAP.

    Do not tell me Vista needs a high-end P4 CPU when you’ve got _engineering flaws_ like this in your device ABI.  This is atrocious.  Back to what I said originally: someone, or some people, need to get FIRED over this.

  70. Dick Martin Shorter says:

    Mark, I commend to your notice the eHome tool VPT – contact John Pennock for details. Using this tool, I generated an ~85Mbps outbound TCP stream (on a 100bT network), which immediately fell to ~42Mbps when a local media file was played. This seems a bit more intrusive than your analysis indicated.

    VPT allows you to drive configurable bit-rates, as well, both in TCP and UDP protocols, using your choice of packet lengths (or msg sizes for some tests).

    You might find it interesting that playing media on the sending side of this flood test results in a larger performance penalty than playing on the receiving side. (That’s what’s described above) Also, increasing the message size to ~10 MTU almost completely eliminates the effect, at least on the receiving side.

    Hope this tool helps shed some light on the matter, when used by more capable hands than mine.

  71. Tim B says:

    Sounds like MS still don’t understand scheduling… Or just presume that no one watches a DVD while doing other things.

    I tend to agree with "I like sausages"… The people responsible for this should be fired, or at least demoted to a level where they don’t touch code at all (janitor, marketing, graphical people, I don’t care).

  72. samwyse says:

    "Tests of MMCSS during Vista development showed that, even with thread-priority boosting, heavy network traffic can cause enough long-running DPCs to prevent playback threads from keeping up with their media streaming requirements, resulting in glitching."  I’m going to assume that it wasn’t "normal" media streaming, but DRM-protected media.  And probably not music, but video.  However, the biggest question that I have is who designed the IP stack so that it handles packets for the TCP/IP driver.  Both TCP and UDP are designed to throttle back if the CPU can’t keep up with the traffic.  The DPC should be moving the packet into a buffer and then returning, not calculating TCP checksums.  Let someone interrupible do that; if the buffer fills up in the meantime, let the retransmit algoithms deal with it.

  73. Adam Zey says:

    I wish the limit was lower yet. On my wireless network, transferring data at even moderate speeds (a few megabits per second) can cause glitching in audio playback. And this is with a fast dual-core processor.

  74. Chris says:

    @Erik:

    We had several Dual Processor P4’s with Hyperthreading, and we were trying to set up MS ISA 2004 on Windows 2003 SP1.  The servers had multiple NIC’s, and we ran network throughput testing software between the servers either side of the ISA servers we were testing.

    I think it was the combination of using NIC bonding (aka Gigabit-etherchannel) as well as having multiple processors.

    It seemed like the nics were continually being handled by different CPU’s with some sort of context switch penalty when this happened.  We could only manage something like 70Mbits through these servers when the theoretical maximum should have been 2 Gigabit.

    We ran out of time and so replaced the servers with linux which did not exhibit the problem, (could exceed at least 1 gigabit throughput)

    However afterwards I read somewhere that there is a setting somehow to enforce each nic to a particular cpu only, which may give better performance.

    http://support.microsoft.com/kb/252867 explains  how to set the processor affinity to a single nic, as per the ISA recommendation in KB293640.  

    My point to Zachary is that it’s not about optimizing code specifically for Hyperthreading/SMP, but more that since it’s very common now to have HT or SMP that something would need to be done to counteract this symptom.  Perhaps the TCP/IP stack was fixed past Windows 2003 SP1 or in Vista so this is no longer applicable…

  75. Jeff Brown says:

    The whole trick is to enforce unfair rules in a way the user expects.  Prioritizing audio/video playback over network traffic is very reasonable indeed!  The throttling leaves somewhat to be desired though…

    DPCs in the TCP/IP stack being long-running and non-preemptible (except by other interrupts) seems a little problematic.  Once the critical sections of the interrupt handlers are finished, is there any reason not to queue up the remaining work so it can be scheduled fairly against other processes?

    The current approach essentially results in priority inversion on behalf of the processes that are doing all of the heavy I/O because the time spent in the kernel doing interrupt processing is not charged against them.  When that happens scheduler priority classes become largely irrelevant and the rest of the system starves.  It might also undermine rate management algorithms depending on how much buffering is going on.

    Is it possible to prevent this case through more accurate bookkeeping that takes into account the amount of time spent performing kernel services "on behalf of" a user-space process?  If the consuming process can be identified early enough in the processing chain, then subsequent processing could be deferred unless the process has enough available CPU cycles in its budget for it to proceed.  Thus a process that performed very heavy network I/O could only starve other processes if it resided in a sufficiently elevated priority class.

    Essentially, that would make "realtime" priority classes meaningful in the face of DPCs.  But it sounds like a whole lot of work.

    So…

    What happens when using multi-core CPUs?  Are ordinary processes allowed to run concurrently with DPCs?  If so, can this property be used as part of the algorithm to tune throttling?

  76. tck says:

    Surely setting multimedia playback to the highest possible priority means that it takes all the CPU time it needs and the rest can be used for other processes (i.e., handling network traffic). What’s the point of having prioritized processes otherwise? It seems like there’s some kind of fundamental flaw in Vista process scheduling. God help us all if anybody tries to use Vista for important stuff like medical equipment monitoring.

  77. james says:

    At such a low speed this might be due to something else.

  78. ccj says:

    Obviously, if network was not demanding so much CPU, you could give more to the audio subsystem. But Vista would not probably be Vista if it did.

  79. Tom M says:

    I very much disagree with "I like sausages".

    Someone tells a story (and I can’t remember who) about a programmer who makes a mistake which costs his employer a huge sum of money (millions, maybe even 10s of millions). He gets called into see the boss, fully expecting to be fired, and gets a thorough roasting. At the end of the meeting he asks the boss about handing in his security badge, but the boss replies "I just spent x-million on your education. Why would I want to fire you?"

  80. Jerry Schneider says:

    "Heads must roll?".  Before you pass out from over-ranting, go learn about the difference between a real-time OS and one that schedules threads of various priority.

    Oversimplifying, real-time means being able to GUARANTEE a RT thread will receive cycles either periodically and/or for a specified period.  A trivial example is the engine computer in your new car — it must be able to schedule the proper sparkplug to fire within a few microseconds at high rpm.

    OSes designed for real-time don’t make for very good interactive, graphical user experiences, since it’s not important, for example, for all threads, particular the majority that do things like draw pixel, to run at a precisely specified time.  For this use, you want an OS that can schedule threads using variable priority, while avoiding priority inversion and such, to get a responsive interface.

    The basic desktop PC OSes (MS, *nix) use the latter kernel scheduling design because you want a nice user experience.  As you can probably deduce, there are a few types of functions running on the desktop OS that require GUARANTEED cycles at specific times, and if they don’t get their "tick" EVERY time, you hear a pop or click or video streak or ….

    So how do you implement such a hybrid OS?  The old way (before Vista) was to schedule and run the RT-critical stuff in the Interrupt/DPC arena, leaving the scheduler to handle everything else as interruptible threads.  The problem is that as your add more and more time-critical processes that need guaranteed cycles, you start writing a mini-scheduler that controls the stuff in the IRP/DPC world, and it’s really difficult to sync and interact this with the other world of scheduled threads.

    Vista, using MMCSS and new dynamic thread priority/importance mechanism, is attempting to move this Real-Time stuff back under a generic thread-scheduling design. One could argue that all of the DPC stuff in Vista should be converted to scheduled RT threds, and that would prevent anything except uncontrolled interrupt handlers from delaying RT functionality.

    In RTM Vista, it looks like the network hardware stuff is still running in DPCs that are capable of stalling the RealTime stuff.  The obvious question is why are the DPCs doing so much?  If you think it’s easy to do things like avoid deadlocks when you have DPCs and RT threads sharing data, then it won’t be obvious to you why a network DPC may have to do more than minimal operation on a chunk of data.  

    Blaming "some idiot FTE" for these design problems isn’t pointing the gun in the right direction.  Rather, the problem is how to dynamically limit the network DPC activity based on how likely the RT threads are to miss their scheduled time.  You obviously don’t want to wait until you hear a pop/click before throttling back, so you must design an anticipatory algorithm to prevent interference by throttling the rate of interrupts that run DPCs. (This is extremely hard to do reliably, across all performance levels of CPUs and network hardware, making the "hack" look more appealing.)

    My guess is that future Vista scheduling mechanisms will move most or all of the remaining DPC-level code into real-time threads like the multimedia is now, but until that happens, the next best thing will be to come up with a more effective algorithm to dynamically limit the DPC activity based on how likely the RT threads are to miss their scheduled time.

    Read Larry Osterman’s blog quoted a few comments above about how isochronous streams require RT scheduling, and realize that major redesign of the kernel is not gonna happen in a SP, leaving the likely near-term solution to be limited dynamic network throttling based on a better anticipatory algorithm for the RT stuff.  (I’m glad I’m not the one who has to come up with the solution — I’ve been there and done similar, and it’s a brain-twister.) However, some time in the future, the beauty of converting DPC tasks to RT threads is that threads scale very nicely to dual/quad/heptapenta core processors.  

    And yes, I realize that Linux doesn’t have this problem — their equivalent of kernel DPCs is handled in the scheduler (which is the main reason it was so tough to get it right).

    (I don’t work for MS — in fact, I often don’t agree with some of the design approaches they have taken, but I hate to see great software engineers flamed just because understanding the problem is beyond the grasp of some.)

    As us old kernel farts used to say, flames to /dev/null  (google it if you don’t grok)

  81. Erno says:

    I am perplexed by the claim that Internet traffic is not affected by this, and even more perplexed by its reasoning. Is the writer perhaps unaware that people with beefy Internet connections get 9 gigabits per second of bandwidth in long distance benchmarks, and still much more than 1 gigabits in real world applications?

  82. LinuxGuy says:

    Jerry Schneider – Is it not that DPCs consuming large amount of CPU is the main short term problem for MSFT to resolve? If the DPCs operated efficiently (i.e. they did not consume more than few percent CPU) would not the scheduler have more time to give to the RT Audio task and things be fine? Why would one give up on optimizing the DPCs? With modern NICs with offloads and interrupt coalescing and  such sophisticated features DPCs have no business using so much CPU.

    Can anyone explain why optimizing DPCs would not work?

  83. Paul G. says:

    Yikes!!!

    Why is it that whenever someone from Microsoft gives some decent technical information on a recent problem, especially ones that haven’t yet been fixed, everyone seems to pile-in with a "MS-have-catfood-for-brains" mentality?

    I have some ideas about these demographic groups, but no substantive evidence. However, I think the groups are:

    1) Generally ignorant people

    2) *nix advocates

    3) People who maybe didn’t pass an MS interview, and don’t feel it fair… (and if you’re in that category and have a fix, now’s the time to let the channels know…)

    Of course, if you really hate a particular OS behaviour, you can always switch to an alternative, or roll your own. (And if you choose the last option, then scheduling network and MM code will be the least of your worries!!)

  84. Leo Davidson says:

    Bogdan,

    Regarding your 1st point:

    As stated in Larry’s blog and his further comments below it, MS are already getting stick for the latency of the sound system in Vista. Adding *more* latency like you suggest, so that the audio thread doesn’t have to wake up as often, is not an option. Larry mentions voice communication as being one thing which is particularly sensitive to latency. (Playing musical instruments through your computer is another example.)

    (I don’t know enough about the other two points to comment.)

    Not directed at Bogdan:

    A lot of people are still talking about this issue here and elsewhere as if it were one of CPU usage. It isn’t. It’s not about not being able to do enough on the CPU; it’s about being able to schedule things often enough that they don’t glitch. The audio thread wakes up once every 10ms and does a miniscule amount of work. It needs very little CPU to work but if it misses that 10ms wake-up call then everything falls apart.

    Larry also states that it is very easy to make XP’s audio system glitch under load.

  85. Brian LaPierre says:

    "Why does it need to verify each packet, one would think that can be done on the network card."

    Wouldn’t integrated network cards tend to use the CPU to do calculations like this? I know that with Integrated graphics, and audio cards they tend use the CPU for calculations. Perhaps the same thing happens with the integrated NICs?

  86. Peter says:

    So…I’ve been poking at this for a few hours now — when you say that it "causes NDIS to “indicate”, or pass along, at most 10 packets per millisecond" — what __exactly__ is it setting?

    That is, I want simulate the same thing by making a call to DeviceIoControl(), presumably passing in an "OID" code and a new value — but what value? And can I call DeviceIoControl() with IOCTL_NDIS_QUERY_GLOBAL_STATS and an OID code to retrieve the value that’s being set to 10 packets per millisecond?  There’s an OID_GEN_LINK_SPEED, but that seems to be the maximum physical speed and not the throttled speed.

    And I can’t find (anywhere!) any documentation where the verb "indicates" is used to mean "throttle".  Can you point to any (other) place where the word "indicate" is used this way?

    Inquiring minds want to know!

  87. Peter says:

    samic — you’ve never seen TV shows where the heart monitor goes "beep—beep—beep—"?  Nor seen medical equipment that produces high-res video? (hint: uber-common ultrasound equipment).

    The world has moved on, my friend.  There are more uses for audio than listening to the latest Mick Jagger single.

  88. Glen Turner says:

    Mark wrote:

    "Despite even this level of throttling, Internet traffic, even on the best broadband connection, won’t be affected. That’s because the multiplicity of intermediate connections between your system and another one on the Internet fragments packets and slows down packet travel, and therefore reduces the rate at which systems transfer data."

    This is incorrect. The packet burst that occurs with a TCP segment transmission will be dispersed depending upon the amount of load on the links interconnecting the router’s on the path. If there is not much load then little dispersion will occur. So claiming the "best broadband" connection will not suffer from this bug is overstating the case — the "best" connection will have a lightly loaded upstream network, especially outside of peak times.

    There are also attributes of TCP that will lower the performance. The added jitter to the path from waiting for audio reduces the accuracy of TCP’s RTT estimate, and thus the TCP transmitter lowers its sending rate.

    It is also believed that quantised jitter causes the quality of the TCP RTT estimate to be degraded much more severely than random jitter. Your fine explanation of the audio subsystem scheduling suggests that a high level of quantised jitter will be introduced to the TCP stream.

    Best wishes solving the problem, Glen.

  89. dvhh says:

    some website forum advise to deactivate the mmcs seevice.

    isn`t this service a bit of overdoing in vista.

    I`m sure that a new low latency audio stack is necessary for windows. but shouldn`t the os provide it only to application that specifically request it. and provide a less cpu consuming / high buffer stack to other.

  90. dawnh says:

    Perhaps not only the network stack scheduling problem but the audio scheduling?

    Notice these :

    1,In the Vista arch desin,audio subsystem has been moved from kernerl to user land.See this link from Larry’s blog:http://blogs.msdn.com/larryosterman/archive/2006/03/07/545451.aspx

    2,In some early Vista version,perhaps,beta2 or RC1,we always  expirenced  "pop" or "stutter",not only high network usage ,but also other operation,even when we scroll the IE browser.So the audio schedling problem appears then,perhaps abount the 10ms DPC?This problem also can be seen now,in the windows server 2008 beta3.

    3,In Vista RTM,we can only find the slow network performance,other problem seems being resloved.So we can image that the "magic number" not only in network subsystem,but others? for example video subsystem?And these "magic number" may be diffenrent between Vista and Server 2008.

  91. mike says:

    I’ve been using Vista for three months now.

    I have brand-new hardware, updated drivers, and all the latest patches. It still runs like a turd in desert heat.

    This MMCS issue is the last straw — XP and Linux do not have this problem.

    Tomorrow, I’m finally switching to Ubuntu. Screw Microsoft and their arrogance.

  92. Andrew says:

    Mike: I feel exactly your pain. I’ve been using vista at home for about 3 months. It feels slow compared to my XP box at work. I don’t do much network transfer speeds at home, so could live with it.

    Now I upgraded to Vista at work where I do a lot of network transfers from mapped drives and it’s just horrible.

    I’ll be going back to XP anyday now.

  93. Nameless says:

    Why has this site (main page of Mark’s blog) suddenly started asking me for a certificate? I frequented it regularly in the last few days and it hasn’t done so. It started today. I use Firefox 2.xx. Sorry for off-topic.

  94. a student says:

    On the whole the many comments above have been quite educational for me and I thank the commenters. I would have welcomed an explanation of why Vista’s TCP processing is taking over six times longer to execute than Linux’s (consuming 40% of available clock cycles compared to 7%). The absence of an effort to explain it suggests that people simply don’t know why. In the end it’s the main question.

  95. Daniel says:

    CAVEAT: I stopped being a computer engineer a while ago, so it is possible that I am missing some obvious things. If anyone would chip in it would be greatly appreciated.

    There are several things I don’t understand.

    1. What is the point of contention? So far it looks like the only contention is computing power. Then this problem should not happen on a multi core processor – one core processes network packets while another core handles multi media. I am being led to believe that either the point of contention is not computing power (but then what is the constraint?) or the multimedia system used a quick hack to cope with the current hardware, and failed to put in place a fix for future hardware.

    2. So the multi media engineer goes tries to solve the glitch solution. It’s a hard problem, he or she thinks a long time about this and comes up with the only answer: throttle the network. Not only that, but she or he puts in a hard coded limit. What I don’t understand is why is this situation a surprise at all? According to Larry Osterman’s blog it took a team of engineers from Tuesday morning until Friday afternoon to find the root cause. I don’t get it!!! I can think of two possible answers: 1) the engineer that made the change left the company and nobody knew about this dependency. 2) components in Windows are so interconnected that a change in one component shows up in a completely different component. For Window’s long term future as well as the prestige of the Windows engineering team I hope the right answer is answer 1: the engineer left the company.

    All in all I am left with a bad taste regarding this particular engineering team. They clearly dropped the ball on this one.

    Now I want to offer some un-asked advice:

    1. fire the engineer to made the design. If the engineer received a code review, fire the code reviewer as well. If said engineers are hot shots then it won’t matter to them – she or he will get another job easily. If the engineers are not so hot, then it’s better for the team to make room for better qualified professionals. Either way, it will send a strong signal to the rest of the team: think hard about what solutions you give and be more careful when giving code reviews. I remember the last time I got a code review in which I hard coded a value. The hard coded value would have probably stopped working a year after putting the product on the market. I had to come back in the weekend to put in place the proper solution. This is a job, not some kid’s game.

    2. If there are any college kids reading, here is something I learned in my networking  class: in general, when investigating latencies focus on high throughput networks. Low throughput networks are too coarse to identify incidents – it takes more to move the needle on a low throughput network than it does on a high thouroghput network.

    Daniel.

  96. KRS says:

    If playing ANY audio will degrade network performance, then let’s imagine a scenario where a telemedicine software is installed on a Vista machine… the high bandwidth network interface supplies both audio and video; so Vista it appears the quality of both video AND audio will suffer.

    I think hospitals acquiring new PCs must now consider upgrading to XP which doesn’t have these problems!

  97. Brian LaPierre says:

    "If anyone would chip in it would be greatly appreciated."

    Gladly.

    "What is the point of contention? So far it looks like the only contention is computing power. Then this problem should not happen on a multi core processor"

    The point of contention is NOT computing power. The point of contention is making sure that every 10ms the audio buffer gets refilled. The problem they noticed what that network packet arrival caused these DPCs to be created that were long running, and were at a higher priority then the audio thread, so the audio thread would be stalled. Remember not everyone has multiple core CPUs, and Vista was designed for everyone.

    Secondly, how can you come in with an attidude like that? It took then Tuesday-Friday to come up with the answer? That’s incredibly fast. They had to reproduce the initial problem. Then there was probably several meetings over the course of the 4 days to catch up, discuss current findings and make sure everyone is on the same page. Then they have to verify exactly what is taking place, and decide how to approach it. And all of this as well as what they need to do for any other projects that might be going on.

    I can’t recall where I heard this but I thought I had read somewhere that this wasn’t supposed to be hard coded, and it was intended to be in the registry. That’s not a design problem, that ‘s an implementation problem. And hard coding doesn’t necessarily mean the value was written directly in the code right there. It could have been some constant in another realm somewhere, and there was a disconnect between the two sub systems for what was actually going on.

    As for the employee who wrote this, they might have been on vacation, transfered to another group, working on other projects, (Most likely it wasn’t that everyone in the networking and MM group was trouble shooting this). Event then, they might not have made the link instantaneously.

    Should they be fired over this? Absolutely not. It was a mistake. Plain and simple. I’m sure you’ve made them before. If not, I’ve got a job for you. I want to pay you 100k a year to invest 200k and double it. What you don’t want to? But you never make mistakes, SURELY you can do it.

  98. Jordan Mueller says:

    I experimented with the removal of the dependency of the MMCSS Service. I copied some stuff from one encrypted drive to another (CPU hog). I play some MP3 in the background. This always worked on XP without stuttering. Now, on Vista, the sound gets heavily distorted. I’m on Vista 64 and have all the latest drivers, BIOS and have not. It seems like a bad joke. Just read slashdot. People pump out data on really outdated machines at whatever speed the machine can handle and play MP3 or Video at the same time. On Linux. No stuttering.

  99. Daniel says:

    Brian – thank you for taking the time to respond.

    Like I said, I have been out of the field for a while. But…

    "The point of contention is NOT computing power. The point of contention is making sure that every 10ms the audio buffer gets refilled. The problem they noticed what that network packet arrival caused these DPCs to be created that were long running, and were at a higher priority then the audio thread, so the audio thread would be stalled."

    According to you, the problem is that the audio thread pre-empts the networking thread. To me this implies that the point of contention is the CPU – i.e. computing power.

    "Remember not everyone has multiple core CPUs, and Vista was designed for everyone."

    Vista is (was?) supposed to be the operating system for the next 10 years. As such it should have planned for technology improvements. If your comment above reflects the actual thinking during Vista design process, then I am at loss for words.

    "Secondly, how can you come in with an attidude like that?"

    (Left in the original mis-spelled word) It’s just my opinion, take it or leave it. But when your media design explicitly throttles the network and then it takes the engineering team the better half of a  week to find that out then either:

    1. there is a serious disconnect between different teams inside Vista. I am thinking something along the lines of work done by media team affecting the networking team and no communication between them.

    2. the Vista design is fragile. I would liken it to a balloon – you push it in one spot and then another random spot expands. You make a change in one spot and it affects another random spot.

    3. the engineering team that did the investigation is just not up to speed. I mean how much more clear than this can it get: media playing throttles the network. There are reports that media playing affects network. And nobody knows what is going on. THIS IS LAME!!!!

    In any of these cases is true there is room for improvement for the Vista engineering team. My comment stands.

    "I can’t recall where I heard this but I thought I had read somewhere that this wasn’t supposed to be hard coded, and it was intended to be in the registry. That’s not a design problem, that ‘s an implementation problem."

    I hope you realize that your "solution" is nothing more than a band-aid. And I hope you don’t push this as sound design – some college kids may be still reading this thread.

    "Should they be fired over this? Absolutely not. It was a mistake. Plain and simple."

    Let’s not fire them. And leave the message that you can seriously screw up and get away with it. But seriously now: I don’t know why you would advocate sending this message.

    I was thinking trying something different: instead of firing the engineers that came up with the design, make them work for free until they fix it :). Hey, if they are hot-shots it should take no time at all. Actually, let’s expand this idea: engineers should not be paid for the time it takes to fix any bug. There should be an economic penalty for screwing things up.

    "I’m sure you’ve made them before. If not, I’ve got a job for you. I want to pay you 100k a year to invest 200k and double it. What you don’t want to? But you never make mistakes, SURELY you can do it."

    Why would I want to take this job? I already make more than what you mention with way smaller targets. Honestly, you would have to up your offer to be taken seriously. Much like some of your other comments…

  100. Daniel says:

    It’s me again.

    Larry Osterman’s blog has more technical information – look around
    http://blogs.msdn.com/larryosterman/archive/2007/08/28/windows-vista-sound-causes-network-throughput-slowdowns.aspx#4615267.

    Here is the contention: in Vista interrupts run only on CPU0. Both networking and media systems are driven by interrupts so they have to share CPU0. And in Vista media trumps network.

    It seems that other OS’es are able to distribute interrupts across different cores – see comment in Lary Osterman’s blog:

    http://blogs.msdn.com/larryosterman/archive/2007/08/28/windows-vista-sound-causes-network-throughput-slowdowns.aspx#4619393

    Well, I guess Vista can’t do everything.

    This was an interesting problem to look at. Good geeky start to the Labor Day weekend.

  101. Triangle says:

    @atglabs: "But what if that thread is swapped with a higher priority thread while IRPTs are disabled?  It’s a really complicated problem to solve correctly."

    I fail to see how you could preempt a thread that has interrupts disabled.

    @everyone else: It has been mentioned that disabling MMCSS seems to solve this problem quite well, so if you are being effected by it I would suggest using that as a workaround. Also, experimentation leads me to believe that disabling every service except DHCP and DNSCache lead to the best system performance.

  102. chr0m says:

    On my system

    Athlon X2 4600+

    4Gb Ram

    Vista 64

    My audio stutters like crazy whenever there is any hdd activity on the primary drive. I’ve been tearing my hair out trying to sort it out.

    A PC with this power on a modern OS should not have issues playing mp3s and using the hdd at the same time!

  103. Shelby Cain says:

    I can’t even begin to understand how anyone could have thought this was a good idea… much less got enough buy in to the idea that it actually ended up as a (mis)feature in Microsoft’s new flagship operating system.

    For instance, simply having an application that like Steam (a common gaming service that you would normally leave minimized in your system tray) that relies on multimedia functionality causes Vista’s network throttling to kick in as well.  I spent hours trying to figure out why my gigabit nic was transferring at a whopping 7 MB/s while I wasn’t playing audio.

  104. james says:

    The really sad thing here? Most of the times the throttling will be triggered and make a difference, it would have been better all round to use a bigger buffer instead: when simply playing back ripped CDs, for example, there is absolutely no need for the 10ms response times – except when skipping tracks, when you can simply flush the buffer, you know not just seconds but whole *minutes* in advance what is going to be played, and can buffer accordingly! (Something the iPod exploits to cut power consumption, by reading several minutes’ worth of music in a single burst then powering the hard drive down until the next burst is needed, minutes later.)

    Yes, for game sound effects you need fast response times: you don’t always know in advance when to generate explosion sound effect, so you need to work with short buffers and fast response times – but for music playing, using 10ms segments is not just pointless, it’s a net loss, both in efficiency (you’re performing 100 times as many context switches as a 1s buffer would require!) and in the detrimental effects we’ve already seen on other Windows components.

  105. Tr0n says:

    So, what’s the verdict?

    How do I fix this?

    Wait for SP1?

  106. Chris says:

    I am having this Vista network slow down too. I have tried all of the fixes I have found on the net but had no success. I have a gigabit network adapter but it is only connected at 100mbps and I get a transfer speed of about 600KB/s.

    The thing is that I am not playing any music on my system, I have disabled Multimedia Scheduler Service and Windows Audio, and a load of other services but that did not help.

    Is there something which is using the audio service (without my knowledge) which causing my speed cap to be enabled?

  107. Mike Diack says:

    I know that this isn’t related to the sound issue.

    But can I make a request for myself and others:

    Many many people, including me and many others (googling for "TrustedInstaller.exe" will show you!) are finding on Vista that this process totally hogs the CPU and disk for minutes at a time even when no activity is going on at Vista (typically at startup, rendering the desktop performance very poor for minutes after bootup).

    Could Mark investigate this?

    Cheers,

    Mike Diack

    mike_diack@hotmail.com

  108. Frymaster says:

    > I am having this Vista network slow down too

    No, you’re not.

    The symptoms don’t match in the slightest.  If you aren’t playing music, and you’ve disabled MMCSS then this issue can’t possibly apply to you.  Also, the audio issue doesn’t affect what speed you’re connected at, just how many packets you can process.

    Also, the speeds you’re getting are waaaaaaay lower than usual- or almost even worst-case scenarios for the audio throttling problem (unless all your packets are _meant_ to be around 62 bytes, that is)

    There’s something else going on in your system, I’m afraid.  Try netstat -s and looking for packet loss symptoms

  109. Andreas K. says:

    This issue can have a heavy influence on systems with 100MBit network connections too!

    My database application usually has a network throughput of 1800 packets/s. Starting Media Player will decrease this to 800 packets/s!

    Database is connected by ODBC via TCP/IP to SQL-Server. It’s the same effect using Sybase/Anywhere database instead.

    Quicktime and MusicMatch player showing this effect on startup, on WMP you have to play something to get it.

    Copying files on the network is not effected by this issue on my 100MBit connection.

  110. Prime says:

    So what if I use Winamp to play MP3s on my Vista machine. Will I get hit by the network throttle problem too?

  111. Susan Black says:

    MS is a first class, leading edge software company.

    The real problem here is the extreme difficulty

    of designing highly complex software that works effectively. I am sure that MS will sort it. There were many difficulties with xp and it needed 2 service packs.

    Thank you Mark for the usual interesting and informative article.

  112. Paul Sanders says:

    Could it be that the reason why this problem appears to apply only to Vista and not to XP is the redesign of the Vista audio stack?  The new audio stack in Vista has a lot more components, including something called (from memory) audiodg.exe, and seems to do more of its processing in user mode than XP does. This would explain why lengthy processing in a DPC (which pre-empts any user mode code) would cause audio playback (or recording) to stutter.

    Anyway, I agree with other posters that throttling network bandwith in this way is a horrendous hack on Microsoft’s part, be they or be they not a ‘leading edge software company’, and thank you Mark for your article.

  113. benb says:

    what a disgraceful train wreck. While I appreciate the post and it not being covered up, I am really sick of post after post talking about these sorts of issues cast as feature related. It is absurd to the point of negligent. As an Ultimate owner I cannot even copy a file from my desktop hard drive over USB to the USB backup drive while listening to music! All while running 4GB of ram, dual core,great graphics card, and the list goes on.

    It is not acceptable to release what amounts to garbage and charge for it. These things should have been fixed immediately! Not nearly a year after release.

    Most users do not care what the reason is, new features, rewrites etc. what they care about is basic funtionality that has been working for multiple generations not working any longer. There is no excuse or reason that will make that acceptable.

    This is all coming from a long time Microsoft customer and one who understands it is a complex business. It is not complex, however, to understand that certain basic functionality should run without issue, especially when it has been generally available since 1995.

  114. CR says:

    Say again?  This affects network transfers, how on earth would it affect your USB transfer?

  115. J.M. van der Kolk says:

    I would think throttling should be dynamic, in the sense that the NDIS driver only needs to be notified if Windows Media notices that it gets close to getting into trouble. The amount of throttling should never be fixed even within the one system.

  116. Frank M. Whitman says:

    If they can’t fix the sound problems, Microsoft should give us a copy of XP since it "Just Works", unlike Vista that does nothing but waste our time.  

  117. TMaxim says:

    Users expect multimedia applications, including music and video players, to offer a seamless playback experience. However, demand for the CPU by other concurrently running applications, like antivirus, content indexing, or even the mail client, can result in unpleasant hiccups. To provide a better playback experience, Windows Vista introduces MMCSS to manage the CPU priorities of multimedia threads.

  118. bill says:

    The funniest part of all of this?  Pro Audio applications that actually register with MMCSS still perform very badly under Vista.  As a result, many of us who need to stream audio at low latency have dropped Vista and returned to XP.  Low latency audio in Vista is glitchy as hell, despite MMCSS.  So there are no winners here.  From a pro audio editing/mixing perspective, Vista is an abortion of an OS.  The moving of most of the audio stack from kernel mode to user mode has a lot to do with the problem, IMHO.  Microsoft’s reasoning was that a bad audio driver can cause a BSOD.  My goodness, the carnage caused by this move makes a BSOD seem like a walk in the park.

  119. Fabio says:

    I too am having the same problems like chris , not playing audio or streaming video , brand new core2  duo 3gh 2 gig ram running vista ultimate 64bit OS.

    I have tried doing transfers from a xp machine to vista of mainly a image files. I always get 5% utilisation on the gigabit network  , Vista reports 8.44MB/sec , but if I dump to the xp box i can get speeds of 40MB/sec.

    Previosuly i could get speeds of 22MB/sec when i was using xp with a lower spec system.

    I have just tried , swapping out gigabit network switch for a different switch and connecting with new purchased cables , same result.

    I should mention i’m doing all this on a scsi u320  drive system and its not a primary drive that i dump to , on either systems.

    I have run all the fixes, changed Mb nics even disable my bluetooth adapter and uninstalled my antivirus program , i still get 5%-9% utilization .

    Anyone else got any ideas on what to do bedides going back to xp ?

    regards

    fabio

  120. Samsonite says:

    Has someone tested the behavior with SP1?

    Is it really solved?

  121. TooFunny says:

    Regardless of whether or not there is a multimedia/throttling issue going on. Anyone with a 1 gig router will suffer horribly as I do with internal file transfer speeds that will not under any circumstance go beyond 100Mb/s (89Mb/s to be exact).

    This is unnaceptable for any OS as I can imagine if it was a 100Mb router my internal speeds would be around 8Mbs.

    And this holds true on 6 different routing devices and not only in Vista 32bit, also in Vista 64 bit.

    The whole network throughput thing is not singularily caused by media, there is a deeper more pervasive problem at the kernel core that needs to be addressed and re-worked all the way up to the tcpip.sys driver to get the throughput problem addressed and fixed.

    Say that five times fast.

    If this issue isn’t addressed soon, I will change to linux as MS has proven beyond a doubt that they have no interest in delivering a tested proven product to its customers and the dollar is their bottom line.

    OUT!

  122. Proflogistics says:

    i searched the whole web for others having the same problem since weeks, im so happy to find some people with the same issue

    I get some really good transferrates with a fresh booted vista:

    Packet size 32k bytes: 115145 KByte/s Tx, 113827 KByte/s Rx.

    Which is almost the best you can get on 1GBps .. but if im running winamp, windows media player or even QIP (a messenger) i get those results:

    Packet size 32k bytes: 17100 KByte/s Tx, 7267 KByte/s Rx.

    Which is only about 10% of the available bandwidth  

    I tried almost every parameter you can change… Auto-tuning off, Flow control enabled etc. i dont think its depending on your software setup, my second computer using Vista (completely different hardware) has the same issue.

    Anyone tried Windows Vista 64bit ? I only could try 32

    My concerned Hardware:

    Gigabyte GA-965P-DS4 using onboard Realtek RTL8168B/8111B Family PCI-E Gigabit Ethernet NIC, intel Core2Duo 6600, 2 GIG DDR 800 RAM and a Creative X-Fi Extrememusic soundcard.

    the second pc:

    ASRock 939Dual-SATA2, AMD X2 4200, 2GIG DDR1 RAM, Intel Pro 1000 GT Ethernet NIC

    SWITCH: Netgear GS108, 8 Port, 10/100/1000Mbit

  123. Dick Carlson says:

    Despite the improvement, I’m wondering if this is limited to high-speed networks.  Over my small home/business network I’m not really seeing much of a difference.

  124. Sicaine says:

    Hm bad for me: I have(in germany) a broadbandconnection with 50mbit/10mbit(down/up) and iptv which is streaming with multicasttechnologie. So playing musik or video i have only a 0.1/0.02mbit connection. That is not really funny.

    As long as im a Softwaredeveloper and the problem exists, the first Person, who is asked from the family and co by computerproblems, will get a alternative os or as backfall windows xp(from ebay oem version(german law :P)

  125. quazee says:

    Vista SP1 RC1 does not fix the issue either 🙁

    I would expect *at least* a registry key to increase the 10000 packet limit to a value I can choose.

    And disabling MMCSS is not a viable option either, as the new user-mode audio mixer apparently uses very small buffer sizes, casing glitches even on a dual-core system without MMCSS enabled.

  126. HN says:

    Daniel – What a nasty attitude you got…

    quazee – Haven’t tested SP1 RC but I really think this would be a major problem for MS if some kind of fix for this is not implemented in the final release.

  127. Stefan (Germany) says:

    Hi,

    same problem there. If i dont play music, i got 100-110 MB/sec. over my gigabit network. if i play music, speed drops to 5-8 MB/sec.

    are a solution or a workaround available?

    i test SP1 RC but it hasn’t fix this.

    so, im back on xp64 now. back on vista if the problem are fixed or a working workaround available.

    be stay tuned in this thread

  128. Trevor Morris says:

    Starting a large (1GB) file copy from an XP box to a Vista box or another XP box on a gigabit LAN I get very respectable (and similar) performance.  Starting a copy of the same file from a Vista box  to another Vista box or to an XP box I get only 3-4MB/sec instead of ~12MB/sec.

    If I play an audio file at the same time as the copy tests from Vista, performance drops even further. (to <1MB/sec).

    It boggles the mind that Microsoft’s developers hadn’t discovered these problems long, long ago.  Don’t they use gigabit networks?  Don’t they listen to music while they work?

  129. Jon says:

    To start off, I’m not a huge wizard with computers, but somehow I found this thread. I wanted to say thank you to the author for a well-explained article. I also wanted to state my problem, although I’m sure it’s been mentioned in here at least 100 times. XP worked fine for me before, and now I’m having audio and video trouble with Vista. I invested plenty of money into this problem that I thought was just needing upgrades. I purchased 4gb of ram and a new wireless USB N1 adapter for my computer because I have the slim line and there are no desktop cards out there that are compatible with Vista and fit the slim. It just ticks me off knowing how much money and effort I put into this computer, all to be futile. Thank you for this forum though. I’ve certainly learned my lesson. Vista blows and Google is my best friend when it comes to information. Thanks to all of you.

  130. Kenneth Green says:

    You can just about double your speed to 18 megs per second if you go to registry setting

    HKLMSOFTWAREMicrosoftWindows NOTCurrentVersionMultimediaSystemProfileTasksAudio

    Then set Clock Rate to be 5000 instead of 10000.  I don’t have a clue why this works but it doubled my download speed from 9 megs to 18 megs.

  131. David Whitney says:

    I have read in another blog the story of someone else with a home Gigabit network suffering slow (relatively speaking) transfer rates even with *no* media files playing, and he (after an incredibly length and detailed description of his investigation) discovered that it boiled down the the NIC’s support of Jumbo Frames. Not all cards support them, but if they do, they’re not always turned on in the driver by default. When he installed NICs with drivers that had Jumbo Frames turned on, he saw what he called "disk transfer rate speeds on the wire," and was a happy camper.

    In this same article, he mentioned that contact with another hardware type had relayed to him to watch out for Gigabit ethernet adapters that are integrated onto newer motherboards – some of them are tied to the PCI-33 interface even with PCI-Express present. I regret that I do not recall more specific details of this aspect of the problem (which did not affect the subject throughput issue in this case).

    Bottom line, if you’re on a Gigabit network, and are having file transfer "issues" (or just subpar performance), check out support for Jumbo Frames.

    -David

  132. Craig K says:

    Microsoft has a KB article that allows you to configure how much throttling happens on the network or just turn it off.  It only applies to Sp1 and I haven’t tested it yet but will when I get back home.  

    http://support.microsoft.com/kb/948066

  133. kuba says:

    multimedia playback is still broken in SP1.

    the idea that vista should play mp3s with 10ms latency, was really brilliant.

    Honestly anybody who approved this design should be fired.

    btw. are Steve Ball and Larry Olsterman still in MSFT?

  134. jim jones says:

    WOW… thx for the thread Mark… I was seriously going nuts thinking I had a h/w problem.. After re-installing vista ultimate from enterprise the problem persisted (of course)

    but what did change in my setup since this became a problem was when I went from a LAN setup to pure Wireless (and yes there are torrents that do run and cause havoc)  With LAN and 100Mbps there were no audio glitches even with a fully loaded uTorrent — with 802.11(pre-n) and (g) the problem is very apparent.  If I kill uTorrent and do not web surf the problem vanishes.

    most definitely a bug in the OS…i’m not very happy I live with music 🙁 — maybe it is time to run something else to please my audio needs

  135. J. S. Douglas says:

    Mark,

    Thanks for your fine article and he blog. I came across this while searching for the reason Vista uses so much CPU when processing audio. I’m one of those weird people that would rather listen to audio and work/play on my computer than watch the latest "reality
    TV" and talent shows. To that end, I’ve ripped thousands of CD’s to .flac audio files and transcoded some for use in my MP3 player. My system is more than adequate featuring an AMD 4600+ x2, 8GB DDR-II 800, ATI 3870 video card, a Creative X-Fi sound cad and
    4 HDD’s running 2 RAID-0 stripes. Note that in fact I’ve tried 3 different sound cards (ADI 1988 inegrated, RealTek ALC888 integrated and the X-Fi and on all 3 I note that audio playback is taking 8~38% of the CPU usage in the system. Frankly I wish Micosoft
    would just put DirectSound back in Vista and be done with it.

    All of this reminds me of a note I read some time ago and in retrospect it’s really staring to look as if this fellow should be hired as a consultant for MS.

    http://www.cs.auckland.ac.nz/~pgut001/pubs/vista_cost.html

    Thanks again for your time and efforts.

    JD

  136. Steve... says:

    FWIW on the subject of media playback, video, audio etc. I was having problems with music tracks and video (audio) skipping in Vista Ult sp1, Core 2 duo, 2Gb ram.

    Music playback skipping in iTunes and WinAmp, Video (audio) skipping in Win Media Player and Quicktime. So I read through this thread (thxs Mark for the article and insight), tried a few things with no success, still audio playback skipping). Then, I disabled my ‘wireless NIC’ and the audio skipping problem disappeared, re-enabled it and the audio skipping came back, disabled it and the skipping went.

    So at the moment the audio playback skipping has ceased. Not looked into the network throughput issue yet.

    Does anyone else get similar behaviour to this? The audio playback skipping was driving me to distraction and after browsing for solutions, I couldn’t find any that worked for me.

  137. Tarrant2007 says:

    Ahh, I think you solved my mystery below:

    I have a Vista Ultimate based HTPC (which used to run MCE 2005 prior to my doing a clean Vista install) and a Win XP Pro desktop on the network. Both machines have Gigabit ethernet, and are connected via Cat 6 cable across a D-Link DGL-4300 Gigabit router. Jumbo packets are disabled on both ends. Both have latest chipset drivers, NIC drivers, Vista patches, etc installed.

    Prior to clean-installing Vista, I was getting transfer speeds of 35 MBytes/sec between the HTPC and the WinXP Pro desktop. After clean-installing Vista, I noticed that this dropped to a measley 11 MBytes/sec!

    I’ve tried several solutions, including turning off Vista’s firewall (transfer rates went 11 MB/s -> 16 MB/s). I also tried turning off autotuning (no difference) and turning off remote differential compression (minimal difference).

    Finally, I realized something important:

    If I had VMC running in the background (idling mind you, ie. not playing video, not accessing network, etc), then my transfer rates were limited to 16 MB/sec with Vista’s firewall off. However, if I exited completely out of VMC, then my transfer speeds shot back up to 35 MB/sec to the WinXP Pro desktop!

    I think this is the reason why loading ISO files from within MyMovie’s VMC interface (auto-mounted through Daemon Tools) takes FOREVER with Vista (45-60 secs), but was relatively fast in MCE 2005 (~10 secs).

    Anyone have any ideas why VMC running in the background would cause a > 50% drop in network speeds, even though VMC isn’t doing anything with the network?

  138. A. Mustermann says:

    Leo Davidson commented on August 28, 2007: "MS (or at least two MS employees in their blogs) are being quite open about the fact there is a problem that needs fixing."

    Yet here I am in late June 2008, with Vista SP1+Hotfixes and the problem is as bad as it has always been.

    Firing the guys who invented this would be too good for them. I’m not sure if "hanged, drawn and quartered" would be appropriate, but it would be a start – considering what they have done.

  139. olle says:

    Hi

    I use a sat receiver to watch HD streams.

    That constantly has CPU usage between 50% – 70%

    In xp I just gave the sat software priority of high (13) that provided a glitch free experience.

    Even if a antivirus scanner was running at the same time

    But in vista there were glitches.

    Have tried many different things but nothing worked.

    until i disabled MMCSS.

    Now I have a glitch free experience once again

  140. ich says:

    This is one of the silly features, why vista has such a bad reputation.

    It seems that Vista is designed for a PIII with 100mbit network. This cant be serious !!!

  141. Paul H says:

    I have a problem that is the opposite to this. When ever there is network activity video and music playback is affected it is ‘jumpy’ and ‘laggy’ even when playing flash videos in the firefox, I have to wait until the video has buffered fully before I attempt to watch it. This doesn’t happen as soon as I turn on my laptop but after about an hour it starts.

  142. Phil says:

    Exact same problem, after a clean reboot, my system can play vids and download 500kb/s easy, but after an hour or two, any network activity causes media to jump, and also my mouse to jump and basically my system becomes unusuable at around 500 kb/s activity.  Basically my system is unusable to download torrents or any file sharing. Pretty rediculous.

  143. David Zheng says:

    Hi Mark,

    My Laptop have the similar problem: Most of MP3 or CDs play properly in XP or earlier windows PCs, but they always have some disruption problem with my new Vista PC, either "glitch, sputter or something else". It’s very annoying as I always enjoy listening musics from my PC (and seldom care about network speeds).

    When the wireless network was switched off, music playback seems getting smoother.

    But if I chose to disable the MMSC, the laptop can not even play any musics.

    Can you or anyone help on this? Thanks in advance.

  144. jimbonbon says:

    Ever since i’ve had Vista i’ve had this issue in varying degrees. When I used to use a USB adapter (Wireless N) I would experience music slowdown every minute on the dot…. very annoying. I assume it was polling the AP.

    I went back to a wired connection (with a bastard long cable) and everything was fine up until my network traffic became high, and then the music would once again become stuttery.

    Very very glad to have finally come across an answer, by luck of all things. Will have a play around with disabling MMSC. Its funny, now I have read this, even keeping an eye on NetMeter during playback of music proves the point…

  145. LAMER_CZ says:

    Hmmm, it’s quite funny … I have Vista SP1 with the same problem. I lost many hours with tuning stupid speaker fill with no success. Rear speakers just doesn’t work anymore in Vista. Also I spend many hours with this stupid LAN problem … and the solution is just f***ing easy! Disable MMCSS! It should be by the default in SP1 but it isn’t. Then it should be at least on Microsoft homepage but it isn’t. Vista is out about 2 years and nobody fixed this? Well I’m quite dissappointed … Aero is cool, but it cannot be altered that suxx. Taskbar colors are absolutely flat and hard to distinguish active application window … alttab is absolutely awkward, hopefully there is undocumented AltTabSettings which disables that "cool & stupid" thumbnails … if I would decide now if I spend tens of hours solving Vista stupidity (like missing standard desktop themes which could cost 10kb or less on installation DVD?), I would stay on XP … maybe with Windows 7 I will be so dissatisfied that I will finally "upgrade" to XP again …

  146. hvacr says:

    I have tried this and every other tweak I could find. nothing has fixed this problem for me. I hate vista

  147. sage44 says:

    I have tried it on vista x64. I noticed that the network throughput tripled!! (from 12MB/s to 37MB/s) when media player was running. I have Intel Quad Penryn processor and video playback performance is not a problem at all. Whoever programed this stupid prioritizing in vista deserves the worst programmer of the year title. his manager too.

  148. azverkan says:

    I’ve been having horribly corrupted / hiccuping / crackling / popping audio under Windows Vista on Quad Core laptop.

    So far, disabling processor speed scaling seems to have solved the problem.

    I’ve tried tweaking both the SystemResponsiveness and NetworkThrottlingIndex register variables that were added to Vista SP1’s MMCSS.

    The MMCSS service seems to be a badly designed nasty hack that was thrown in at the last minute rather than a real attempt to fix the various problems with audio scheduling in Vista.

  149. A Swede says:

    I have got the stuttering audio problem in Vista 64, sp1. Turning off MMCSS didn’t work at all.

    How do I tweak Systemresponsiveness and NetworkThrottlingIndex register variables(what is the registry hack)?

    This is making me hate Vista once again! Will probably have to return to xp or even xp64 and cross my fingers that these things will work in Windows 7.

    Vista has been out for more than two years now, and still this incredibly simple thing like playing back some audio (or video) doesn’t work!

    It’s beyond belief how bad it is, no wonder Vista tanked so much that even Ballmer admitted it was a failure!

  150. A Swede says:

    I found how to tweak the Systemresponsiveness and NetworkThrottlingIndex, but that didn’t work either.

    Nothing works! Sound still stutters terribly whenever I do some torrenting or something else involving networking activity.

    I have a quadcore(penryn) cpu, 4 gigs of ram, a (disabled for testing now) wireless network usb-adapter, two network cards ,one for wired internet via the router and one attached to a network printer(which i will attach to my router, but it has stayed directly attached for a while due to my laziness)

    And I don’t use an integrated audio card, instead I use a professional sound card, the ESI Juli@.

    Strange thing is that it has worked flawlessly for half a year (since I started using vista 64) up until this week.

    strange……

  151. C Dickey says:

    Thanks Mark for the explanation and the work the Vista and Win 7 teams are doing to make it possible to do Pro Audio on the Windows platform.

    It appears that the best approach would allow enough settings to be expert user adjustable so that a computer can be optimized for a particular use if needed. In my case I use Vista x64 to run Cakewalk Sonar 8.3.1 for music recording and mixing. Above average audio performance with gauranteed low latencies is needed for that type of application. When multiple tracks of audio are being processed and monitored to produce a glitch free music mix, I certainly don’t care if the network slows down. I can also understand that a casual MP3 listener doesn’t want their network to slow down in order to prevent an occaisonal audio click. These 2 instances of Windows audio usage are certainly on opposite ends of the low latency requirements spectrum.

    Please give expert Vista and Win 7 users the options needed to tune their own audio destiny to perfection. Thanks.

  152. Zed says:

    I changed my NetworkThrottlingIndex to ffffffff (which disables throttling) and i now enjoy full network speeds on my gigbit network while playing audio and video. I haven’t noticed any Problems with audio stutters. However I do have a dedicated Soundblaset XFI so that may account for it.

  153. pandine says:

    Setting NetworkThrottlingIndex to 0xffffffff works fine with my Vista x64. 🙂

    Thanks!

  154. Brandon says:

    I can’t help but wondering why this registry entry "HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesAudioSrv

    Key: DependOnService"

    includes MMCSS. Why do you have a service required if it’s not actually needed for the proper function of the services descending from it? I’m willing to do a quick disabling of this service in all my installs but why do I have to edit the registry before you’ll let me stop a non-essential service without completely disabling my audio?

    Can we get this fixed in the next service pack so people who don’t want this service (it interferes with running media from a network share @ 100mb speed and made my life miserable when I tried to provide some training videos from File Sharing) can disable it easily.

  155. Mark says:

    Can anyone comment on whether this issue affects Windows 7 at all. Was the throttling algorithm redesigned or removed?

    Cheers,

    Mark

  156. infinity downline says:

    This is a great blog post. . It's really nice and

    informative post. Thanks so much.