People, Process and Technology sometimes collide

I'm the people part of the equation.  Recently the process and technology ran right over me and turned me into IT road kill.  So what happened?  Could it have been avoided? Definitely...

Let's set this up by describing my typical work environment.  Like many field employees, I am a mobile worker.  When I am traveling I use a variety of internet connections.  When I am home, I use my broadband isp connection.  I am in the office about 3-5% of the time so it's rare to have a corporate connection.

Know Your Users

When designing a patch management strategy for mobile workers, where would you have them get software updates?  What technology do you use?

In the case of the Microsoft IT org, they use SMS for inventory, patch and software push/pull distribution.  With the new version of Windows Software Update Server (WSUS) v3 getting ready to ship, I guess they were anxious to start using it.  However, they got a little ahead of themselves and it tripped me and a few other folks up.

Symptoms Surface

In early March, I noticed my secondary work laptop stopped getting updates from windowsupdate.microsoft.com.  It is running Windows Vista Enterprise and is joined to one of our corporate domains.  As such, it must adhere to the corporate desktop standards for anti virus, malware prevention, etc.  Some of those standards are implemented via Group Policies that flow from our Active Directory (AD) domain controllers.  One of the group policies I received updated my Windows Update client to point to the internal WSUS servers.

Although the initial problem is probably apparent at this point, it's actually worse than that.  You see, I received that GPO update, but I didn't really see the issue until much later.  By the time I noticed I wasn't getting updates (see screen shot above) from the public update site, it was too late.  Our VPN quarantine process already had several patches that were required to pass quarantine but guess what, I can't get them because they are on the internal WSUS servers.  That's a catch 22.  I'm screwed.

Or am I?

March was a super busy month for me as you know.  We wrapped the Vista launch events, I developed the Longhorn content for Q4 and then headed to Orlando to deliver sessions at TechMentor and Windows Connections.  However, while there, I saw Chris Henley hit the same issues with his laptops.  That put a fresh perspective on the issue and I knew it wasn't just me.  Time to start hacking into this.

So when I got home, I started sand boxing the problem.  Good trouble shooters learn to draw boxes around the problems and isolate components.  Excluding components makes it easier to find the faulty component.  The error message displayed doesn't tell much so you have to "Get Help".  That ends up taking you to some help that advises you to make sure you aren't blocking the windowsupdate servers with a firewall, and to also add their URL's into the browser safe sites list.  Seems reasonable.  This starts to clue you in it's a destination unreachable condition.  We just don't tell you what destination. 

Destination Unknown

To get the destination, you have to dig into the Windows Update client log.  Not many users are going to go down that path.  But I'm a hacker like you so I head on over to  c:\windows\WindowsUpdate.log.  I see a bunch of log entries pointing to Microsoft internal IP addresses.  I smell a rat.  I now know what is going on.  So I dig a little further.  Sure enough, the group policy update set the following registry entries:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate]
"WUServer"=

"WUStatusServer"=

[HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate\AU]
"UseWUServer"=dword:00000001

I have removed the ip addresses and ports to protect the guilty.  Ok, now I know exactly what is going on.  My Windows Update client is pointed at some internal servers for updates and I can't get updates.  So I deleted the registry entries and grabbed my updates from windowsupdate.microsoft.com.  I know, that's cheating.  But hold on, it gets better.

Bug or "By Design" ???

So being the good corporate citizen I am, I report the issue first to our internal Windows Vista discussion alias.  I don't know if what I am seeing is actually a bug yet or not.  I suspected it's just a case of the MSIT org not understanding how field employees work and under estimating the impact of re-homing the update client to an internal WSUS server as opposed to using the public cluster.

In the ensuing discussion on the alias, I find out that some new WSUS clients are being pushed out.  Sure enough, I receive the WSUS v3 client (see the screen shot above).  Notice anything different?

Windows Update (WU) Client v3

If you notice at the bottom of the text, there is now a link that you can use to pick up updates from the public Microsoft Update Service.  Very nice.  It would have been nice to have this client pushed to my machine back in early March before the GPO updated me to point to the internal servers. 

See what I mean?  This wasn't a technology issue at all.  This was a people and process issue.  This happens all too frequently.  Fortunately, it doesn't happen often in Microsoft.  So what's next on this issue?

The End Game

Well, from the technology perspective I would have preferred the WU client to time out trying the internal servers then fail over to the Microsoft Update Service.  This is what my anti virus tool does.  It's tries the list of servers and they can be a mix of locations and protocols (ftp, smb, http).  Maybe we'll see that in the next versions of the WU Client.

So the moral of this story is to make sure you aren't creating end user IT road kill.  Put on their hat and assess their needs fully.  In our case, we might need to assess creating a mobile user AD domain, OU, or backing this change back out.  I think our helpdesk folks are just going to get too many phone calls on update failures. 

Happy Easter !!!

[UPDATE]   The Windows Software Update Server (WSUS) registry settings are documented at https://support.microsoft.com/kb/328010.