Advanced hiding techniques: The mystery of the trojaned Winlogon.exe

So the war between the miscreants and the first responders / incident responders is just that - it's a war with casulaties (servers, workstations, work life / home life balance) and it is complete with an arms race in the form of stealthing (miscreants) and detection (incident responders) technologies.  They hack and hide - we try to find them and recover the servers.  My goal with this blog is to expose the techniques used by the miscreants to educate the incident responders of the world to keep things at parity.

Over the weekend I got a phone call from Kenny - an engineer on my team out west who had a hacking case he wanted me to review with him.  He shared out the output of our incident response toolkit (called WOLF - Windows Online Forensics).  He had already been reviewing the output and needed some advice.  Kenny had done a great job of identifying a series of intrusions but this intrusion was un-like the hundreds of others we examine each year - with this one we stumbled on to something new and interesting.

The call came in the same way they always do - the customer noticed some strange behavior on the box and his Symantec AV software had mysteriously started flagging some files on his machine as a backdoor.  Here are the relevant event log entries from Symantec (where XXX was the machine name which I have removed to protect the innocent):

1/15/2005 1:57:00 AM 1 0 5 Norton AntiVirus N/A XXX Virus Found!Virus name: Backdoor.Trojan in File: C:\WINNT\system32\users.dll by: Realtime Protection scan.  Action: Clean failed : Quarantine failed : Access denied                       
1/15/2005 1:57:36 AM 1 0 5 Norton AntiVirus N/A XXX Virus Found!Virus name: Backdoor.Trojan in File: C:\WINNT\system32\users.dll by: Realtime Protection scan.  Action: Clean failed : Quarantine failed : Access denied                       
1/15/2005 2:12:20 AM 1 0 5 Norton AntiVirus N/A XXX Virus Found!Virus name: Backdoor.Trojan in File: C:\WINNT\system32\users.dll by: Realtime Protection scan.  Action: Clean failed : Quarantine failed : Access denied                       
1/15/2005 2:15:56 AM 1 0 5 Norton AntiVirus N/A XXX Virus Found!Virus name: Backdoor.Trojan in File: C:\WINNT\system32\users.dll by: Realtime Protection scan.  Action: Clean failed : Quarantine failed : Access denied

Norton Antivirus had pulled down updated signatures and as soon as they went into effect it identified a 'Backdoor.Trojan' (not very helpful) and moreover - it could not clearn or quarantine the file (even less helpful - but hey, at least you know something's up!).  So the customer called Microsoft and got routed to our team for some investigative work on a possible hacking case.

What our team does in cases like this where there is a suspected intrusion is we talk to customers about working with law enforcement and we encourage them to work with law enforcement on cases where they suspect customer data may have been stolen or tremendous financial loss can or will occur as the result of the intrusion etc.  We also tell them if they're going to work with law enforcement they don't want us touching the box with any of our tools . . . the vast majority of our customers choose not to work with law enforcement and instead would rather work with us.  Law enforcement agencies are notoriously picky about who they will work with on investigations and who they trust as it is all to easy for some hoo-ha (technical term) to come in with guns ablazin' and trample all over critical evidence that may be found on the server in places like the slack space or free space of the hard drive.  I remember vividly over 3 years ago I was en-route to the airport to work on-site with a customer (after having worked with him unsuccessfully for a few hous on the phone) on a very important security case when I got a call from the president of the company I was about to visit telling me to turn around and go home - I wasn't welcome anymore.  I was a bit upset since I had already purchased travel and invested time on the case.  The reason the customer gave?  They had involved the FBI and my response time was . . . ahem, a bit better than theirs.  When he informed the FBI that I was to arrive onsite within 4 hours - they said "it's either him, or us - you decide who you want to work this case" . . . and that was the end of my involvement (and it was fortunate too as I later learned the FBI managed to aprehend the fellow responsible).  I use this story all the time when educating customers on the reasons they may want to get law enforcement involved and why if they do so - I should NOT be involved.

If our customers decide to work with us we send them our IR toolkit known as WOLF which is more of a data collection agent.  This runs on a live system and collects volatile and non-volatile data from the box and it places all of the data in a .CAB file for analysis back here at Microsoft.  We then have a special viewer we've written in C# to make analysis and searching very fast and efficient.  What used to take us 3-4 hours per server can now be done in 15 minutes or less usually.

What we find in the vas majority of cases is that customers who have been hacked once, have usually been hacked more than once and they never knew about previous intrusions.  The latest guy to hack you is usually simply the one who got 'sloppy' and got caught and ruined it for everyone.  This was to be the case with this particular machine.

During the investigation Kenny found a copy of Hacker Defender that had been dropped on the machine back in May of last year (the customers antivirus softawre had still not detected it as of the time of writing and we manually removed it). 
It had been configured to run as the following service:

Dynamic Storage Allocation - [running]
Provides multiple storage heaps in data transactions for dynamic allocation. If this service is disabled, simultaneous data transactions across the local system may fail.


The files had been dropped on the box:

Files containing instances of 'msdsa.exe'
Number of Files Searched: 6
Time to Search Files: 1 seconds

Directory of c:\WINNT\system32
05/16/2004 03:38a 36,352 msdsa.exe

Directory of c:\WINNT\system32
01/15/2005 01:23p 36,352 msdsa.exe

Directory of c:\WINNT\system32
05/14/2004 02:23p 36,352 msdsa.exe

Looking at other activity around that time, the files appear as if from out of nowhere in the wee hours of the morning and a new user profile directory was created (TSInternaluser, which is also a member of the administrators group - wonder what the password is?) as if it had been used for a local logon shortly thereafter:

2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 13,824 raddrv.dll
2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 15 inst_rad
2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 15 inst_rk
2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 183,296 msdfm.exe
2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 38,912 admdll.dll
2004-05-16 03:37:00 | dir_creation_time_C_drive.txt | c:\WINNT\Temp - 364,077 inst_rk.exe
2004-05-16 03:37:00 | dir_last_access_time_C_drive.txt | c:\WINNT\system32 - 13,824 raddrv.dll
2004-05-16 03:37:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 15 inst_rad
2004-05-16 03:37:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 15 inst_rk
2004-05-16 03:37:00 | dir_last_write_time_C_drive.txt | c:\WINNT\Temp - 364,077 inst_rk.exe
2004-05-16 03:38:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 15 inst_hxd
2004-05-16 03:38:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 3,353 msdsa.ini
2004-05-16 03:38:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 36,352 msdsa.exe
2004-05-16 03:38:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 4,912 msdsadrv.sys
2004-05-16 03:38:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 15 inst_hxd
2004-05-16 03:38:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 3,353 msdsa.ini
2004-05-16 03:40:00 | dir_creation_time_C_drive.txt | c:\Documents and Settings - <DIR> tsinte~1 tsinternaluser

The properties of this account lead us to believe that it was there prior to 5/16/2004 - indicating perhaps it was the administrator account that was used to install the malware (which may indicate it had a weak or easily guessed password):

FullName TsInternalUser
AccountType User
Comment This user account is used by Terminal Services
PswdCanBeChanged Yes
PswdLastSetTime 4/28/2004 12:14 AM
PswdRequired Yes
PswdExpires Yes
PswdExpiresTime 4/28/2005 12:14 AM
AcctDisabled No
AcctLockedOut No
AcctExpiresTime Never
TrueLastLogonTime Never
LastLogonServer XXX
LogonHours All
RasDialin No
RasCallback None

Given the circumstances under which this malware was installed our best guess based on the evidence we have is that this malware was copied to the machine using the default admin shares (C$ / admin$ etc.) using a local administrator account (TSInternaluser) with a weak password that is only set to expire once a year.

Now that's not what I came here to talk about - this is merely an example of a prior intrusion that had gone un-noticed by this particular customer for over 7 months.  The latest intrusion was caught by Symantec and is the reason the customer called us up.  Let's have a look at that.  We have a starting point, which I like to call "a lead" - and that is the file named by Symantec - USERS.DLL in SYSTEM32.

Let's figure out the MAC times on this file:

Files containing instances of 'users.dll'
Number of Files Searched: 6
Time to Search Files: 0 seconds
Directory of c:\WINNT\system32
12/20/2004 12:39p 54,272 users.dll

Directory of c:\WINNT\system32
01/15/2005 01:21p 54,272 users.dll

Directory of c:\WINNT\system32
12/20/2004 12:39p 54,272 users.dll

So it appears that this file was dropped on 12/20/2004 - what else happened on 12/20/2004 - let's have a look:

2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\KVBRPZ64 - 26,112 bits_1~1.gif bits[1].gif
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C - 55,296 list_1~1.gif list[1].gif
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C - 77,824 kill_1~1.gif kill[1].gif
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 26,112 userbin.dll
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 54,272 users.dll
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 55,296 list.gif
2004-12-20 12:39:00 | dir_creation_time_C_drive.txt | c:\WINNT\system32 - 77,824 kill.gif
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\KVBRPZ64 - 26,112 bits_1~1.gif bits[1].gif
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C - 55,296 list_1~1.gif list[1].gif
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C - 77,824 kill_1~1.gif kill[1].gif
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 26,112 userbin.dll
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 54,272 users.dll
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 55,296 list.gif
2004-12-20 12:39:00 | dir_last_write_time_C_drive.txt | c:\WINNT\system32 - 77,824 kill.gif

XXX was the username of an administrator on the box and has of course been changed to protect the innnocent.

Notice we have some ".GIF" files and a couple DLL's that were dropped around the same time . . . which brings me to my next miscreant hiding technique.  Some miscreants favor active stealthing using rootkits to hide from administrators.  Others prefer to hide in plain site using obfuscation techniques . . . this is an example of the latter.  Do you think those are .GIF's you're looking at?  Do you think that's air your breathing?

Our first clue that these are not really .GIF files comes from a tool written by Matt, an engineer on our team that is designed to find binaries (executable files) that have been renamed to non-executable extensions. (More on this in a minute).  Matt's tool - flags these as executables:

Copied from: findbin_C_drive.txt
POSSIBLE EXECUTABLE: C:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\KVBRPZ64\bits[1].gif
POSSIBLE EXECUTABLE: C:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C\kill[1].gif
POSSIBLE EXECUTABLE: C:\Documents and Settings\XXX\Local Settings\Temporary Internet Files\Content.IE5\YPIXMQ8C\list[1].gif

There are various techniques one can use to spot an executable . . . you can use a hex editor . . . you can use something like dumpbin.exe . . . or you can use strings from Sysinternals to pull strings out of the file and have a look.  You can even use Notepad and look for 'MZ' as the first two characters you see when you open the file.

Here are some strings from 'kill.gif':

Strings v2.1
Copyright (C) 1999-2003 Mark Russinovich
Systems Internals -

         (((((                  H
Systems Internals
Systems Internals pkill
!This program cannot be run in DOS mode.

This is obviously not a .GIF.  So why would a miscreant take 'pkill.exe' from Sysinternals and rename it to a .GIF?  The answer may surprise you - did you know that in Windows you can actually run a .GIF?  That's right - take any .EXE and rename it to .GIF . . . now open a command shell (cmd.exe) and navigate to the directory where you placed the renamed .EXE.  Now type <filename>.gif and watch the EXE run.  This trick won't work in Explorer for various reasons I won't get into here - but you can run any file, regardless of its extension from inside a command shell if its executable.  We tend to find that the more advanced miscreants favor 'stealthing' using this technique rather than using rootkits (which they know can be easily detected using a variety of anti-rootkit tools).

On an interesting side note:  I once was involved in a case where the miscreants had hacked a large number of servers . . . we had sent resources on-site and were able to develop a tool that could detect the rootkit which was a new, custom rootkit that I don't think anyone had previously seen or detected before.  We refined our tool to the point where it worked amazingly well and we started pushing it down to machines via logon scripts and startup scripts.  At some point the miscreant got wise to our activities and stopped using the rootkit to hide his backdoor - instead he renamed his backdoor DLL (being hidden by the rootkit previously) to a non-executable extension (like .GIF) and stopped copying the rootkit files to the box.  Yes, I realize that we wouldn't have had that problem had the boxes been disconnected from the Internet during this investigation - but that was advice that was ignored by this customer due to financial costs and SLA's - as is usually the case.

Okay - so we know what the .GIF's are - but what about the DLL's.  At first glance these seem to be legitimate Windows files based on the string version information compiled into the DLL's:

Strings v2.1
Copyright (C) 1999-2003 Mark Russinovich
Systems Internals -

Microsoft Corporation
Copyright (C) Microsoft Corp. 1981-1999
Microsoft(R) Windows (R) 2000 Operating System
1, 0, 0, 1
!This program cannot be run in DOS mode.

But the "PEC2" string right above the .RSRC and .RELOC strings above indicate that this DLL has been compacted / compressed / packed using a utility called PECompact to make it smaller.  Miscreants like to 'pack' their malware for many reasons:

  1. If the packer is new enough - it may be enough to allow 'well known' malware to fly under the radar - avoiding detection by AV software that does not yet have support for this new packer / crypter.
  2. It usually makes the files smaller, much smaller - allowing for faster transfers on those slow links from foreign countries. 🙂
  3. It makes it harder for skilled investigators to investigate using tools like 'strings' from Sysinternals - they will have to either un-pack the file (if there is an unpacker available) or study it under a debugger which requires an entirely different set of skills.

There are some tools out there to assist incident responders in determining whether a file was packed and if so what it was packed with - one such tool is PeID that I have used with much success.  WARNING:  Studying malware with PeID is extremely dangerous - some of the plug-ins for PeID will actually load the malware to attempt to un-pack it - do not use this tool or others like it on a machine of any importance unless you like living dangerously.

At this point - there are not many interesting strings in the DLL since it has been compressed (it will decompress at runtime as it loads into memory) - so the DLL has been submitted to the antivirus vendors via our relationship with them (the Virus Information Alliance) and they will be pushing out signature updates as needed.  Even though the DLL was packed - there is enough information in the strings output to lead me to believe this is a new version of YYT_HAC's backdoor trojan (for more information on YYT_HAC, if you can read Chinese visit:  Here is the relevant string that tipped me off:

I recognize that as the default 'prompt' you get when connecting up to YYT_HAC's backdoor.  This is not the first time I've come across malware from this author being used in the wild.

Now - up until this point we haven't really covered anything really 'new' . . . renaming executables to non-executable extensions, packing binaries, hacking boxes - it's all been done before (woohoohooo!) . . . what I'm about to show you is a development that is fairly new to Windows and signifies perhaps a new technique that is being used by the miscreants to hide from administrators and maintain control of their boxes. 

But first some more background on this investigation.  So we've identified some suspicious files on the disk - but we don't know how or where they are being used on the system and for what purpose (assume we haven't collected the files or run strings on them yet).  Let's find out how these DLL's are being used (i.e. how they get loaded).

,WINLOGON.EXE,228,C:\WINNT\SYSTEM32\USERS.DLL,SYMBOLS_NO,0,0,"December 31, 1969 18:00:00",0,,,,,,,,,,,,,

According to this output - this DLL is loaded inside of one process and only one process: WINLOGON.EXE. 
There are a few reasons a miscreant would want to load a DLL inside of WINLOGON.EXE.

  1. It runs as SYSTEM and is a required process - if you get your code in it (i.e. your backdoor) - you are running as SYSTEM too
  2. You can steal passwords from inside of WINLOGON
  3. If you try to kill WINLOGON you will bugcheck the box with a 0xC0000021a and the system will reboot.  For more information on the C21a stop code you can read the following KB article I wrote a while back: 156669 How to troubleshoot a "STOP 0xC000021A" error

The down-side to running your code inside of WINLOGON is that it has to be really clean code.  Any unhandled exceptions in WINLOGON.EXE will cause the process to crash which will cause the kernel to shutdown the machine with the STOP code mentioned above.

So there are numerous reasons why a miscreant would want to run code inside of WINLOGON (with some associated risk) and to date there are only a couple of techniques for getting that code there:

  1. You can use the GINADLL registry value to load a DLL inside of WINLOGON (easy)
  2. You can inject a thread into WINLOGON and forcibly load your DLL into WINLOGON post-boot. (advanced)

Option one listed above simply requires adding a registry entry for your DLL to a specific well known registry location discussed in my KB article and is easy to detect.
Option two listed above requires some OTHER malware to auto-start somehow and then inject code into WINLOGON and is harder to detect (you have to find the other malware responsible for injecting the DLL into winlogon and what if that other malware is hidden by a rootkit?)

But what if the criteria for option 1 and option 2 above were not met?  What if the GINADLL registry value was empty?  What if there was no other malware running on the machine that could inject your backdoor DLL into WINLOGON? 

This is the situation we were faced with this weekend.

I'm going to take a Quentin Tarantino like tangent here and we're going to jump to a completely different scene before we revisit this thought and conclude this blog post.

There were 2 DLL's we found that were dropped on this customers machine.  There was USERS.DLL and there was another called USERBIN.DLL.
Let's see if we can find them in the registry.

Here are the search results for USERS.DLL against this customer's registry:

Files containing instances of 'users.dll'
Number of Files Searched: 1
Time to Search Files: 0 seconds

Here are the search results for USERBIN.DLL against this customer's registry:

Files containing instances of 'userbin.dll'
Number of Files Searched: 1
Time to Search Files: 0 seconds

What's this?  This is a variation of a hiding technique I've discussed in a previous blog post but with a twist.  As it turns out - on Windows 2000 and later operating systems you can implement a service that runs as either an EXE or as a DLL.  If you choose to implement a service as a DLL it will run in what's called a 'service host' which are those 'svchost.exe' processes you see running on all Windows boxes.  This particular miscreant has written a service dll and has hijacked the DLL used by the BITS service and replaced it with his own entry.  If you were to look in the Services applet in control panel / administrative tools - the BITS service would appear normal (that is it's still pointing to: %SystemRoot%\system32\svchost.exe -k netsvcs).  However, below the BITS service registry entry is a 'Parameters' key and in there is a ServiceDLL value and this registry value is SUPPOSED to point to QMGR.DLL - not userbin.dll.

So while this was rather clever - it was easily detected by our analyst Kenny and he restored the registry value to its proper setting.  In addition to restoring this registry value to its proper setting, he also renamed USERS.DLL to a different file name since we had yet to determine how this DLL was getting loaded inside of WINLOGON.EXE.  He then told the customer to restart and when he did the customers box went into a 'reboot' loop at the place where you would normally see the CTRL-ALT-DELETE screen and the customer could only boot to the recovery console.  At the recovery console Kenny had the customer restore 'USERS.DLL' in the SYSTEM32 directory and reboot - and to his surprise the machine booted successfully.

And now we're back to the point where we previously diverged (I hope I haven't lost anyone).  There were no registry entries for USERS.DLL (assuming no rootkit was hiding their presence) - one might assume then that perhaps USERBIN.DLL, which was starting via the hijacked BITS service registry entry was injecting USERS.DLL into WINLOGON when it started . . . but this is not the case as simply restoring USERS.DLL (and leaving the BITS service fixed and pointing to the proper DLL) was enough to get the box to boot successfully.

At this point - Kenny and I theorized that somehow WINLOGON had become 'dependant' on this DLL to function and that if this DLL was removed, WINLOGON would crash with a C21a and the box would get stuck in an infinite reboot -> C21a cycle.

So how does one make WINLOGON dependant on some 3rd party backdoor DLL?  We were stumped . . . until Kenny noticed something intresting in the WOLF output from MBSA.

Warning MS04-011 835732
File version is greater than expected.
[C:\WINNT\system32\winlogon.exe, 5.0.2195.6970 > 5.0.2195.6898]
Note MS04-016 839643
Please refer to 306460 for a detailed explanation.
Patch NOT Found MS04-031 841533
File has an invalid checksum. [C:\WINNT\system32\winlogon.exe]

This MBSA output is confusing because it checks the version of WINLOGON.EXE against two different patches (normally MBSA would only complain about the most recent missing security update).  MBSA claims that the MS04-011 patch has a version is greater than what it was expecting.  This can be considered normal as sometimes there are some non-security related hotfixes that will redistribute files (like WINLOGON.EXE) that customers can apply to fix non-security related issues.  These 'hotfixes' can actually contain files that supersede, or are newer than, publicly released security updates (which are broadly distributed as opposed to hotfixes which are not).  In that sense - this MBSA output is 'normal' or can be explained in the presence of a non-security hotfix - but then we get to the MS04-031 check and MBSA complains of an invalid checksum?  On a completely differnet update?

First off - that IS the version of WINLOGON.EXE that shipped with MS04-031, but MBSA is saying the CRC32 based checksum is invalid.  There is a limitation in the current version of MBSA where it presently doesn't support multiple checksums for multiple versions of the same file contained in a security update.  This poses a problem on multi-proc machines and security updates where there are two versions of the same file in the security update (one for uniproc machines and one for multiproc machines).  The MSSECURE.XML for now always contains the checksum for the uniproc binaries and it will generate the invalid checksum message on multiproc machines that have the multiproc binaries installed.  This would be readily explainable IF this machine were multi-proc . . . but it is not.

I think you see where I'm going with this - it appears that WINLOGON.EXE itself was modified or replaced.  It is no longer the legitimate copy of WINLOGON.EXE that came with MS04-031 on this customers machine.  This would certainly explain the hard-coded dependancy on USERS.DLL and would explain why WINLOGON.EXE would crash when we deleted / renamed USERS.DLL.

As a final proof point we collected MD5/SHA-1 digests of the suspect WINLOGON.EXE and verified that it no longer matches up with any of the ones we ship!

For those who are curious - here are the MD5/SHA-1 digests of the suspect WINLOGON.EXE

MD5:    3A0F4A1C9FA13EE9D5BD88CA76A20758
SHA-1: 7B8F74B78959E266B91C70283292AB39367B2AC0

Now - some of the more clueful readers may wonder why SFP (System File Protection) did not replace WINLOGON.EXE with the original version or complain loudly about this imposter?  This is a reasonable question - for years I have watched in amusement as the miscreants try to overwrite system file only to have SFP put them back immediately.  I was saddened when @Stake released a tool that would temporarily disable SFP (until the next reboot) allowing miscreants to finally overwrite SFP protected files (the imposters would still be detected at the next SFC.EXE scan).  I was informed by our product group that SFP is a *reliability* feature and not a *security* feature and thus it was never intended to be 'hack proof' - the fact that it had been subverted was not a surprise to any of our security developers who said 'it was only a matter of time'.  I've always wondered why on Windows we don't see more trojaning of system binaries and I've always had two theories on why this almost never happens.

  1. We are a closed source company - it's very hard to take something like WINLOGON a core OS subsystem and write it from scratch without access to the source code and make it work in a way that would not break Windows.
  2. System File Protection, despite being only a reliability feature - was actually a formidable barrier for a while that protected the OS from arbitrary system file overwrites.

It is worth pointing out, however, that both of these barriers have been overcome.  As you may or may not know parts of the Windows 2000 operating system were stolen last year from a partner who had legal access to our source code.  In addition it has been known for over a year how to disable System File Protection during the current boot - allowing anyone with admin rights to replace SFP protected system binaries.

As a final thought - the process responsible for implementing System File Protection is none other than our very own WINLOGON.EXE a process on this customers machine which I no longer trust.

UPDATE:  The antivirus vendors have analyzed these two DLL's (USERS.DLL and USERBIN.DLL) and have come up with the following names for them:
USERS.DLL - Backdoor:Win32/Hackdoor.B
USERBIN.DLL - Backdoor:Win32/Bits.B

Win32/Hackdoor.B is an interesting backdoor.  When you run it you can specify a process you want it to "attach" to.  It will then modify that processes imports table by modifying the binary itself (thus confirming our theory that WINLOGON.EXE had been modified) - thus ensuring that when the process starts the DLL is loaded.  It does this rather well and manages to keep the file size the same making it necessary to generate message digests of the files to tell if they were modified.  When the DLL loads it also loads a driver which can intercept all incoming network traffic so that the backdoor can hijack an existing port (it doesn't need to bind to a new port - it can use any existing port).  Hacker Defender 1.0 has similar capabilities.

Comments (16)

  1. smidgeonsoft says:

    Thank you very much for these posts! Very educational and entertaining! It is information like this that lends credence to Microsoft’s commitment to security. I look forward to further installments.

  2. Chris Quirke says:

    Sobering stuff. I work with home PCs that are hopefully not exposed to this detailed attack, and one thing I do is kill admin shares such as c$. I also disable "automatically restart on errors", so halting instead of rebooting on a STOP error gives a chance to see what’s up.

    IMO the OS should never restart on errors if they occur during OS boot (or if you like, a Safe Mode OS boot). There’s no point in endlessly rebooting an unattended PC that never reaches a point it can be remotely admin’d; you just shred the file system.

    In today’s world of mainly stand-alone malware, it’s easy to forget intrafile infectors exist <g>

    Finally, I notice some common drive-by malware drop a non-malware IDE driver, but what I’ve read doesn’t explain why. It could be to uniquely id the HD (more useful than the session IP) but I’d worry about raw disk access, below NTFS’s abstraction layer.

  3. Nektar says:

    What I do not understand is if the system is compromised and so might likely provide false information to a security researcher, how come your wolf analysis program be able to gather all these info and be able to trust it.

  4. Nektar says:

    I believe that Micosoft should provide security guide in the spirit of this weblog instead of their current hardening guides which are very beginner like.

  5. Robert Hensing says:

    In response to Nektars comment about being able to trust data captured from a possibly rootkit’d machine. This is a very real and very valid concern. We spend a lot of time researching rootkits and collecting them off of customers systems. We feel we have a very strong knowledge in this area and our IR toolkit has tools to be able to detect the presence of all known rootkits (and probably even ones we don’t know about). In the event that our rootkit tools come back ‘clean’ and we still can’t detect the presence of a rootkit but feel that one must be there – we request an image of the system and we restore that image in a lab and we do our analysis there.

  6. Robert Hensing says:

    And in response to Nektars later comment about security guidance being in blog form vs. guide form – actually we feel exactly the opposite. Our ONLY concern with blogs right now is that they may ‘dilute’ our security guidance. If it’s worthy of being blogged about and its stuff that customers need to know – our thinking is that it should be in a security best practices document / guide vs. in a blog, this way all customers have equal access to it.

  7. AT says:

    No needs to be able contact MS Product groups to find out that WFP is not a security measure.

    I’ve discussed this about 2 years ago with Secure@microsoft team.

    "This implies WFP can be circumvented or cancelled; it should not be presumed a replacement for having properly restricted user roles and a locked down system. So your assertion that WFP is sesigned to keep the system stable is correct – it is not designed as a security feature."

    Unfortunately – some of my security ideas I’ve discussed with Microsoft – still not implemented 🙁

  8. Lee S says:

    NOt sure how you fixed this for sure, but we had a similar problem today… Symantec found the dlls noted, but no fix. Tech had tried removing the dll and/or renaming and then it blue screened and couldn’t get back. Ended up restoring (which wouldn’t have had to do) and getting a fresh infected system. Files were dropped 12/19/2004. Booted in safe mode, deleted or renamed the dlls, deleted gifs noted, and copied winlogon.exe from the dllcache folder over. Found a user that had been created by someone else in admins group, removed this and we were in business. Thanks for the interesting reading.

  9. Robert Hensing says:

    Sorry I guess I should have called that out in the blog. Since winlogon.exe has been modified on disk to recover the system you need to boot to the recovery console and copy a good copy back to system32 (i.e. from the dllcache or from the latest hotfix uninstall directory). You may even be able to rename winlogon.exe while the system is booted to something like winlogon.bad and then copy the one from the latest security update back to system32

  10. Martin says:

    We just had a similar problem but we solved it in a different way. One of our W2K Servers was behaving "strange" but there were absolutely NO visible problems. All known virus scanners found NO problems. But there was one, we felt it.

    The solution was tricky :

    We just dig a defrag on the server. In the report the server said : …was unable to defrag the following files… -> There was a listing of non visible files. Their location was in %SYSROOT%SYSTEM32driversetcfonts{00 08…}

    Impossible to browse, impossible to delete. There where also some services, not showing up in the service list, only visible via msinfo32.

    None of the services was visible with regedit.

    The only way to stop them was to rename them with another server (visible) and restart the infected one. Then it was also possible to delete the files via the command line (looked like a complete operating system with storage in RECYCLER.BIN).

    We deleted the invisible services with another server (network registry). The name of the services : Atarpisrv (calls ATA122.exe)



    Thank you for your interesting article. Best regards from Martin

  11. Robert Hensing says:

    You did some pretty good detective work – it sounds like you had a rootkit on your box that was hiding the services. Most if not all current rootkits can usually be spotted by connecting up to the affected machine from a ‘known good’ machine across the network and examining either the file system, the registry or both. The rootkits usually only hide stuff locally – they haven’t bothered hooking the redirector yet.

    This is precisely the kind of stuff my team deals with every day – if you ever want help or a second opinion in the future don’t hesitate to give us a call.

  12. Martin says:

    Hello Robert,

    the only thing i’m wondering about is : We once (1 year ago) had some weak passwords but we cleared this problem. Since then we have no weak passwords.

    How do the intruders get into the machine with no weak passwords ?

    Best regards from Martin

  13. Robert Hensing says:

    Intruders get into your machines by using the philosophy of the "3 p’s".




    If you have exposed ports (i.e. no firewall, weak router ACL’s etc.) then they will compromise your machine remotely using either weak passwords or missing patches.

    You say you fixed your weak password problem; but everyones definition of ‘strong’ varies. Is ‘Password;1’ strong? technically it meets length and complexity requirements at 99% of IT shops in the world – but it’s one of the weakest / lamest passwords ever – don’t ever use that!

    So I guess what I’m saying is – if you changed all your admin passowrds and made them ‘stronger’ and you are still getting hacked remotely then you have a problme with:

    1. Ports – check your firewall rules

    2. Passwords – provide an example of a ‘strong’ password you have used in the past

    3. Patches – what’s your security patch policy like? How long does it take to deploy all critical security patches for your OS and applications?

    Finally – it is possible that when the attackers had access previously they installed a backdoor hidden by a rootkit which you now cannot see – this backdoor server would run as SYSTEM and probably allow them to connect up remotely, anonymously, without using an NT password – thus it wouldn’t matter what you changed your admin passwords too – if they’ve installed a backdoor server process on your windows box you can change passwords till your blue in the face and they will still have access.

    You should have someone come in and verify the security of your servers (my team for example could perform a security check of the box that was previously hacked to see if we can find any evidence of persistent malware). Keep in mind that no incident response analysis will ever be 100% guaranteed – if your box was hacked and you have evidence that the attackers have continued to gain access you should immediately flatten the box and rebuild it securely.

    How do you rebuild securely?

    1. You buy Windows Server 2003 (more on why i a second).

    2. You un-plug the network cable from the machine

    3. You install the OS by booting from the CD.

    4. During setup you either leave the local admin password blank (preventing the use of this account for network authN) or you create a really strong 30 character or more pass-phrase on the account.

    5. after setup completes and you reboot – you login as the admin and you immediately enable the firewall on all adapters.

    6. You then plug the network cable in, configure your Internet access settings and hit Windows Update.

    7. You let WU fully update the box and reboot as needed.

    8. You then consider locking down the security even further using security templates from our Windows 2003 security guidance based on the role of the sever.

    In the near future we will be releasing Win2k3 SP1 which I would recommend slip-streaming into a build of Win2k3 you host on a network share. This way you can install Win2k3 over the network using something like RIS or a network boot disk and you can have a Win2k3 SP1 box during setup – this will greatly improve your security as well.

    If you can do all of that – your machine won’t get hacked during or shortly after setup. If you build a Windows 2000 machine and leave the cable plugged in to your internet presence I can’t guarantee it won’t get exploited during hte latter half of setup (after the network stack initializes) or after you reboot for the first time. Win2k has no built-in firewall and doesn’t make you think about good admin passwords during setup.

  14. Martin says:

    Hello Robert,

    i did some further investigation on this topic, here’s the result (strange but true) :

    On Jan 8th at 2.16am we had an AUTOMATIC update with a MS Servicepack (KB 870763 WINS). Strange because we have disabled automatic updates on all machines.

    Then we found serveral files that where copied to the machine with the same timestamp : 2.16am, including http://ftp.exe tftp.exe aso. Also the "Computer" that was hidden behind the %SYSVOL%SYSTEM32etcfonts.{00 08..} was created at that time. We made some copies of the .ini files of the proud intruders.

    Can you tell me how it is possible to start an automatic update without any user interaction and the automatic update service disabled ?

    I think the problem is might be bigger than we think…

    Regards from Martin from the snowy Austrian mountians.

  15. Robert Hensing says:

    Why do you think it was automatic updates that updated your machine? It most assuredly was NOT automatic updates – it was a miscreant patching your machine to close the hole they used to exploit it. It is very common for the miscreants to patch and then harden your machine once they get on it to prevent ‘re-hacking’ or ‘leech hacking’ as they like to call it by other crews.

    You were more than likely exploited by the WINS vulnerability on this server becuase you didn’t have the patch installed – there has been public WINS exploit code available since shortly after we released the bulletin. This is why it’s so critical that organizations perfect their patch management policies and adopt the ability to deploy all critical patches within 24 hours.

    If you think it was automatic updates that installed the patch becuase the account that did the installing was SYSTEM that’s a red herring. While the automatic updates service does run as SYSTEM by default, so does WINS and that’s the service that was most likely exploited here.

    In the future when you find something is strange on your machine, please don’t hesitate to give us a call. 🙂

Skip to main content