The Case of the Malicious Autostart


Given that my novel, Zero Day, will be published in a few weeks and is based on malware’s use as a weapon by terrorists, I thought it appropriate to post a case that deals with malware cleanup with the Sysinternals tools. This one starts when Microsoft support got a call from a customer representing a large US hospital network reporting that they had been hit with an infestation of the Marioforever virus. They discovered the virus when their printers started getting barraged with giant print jobs of garbage text, causing their network to slow and the printers to run out of paper. Their antivirus software identified a file named Marioforever.exe in the %SystemRoot% folder of one of the machines spewing files to the printers as suspicious, but deleting the file just resulted in it reappearing at the subsequent reboot. Other antivirus programs failed to flag the file at all.

The Microsoft support engineer assigned the case, started looking for clues by seeing if there were additional suspicious files in the %SystemRoot% directory of one of the infected systems. One file, a DLL named Nvrsma.dll, had a recent timestamp and although it was named similarly to Nvidia display driver components, the computer in question didn’t have an Nvidia display adapter. When he tried to delete or rename the file, he got a sharing violation error, which meant that some process had the file open and was preventing others from opening it. There are several Sysinternals tools that will list the processes that have a file open or a DLL loaded, including Process Explorer and Handle. Because the file was a DLL, though, the engineer decided on the Sysinternals Listdlls utility, which showed that the DLL was loaded by one process, Winlogon:

image

Winlogon is the core system process responsible for managing interactive logon sessions, and in this case was also the host for a malicious DLL. The next step was to determine how the DLL was configured to load into Winlogon. That had to be via an autostart location, so he ran the Autoruns utility, but there was no sign of Nvrsma.dll and all the autostart entries were either Windows components or legitimate third-party components. That appeared to be a dead end.

If he could watch Winlogon’s startup with a file system and registry monitoring utility like Process Monitor, he might be able to determine the magic that got Winlogon to load Nvrsma.dll. Winlogon starts during the boot process, however, so he had to use Process Monitor’s boot logging feature. When you configure Process Monitor to log boot activity, it installs its driver so that the driver loads early in the boot process and begins monitoring, recording activity to a file named %SystemRoot%\Procmon.pmb. The driver stops logging data to the file either when someone launches the Process Monitor executable or until the system shuts down.

After configuring Process Monitor to capture boot activity and rebooting the system, the engineer ran Process Monitor and loaded the boot log. He searched for “nvrsma” and found this query by Winlogon of the registry value HKLM\Software\Microsoft\Windows NT\CurrentVersion\bwpInit_DLLs that returned the string “nvrsma”:

image

The engineer had never seen a value named bwpInit_DLLs, but the name was strikingly similar to an autostart entry point he did know of, AppInit_DLLs. The AppInit_DLLs value is one that User32.dll, the main window manager DLL, reads when it loads into a process. User32.dll loads any DLLs referenced in the value, so any Windows application that has a user-interface (as opposed to being command-line oriented) loads the DLLs listed in the value. Sure enough, a few operations later in the trace he saw Winlogon load Nvrsma.dll:

image

Its power to cause a DLL to get loaded into virtually every process has made AppInit_DLLs a favorite of malware authors. In fact, it’s become such a nuisance that in Windows 7 the default policy requires that DLLs listed in the value be code-signed to be loaded.

The boot trace had no reference to AppInit_DLLs, making it obvious that the malware had somehow coerced User32.dll into querying the alternate location. It also explained why the entry hadn’t shown up in the Autoruns scan. One question he had was why no other process had Nvrsma.dll loaded into it, but further into the trace he saw that an attempt to load the DLL by another process resulted in the same sharing violation error he’d encountered:

image

Simply loading a DLL won’t cause a handle to remain open and cause this kind of error, so he searched backward, looking for other CreateFile operations on the DLL that had no corresponding CloseFile operation. The last such operation before the sharing violation was performed by Winlogon:

image

The stack of the operation, which he viewed by double-clicking on the operation to open the properties dialog and then clicking on the Stack tab, showed that it was Nvrsma.dll itself that opened the file, presumably to protect itself from being deleted and to prevent itself from loading into other processes:

image

Now he had to determine how User32.dll was compromised. User32.dll is one of the system “Known DLLs”, which means that as a performance optimization Windows creates a file mapping at boot time that can then be used by any process that loads the DLL. These known DLLs are listed in a registry key that Autoruns lists in the KnownDLLs tab, so the engineer went back to the Autoruns scan to take a closer look. The most effective way to spot potential malware when using Autoruns is to run it with the Verify Code Signatures option set, which has Autoruns check the digital signature of the images it finds. Upon closer inspection, the engineer noticed that User32.dll, unlike the rest of the Known DLLs, did not have a valid digital signature:

image

The compromised User32.dll behaved almost identically to the actual User32.dll, otherwise applications with user-interfaces would fail, but it seemed to be different enough to cause it to query the alternate registry location. To verify this, he ran the Sysinternals Sigcheck utility on the tweaked copy and on one from a different, uninfected, system that was running the same release of Windows. A side-by-side comparison of the output, which includes MD5, SHA-1 and SHA-256 cryptographic hashes of the file, confirmed they were different:

image

As a final check to make sure that the difference was indeed responsible for the different behavior, the engineer decided to scan the strings in the DLL. Any registry keys and values, as well as file names, used by an executable will be stored in the executable’s image file and be visible to a string-scanning tool. He tried using the Sysinsternals Strings utility, but the sharing violation error prevented Strings from opening the compromised User32.dll, so he turned to Process Explorer. When you open the DLL view for a process and open the properties of a DLL, Process Explorer shows the printable strings on the Strings tab. The results, which revealed the modified APPInit_DLLs string, validated his theory:

clip_image002[4]    image

With the knowledge of exactly how the malware’s primary DLL activated, the engineer set out to clean the malware off the system. Because User32.dll would be locked by the malware whenever Windows was online (otherwise you can rename the file and replace it, which is what the malware did), he booted the Windows Preinstallation Environment (WinPE) off a CD-ROM and from there copied a clean User32.dll over the malicious version. Then he deleted the associated malware files he’d discovered in his investigation. When through, he rebooted the system and verified that the system was clean. He closed the case by giving the hospital network administrators the cleaning steps he’d followed and submitted the malware to the Microsoft antimalware team so that they could incorporate automated cleaning into Forefront and the Malicious Software Removal Toolkit. He’d solved a seemingly impossible case by applying several Sysinternals utilities and helped the hospital get back to normal operation.


Comments (43)

  1. Anonymous says:

    Anti-virus doesn't flag an unverified dll, gotta love the usefullness of A/V.

  2. Anonymous says:

    @zzz Great points: it is possible to really lock things down with secure launch rooted in tamper-resistenc firmware. Even that's not a magic bullet against malware infection, though, and Windows is a general purpose open operating system with a tremendous amount of application compatibility, not a console.

  3. Anonymous says:

    @ advcom: A rootkit can hide a virus completely (if it is weel written).  And yes a virus can be verified – it just take $70 to buy a signing cert.  The majority of virus' are not signed though.

  4. Anonymous says:

    @Yaron, @Eric: I clarified the closing paragraph to make it clear that it's the malware's open handle that locks the file. Under normal circumstances you can rename system DLLs, since the file mappings created by the loader don't keep the file handles open.

  5. Anonymous says:

    @Decheng Thanks for the feedback and correction!

  6. Anonymous says:

    Mark,

    I do about 10 or so virus removals per week and I refuse to reinstall unless there is corruption to the os. I can remove most virus in an hour, but occasionally I will run into a virus that keeps me up half the night. My question is this. Can a rootkit completely hide it's autostart entries from Autoruns. Does a rootkit hide it's malicious files from all software or just software it is programmed to be aware of? And my last question. Can a virus show as "verified" in Autoruns? In other words, can I trust all files that show as verified in Autoruns? Just things I have wondered and never really got a straight answer on.

    Thanks,

    Miles

    Advanced Computing

  7. Anonymous says:

    @Craig, @Aram: there's no magic bullet here. Windows does try and prevent tampering: in Windows XP Windows File Protection replaces modified versions with one from a backup location (DllCache), but the malware modifies that version as well. On Windows Vista the files are locked down so that only the Trusted Installer account. But the fundamental problem remains: if malware is executing with admin rights, it can circumvent any protection.

  8. Anonymous says:

    @Mihailik Unfortunately, that's a question just about anyone fighting a new malware infection will have a near impossible time of determining. Unless you actually see the infection as it takes place, you can't know – it could have been someone executing a malicious email attachment, opening an infected document, or via a network-spreading worm. Just because you don't know for sure how the infection happened doesn't mean that you shouldn't attempt a cleanup. And in an ideal world, once you find an infection you would repave every system on your network. Unfortunately, that's just not practical in most cases and even then the infection could occur again, if for example it spread from outside via a network vulnerability or the user that caused it in the first place launched the attachment again.

  9. Anonymous says:

    @Sol I'm not sure you read what I wrote: "if malware is executing with admin rights, it can circumvent any protection".

  10. Anonymous says:

    @ComputerGeek01: Well, the way Mark laid it out to us, yes, you are right. But on scene, things aren't simple and straightforward as they look from here.

    Still, if the things on scene were as they look here, … well, you'd have been right…

  11. Kane says:

    Bravo! That was an awsome case of great detective work and demonstration the use of different tools. I am very impress. Kudos to the tech.

  12. Good article, but how did the malware replace User32.dll while Windows

    was running in the first place?  As you stated above, Windows will

    always be running this dll.  If there is a way to do this can you

    perform the same method instead of dropping into WinPE?

    Just curious…

  13. Yaron says:

    Awesome work indeed! Can you please elaborate how the malware infected the original User32.dll if it's locked whenever Windows is online (MoveFileEx?)?

  14. Jeff Allen says:

    Because bwpInit and appInit are the same length, I'm assuming they used an automated binary editor to edit the string inside the dll? Reminds me of a hack we used a few jobs back where every time we would exit the fax application, it would return some unexplainable error. Other than that, the app worked perfectly. So I edit the DLL with that error message and replaced the string with a "Thanks for using FaxWorks!" message that was exactly the same length string! :)

  15. tam says:

    A procmon pmb file will survive a bsod and a pml won't right?  Is there a way to have procmon log to a pmb instead of a pml without going through boot logging?

  16. Peter Hahndorf says:

    Couldn't the malware also use the MoveFileEx-API and Mark's own MoveFile.exe to replace files? It could initiate a reboot or just wait for the user to reboot.

  17. Eric Wilson says:

    I thought Windows protected core files like User32.dll from being replaced.    Was this disabled in some way?  Is user32.dll not protected?

  18. Harry F says:

    Wouldn't whitelisting solutions be a better alternative to this attack?

  19. Aram Hăvărneanu says:

    What is not explained is how User32.dll got patched in the first place.

    Also I find is strange why the malware author bothered with creating a payload dll that he loaded with a patched User32.dll. Surely if he found a way to patch User32.dll he could drop his payload directly there, why bother with a different dll.

  20. Craig says:

    Why don't we give Windows stronger tamper resistance or detection – at least for objects like knownDLLs?

    Where was WFP during all of this? Granted WFP can be disabled, but couldn't Windows could at least detect the appearance of an unsigned or unverifiable knownDLL and log/alert on this?

  21. Jeff Allen says:

    Aram,

    If the size of the original DLL changes, it will refuse to load or crash.

  22. Sol Birn says:

    How about digital signatures and hash checking (at least added to security essentials if not windows itself)?

  23. Shri Ganesh says:

    Hi Mark, great post! I was just wondering how to wondering how to fight malware which doesn't have an associated autorun entry! Recently I read somewhere where a rootkit modifies a windows kernel mode driver and every time, winodws loads with no autorun entry, the malware also loads with windows! Moreover the modified file and the original file also were of exact same length.

    I guess only CRC/MD5 check will find the difference. In such a case, will the verify code signatures find the difference? I think it has started to become more and more difficult clean malware without reinstalling the OS!

    Also could you please tell me, if such a rootkit loads with a kernel mode driver, is it possible for that rootkit fool all the tools and conceal it's presence? Any chance to find it and clean it in WinXP and Win7?

  24. Sol Birn says:

    Point taken. Can the system be re-structured, so as to limit the admin rights with regards to core system files; something like a super-system account, for internal os use only (at least w/o an "pre"-boot intervention)? Can we expect future security development in the system files area?

  25. Daniel says:

    Would a mainboard with TPM and white-listing all the signed apps in a PC protect against this attack?

  26. Craig says:

    ..understand the malware problem is unsolved; I'm thinking more along the lines of survivability and scalable detection & response. Why not incorporate and automate some of the Sysinternals tools into Windows and make Windows more tamper evident? For example, detect and generate an alert when an unverifiable knownDLL shows up..or an unsigned program file in system32..run process monitor in the background with circular logging..and while we're at it, make it easier to natively export logs and events out of windows to a log aggregation and/or threat management tool

  27. Mihailik says:

    Mark, the question is not whether corrupt user32.dll could or could not get onto the system. The question is how exactly did it happen?

    If you don't answer that, you can't reasonalby close the case. 20 minutes after you clean the system it might get corrupted again through the same way it was infected before.

    You might still be riding the elevator approaching the ground floor to exit the hospital — and that very moment the user32.dll gets replaced, computers restarted, the winlogong pulls the traitor in and the printers wake up to produce more garbage.

  28. zzz says:

    @Mark "I'm not sure you read what I wrote: "if malware is executing with admin rights, it can circumvent any protection"."

    HDD or motherboard firmware can calculate a hash of every byte (not file) loaded during the boot process and compare this on every boot. This could be achieved by putting all the critical and rarely changing boot & OS sectors in one continous area and when the firmware detects that the read is happening inside this area it will perform the hashing, so that often changing files are not included in the hash.

    Going with the "KISS" principle, the BIOS should simply either stop the boot and ask the user if the user performed a update involving system files, if user did not then it was malware. For IT-scenarios there'd need to be bit more involved process, possibly with the BIOS talking to a service managing the computers in the network.

    Also your statement suggests that anyone running with admin right can instantly circumvent any protection. Reality check: Only poor protections are cracked by quick by "career crackers". Read up on cracking the consoles and certain PC music software which took years – enough time to overhaul the protection for next major version if they wanted (but they didn't so it was cracked quicker).

    So realistically if Windows had proper protections, just getting to Admin wouldn't mean a total compromise, since ANY GOOD protection isn't just ONE layer (being admin) but a matrix of several layers of checksum checks and boundaries.

    This is just a case of MS putting hands up "if they get admin game over" well obviously that is false truth that's repeated a lot by tech press but I know there's few people beside me that know better. The issue with TPM & Intel is the way it was marketed and published by tech press. It didn't come out as a security solution to protect users from bad things, it came as a way to lock people out of their computers and to protect IP & rights – what better way to get tech press to shoot you down. And of course it wasn't really thought out well – having such thing baked into CPU without choice isn't good due to perception that causes. It needs to be optional and in the motherboard (firmware). Of course later it turned out the TPM had problems but it was doomed even before the implementation issues came to light.

  29. zzz says:

    Slight correction, I said "on every boot" but:

    The firmware must also hash the boot & OS boot/security file sectors prior to starting the OS update process, not just on boot and then invoke the update to happen from the re-certified files.

    There's no doubt other details to be aware of but the critical OS files can be secured, both on disk and in memory. This is prob why Intel bought McAfee, they want to incorporate hash checks during execution. McAfee prob. had patents related to something along these lines.

  30. Mihailik says:

    In order to replace user32.dll under the nose of all that Windows Protection one has to 'accept' quite a few nagging message box popups. That activity is usually reflected in Event Logs too.

    Given the date/time of nv*.dll you could reasonably narrow down the opportunity window and look there in the logs for the traces.

    Or maybe Windows Protection was completely switched off in that hospital? Shouldn't you recommend switching it back on then?

    Hopefully you've done all that investigation and just skipped it here for brewity :-)

  31. Hector Santos says:

    Great Detective work. Eager to read Zero Day (I had ambitions of writing a software-crime story/book for a long time, so maybe your book will be a good basis for what works.  :))

    Its seems obvious the infection, as nearly always, with promiscuous and/or unauthorized loading of downloaded software.  This generation is too trusting. I don't even think its part of their mindset to have any kind of suspicion – they just click.  But too much time is increasingly being spent on this sort of complex detective work and frankly I hope it doesn't become the new norm (as a means to solve the entry point problem).

    BTW,  great work with Microsoft!

    Hector Santos, CTO

    Santronics Software, Inc.

  32. Zack says:

    Great..

    At the moment I only use 4 tools with sysinternals, but I guess its time to increase that number.

    Mark, please blog more incident response that uses sysinternal tools.

    We would love to read and test it.

    Cheers

  33. Decheng (Robbie) Fan says:

    Nice article containing so many good use of the SysInternals tools. One small typo: "making if obvious" => "making it obvious". :-)

  34. Edgar says:

    Mark, great war story as always!!

    As I was reading through the comments I saw you remark of how it is nearly impossible to identify the initial attack vector. I found that interesting as that is something we always struggle with in our engagements, and while sometimes there is enough evidence left behind to give you a reasonable guess as to the initial vector, that by no means always the case.

    So to try to tackle that problem we developed a sensor for our customers that tracks execution on a system, and stores that information. I like to think of it as a Tivo, that allows us to rewind "the show" and see what what happened in slow motion. As you can imagine there have been some hurdles, such as making sense of the huge amount of data collected, but with some trial and error we have been able to tweak it so that the data collected is manageable, and it doesn't impact the user's experience.

    While it is still in its beta stage, we have used it in a couple IR incidents and found it to be effective and efficient for identification and remediation and more importantly, has saved us a bunch of time… the one problem with it is…well it has to be installed first :-)

    Again thanks for sharing the case, they are always interesting reads.

    Edgar Sevilla, CIO

    Kyrus Technology

  35. ComputerGeek01 says:

    It's great reading stuff like this on your blog, although I can't help but to feel that the support engineer had his answer a few steps before he executed a solution. Was this a rational precaution as in better safe then sorry?

    I have to say though the comments on this one leave me a bit surprised. It was pretty clear from the start to me at least that this hospitol was running a dated OS (XP I would guess), yet some of the posters don't seem to realise how far along MS Securtiy Center has come in its past few iterations. Also you would think that fellow Sys Admins would know some of the basics behind stuff like this: en.wikipedia.org/…/DLL_injection

  36. wanderSick says:

    What I get from reading this (hopefully right!): If we see an unverified user32.dll on the Known_DLLs tab in Autoruns, Appinit could have been tampered and we should not trust what is shown on the Appinit tab anymore. Instead, we should inspect the Appinit registry entry manually.

  37. Hector Santos says:

    Probably a side note or note for a new topic,  but pstools has been and now the suite is part of my everyday usage.  Is it possible to make it a 1 time "I AGREE" for all the tools rather than have it for each tool, for every machine, for every profile they are run under?  It is so ignored anyway and a few time I got stuck with a passive psexec hung on another machine waiting for the "I AGREE" user input.

  38. wanderSick says:

    @Hector Santos

    For PsExec, there's the switch  -accepteula.

    To accept all EULAs for all tools at once,  you may 'Bing' this script by a Sysinternal forum member: "syseula.cmd". It will set the AcceptEula key in the registry for all tools.

    Alternatively, there's a third party automated tool to install all Sysinternals tools, accept the EULA, update system path, unattended, all at once. You may 'Bing' 'SSIBuild'. It is a tool made by DarkShadows of the MSFN forum.

    HTH

  39. Keith says:

    I am very impressed with story, the tech and the the description of tools used. It was a very interesting and entertaining read as well as informative. It was loaded with information regarding the registry and what happens at Windows start-up. I really enjoyed it, thank you.

  40. mohdtarmizie says:

    Good job done! What an awesome tech article! :)

  41. Ncage says:

    Curious if at the time of infection (if it was windows 7) the user would have got a UAC dialog. Moral of the story is don't run under and administrative account. I wish i could heed this device but i can't. I'm a developer and i just run into to many problems IIS, ect….but running under a limit account. God knows i've tried. My solution is just to run utilities like sandboxie & appguard though i know these things are just to dang complicated for the general user.

  42. Chris Hutchcroft says:

    Great story, every one I read I end up using one more of the tools on a regular basis. I hadn't even thought of using boot time logging with procmon until I saw it at Tech-Ed last year.

    @wanderSick thanks for the tip on the EULA batch file

  43. Ramesh says:

    Great article! Reminds me about hex editing of "Regedit.exe" to circumvent the "DisableRegistryTools" Policy.