Recently while I was analyzing a bunch of samples packed by custom packers, one of them struck me as a bit different than any others I saw before. At first glance, the outer layer of packing is a UPX stub, which is commonly used in malware. Especially when combined with a custom packer, UPX can provide an excellent compression ratio.
Since it's packed by UPX, I first unpacked it with a static unpacker and examined the dump. The heavily obfuscated code at the entry point easily leads me to think there is another layer of packing, so let's trace a bit to see what happens. The inner packing eventually ends up with a dead loop, which is calling Kernel32!GetUserDefaultLCID in each iteration.
Figure 1. The obfuscated dead loop
From the above figure, it's very clear that this is a closed loop without an exit condition and the code is mainly useless. But the blackbox analysis shows it's actually doing something. Going backwards to the entry point, I carefully look through the code and find two interesting spots.
Figure 2. The first check by UPX
Although there is plenty of code before the 'or eax, eax' instruction in Figure 2 above, none of them change the value of register EAX, but this UPX packer assigns a pointer from the stack to EAX before jumping to OEP (see Figure 3 below). So, on the versions of Windows where the register EAX is zero when the file loads, if the file is unpacked and then run, the sample will know it and jump to the closed loop.
Figure 3. The initialization of EAX in UPX
Another code snippet from the inner packer does additional checks against the presence of UPX. To satisfy this check, the EAX must equal to ESP + 0x11C, which matches the EAX value initialized by the UPX stub code.
Figure 4. Another check by UPX
If either of the checks fail, it leads to a dead loop or an unhandled exception. Now we know that to debug or emulate this sample, the UPX stub code can't be skipped by a static unpacker. By bypassing or including the UPX stub code in the execution, the malware underneath is finally revealed to be a member of the Bamital malware family. I also compared the size of UPX, packed and unpacked, and notice a big difference in file sizes, so I decide to dig more. The encrypted file underneath occupies a big portion of the file and looks to be filled with a lot of padding. By looking through the encryption this packer uses, it's clear that it appends tons of useless padding bytes (0x6A in this case) to the compressed code and re-orders the whole buffer, so that the compressed code blends into the padding, making it look like junk data that can avoid entropy checking and other similar techniques.
Figure 5. Mixed compressed code with padding
Figure 6. Compressed code appended by padding
The malware author may have tried out this technique to avoid detection, but we're on to them. With our antivirus products such as Microsoft Security Essentials, this trick is handled well and we detect the malware underneath as Trojan:Win32/Bamital.I.