Pre-mortem debug analysis

We’ve looked at generating dumps of processes, the kernel or the entire set of used physical memory pages – but there is another method to do debug analysis on the target directly rather than with a “snapshot” of what it looked like at one point in time, and sometimes this is very useful.

The “live” debug is where we attach a debugger directly to a process (running on the same machine) or a port (on a different machine) to communicate with the kernel.
The latter is very interesting, and until the days of virtualization was very tricky to set up, and was very slow.

A live kernel debug requires the target (debuggee) is running in debug mode – this instructs the kernel to listen on the specified port for instructions from a debugger running on another machine.
If a break instruction is received, the debuggee is then frozen and control is passed to the debugger.

Virtually serial

A virtual machine running on Hyper-V has 2 virtual COM ports available to it, but these cannot map to physical serial ports on the host (which might not even have any), so what use are they?
You can specify a name for a a “named pipe” with which any process running on the host or even a remote machine can communicate with the “COM” port of the guest.

For example, if we have a Windows Server 2008 guest machine and set its COM1 to use the name “w2k8”, the named pipe path as accessed by the host itself would be:
\\.\pipe\w2k8

Making the kernel listen

Inside the guest machine, the BCD needs to hold the configuration that indicates we want to enable kernel debugging, and the settings to use, so the following 2 command at an elevated command prompt will enable this:

bcdedit /dbgsettings SERIAL DEBUGPORT:1 BAUDRATE:115200
bcdedit /debug ON

The first command indicates kernel debugging is being done via a serial port, which is COM1, at 115,200 baud.
The second command actually enables kernel debug mode during boot.

(Windows versions before Vista had debug settings configured in BOOT.INI instead.)

Taking control

On the Hyper-V host you can now launch an elevated WinDbg, select File / Kernel Debug and point to the named pipe path (ensuring “Pipe” is checked and “Baud Rate” is set to 115200).

So long as the host has access to download symbols, you should now be able to handshake with the debuggee kernel to get the version string, and hitting CTRL-BREAK will suspend the debuggee and turn control over to the debugger (until ‘g’ is entered to resume).

While the debugger is connected you will see all the debug spew that the kernel would normally hide from you (if you are running instrumented binaries then this might be a lot).
While the debugger has control the target machine is effectively inaccessible to everyone else, and you can do all the things you can with a .DMP File and more – if you want to crash the target machine to create a memory dump at any time, enter the command ’ .crash’.

The real power of live debugging, however, is the ability to set breakpoints – allowing the kernel to run as normal until it hits a certain criteria (CPU register contains a specific value, a memory address is accessed or a point in a function is reached).
As soon as it hits a breakpoint the debugger is passed control immediately – for you to run further commands to investigate a problem, or to have it automatically script some actions and resume if it will be hit a lot.

The command ‘bl’ will list the currently-set breakpoints that you have defined (lost each time the target is reset), and ‘bp’ sets an unconditional breakpoint at the address passed to it.
e.g. To make the debugger break in every time any file is opened:
bp nt!NtCreateFile

To clear (remove) a breakpoint, use ‘bc’ and the number assigned to it (visible via ‘bl’).
To disable but leave defined a breakpoint, use ‘bd’ and its number.
In either of the 2 above cases, an asterisk is a wildcard meaning “all breakpoints”.

There are also “events” that the kernel can send to the debugger which it can use for control instead of explicit breakpoints, these are configurable in WinDbg through Debug / Event Filters – you can monitor the creation and exit of threads & processes, load and unloading of modules, various normally handled exceptions, and much more.

As a live debug has access to the entire still-running system, you can look in the kernel address space or the user-mode address space of a particular process – it is important to force a reload when switching context, the way I always use this is (for fictional PID 12345678):

.process /p /r 12345678

This will set the debugger context in the user-mode portion of this process so that ‘ !peb’ will work and the ‘k’ commands will show the function names (symbols permitting) of the user-mode modules instead of memory addresses.