Introduction

This document introduces the tools used to test Windows Kernel Drivers. These are not functional tests, these tools know nothing about the intended function of the driver, they test its fitness from a Windows perspective.

Build:

The driver will need to be built in debug mode so optimisations are off, ie, the code lines up with the binary, making debugging easier.

Install:

Secure boot need disabling, and the target putting into test mode, with

bcdedit /set TESTSIGNING ON
bcdedit /set loadoptions DDISABLE_INTEGRITY_CHECKS

On reboot 'TestMode' should appear bottom right.

Using certmgr.exe, the .cer file used in signing will need installing on the target machine in:

Trusted Root

This allows the test signed driver to install. Otherwise a release, and MSFT signed, or attestation signed, driver has to be used.

https://learn.microsoft.com/en-us/windows-hardware/drivers/install/introduction-to-test-signing

Enabling debug out:

By default Vista and later versions of Windows wont produce debug output.
To enable this, go to HKLM\Sys\CCS\Control\Session Manager\Debug Print Filter (or create the key)Create a DWORD called DEFAULT set to 0xffffffff to enable debug out for KdPrint

Create IHVDRIVER with a value of 1 for ERROR 4 for TRACE to enable KdPrintEx ERROR or TRACE

https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/reading-and-filtering-debugging-messages

Test Tools Available

Driver Verifier. Built into Windows
WinDbg
DbgView
Device Path Exerciser
WHQL (including Device Path Exerciser)

Driver Verifier:

Built into Windows, it is an administrators and developers tool, as it can help identify badly written third party drivers and find issues during development. https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/driver-verifier

It works in a number of ways, but as an example, memory allocation. When Verified, a driver request for memory will create a chunk that is surrounded by guard memory, which has a pattern written to it. When that memory is freed, any under or overwrite is detected, and the machine crashed, with a Verifier error message, and the driver code pointed to.

Another example if the test for accessing paged code or data at elevated levels, such as during an interrupt or timer expiry. Very often this wont cause a page fault, as the memory with the pageable data hasn't been paged. But if memory starts becoming scare, you get a crash. Verifier will check for this even though the memory hasn't been paged yet, and crash the machine, with the relevant error message and the driver code pointed to.

Therefore Verifier should be run on any driver being stress tested, or tested with python scripts, or any other functional testing.

Verifier is very easy to set up and should be used every time a driver is tested, and developed. it is set up by running verifier.exe from a command prompt, and selecting:

Open

Then
selecting the standard settings, including Miscellaneous checks,

Open

Manually select drivers

And either select the driver if already loaded, or select its .sys file from the hard disk

Then do a reboot.

Verifier will be activated for that driver.

One word of note, don’t verify more than two drivers at a time, perhaps
three. Resources can be scare, and if there are insufficient
Verifier will not be able to do its job.

Any error will result in a BSOD, Blue Screen of Death, and a memory dump.
Depending on how the target machine is set up (control panel →
system → advanced system settings → start-up and recovery) this
could be a mini dump, or a memory dump.

These dump files can be opened with WinDbg...

WinDbg:

Windbg is available from a number of places, https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools and allows crash dumps to be opened.

When the symbol and code paths are specified, it will give you the line of code that caused the error.

Open a crash dump, set the paths, and when loaded, tun !analyze -v in the command window at the bottom. This will auto analyse the crash.

WinDbg can also be used to live debug a system. The ARM set up is a little different, it uses a Type C to A USB cable, connecting to the port nearest the on /off button, to any port on the host PC.

Setting up the target in debug mode is discussed here: https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-up-kernel-mode-debugging-over-usb-eem-arm-kdnet

When live debugging it will pick up debug traces, depending on the settings on the Target, and any crash will immediately break into the debugger, identifying the offending code.

WinDbg is very powerful, has a very good tutorial shipped with it, and a very good help file. It is one of Microsofts best products in fact.

DbgView:

Another really useful tool is DbgView from Sysinternals, now bought out by Microsoft, it allows debug messages to be picked up on the target machine, via a Kernel hook. So run it in Admin mode to install that hook the first time Kernel capture is enabled.

Traces are again dependant on the settings on the Target

Device Path Exerciser:

Device path exerciser is a tool that stress tests a driver, in our case, we will use it on the IOCTL handler as that is the only interface the driver has. It is available from the 7600 DDK, the last pre Visual Studio one, as a standalone exe. The x86 build runs ok on ARM64, as it has WoW functionality, a kind of VM.

Run devpathexer

On the next page set these values:

Then select all IOCTL tests on the next

After that accept the default logging and start the test.

Ideally have the test machine on WinDbg, so any BSODs can be picked up immediately.

Device path exerciser and this configuration, DevPathExerConfig.XML run devpathexer /cfg, are attached

WHQL:

WHQL is an immense subject, with a varied and complex setup involving Windows Server machines, and various clients in many configurations depending on the kind of driver being tested

HLK Arm64 Getting Started Guide links to Windows HLK Getting Started

It is a very lengthy and thorough test, and will run Verifier on the driver automatically, and will also run tools like Device Path Exerciser, as well as all sorts of system tests such as handling PnP and Power requests.

If a driver passes WHQL, it is considered to be pretty much bullet proof, and Microsoft will sign it for release.

The setup for a driver like WindowsPerf will need one client, network drivers can need two or three, so it is fairly simple. The instructions in the link above are straightforward, a Windows Server is needed for the Host machine, the client needs to be on the same network. The client pack is installed from the server.

My own experience is that the server and client can get out of step with each other, so I have used VMs for the server, so I can revert to a known good state, ie, with the client visible, the client pack installed, and the driver to be tested visible, with the machine pools set up etc.