Introduction
Build:
The driver will need to be built in debug mode so optimisations are off, ie, the code lines up with the binary, making debugging easier.
Install:
Secure boot need disabling, and the target putting into test mode, with
bcdedit /set TESTSIGNING ON
bcdedit /set loadoptions DDISABLE_INTEGRITY_CHECKS
On reboot 'TestMode' should appear bottom right.
Using certmgr.exe, the .cer file used in signing will need installing on the target machine in:
Trusted Root
This allows the test signed driver to install. Otherwise a release, and MSFT signed, or attestation signed, driver has to be used.
https://learn.microsoft.com/en-us/windows-hardware/drivers/install/introduction-to-test-signing
Enabling debug out:
By default Vista and later versions of Windows wont produce debug output.
To enable this, go to HKLM\Sys\CCS\Control\Session Manager\Debug Print Filter (or create the key)Create a DWORD called DEFAULT set to 0xffffffff to enable debug out for KdPrint
Create IHVDRIVER with a value of 1 for ERROR 4 for TRACE to enable KdPrintEx ERROR or TRACE
Test tools available:
Driver Verifier. Built into Windows
WinDbg
DbgView
WHQL (including Device Path Exerciser)
Driver Verifier:
Built into Windows, it is an administrators and developers tool, as it can help identify badly written third party drivers and find issues during development. https://learn.microsoft.com/en-us/windows-hardware/drivers/devtest/driver-verifier
It works in a number of ways, but as an example, memory allocation. When Verified, a driver request for memory will create a chunk that is surrounded by guard memory, which has a pattern written to it. When that memory is freed, any under or overwrite is detected, and the machine crashed, with a Verifier error message, and the driver code pointed to.
Another test is for accessing paged code or data at elevated levels, such as during an interrupt or timer expiry. Very often this wont cause a page fault, as the memory with the pageable data hasn't been paged. But if memory starts becoming scare, you get a crash. Verifier will check for this even though the memory hasn't been paged yet, and crash the machine, with the relevant error message and the driver code pointed to.
Verifier is very easy to set up and should be used every time a driver is tested, and developed. it is set up by running verifier.exe from a command prompt, and selecting:
Then
selecting the standard settings, including Miscellaneous checks,
Manually select drivers
And either select the driver if already loaded, or select its .sys file from the hard disk
Then do a reboot.
Verifier will be activated for that driver.
One word of note, don’t verify more than two drivers at a time, perhaps
three. Resources can be scare, and if there are insufficient
Verifier will not be able to do its job.
Any error will result in a BSOD, Blue Screen of Death, and a memory dump.
Depending on how the target machine is set up (control panel →
system → advanced system settings → start-up and recovery) this
could be a mini dump, or a memory dump.
These dump files can be opened with WinDbg...
WinDbg
Windbg is available from a number of places, https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/debugger-download-tools and allows crash dumps to be opened.
When the symbol and code paths are specified, it will give you the line of code that caused the error.
Open a crash dump, set the paths, and when loaded, tun !analyze -v in the command window at the bottom. This will auto analyse the crash.
WinDbg can also be used to live debug a system. The ARM set up is a little different, it uses a Type C to A USB cable, connecting to the port nearest the on /off button, to any port on the host PC.
Setting up the target in debug mode is discussed here: https://learn.microsoft.com/en-us/windows-hardware/drivers/debugger/setting-up-kernel-mode-debugging-over-usb-eem-arm-kdnet
When live debugging it will pick up debug traces, depending on the settings on the Target, and any crash will immediately break into the debugger, identifying the offending code.
WinDbg is very powerful, has a very good tutorial shipped with it, and a very good help file. It is one of Microsofts best products in fact.
DbgView
Another really useful tool is DbgView from Sysinternals, now bought out by Microsoft, it allows debug messages to be picked up on the target machine, via a Kernel hook. So run it in Admin mode to install that hook the first time Kernel capture is enabled.
Traces are again dependant on the settings on the Target
WHQL
WHQL is an immense subject, with a varied and complex setup involving Windows Server machines, and various clients in many configurations depending on the kind of driver being tested
https://learn.microsoft.com/en-us/windows-hardware/test/hlk/getstarted/hlk-arm64-getting-started-guide links to https://learn.microsoft.com/en-us/windows-hardware/test/hlk/getstarted/windows-hlk-getting-started
It is a very lengthy and thorough test, and will run Verifier on the driver automatically, and will also run tools like Device Path Exerciser, as well as all sorts of system tests such as handling PnP and Power requests.
If a driver passes WHQL, it is considered to be pretty much bullet proof, and Microsoft will sign it for release.
The setup for a driver like WindowsPerf will need one client, network drivers can need two or three, so it is fairly simple. The instructions in the link above are straightforward, a Windows Server is needed for the Host machine, the client needs to be on the same network. The client pack is installed from the server.
My own experience is that the server and client can get out of step with each other, so I have used VMs for the server, so I can revert to a known good state, ie, with the client visible, the client pack installed, and the driver to be tested visible, with the machine pools set up etc.