Event Tracing for Windows

Event tracing in its simplest form uses the system Event Log (Computer Management, Event Viewer).

A more complex use is WPA/WPR Windows Performance Recorder and Analyser, available on the Windows Performance Toolkit Windows Performance Toolkit

Download Windows Performance Toolkit

Note: You can download the Windows Performance Toolkit through Download and install the Windows ADK.

Do not forget to read What's new in the Windows ADK and ADK tools | Microsoft Learn article for more details.

Architecture

A driver which produces such logs is registered with the system with an XML file, which identifies the source of the logs, and the form they take (data types), names, and other details.

The XML file is in the project Resource Files folder.

The XML is processed by the Message Compiler by adding this to the project file in each ItemDefinitionGroup (ie, Debug|ARM64 and Release|ARM64)

<MessageCompile> <HeaderFilePath>.\\</HeaderFilePath> <RCFilePath>.\\</RCFilePath> <GenerateKernelModeLoggingMacros>true</GenerateKernelModeLoggingMacros> <UseBaseNameOfInput>true</UseBaseNameOfInput> </MessageCompile>


The generated header file is then included in the source code, and the generated binary file is added to the end of the resources file like this:

LANGUAGE 0x9,0x1 1 11 "Wperf_DriverETW_schema_MSG00001.bin" 1 WEVT_TEMPLATE "Wperf_driverETW_schemaTEMP.BIN"

Registering


This XML file is copied with the driver to the system32\drivers directory, and needs to be registered with the system on install with wevtutil.exe im Wperf_DriverETW_schema.xml from the drivers directory.
To unregister call wevtutil.exe um Wperf_DriverETW_schema.xml.

Viewing logs

Open Event Viewer. Right click Custom View and select Create Custom View.

 


Click OK, give it a useful name, and OK again.

You will be presented with logs like this

 



Windows Performance Recorder / Windows Performance Analyzer

Windows Performance Recorder is a tool to enable/disable the recording of trace acquisition. You can configure it either via a GUI or CLI wpr.

image-20240124-192024.png

You can either use default profiles for recording or create your own custom profiles which use the .wprp extension but are just normal xml files.

After selecting the set of profiles just click Start and it will start tracing.

image-20240124-192553.png

After the workload runs you can click Save which will create a .etl file that you can open on WPA. It will somewhat look like this

image-20240124-192748.png

Adding ETW support to the driver

You need to add an Instrumentation Manifest to all applications that make calls to EventWrite* outputting events to ETW or a log channel. The manifest defines all the type of events and providers. You should compile your manifest with the Message Compiler (Message Compiler (MC.exe) - Win32 apps | Microsoft Learn)

Documentation says that inside the Windows SDK there is the tool ECManGen.exe which is a GUI to facilitate writing mainfests however this link Windows 10 SDK Preview Build 16267 and Mobile Emulator Build 15240 Released - Windows Developer Blog shows that the tool was removed.

Sample manifest

<instrumentationManifest xmlns="http://schemas.microsoft.com/win/2004/08/events" xmlns:win="http://manifests.microsoft.com/win/2004/08/windows/events" xmlns:xs="http://www.w3.org/2001/XMLSchema" > <instrumentation> <events> <provider ...> <channels> <importChannel .../> <channel .../> </channels> <levels> <level .../> </levels> <tasks> <task .../> </tasks> <opcodes> <opcode .../> </opcodes> <keywords> <keyword .../> </keywords> <filters> <filter .../> </filters> <maps> <valueMap ...> <map .../> </valueMap> <bitMap ...> <map .../> </bitMap> </maps> <templates> <template ...> <data .../> <UserData> <!-- valid XML fragment --> </UserData> </template> </templates> <events> <event .../> </events> </provider> </events> </instrumentation> <localization> <resources ...> <stringTable> <string id="" value=""/> </stringTable> </resources> </localization> </instrumentationManifest>

Some of this entries are optional e.g. tasks and opcodes are for organizational purposes.

Defining events

DRAFT

“If administrators will be consuming your events using a tool like Windows Event Viewer, then you must set the channel attribute.”

Events that have more than a single message seem to need a message template. You can define an event template like so

<template tid="t4"> <data name="FilesCount" inType="win:UInt16" /> <data name="Files" inType="win:UnicodeString" count="FilesCount"/> <data name="Path" inType="win:UnicodeString" /> </template>

And the event to use it like so

<event value="3" level="NotValid" keywords="Local Write" task="Validate" opcode="Cleanup" template="t4" channel="c2" symbol="TEMPFILE_CLEANUP_EVENT" message ="$(string.Event.TempFilesNotDeleted)"/>

Custom .wprpand how to plug the driver

<?xml version="1.0" encoding="utf-8"?> <WindowsPerformanceRecorder Version="1.0" Author="Linaro"> <Profiles> <SystemCollector Id="SystemCollector_Wpr" Name="WprTest_SystemLogger"> <BufferSize Value="1024"/> <Buffers Value="32"/> </SystemCollector> <EventCollector Id="LinaroEventCollectorID" Name="LinaroEventCollectorName"> <BufferSize Value="256"/> <!-- Optional. Default value is 128--> <Buffers Value="64"/> <!-- Optional. Default value is 64 --> </EventCollector> <SystemProvider Id="SystemProvider_Base" Base=""> <Keywords> <Keyword Value="ProcessThread"/> <Keyword Value="Loader"/> </Keywords> </SystemProvider> <EventProvider Id="WperfProviderID" Name="Wperf_Driver" Stack="false" NonPagedMemory="true"> </EventProvider> <Profile Id="TraceBus.Verbose.File" Name="TraceBus" DetailLevel="Verbose" LoggingMode="File" Description="TraceBus script"> <Collectors> <SystemCollectorId Value="SystemCollector_Wpr" > <SystemProviderId Value="SystemProvider_Base"></SystemProviderId> </SystemCollectorId> <EventCollectorId Value="LinaroEventCollectorID"> <EventProviders> <EventProviderId Value="WperfProviderID"/> </EventProviders> </EventCollectorId> </Collectors> </Profile> </Profiles> </WindowsPerformanceRecorder>

Running traces for PMU counters

WPR PMU sources on Volterra using wpr -pmcsources

Maximum selectable profile sources: 5. Id Name Interval Min Max -------------------------------------------------------------- 0 Timer 10000 1221 1000000 2 TotalIssues 65536 4096 2147483647 8 DcacheMisses 65536 4096 2147483647 9 IcacheMisses 65536 4096 2147483647 11 BranchMispredictions 65536 4096 2147483647 19 TotalCycles 65536 4096 2147483647 21 DcacheAccesses 65536 4096 2147483647 25 SoftwareIncrement 65536 4096 2147483647 26 ICacheRefill 65536 4096 2147483647 27 InstructionTLBRefill 65536 4096 2147483647 28 DCacheRefill 65536 4096 2147483647 29 DCacheAccess 65536 4096 2147483647 30 MemoryTLBRefill 65536 4096 2147483647 33 InstructionRetired 65536 4096 2147483647 34 ExceptionTaken 65536 4096 2147483647 35 ExceptionReturn 65536 4096 2147483647 36 WriteContextID 65536 4096 2147483647 41 BranchMisprediction 65536 4096 2147483647 42 CycleCount 65536 4096 2147483647 43 BranchPrediction 65536 4096 2147483647 64 L2DCacheRefill 65536 4096 2147483647 65 L2DCacheAccess 65536 4096 2147483647 66 L1ICacheAccess 65536 4096 2147483647 67 BusCycles 65536 4096 2147483647 70 L3CacheAccess 65536 4096 2147483647 71 L3CacheRefill 65536 4096 2147483647

Counter allocation

Xperf does not support multiplexing as on Volterra the following command xperf -on proc_thread+loader+cswitch+dpc+interrupt+pmc_profile -pmcprofile instructionretired,dcachemisses,branchprediction,timer,totalcycles,buscycles -f kernel_03.etl -stackwalk pmcinterrupt returns

PS C:\Users\tcwg> xperf -on proc_thread+loader+cswitch+dpc+interrupt+pmc_profile -pmcprofile instructionretired,dcachemisses,branchprediction,timer,totalcycles,buscycles -f kernel_03.etl -stackwalk pmcinterrupt xperf: error: Failed to configure counters

References

  1. Recording Hardware Performance (PMU) Events | Microsoft Learn

  2. Recording with Custom Profiles | Microsoft Learn

  3. Record a Heap Snapshot | Microsoft Learn

  4. Recording Profiles | Microsoft Learn

  5. Using the Windows Performance Toolkit (WPT) with WDF - Windows drivers | Microsoft Learn

  6. Writing an Instrumentation Manifest - Win32 apps | Microsoft Learn