Digging into Windows Internals

Although much of the information in this book is based on reading the Windows source code and talking to the developers, you don't have to take everything on faith. Many details about the internals of Windows can be exposed and demonstrated by using a variety of available tools, such as those that come with Windows, the Windows Support Tools, the Windows resource kit tools, and the Windows debugging tools. These tool packages are briefly described later in this section.

To encourage your exploration of Windows internals, we've included "Experiment" sidebars throughout the book that describe steps you can take to examine a particular aspect of Windows internal behavior. (You already saw one of these sections earlier in this chapter.) We encourage you to try these experiments so that you can see in action many of the internals topics described in this book.

Table 1-3 shows a list of the tools used in this book and where they come from.

Performance Tool

We'll refer to the Performance tool found in the Administrative Tools folder on the Start menu (or via Control Panel) throughout this book. The Performance tool has three functions: system monitoring, viewing performance counter logs, and setting alerts. For simplicity, when we refer to the Performance tool, we are referring to the System Monitor function within the tool.

The Performance tool can provide more information about how your system is operating than any other single utility. It includes hundreds of counters for various objects. For each major topic described in this book, a table of the relevant Windows performance counters is included.

The Performance tool contains a brief description for each counter. To see the descriptions, select a counter in the Add Counter window and click the Explain button. Or open the Performance Counter Reference help file in the resource kit. For information on how to interpret these counters to detect bottlenecks or plan capacity, see the section "Performance Monitoring" in the Windows 2000 Server Operations Guide, which is part of the Windows 2000 Server Resource Kit. These chapters provide an excellent description to anyone seriously interested in understanding Windows performance. For Windows XP and Windows Server 2003, see the Windows Server 2003 Resource Kit Performance Counters Reference documentation (available online at http://www.microsoft.com).

Note that all the Windows performance counters are accessible programmatically. The section "HKEY_PERFORMANCE_DATA" in Chapter 4 has a brief description of the components involved in retrieving performance counters through the Windows API.

Windows Support Tools

The Windows Support Tools consist of about 40 tools useful in administering and troubleshooting Windows systems. Many of these tools were formerly part of the Windows NT 4 resource kits.

You can install the Support Tools by running Setup.exe in the /Support/Tools folder on any Windows product distribution media. For Windows 2000, the Support Tools are the same on Windows 2000 Professional, Server, Advanced Server, and Datacenter Server. Windows XP has its own version of the Support Tools, as does Windows Server 2003.

Windows Resource Kits

The Windows resource kits supplement the Support Tools, adding additional tools for system administration and support. The Windows 2003 Resource Kit tools are freely downloadable from http://www.microsoft.com (by searching for "resource kit tools"). They can be installed on Windows XP or Windows Server 2003.

There are two editions of the Windows 2000 resource kits: the Windows 2000 Professional Resource Kit and the Windows 2000 Server Resource Kit. (Supplement 1 is the most recent version.) Although the latter kit is a superset of the former and can be installed on Windows 2000 Professional systems, none of the experiments in this book use the tools that are included only with the Windows 2000 Server Resource Kit. Unlike the Windows Server 2003 Resource Kit, these tools are not freely downloadable. However, the Windows 2000 Server Resource Kit is included with the MSDN and TechNet subscriptions.

Kernel Debugging

Kernel debugging means examining internal kernel data structures and/or stepping through functions in the kernel. It is a useful way to investigate Windows internals because you can display internal system information not available through any other tools and get a clearer idea of code flows within the kernel.

Kernel debugging can be performed with a variety of tools: the Windows Debugging Tools from Microsoft, LiveKD from http://www.sysinternals.com, or SoftIce from Compuware NuMega. Before describing these tools, let's examine a file that you'll need to perform any type of kernel debugging.

Symbols for Kernel Debugging

Symbol files contain the names of functions and variables. They are generated by the linker and used debuggers to reference and display these names during a debug session. This information is not usually stored in the binary image because it is not needed to execute the code. This means that binaries are smaller and faster. However, this means that when debugging, you must make sure that the debugger can access the symbol files that are associated with the images you are referencing during a debugging session.

To use any of the kernel debugging tools to examine internal Windows kernel data structures (such as the process list, thread blocks, loaded driver list, memory usage information, and so on), you must have the correct symbol files for at least the kernel image, Ntoskrnl.exe. (The section "Architecture Overview" in Chapter 2 explains more about this file.) Symbol table files must match the version of the image they were taken from. For example, if you install a Windows Service Pack or hot fix, you must obtain the matching, updated symbol files for at least the kernel image; otherwise, you'll get a checksum error when you try to load them with the kernel debugger.

While it is possible to download and install symbols for various versions of Windows, updated symbols for hot fixes are not always available. The easiest solution to obtain the correct version of symbols for debugging is to use the Microsoft on-demand symbol server by using a special syntax for the symbol path that you specify in the debugger. For example, the following symbol path causes the debugging tools to load required symbols from the Internet symbol server and keep a local copy in the c:/symbols folder:

srv*c:/symbols*http://msdl.microsoft.com/download/symbols

For detailed instructions on how to use the symbol server, see the Debugging Tools help file or the Web page http://www.microsoft.com/whdc/ddk/debugging/symbols.mspx.

Windows Debugging Tools

The Windows Debugging Tools package contains advanced debugging tools used in this book to explore Windows internals. You can find the latest version at http://www.microsoft.com/whdc/ddk/debugging. These tools can be used to debug user-mode processes as well as the kernel. (See the following sidebar.)

Note

The Windows Debugging Tools are updated frequently and released independently of Windows operating system versions, so check often for new versions.

There are two primary variants of the Microsoft debuggers that can be used for kernel debugging: a command-line version (Kd.exe) and a graphical user interface (GUI) version (Windbg.exe). Both provide the same set of commands, so which you choose is a matter of personal preference. You can perform three types of kernel debugging with these tools:

• Open a crash dump file created as a result of a Windows system crash. (See Chapter 14 for more information on crash dumps.)

• Connect to a live, running system and examine the system state (or set breakpoints if you're debugging device driver code). This operation requires two computers—a target and a host. The target is the system being debugged, and the host is the system running the debugger. The target system can be either local (connected to the host via a null modem or IEEE 1394 cable) or remote (connected to the host via a modem). The target system must be booted with the /DEBUG qualifier (either by pressing F8 during the boot process and selecting Debug Mode or by adding a boot selection entry in Boot.ini).

• For Windows XP and Windows Server 2003 systems, connect to the local system and examine the system state. This is called local kernel debugging. To initiate local kernel debugging, select the menu item File, select Kernel Debug, click on the Local tab, and click OK. An example output screen is shown in Figure 1-7. Some kernel debugger commands do not work when used in local kernel debugging mode (such as viewing kernel stacks and creating a memory dump with the .dump command). However, you can use the free LiveKd tool from http://www.sysinternals.com in cases where the native local debugging support does not work. (See the next section.)

   

  Figure 1-7. Local kernel debugging

   

   

  dt nt!_* in the kernel debugger. A sample partial output is shown below:   lkd> dt nt!_* nt!_LIST_ENTRY nt!_LIST_ENTRY nt!_IMAGE_NT_HEADERS nt!_IMAGE_FILE_HEADER nt!_IMAGE_OPTIONAL_HEADER nt!_IMAGE_NT_HEADERS nt!_LARGE_INTEGER You can also use the dt command to search for specific structures by using its wildcard lookup capability. For example, if you were looking for the structure name for an interrupt object, type dt nt!_*interrupt*: lkd> dt nt!_*interrupt* nt!_KINTERRUPT nt!_KINTERRUPT_MODE Then, you can use dt to format a specific structure as shown below: lkd> dt nt!_kinterrupt nt!_KINTERRUPT +0x000 Type : Int2B +0x002 Size : Int2B +0x004 InterruptListEntry : _LIST_ENTRY +0x00c ServiceRoutine : Ptr32 +0x010 ServiceContext : Ptr32 Void +0x014 SpinLock : Uint4B +0x018 TickCount : Uint4B +0x01c ActualLock : Ptr32 Uint4B +0x020 DispatchAddress : Ptr32 +0x024 Vector : Uint4B +0x028 Irql : UChar +0x029 SynchronizeIrql : UChar +0x02a FloatingSave : UChar +0x02b Connected : UChar +0x02c Number : Char +0x02d ShareVector : UChar +0x030 Mode : _KINTERRUPT_MODE +0x034 ServiceCount : Uint4B +0x038 DispatchCount : Uint4B +0x03c DispatchCode : [106] Uint4B Note that dt does not show substructures (structures within structures) by default. To recurse through substructures, use the "-r" switch. For example, using this switch to display the kernel interrupt object shows the format of the _LIST_ENTRY structure stored at the InterruptListEntry field: lkd> dt nt!_kinterrupt -r nt!_KINTERRUPT +0x000 Type : Int2B +0x002 Size : Int2B +0x004 InterruptListEntry : +0x000 Flink : Ptr32 +0x000 Flink : Ptr32 _LIST_ENTRY +0x004 Blink : Ptr32 _LIST_ENTRY +0x004 Blink : Ptr32 +0x000 Flink : Ptr32 _LIST_ENTRY +0x004 Blink : Ptr32 _LIST_ENTRY +0x00c ServiceRoutine : Ptr32

  
  

  The Windows Debugging Tools help file explains how to set up and use the kernel debuggers. Additional details on using the kernel debuggers that are aimed primarily at device driver writers can be found in the Windows DDK documentation. There are also several useful Knowledge Base articles on the kernel debugger. Search for "debugref" in the Windows Knowledge Base (an online database of technical articles) on support.microsoft.com.

  LiveKd Tool

  LiveKd is a free tool from http://www.sysinternals.com that allows you to use the standard Microsoft kernel debuggers just described to examine the running system without requiring a second computer to act as the host (via a null modem cable). While the built-in support for local kernel debugging works only on Windows XP and Windows Server 2003, LiveKd permits local kernel debugging on Windows NT 4.0, Windows 2000, Windows XP, and Windows Server 2003.

  You run LiveKd just as you would Windbg or Kd. LiveKd passes any command-line options you specify to the debugger you select. By default, LiveKd runs the new command-line kernel debugger (Kd). To run the GUI debugger (Windbg), specify the –w switch. To see the help files on the switches for LiveKd, specify the –? switch.

  LiveKd presents a simulated crash dump file to the debugger, so you can perform any operations in LiveKd that are supported on a crash dump. Because LiveKd is relying on physical memory to back the simulated dump, the kernel debugger might run into situations in which data structures are in the middle of being changed by the system and are inconsistent. Each time the debugger is launched, it gets a snapshot of the system state, so if you want to refresh the snapshot, quit the debugger (with the "q" command) and LiveKd will ask you whether you want to start it again. If the debugger gets in a loop in printing output, press Ctrl+C to interrupt the output, quit, and rerun it. If it hangs, press Ctrl+Break, which will terminate the debugger process and ask you whether you want to run the debugger again.

  SoftICE

  Another debugging tool that doesn't require two machines for live kernel debugging is a thirdparty kernel debugger called SoftICE, which you can buy from Compuware NuMega. (See http://www.compuware.com for details.) SoftICE has essentially the same capabilities as the Windows debugging tools, but it also supports stepping between user-mode and kernel-mode code. It also supports the Microsoft kernel extension DLLs, so most of the commands we describe in the book also work in SoftICE. Figure 1-8 shows the SoftICE user interface, which appears in response to the SoftICE activation key (by default, Ctrl+D) as a window on the desktop of the machine on which it's running.

   

  Figure 1-8. The SoftICE interface

   

   

  Platform Software Development Kit (SDK)

  The Platform SDK is part of the MSDN Professional and higher subscription levels, or it can be downloaded for free from msdn.microsoft.com. It contains the documentation, C header files, and libraries necessary to compile and link Windows applications. (Although Microsoft Visual C++ comes with a copy of these header files, the versions contained in the Platform SDK always match the latest version of the Windows operating systems, whereas the version that comes with Visual C++ might be an older version that was current when Visual C++ was released.) From an internals perspective, items of interest in the Platform SDK include the Windows API header files (/Program Files/Microsoft SDK/Include) as well as several utilities (Pfmon.exe, Pstat.exe, Pview.exe, Vadump.exe, and Winobj.exe). Some tools in the Platform SDK also come with the Support Tools and Resource Kits. Finally, a few of these tools are also shipped as example source code in both the Platform SDK and the MSDN Library.

  Device Driver Kit (DDK)

  The Windows DDK is also shipped as part of the MSDN Professional (and higher) subscription levels, but unlike the Platform SDK, it is not available for free download (although you can order the CD-ROM for a minimal cost). The Windows DDK documentation is included in the MSDN Library.

  Although the DDK is aimed at device-driver developers, it is an abundant source of Windowsinternals information. For example, while Chapter 9 describes the I/O system architecture, driver model, and basic device driver data structures, it does not describe the individual kernel support functions in detail. The DDK documentation contains a comprehensive description of all the Windows kernel support functions and mechanisms used by device drivers in both a tutorial and reference form.

  Besides including the documentation, the DDK contains header files (in particular, Ntddk.h and Wdm.h) that define key internal data structures and constants as well as interfaces to many internal system routines. These files are useful when exploring Windows internal data structures with the kernel debugger because although the general layout and content of these structures are shown in this book, detailed field-level descriptions (such as size and data types) are not. A number of these data structures (such as object dispatcher headers, wait blocks, events, mutants, semaphores, and so on) are, however, fully described in the DDK.

  So if you want to dig into the I/O system and driver model beyond what is presented in this book, read the DDK documentation (especially the Kernel-Mode Driver Architecture Design Guide and Reference manuals). Another excellent source is Programming the Microsoft Windows Driver Model, Second Edition (Microsoft Press) by Walt Oney.

  Sysinternals Tools

  Many experiments in this book use freeware tools that you can download from http://www.sysinternals.com. Mark Russinovich, coauthor of this book, wrote most of these tools. The most popular tools include Process Explorer, Filemon, and Regmon. Note that many of these utilities involve the installation and execution of kernel-mode device drivers and thus require administrator privileges.

  •  

   

  
  

   

  
  

Once connected in kernel debugging mode, you can use one of the many debugger extension commands (commands that begin with "!") to display the contents of internal data structures such as threads, processes, I/O request packets, and memory management information. Throughout this book, the relevant kernel debugger commands and output are included as they apply to each topic being discussed. In addition, the dt (display type) command can format over 400 kernel structures because the kernel symbol files for Windows 2000 Service Pack 3, Windows XP, and Windows Server 2003 contain type information that the debugger can use to format structures.