Windows Search Index: The Forensic Artifact You’ve Been Searching For

April 26, 2023 16 mins

Windows Search Index: The Forensic Artifact You’ve Been Searching For

The Windows Search index is a useful artifact for Incident Response investigations or for Digital Forensics cases. Stroz Friedberg has released an open-source CLI tool named SIDR that can be used to analyze the Windows Search index at scale.

Summary

Windows Search Indexer is a service which enables faster searching of files, emails, and other content on Windows systems. The service builds an index that the system refers to whenever a search is run.
Using the Windows Search index, investigators obtain important data about indexed files and user activity, including:
- File metadata
- Limited file contents
- User interaction with files
- URLs accessed
This is a useful artifact for Incident Response investigations or for Digital Forensics cases involving intellectual property theft and departed employees.
Microsoft changed the structure of the Search index in Windows 11, dropping the former ESE database structure and implementing SQLite in its place. Stroz Friedberg’s research into both the old and new structures has revealed how the information in the Windows 10 Search index is mapped in the Windows 11 Search index.
Stroz Friedberg has released an open-source CLI tool named SIDR that can be used to analyze the Windows Search index at scale—especially when working with a large database or with a large number of systems.

Introduction

Windows Search Indexer is a service that records information about files and data types in select directories and enables users to search for these files using the Start Menu and Windows Explorer. Like many features of Windows, Search Indexer was created to enhance the user experience. For Digital Forensics and Incident Response (“DFIR”) practitioners, this service generates valuable information that can be useful during an investigation.

This blog post will cover how data is stored in the Search index in Windows 10 and prior versions, and how it has changed in Windows 11. It will also cover how this data can be useful from a DFIR perspective, and how Stroz Friedberg’s tool, Search Index Database Reporter (“SIDR”), can help gather insights from the Search index for the purpose of DFIR investigations.

Structure of the Windows Search Index

In all versions of Windows except Windows Server, Search Indexer recursively indexes every file and folder present within the following directories by default:

C:\Users\*
- The AppData directory is excluded from indexing.
C:\ProgramData\Microsoft\Windows\Start Menu\Programs\*

Users can change the default configuration and choose which locations are indexed. Search Indexer also indexes URLs accessed using Internet Explorer and Edge, as well as user activity related to some programs, such as WordPad, Notepad, and Excel.

The following sections will describe the structure of the Search index in detail.

Windows 10 and Earlier (Windows.edb)

Beginning from Windows Vista until Windows 10, Windows stores the Search index inside an Extensible Storage Engine (ESE) database located at C:\ProgramData\Microsoft\Search\Data\Applications\Windows\Windows.edb. For Windows Server 2008 until Windows Server 2022, Stroz Friedberg observed that the database was structured the same way, but that Search Indexer was not enabled by default. The service is enabled by default on non-Server Windows versions.

Windows.edb contains several tables, three of which provide the most value to investigators:

SystemIndex_Gthr

This table contains metadata of every file and folder indexed by Search Indexer. Stroz Friedberg identified the following columns as those typically most useful to an investigator:

ScopeID	An integer that can be used to determine the record’s parent folder. This ID is also referenced in the Scope column in the SystemIndex_GthrPth table.
DocumentID	An integer assigned to every file and folder. Assignment of this ID occurs sequentially as files are created. This ID is also referenced in the WorkID column in the SystemIndex_PropertyStore table.
SDID	A Security Descriptor ID that contains information about file ownership and access control.
LastModified	Last modified timestamp of the record, stored in Windows File Time format.
FileName	Name of the file or folder.

Table 1: Columns of interest in SystemIndex_Gthr table. Additional columns are available.

SystemIndex_GthrPth

This table contains parent folders of the files indexed in the SystemIndex_Gthr table:

Scope	An integer assigned to every folder. This can be correlated with ScopeID from SystemIndex_Gthr.
Parent	The Scope of the record’s parent folder. This can be correlated with ScopeID from SystemIndex_Gthr.
Name	The name of the folder.

Table 2: Columns in SystemIndex_GthrPth table.

SystemIndex_PropertyStore

This table contains additional attributes about the indexed files and folders, including the following columns of interest:

WorkID	An integer assigned to the record. Maps to DocumentID in SystemIndex_Gthr table.
System_Search_GatherTime	The time at which the record was indexed in the database, stored in Windows File Time format.
System_Size	The size of the file in bytes.
System_ModifiedTime	The $FN last modified time of the record, stored in Windows File Time format.
System_CreatedTime	The $FN creation time of the record, stored in Windows File Time format.
System_FileOwner	User who created the file, stored as username.
System_ItemPathDisplay	Full path of the record.
System_ItemType	File type of the record based on the extension of the file. If a file does not have an extension, the value will be a single period (".").
System_FileAttributes	Windows file attributes.
System_Search_AutoSummary	Partial contents of the file. Stroz Friedberg was unable to determine a consistent rule for how many bytes were recorded in this property in Windows 10. See further sections for more information on AutoSummary.

Table 3: Columns of interest in SystemIndex_PropertyStore table. Additional columns are also available.

The screenshot below illustrates a sample from the Windows 10 SystemIndex_PropertyStore table when viewed with ESEDatabaseView. The highlighted record shows an example of a text file where partial content was indexed by the service.

Representation of indexed records in SystemIndex_PropertyStore table in Windows 10

Changes in Windows 11

In Windows 11, this data is stored in the same directory, but the single ESE database is replaced by SQLite database files called Windows.db and Windows-gather.db, discussed in further detail below. Because Windows-usn.db, a third database associated with the Search index on Windows 11, has less forensic value, it is not covered in this post.

The SystemIndex_Gthr and SystemIndex_GthrPth tables from the Windows 10 ESE database were placed in Windows-gather.db, and the content of SystemIndex_PropertyStore was placed into a table named SystemIndex_1_PropertyStore in Windows.db. The graphic below illustrates this change.

Changes in the Windows 10 and Windows 11 Search database format

Despite having a similar name, the new SystemIndex_1_PropertyStore table in Windows 11 is structured differently from its Windows 10 counterpart. Rather than having multiple columns for each file property, the property values are stored as individual rows. The properties are stored as Property IDs, which are mapped to their names in the SystemIndex_1_PropertyStore_Metadata table.

The following graphic illustrates the relationship between SystemIndex_1_PropertyStore_Metadata and SystemIndex_1_PropertyStore in Windows.db.

Illustration of file properties in the SystemIndex_1_PropertyStore and SystemIndex_1_PropertyStore_Metadata tables in Windows 11.

According to this graphic, 13 is the Property ID for System_IsFolder. Therefore, we conclude that the record with WorkId set to 1 is a folder.

To perform meaningful analysis of these properties, investigators will need to design queries with table joins.

Forensic Value of the Windows Search Index

Stroz Friedberg's testing has generated valuable insights on the information indexed in the SystemIndex_PropertyStore in Windows 10 and SystemIndex_1_PropertyStore in Windows 11 tables. The sections below outline the value this table can provide to DFIR practitioners. The data was structured differently in Windows 10 and Windows 11; however, Stroz Friedberg observed no relevant differences in the content of the data that was indexed.

File and Folder Existence, Metadata, and Contents

Stroz Friedberg observed that file creation in any of the indexed directories triggers Windows to create a record for that file in the SystemIndex_PropertyStore and SystemIndex_1_PropertyStore tables. File deletion triggers a reindexing, which is reflected as a mark-for-deletion in the database. The discussion of recovering deleted records continues in the "Deleted Records" section below.

Records indexed in the SystemIndex_PropertyStore and SystemIndex_1_PropertyStore tables will contain metadata such as modified, accessed, and created timestamps, the full file path, and the file owner. This is an important source of information to review for threat actor activity, as C:\Users\* is a common location for threat actors to drop malware and stage files. Malicious startup tasks used by threat actors as a persistence mechanism can also be found in C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Start Up\ and therefore indexed in the SystemIndex_PropertyStore and SystemIndex_1_PropertyStore tables as well.

The time at which the file was indexed is stored in the System_Search_GatherTime property. If the indexed creation time varies from the indexed gather time of the target file, this suggests potential time stamp manipulation.

AutoSummary – Obtain Partial File Contents

Possibly one of the most valuable features of the Windows Search Indexer service is AutoSummary. AutoSummary enables the user to search for a file using the contents of the file. For our purposes, AutoSummary allows the investigator to obtain limited contents of files with select file extensions. Stroz Friedberg's testing has shown that, in most cases, the file extension must match the file type for Auto Summary to record the contents of the file. In other words, for AutoSummary to record the contents of the file, it must be able to correctly parse the text of the recorded file. Therefore, if the file extension is .pdf but the file is a plaintext file, then AutoSummary will parse it incorrectly.

AutoSummary can still parse plaintext files even in the case of a file extension mismatch if the file extension is also that of a plaintext file. For example, AutoSummary will be able to parse the contents of a .txt file renamed to have a .bat extension.

The file contents indexed by the Search Indexer service can be found in the System_Search_AutoSummary field in the SystemIndex_PropertyStore and SystemIndex_1_PropertyStore tables. Stroz Friedberg's testing found that, in Windows 11, partial file contents (up to first 1024 bytes) may be found within this property.

Stroz Friedberg has confirmed that, by default, Search Indexer will record contents from the following file types in the AutoSummary property:

Active Server Page (.asp)
Batch files (.bat)
Command files (.cmd)
Email files (.eml)
Excel files (.xlsx)
HTML files (.html)
Configuration files (.ini)
JavaScript (.js)
OneNote files (.one)
PDF files (.pdf)
Registry files (.reg)
SQLite files (.sql)
Text files (.txt)
Visual Basic files (.vbs)
Word files (.docx)
XML files(.xml)
Zip files (.zip) – Windows 10 only

Deleted Records

In Windows 10, investigators can carve the slack space of Windows.edb to try and recover indexed records of deleted files. Tools like WinSearchDBAnalyzer leverage the Windows Search platform API to recover deleted records. Without these tools, carving and keyword searching may be difficult due to the database encoding and frequent restructuring.

In Windows 11, any changes made to the Windows.db and Windows-gather.db databases will be first written to the write-ahead logs, Windows.db-wal and Windows-gather.db-wal, which are temporary records of recent changes also known as WAL files. When a user deletes a file, the file record will be available in the main database until the changes in the WAL are written to the database upon reboot or checkpoint. This applies to all changes made by the Search Indexer service, including file creation and file renaming. This gives investigators a window of opportunity where the original database may still contain records of items that have been deleted and highlights the importance of getting a forensic image of the current state of a system. Investigators should keep in mind, however, that data found in slack space may not always be intact and may exist without additional context.

IE/Edge Browsing Activity

Stroz Friedberg found that Search index also records all URLs that a user accesses via Internet Explorer and Edge, except those viewed using Private Browsing Mode. In Windows 10 and earlier, visited URLs are stored in the SystemIndex_PropertyStore table in one or more of the following three fields. Stroz Friedberg found that several factors affected how the URL was stored, including whether the URL was valid, and whether the user was connected to the Internet at time of access.

ItemPathDisplay	For valid URLs accessed with an active Internet connection.
Activity_ContentUri	For invalid URLS accessed with or without an active Internet connection.
Activity_Description	For invalid URLS accessed with or without an active Internet connection.

Table 4: Types of URL access stored in SystemIndex_PropertyStore table in Windows 10.

In Windows 11, visited URLs are stored in the System_Link_TargetURL property of the SystemIndex_1_PropertyStore table. In Windows 10 and Windows 11 some invalid URLs may not be indexed.

If the user deletes their browser history, the visited URLs may still be present in the Windows.db file in Windows 11 and can potentially be carved from the Windows.edb file in Windows 10.

User Activity Logging

Another useful feature built into the Search Indexer service is called ActivityHistory. This feature tracks file opening on a per-user basis, which can provide investigators with evidence of file knowledge and user account attribution for suspicious activity such as a user opening files using select programs.

ActivityHistory records will contain the string "ActivityHistoryItem" in the System_ItemType field.

When a user opens any file in an indexed location, several attributes are recorded in the ActivityHistory for the session, including:

System_ItemPathDisplay	Full path of the opened file, which will include the SID of the user who interacted with the file.
ActivityHistory_StartTime and ActivityHistory_EndTime	Start and end times of the activity, which correspond to time of file open/close.
ActivityHistory_AppId	Program used to open the file.

Table 5: Attributes of interest for ActivityHistory records in PropertyStore tables.

Stroz Friedberg's testing has shown that, when a file is deleted or renamed, Search Indexer does not delete any ActivityHistory records associated with that file. Therefore, investigators can use ActivityHistory to attribute file knowledge even when a user deletes or renames a file. Stroz Friedberg is still investigating the size limit of the database and the retention period for ActivityHistory of renamed or deleted files.

Stroz Friedberg observed tracking of the following activity by the Search Indexer service. Further testing may reveal additional tracked activity:

Opening text files (.txt, .bat, .xml, and .js) with the following programs:
- Notepad
- WordPad
- Internet Explorer browser (except .bat files)
- Edge browser (except .bat files)
- Notepad++
Opening PDF files with the following programs:
- Edge browser
- Firefox browser
- Chrome browser
- Adobe Acrobat Reader
Opening Excel files (.xls and .xlsx) files with Microsoft Excel
Opening Document (.doc and .docx) files with Microsoft Word
Opening PowerPoint (.ppt and .pptx) files with Microsoft PowerPoint
Opening image files (.bmp, .png, and .jpg) with Microsoft Photos

Stroz Friedberg observed that activity from PowerShell and 7zip was not tracked. Stroz Friedberg also observed that Search Indexer will record file-specific metadata for other file types, such as subject and recipient in email message files (.eml and .msg).

Reviewing the Content of the Search Index Database

Stroz Friedberg reviewed the Search index on Windows 10 22H2 and Windows 11 22H2 after performing actions on the system such as creating, renaming, and deleting files. The following tools were used to view the Search index:

These tools provide a graphical user interface for the index that can be useful for investigators when looking at a handful of systems, but they do not scale well for investigations that involve large enterprise networks with hundreds or thousands of systems. Additionally, the size of the index can get very large depending on the indexing configuration. In this case, the tools above may struggle to process the entire index. Stroz Friedberg recognizes these challenges and has created a CLI tool called SIDR that can process the Search index in both Windows 10 and Windows 11 at scale. The tool provides the following reports which summarize the most valuable information available in the Search Index.

Windows Search Index File Report

The File Report will contain a list of all the files present in the indexed locations along with metadata such as the full file path, MAC timestamps in UTC, and the file owner.

Windows Search Index Internet Explorer and Edge History Report

The Internet Explorer and Edge History Report will contain URL browsing activity.

Windows Search Index ActivityHistory Report

The ActivityHistory report will contain user account-attributed file access activity.

Conclusion

The Windows Search index is a lesser-known artifact that, if analyzed properly, can supplement analysis for investigations that involve fragmented evidence and defense evasion tactics. By providing user attribution and records of files created on the system, the index serves as an important source of evidence for not only threat actor activity in incident response investigations, but also suspicious user activity for insider threat and intellectual property cases. Stroz Friedberg is proud to release this research and scalable CLI tool to help the DFIR community respond faster and more accurately in future investigations.

Building — Support

Contact our cyber response teams for urgent breach assistance.

Contact Us

Authors

Phalgun Kulkarni

DFIR
Julia Paluch

Solutions Development

Contributors

Partha Alwar

Director, Stroz Friedberg Digital Forensics and Incident Response

Cyber security services are offered by Stroz Friedberg Inc., its subsidiaries and affiliates. Stroz Friedberg is part of Aon’s Cyber Solutions which offers holistic cyber risk management, unsurpassed investigative skills, and proprietary technologies to help clients uncover and quantify cyber risks, protect critical assets, and recover from cyber incidents.

While care has been taken in the preparation of this material and some of the information contained within it has been obtained from sources that Stroz Friedberg believes to be reliable (including third-party sources), Stroz Friedberg does not warrant, represent, or guarantee the accuracy, adequacy, completeness or fitness for any purpose of the article and accepts no liability for any loss incurred in any way whatsoever by any person or organization who may rely upon it. It is for informational purposes only. You should consult with your own professional advisors or IT specialists before implementing any recommendation or following the guidance provided herein. Further, we endeavor to provide accurate and timely information, there can be no guarantee that such information is accurate as of the date it is received or that it will continue to be accurate in the future. Further, this article has been compiled using information available to us up to 04/26/2023.

The contents herein may not be reproduced, reused, reprinted or redistributed without the expressed written consent of Aon, unless otherwise authorized by Aon. To use information contained herein, please write to our team.