Change Journals
An automatic backup application is one example of a program that must check for changes to the state of a volume to perform its task. The brute force method of checking for changes in directories or files is to scan the entire volume. However, this is often not an acceptable approach because of the decrease in system performance it would cause. Another method is for the application to register a directory notification (by calling the FindFirstChangeNotification() or ReadDirectoryChangesW() functions) for the directories to be backed up. This is more efficient than the first method, however, it requires that an application be running at all times. Also, if a large number of directories and files must be backed up, the amount of processing and memory overhead for such an application might also cause the operating system's performance to decrease.
To avoid these disadvantages, the NTFS file system maintains a change journal. When any change is made to a file or directory in a volume, the change journal for that volume is updated with a description of the change and the name of the file or directory. Change journals are also needed to recover file system indexing, for example after a computer or volume failure. The ability to recover indexing means the file system can avoid the time-consuming process of reindexing the entire volume in such cases.
Change Journal Records
As files, directories, and other NTFS file system objects are added, deleted, and modified, the NTFS file system enters change journal records in streams, one for each volume on the computer. Each record indicates the type of change and the object changed. The offset from the beginning of the stream for a particular record is called the update sequence number (USN) for the particular record. New records are appended to the end of the stream. The NTFS file system may delete old records in order to conserve space. If needed records have been deleted, the indexing service recovers by re-indexing the volume, as it does when no change journal exists.
The change journal logs only the fact of a change to a file and the reason for the change (for example, write operations, truncation, lengthening, deletion, and so on). It does not record enough information to allow reversing the change. In addition, multiple changes to the same file may result in only one reason flag being added to the current record. If the same kind of change occurs more than once, the NTFS file system does not write a new record for the changes after the first. For example, several write operations with no intervening close and reopen operations result in only one change record with the reason flag USN_REASON_DATA_OVERWRITE set. To illustrate how the change journal works, suppose a user accesses a file in the following order:
In this case, the NTFS file system takes the following actions in the change journal (where | indicates a bitwise OR operation).
|
Event |
NTFS file system action |
|
Initial write operation |
The NTFS file system writes a new USN record with the USN_REASON_DATA_OVERWRITE reason flag set. |
|
Setting of file time stamp |
The NTFS file system writes a new USN record with the flag setting USN_REASON_DATA_OVERWRITE | USN_REASON_BASIC_INFO_CHANGE. |
|
Second write operation |
The NTFS file system does not write a new USN record. Because USN_REASON_DATA_OVERWRITE is already set for the existing record, no changes are made to the record. |
|
File truncation |
The NTFS file system writes a new USN record with the flag setting USN_REASON_DATA_OVERWRITE | USN_REASON_BASIC_INFO_CHANGE | USN_REASON_DATA_TRUNCATION. |
|
Third write operation |
The NTFS file system does not write a new USN record. Because USN_REASON_DATA_OVERWRITE is already set for the existing record, no changes are made to the record. |
|
Close operation |
If the user making changes is the only user of the file, the NTFS file system writes a new USN record with the following flag setting: USN_REASON_DATA_OVERWRITE | USN_REASON_BASIC_INFO_CHANGE | USN_REASON_DATA_TRUNCATION | USN_REASON_CLOSE. |
The change journal accumulates a series of records between the first opening and last closing of a file. Each record has a new reason flag set, indicating that a new kind of change has occurred. The sequence of records gives a partial history of the file. The final record, created when the file is closed, adds the USN_REASON_CLOSE flag. This record represents a summary of changes to the file, but unlike the prior records, gives no indication of the order of the changes. The next user to access and change the file generates a new USN record with a single reason flag.
Using the Change Journal Identifier
The NTFS file system associates an unsigned 64-bit identifier with each change journal. The journal is stamped with this identifier when it is created. The file system stamps the journal with a new identifier where the existing USN records either are or may be unusable. For example, the NTFS file system re-stamps a change journal with a new identifier when a volume is moved from Windows 2000 to Windows XP and then back to Windows 2000. Such a move can happen in a dual-boot environment or when working with removable media.
To obtain the identifier of the current change journal on a specified volume, use the FSCTL_QUERY_USN_JOURNAL control code. To perform this and all other change journal operations, you must have system administrator privileges. That is, you must be a member of the Administrators group. When an administrator deletes and recreates the change journal, for example when the current USN value approaches the maximum possible USN value, the USN values begin again from zero. When the NTFS file system stamps a journal with a new identifier rather than recreating the journal, it does not reset the USN to zero but continues from the current USN. In either case, all existing USNs are less than any future USNs. When you need information on a specific set of records, use the FSCTL_QUERY_USN_JOURNAL control code to obtain the change journal identifier. Then use the FSCTL_READ_USN_JOURNAL control code to read the journal records of interest. The NTFS file system only returns records that are valid for the journal specified by the identifier. Your application needs both the records' USNs and the identifier to read the journal. This requirement provides an integrity check for cases where your application should ignore the existing records in the file and where records were written in previous instances of the journal for the same volume. To obtain the records in which you are interested, you must start at the oldest record (that is, with the lowest USN) and scan forward until you locate the first record of interest.
Creating, Modifying, and Deleting a Change Journal
Administrators can create, delete, and recreate change journals at will. An administrator should delete a journal when the current USN value approaches the maximum possible USN value, as indicated by the MaxUsn member of the USN_JOURNAL_DATA structure. An administrator might also delete and recreate a change journal to reclaim disk space. To perform this and all other non-programmatic change journal operations, you must have system administrator privileges. That is, you must be a member of the Administrators group. To create or modify a change journal on a specified volume programmatically, use the FSCTL_CREATE_USN_JOURNAL control code. When you create a new change journal or modify an existing one, the NTFS file system sets information for that change journal from information in the CREATE_USN_JOURNAL_DATA structure, which FSCTL_CREATE_USN_JOURNAL takes as input. CREATE_USN_JOURNAL_DATA has the members MaximumSize and AllocationDelta. MaximumSize is the target maximum size for the change journal in bytes. The change journal can grow larger than this value, but at NTFS file system checkpoints the NTFS file system examines the journal and trims it when its size exceeds the value of MaximumSize plus the value of AllocationDelta. (At NTFS file system checkpoints, the operating system writes records to the NTFS file system log file that allow the NTFS file system to determine what processing is required to recover from a failure.)
AllocationDelta is the number of bytes added to the end and removed from the beginning of the change journal each time memory is allocated or deallocated. In other words, allocation and deallocation take place in units of this size. An integer multiple of a cluster size is a reasonable value for this member. If an administrator modifies an existing change journal to have a larger MaximumSize value, for example if a volume is being re-indexed too often, the change journal simply receives new entries until it exceeds the new maximum size. To delete a change journal, use the FSCTL_DELETE_USN_JOURNAL control code. When you use this operation, it walks through all of the files on the volume and resets the USN for each file to zero. The operation then deletes the existing change journal. This operation persists across system restarts until it completes. Any attempt to read, create, or modify the change journal during this process fails with the error code ERROR_JOURNAL_DELETE_IN_PROGRESS. You can also use the FSCTL_DELETE_USN_JOURNAL control code to determine if a deletion started by some other process is in progress. For example, your application, when it is started, can determine if a deletion is in progress. Because journal deletions persist across system restarts, services and applications started at system restart should check for an ongoing deletion. Change journals are not necessarily created at startup. To create a change journal, an administrator may do so explicitly or start another service that requires a change journal.
Obtaining a Volume Handle for Change Journal Operations
To obtain a handle to a volume for use with change journal operations, call the CreateFile() function with the lpFileName parameter set to a string of the following form: \\.\X:
Note that X is the letter that identifies the drive on which the NTFS volume appears. If the volume does not have a drive letter, use the syntax described in Naming a Volume section.
Change Journal Operations
The following list identifies the control codes that work with the NTFS file system change journal.
The following list identifies the structures information that relates to the NTFS file system change journal.