MORE INFORMATION
To understand the methods that are described in this
article, you must understand something about the building blocks of an Exchange
Server mailbox database.
Database Structures
An Exchange Server database file is a set of 4-kilobyte (KB)
pages, numbered in sequence. The first two pages of the database are the
database header. The third page is logical page number 1, the fourth page in
the database is logical page number 2, and so on.
Pages are organized
in balanced trees (b+ trees). These trees are collections of pages that are
linked together logically for fast search and retrieval of information.
Trees are organized in tables. A database table is typically made up
of several trees that serve specialized functions in the table. There is a data
tree, and there may also be index and long-value trees, along with other trees
for specialized system functions. (A long value tree is used to hold data that
cannot fit in its entirety on a single data page.)
Tables are organized
in mailboxes. Each folder in a mailbox is a separate table. Along with tables
that make up mailboxes, there are tables shared between mailboxes, such as the
folders, messages, and attachments tables. The Mailbox table lists all the
mailboxes in the database.
There are also several system tables in
the database. The two most important system tables are the catalog (the
MSysObjects table) and a second copy of the catalog (MsysObjectsShadow). These
catalogs are the "table of tables." If they are both lost, the database does
not know what other tables it contains, or where to find the other tables.
How the Eseutil Utility Repairs Database Files
Eseutil.exe is a multifunction utility that is included with
Exchange Server 5.5 and Exchange 2000 Server. You can use the Eseutil utility
to view database information, defragment the database file, and perform several
other functions, which include repair of a damaged database.
In
repair mode, the Eseutil utility fixes individual tables. The Eseutil utility
does not maintain the relationships between tables. For example, if the Msgs
table, which contains all messages for all mailboxes, is damaged, when the
Eseutil utility repairs the Msgs table, the Eseutil utility may need to discard
some unreadable messages. These messages may still be listed as valid in some
users' Inboxes. To resolve this problem, you have to run the Isinteg.exe
utility, which is also included with Exchange Server. This utility maintains
the relationship between the tables, and removes any discarded items from the
user's list of messages in the Inbox.
Before you run a repair
procedure, it is useful to know what the chances are that the repair will
succeed, and where data is likely to be lost. When you run repair there is
always a chance of catastrophic data loss, and you should always have a backup
of a database file before you attempt repair. It is safer to restore data from
an online backup and roll forward through the transaction logs to bring the
database completely up to date than it is to run repair. Use repair when you
cannot follow this preferred strategy for some reason.
To discover
the purpose of a given page in the database, you need to use both the Eseutil
and Isinteg utilities together. For Isinteg to gather the necessary
information, the database must be startable, and the database must be installed
on a functioning Exchange Server computer, because Isinteg actually starts the
database silently to read the database as a collection of mailboxes, not just a
collection of tables.
The methods in this article apply to both
Exchange Server 5.5 and to Exchange 2000 Server. The database and page
structures in both versions are similar; you can use the version of Eseutil
that is included with Exchange 2000 Server in some read-only modes against an
Exchange Server 5.5 database. Microsoft recommends that you run the Exchange
2000 Server version of Eseutil against Exchange Server 5.5 databases in
read-only modes.
WARNING: Do not use a mismatched version of Eseutil in a mode that causes
changes to the database. These modes include, but are not limited to, use of
the
/r,
/p, and
/d command-line switches. The only mode that is definitely safe is
the
/m mode, which dumps various types of information but does not alter
the database.
Be absolutely certain that you know which version of
Eseutil you are running at all times. If you run Eseutil with no switches, the
version information is displayed on screen, and no database operations are
performed.
To run the Exchange 2000 Server version of Eseutil against
an Exchange Server 5.5 database, copy the Eseutil.exe and Ese.dll files from an
Exchange 2000 Server computer to a convenient location together, start Eseutil
from that folder, and specify the path to the database file.
The Scenario
The rest of this article describes a typical scenario in which a
damaged page is discovered in a database and that page is resolved to a
specific folder.
In this scenario, the following event is logged in
the application event log on an Exchange Server 5.5 computer:
Event Type: Error
Event Source: ESE97
Event
Category: Database Page Cache
Event ID: 200
Date: 5/4/2000
Time: 3:39:05 PM
User: N/A
Computer: EX1
Description:
MSExchangeIS (2312) A read of the database file e:\exchsrvr\MDBDATA\PRIV.EDB
between offsets 0x0000000000212000 and 0x0000000000212FFF failed after 16
failed read attempts with error -1018. There is a software or hardware problem
affecting the database drive that must be corrected to preserve database
integrity. Contact Microsoft Product Support Services.
Determining Which Mailbox Owns a Database Page
This section contains the following information:
- How to Determine Which Database Page Is Damaged
- How to Determine Which Tree Owns a Page
- How to Determine Which Table Owns a Tree
- How to Determine Which Mailbox Owns a Table
How to Determine Which Database Page Is Damaged
Use one of the following methods to determine which page in the
database is damaged:
- Calculate the page number from the offsets in the event
error message. In this example, the damaged page is 0x212000 bytes into the
database. In decimal numbers, this number is 2,170,880. Divide that number by
4,096 and subtract 1 to determine the number of the damaged page, which in this
example is 529. -or-
- Use the Esefile utility to perform the calculation for you
by scanning an entire database and reporting every page that has physical
damage.
NOTE: You can use the Calc.exe utility in scientific mode to easily
convert hexadecimal and decimal numbers.
After you determine that
page 529 is damaged, you need to determine which tree the page belongs to.
How to Determine Which Tree Owns a Page
The Exchange 2000 version of the Eseutil utility introduces the
capability to dump the logical information in the header of any page in the
database. This capability works also against Exchange Server 5.5 databases. To
dump a page, run the following command:
eseutil /m [database_name] /pnnnnn > file_name.txt
where
database_name is
the name of the database,
nnnnn is the number of the
page that is damaged, and
file_name is the name that
you want to give the output file.
For example, if the database name is
Priv1.edb, the page number of the damaged page is 529, and you want to save the
output in the plain text file 529.txt, run the following command:
eseutil /m priv1.edb /p529 > 529.txt
The beginning of the resulting page dump is similar
to the following:
Microsoft(R) Exchange Server(TM) Database Utilities
Version 6.0
Copyright (C) Microsoft Corporation 1991-2000. All Rights Reserved.
Initiating FILE DUMP mode...
Database: priv.edb
Page: 529
pgnoThis <0x02660004 , 4>: 529 (0x00000211)
objidFDP <0x02660018 , 4>: 123 (0x0000007b)
ulChecksumParity <0x02660000 , 4>: 881043383 (0x3483a7b7)
** computed checksum: 2653031709 (0x9e220d1d)
dbtimeDirtied <0x02660008 , 8>: 522872 (0x000000000007fa78)
cbFree <0x0266001c , 2>: 4026 (0x0fba)
ibMicFree <0x02660020 , 2>: 1675 (0x068b)
itagMicFree <0x02660022 , 2>: 2 (0x0002)
cbUncommittedFree <0x0266001e , 2>: 0 (0x0000)
pgnoNext <0x02660014 , 4>: 0 (0x00000000)
pgnoPrev <0x02660010 , 4>: 0 (0x00000000)
fFlags <0x02660024 , 4>: 1 (0x00000001)
Parent of leaf
Internal page
Root page
FDP page
Multiple Extent Space (ParentFDP: 1, pgnoOE: 2341)
Primary page
Locate the line in this output that begins with
objidFDP, which in this example has a number of 123. Each tree in the database
is identified uniquely by an objidFDP number. If you dump all of the single
pages in the database and collect all the pages that have an objidFDP of 123,
those pages compose that particular tree in its entirety.
Even though
the page is damaged, the page dump utility reads whatever information is left
intact. If the page is too damaged to obtain a readable header dump, you can
try to dump the surrounding pages. If the pages that precede and follow the
damaged page have objidFDP values that match each other, the damaged page is
also likely to be part of the same tree.
NOTE: If you want to dump all the pages in a database at once, you can
do it with a command similar to the following; note that this process may take
a long time for a large database
for /l %a in (1,1,nnnnn) do eseutil /m priv1.edb /p%a >>allpages.txt
where
nnnnn is the number
of pages in the database. To determine the number of pages, divide the exact
byte size of the database by 4096, and then subtract 2 (for the database
header). For example, if the database is 56,631,296 bytes in size, it has
13,824 pages.
After you determine that the page belongs to tree 123,
you need to determine which table owns tree 123.
How to Determine Which Table Owns a Tree
You can determine which table owns a tree by using the Exchange
2000 version of the Eseutil utility to dump the space usage in the database. A
space dump lists each table in the database, and lists how many pages each
table is using. The Eseutil utility for Exchange Server 5.5 can also dump space
usage, but the Exchange 2000 version adds a column to the output that
identifies the objidFDP value for each tree in each table.
To use the
Exchange 2000 version of the Eseutil utility against either an Exchange Server
5.5 or Exchange 2000 database to dump the space usage, run the following
command:
eseutil /ms priv1.edb >space.txt
The output is similar to the following (irrelevant
columns have been abbreviated for this example):
Microsoft(R) Exchange Server(TM) Database Utilities
Version 6.0
Copyright (C) Microsoft Corporation 1991-2000. All Rights Reserved.
Initiating FILE DUMP mode...
Database: priv.edb
******************* SPACE DUMP ********************
Name Type ObjidFDP PgnoFDP
===================================================
priv.edb Db 1 1
1-1511 Tbl 98 433
MsgFolderIndex7 Idx 100 434
MsgFolderIndexPtagDel Idx 102 436
RuleMsgFolderIndex Idx 101 435
1-172E Tbl 103 457
MsgFolderIndex7 Idx 105 458
MsgFolderIndexPtagDel Idx 107 460
RuleMsgFolderIndex Idx 106 459
1-194E Tbl 108 465
MsgFolderIndex7 Idx 110 466
MsgFolderIndexPtagDel Idx 112 468
RuleMsgFolderIndex Idx 111 467
1-1B6D Tbl 113 489
MsgFolderIndex7 Idx 115 490
MsgFolderIndexPtagDel Idx 117 492
RuleMsgFolderIndex Idx 116 491
1-1D8A Tbl 118 502
MsgFolderIndex7 Idx 120 503
MsgFolderIndexPtagDel Idx 122 505
RuleMsgFolderIndex Idx 121 504
1-1FA8 Tbl 123 529 <- tree 123
MsgFolderIndex7 Idx 125 530
MsgFolderIndexPtagDel Idx 127 532
RuleMsgFolderIndex Idx 126 531
This output indicates that the tree with the
objidFDP of 123 is part of the data tree for table 1-1FA8.
HINT: When you search for the objidFDP number in a space dump, use a
space before and after the number, rather than searching for only the number,
to reduce multiple matches. For this example, search for " 123 " (without the
quotation marks).
Tables in an Exchange Server database are
identified by either a hyphenated number (such as 1-1FA8) or by a name (such as
Msg). Many system tables are identified by name, but not all of them are. For
example, table 1-24 holds all attachments in the database. All user mailbox
folder tables are numbered, not named.
In this example, the page
number, 529, is also contained in the space dump output. However, this does not
mean that you do not need to dump the page to find the tree number. You cannot
go directly to the space dump and look for the page.
The page that is
listed in the space dump is the root page for the tree. Page 529 happens to be
the root page of the tree in this example, but it might not have been, and in
that case page 529 would not be listed in the space dump.
After you
determine that page 529 belongs to tree 123, which belongs to table 1-1FA8, you
need to determine which mailbox owns the table. If page 529 belongs to the Msgs
table, that indicates that there is some kind of damage to at least one message
in the database. However, because the table that is involved is not named, you
need to determine what type of table 1-1FA8 is.
How to Find the Mailbox That Owns a Table
If the page is not in a system or shared table (such as 1-24 or
Msg), the page probably belongs to a particular mailbox. Isinteg.exe allows you
to dump all folders, and identifies the type of the folders. Therefore, you can
usually tell that a table is a Sent Items or Inbox folder. In with Exchange
Server 5.5 Service Pack 3 and later, Isinteg also dumps the mailbox table,
which enables you to correlate a particular folder with the mailbox that owns
it. To dump the folder and mailbox tables, run the following command with the
version of the Isinteg.exe file that is included with Exchange Server 5.5
Service Pack 3 or later:
isinteg -pri -dump -l isdump.txt
If you are running this command on Exchange 2000,
run the following command:
isinteg -s <servername> -dump -l c:\dump1.txt
The Isinteg dump file is likely to be very
large. The file contains three major sections: the Index Age table, the Folder
table, and the Mailbox table. Find the table that you identified from the
Eseutil space dump in the Folder table section of the Isinteg dump. The
following is the Folder table entry in the Isdump.txt file for 1-1FA8:
[141] Folder FID=0001-000000001FA8
Parent FID=0001-000000001FA5
Root FID=0001-000000001FA4
ACL ID=0000-000000000000
Folder Type=1
Msg Count=859
Msgs Unread=859
Msgs Submitted=0
Rcv Count=4
Subfolders=0
Name=Inbox
Comment=
Restriction=
Search FIDs=
Recursive FIDs=
Search Backlinks=
Categ FIDs=
HINT: The easiest way to search for the folder 1-1FA8 is to simply
search for the suffix 1FA8. In the Isinteg output, the folder ID (FID) of the
table is padded with zeroes to make all IDs the same number of digits, which
makes it impossible to search for 1-1FA8.
Notice that the Name value
for the table is Inbox, which indicates that the bad page is a user's Inbox. In
this example, the root FID that is listed for 1-1FA8 is 1-1FA4 (or
0001-000000001FA4). After you determine the root FID, you can search the
Mailbox table.
The Mailbox table is at the bottom of the dump. The
easiest way to find the beginning of the Mailbox table is to search for the
following words:
When you search the Mailbox table, you will see the following:
[10] RootFID=0001-000000001FA4
Owner DN=???
GUID=A5C4D99B DE518E40 82BD51B8 197571A7
Display Name=David Johnson
Comment=
Sentmail FID=542D-6530362B5136
Subtree=0001-000000001FA5
Inbox=0001-000000001FA6
Outbox=0001-000000001FA7
Sentmail=0001-000000001FA8
Finder=0001-000000001FAA
DAF=0001-000000001FAB
Spooler Q=0001-000000001FAC
Size=(ec:ecNotFound-MAPI_E_NOT_FOUND)
Localized=TRUE
Locale=0x0
Error: 1 database retrieve error on current record.
Ignore the error at the end of the mailbox record;
this error is an anomaly of the Mailbox dump output and is always
displayed.
This Mailbox table indicates that damage was done to David
Johnson's Inbox folder.
Contact the user and ask the user to start
Microsoft Outlook. In this example, David's Inbox says he has 968 messages, but
David cannot see a single one of them.
Assessing the Damage
The page dump in the "How to Determine Which Tree Owns a Page"
section of this article lists several "flags" on the page that indicate the
nature of what is stored on the page.
These flags were:
- Parent of leaf
- Internal page
- Root page
- FDP page
- Multiple Extent Space (ParentFDP: 1, pgnoOE:
2341)
- Primary page
There are several other page types or flags that can be set on
a page, but the only two that you need to be concerned about are the "Internal
page" and "Leaf page" flags.
Internal pages are pages that help
define the structure of each tree in the database. Leaf pages hold the actual
data in the tree.
In most cases, even if all of the single internal
pages for a tree are lost, the tree can still be reconstructed with perfect
fidelity by the Eseutil utility repair function, because each leaf page knows
which tree it belongs to, and the relationships between the pages can be
recalculated.
But if a leaf page is lost, some data is always lost,
unless the leaf is from one of the system catalog tables. In that case, you can
retrieve the page from the other copy of the catalog.
In this
example, even though David cannot see any of the messages, the damage is only
to a single, internal page in the tree, therefore it is likely that repair will
succeed completely.
Only undertake repair with the
eseutil /p command on a copy of the database, not the original database. No
matter how positive all indications are, repair always carries the risk of
permanent data loss.
After you run the
eseutil /p
priv.edb /v /x command on this database, all of the user's messages
are visible again. Because the damage was to an internal page and it is very
likely that repair was 100% successful, you do not need to run Isinteg
afterward. If a leaf page in the user's Inbox folder was damaged, you would
need to run Isinteg to restore the integrity of the relationship between the
user's Inbox and the shared data tables in the database. Refer to the Exchange
Server documentation about Isinteg to determine which Isinteg switches and
tests you need. You can also run Isinteg in an "alltests" mode for a
comprehensive check and fix.
General Notes
- Eseutil can perform an integrity check of the database,
which is more or less a "practice run" of repair. The eseutil /g command may stop working before Eseutil reaches the end of the
database if the database is severely damaged or if one of the system catalog
tables is damaged. If one of the MSysObjects tables is not intact, you must run
repair to restore integrity to the catalog before you can check any of the
other tables. In such a case, you cannot predict in advance of running repair
which tables or pages are damaged.
- The last table that the integrity checker or repair
traverses is the TimedEvents table. If you do not see this table at the bottom
of the Eseutil output in verbose mode (eseutil /g priv.edb /v /x), then the integrity check did not finish.
NOTE: In Exchange 2000, the /v (verbose) and /x (extended) switches are implied and are no longer needed.
Redirect the output of the integrity check to a file (eseutil /g >file.txt) so that you can examine the results. You can check the progress
of the integrity check periodically by examining the file in a text
editor. - By far the two largest tables in a typical database are the
messages (Msg) and attachments (1-24) tables. Therefore, these tables are
statistically more likely than others to be damaged because they comprise so
much of the database. If you perform repair when either of these tables is
damaged, you risk losing all the data in these tables.
- You are much more likely to suffer damage to a leaf page
than to an interior page because most pages in the database are leaf
pages.
- If damage has been done to system tables, such as the
catalog, or other tables that do not contain user data, it is risky to leave a
repaired database in service. The risk from this is relatively low, but it
exists. Even if no problems are immediately noticed, you may encounter problems
such as high CPU usage, or sudden stops of the database as these tables are
accessed. When system tables have been damaged, implement a plan to salvage
data and discard the database as soon as feasible. The easiest way to do this
is to move all the mailboxes to another server, delete the damaged database,
and then move the mailboxes back to a fresh database.