Mailhelp

Directory Search is a standalone Web application that you can give to your users that will allow them to use the Active Directory as a web-based phone book.

Directory Manager is a web-based application that allows a designated user to update other user's information in Active Directory through a easy to use interface. Supported on Windows 2003/2008 x86 or x64. Version 1.3

Directory Update is a "self-service" solution that allows a user to update their own information in the Active Directory. The Web interface is customizable so that a user can provide only valid information (such as city, department, title) and the telephone number format can be validated. Version 1.7. Supported on Windows 2003/2008 x86 or x64.

Use OsaSync to share or synchronize your Outlook contacts, calendar, and tasks with one or more other computers on your network. All changes made to a shared Outlook item are automatically reflected in the item on the other computer(s). Supports fully automatic synchronization--just connect your computer to the network or the internet (if synchronizing via FTP) and start Outlook. OsaSync process all the changes for you. OsaSync is available in PRO (v. 6.3.1) and Lite (v. 8.2) versions and supports Outlook 2000 and later.

MailDetective analyses mail server log files and provides the employer with detailed reports about private and business emails coming to and from the corporate network as well as traffic distribution by users and email addresses. Version 2.2b

Controls and verifies Exchange Mailbox and public folder access rights and permissions. Manager then enables you to audit and update users’ permissions in line with approved security policy and practices, e.g. when an employee changes roles within or leaves the company.

Interrogates the contents of the live Exchange email stores, looking into personal folders, mailboxes and subfolders, local and central PST files, and deleted items to ensure all relevant messages and attachments are discovered.

You may find that recovery operations from online backup fail when the target of a restore is a Recovery Storage Group that exists on an External USB storage device. Most commonly External USB storage is implemented as a temporary measure due to lack of physical disk space on the server. Thus, the storage is added, an RSG is configured and linked, and a restore is attempted. However, there are a number of "gotchas" you need to be made aware of to allow these operations to complete successfully.

The "general inspiration" for this post was taken from a real case I worked within Customer Support Services.

Primary Issue & Symptoms:

As mentioned above, I recently worked with a customer who was attempting to restore an Exchange 2003 Mailbox Database to a Recovery Storage Group that had been configured on an External USB Hard-Drive. This was done because the available disk space on the production server was not sufficient for restoring the complete mailbox database from backup. Once restored, the customer was going to perform any number of additional recovery operations (single item recovery, complete mailbox, complete database, etc.). What we found was that the restore job would initialize successfully but the byte count never increased above 238 bytes. The restore operation would eventually timeout and fail.

A review of the restore log contained the following information:

Job Completion Status
Job Ended: <time stamp>
Completed Status:  Failed
Final Error:  0xa0008488 -- Access is Denied

Troubleshooting:

At first glance the "Access is Denied" error would suggest a permissions related problem. I initially began the call with a review of the Recovery Storage Group's configuration. The configuration clearly showed that the linked mailbox store was being written to the USB drive, which was being provided to the system as the f:\ volume. I performed a compare and contrast of the permissions that had been written to the volume with the other drives and did not notice any specific NTFS permissions anomalies.

I then proceeded to perform the following physical tests:

• Verified that I could traverse the f:\ via Windows Explorer as the user performing the restore.
• Verified that I could manually create files and folders on the f:\ as this user.
• I tested building a new mailbox store on the f:\, built a single mailbox, sent a test message and then verified I could backup the mail store on the f:\.

All of these operations completed successfully. I then proceeded to add the "Test Mailbox Store" to the RSG as a linked database (I actually built "Test Mailbox Store" within the same production storage group that he was trying to restore), and attempted to restore it to the f:\ (e.g. the USB Storage Device). This failed with the identical errors as before. I then reviewed the application log which contained the following Error:

Event Type:  Error
Event Source: ESE Backup
Event ID: 950
Description:  Unexpected file system error 53 encountered while opening the restore environment file.

This event clearly shows that the nature of the failure was during the creation of the Exchange Restore Environment file (restore.env). The System Error Code of 53 equates to "ERROR_BAD_NETPATH" or more commonly referred to as "the network path was not found".

I decided to try testing whether or not my "Test Mailbox Store" could be restored to a physical (non-USB device). I did this by removing the RSG from the USB device, recreating it, and testing the restore. The result was that the restore was successful.

Troubleshooting Conclusions:

1. The fact that the "Test Mailbox Store" could be restored to an RSG that did not exist on USB storage, immediately made the USB drive suspect.

2. The fact that the restore could complete successfully suggested no problems with the backup/restore agent or with the files actually being restored.

So outside of the f:\ simply being a USB device, what made it unique?

Backup and Restore Agents:

When any valid Backup/Restore agent (e.g. one that abides to the rules and methods of the APIs responsible for Exchange aware backup/restore), actually restores an Exchange Database (regardless of whether or not it resides within a Recovery Storage Group or a production directory), the Extensible Storage Engine needs to be able to physical map the path specified as the Temporary Location for the restore. It does this via a UNC path that includes within the path the Admin Share for the target drive.

\\Server Name\<AdminShare of Drive>\Temp Directory Specified For Log Files

Example: \\MyServer\f$\MyRestore

So to verify that the shares were configured properly, I had the customer open a command line and enter: NET SHARE

NET SHARE will list all configured shares on a system. In my case, a review of the output clearly showed that the default Admin Share had not been created for the f:\ yet was present for all other drives. This explained why a restore to a separate physical disk worked in my previous troubleshooting step. The fact that share was not present also provided a logically sound reason for why System Error 53 was being documented in the 950 Event.

Once, I had made this observation, I initially tried to create the share "manually". To do so, I shared the f:\ volume as f$ and manually applied all the permissions via a compare and contrast of a properly initialized Admin Share. Once done, I re-ran NET SHARE. Although the share now showed as present, the share itself did have a few noticeable differences (e.g. was not a Default Admin share), etc. (which in this instance would be expected as the share itself had not been actually created by the SERVER service).

I proceeded to retry a restore, which failed with the identical errors.

Troubleshooting Conclusions 2:

The fact that the Default Admin share for the f:\ was missing suggested the following:

• An Administrator had manually removed the Admin Share (seemed very unlikely).
• The USB drive was not fully initialized when the SERVER service had completed startup (seemed very likely).
• "Manually" attempting to create the share has no net effect on providing a resolution to the issue.

As previously noted, Admin Shares (drive$) are created during startup when the SERVER service fully initializes. My suspicion was that when the SERVER service created the shares the USB device itself had not been fully initialized by the Disk Management System. Thus, no Admin Share would have been configured for the f:\ because it would not have been technically valid at the time of assignment by the SERVER service.

I tested this hypothesis on my own system and witnessed the identical behavior. If you simply attach an external USB drive you will notice that a Default Admin Share is not created because the SERVER service is already started.

Taking this a step further, I restarted the SERVER service on my own workstation then re-ran NET SHARE once the service was back online. The result was that a Default Share had been created for the USB drive on my system.

The next logical step was to restart the SERVER service on the Exchange Server. There are two primary Exchange Services that maintain dependencies on the SERVER service (as well as the Computer Browser service which is also dependent on the SERVER service):

• The Exchange Information Store service (MSExchangeIS)
• The Exchange System Attendant service (MSExchangeSA)

Prior to restarting the SERVER service, I had the customer manually dismount all production stores which was done to force all pending writes into the store. Once done, I manually stopped the Information Store and System Attendant services, then restarted the SERVER service:

From command line (cmd):

net stop srv

net start srv

With the SERVER service back online, I proceeded to bring the Computer Browser, Information Store and System Attendant services back online. Once done, I re-ran NET SHARE to verify the Default Admin share had been created successfully:

NET SHARE

Share name   Resource                        Remark
----------------------------------------------------------
F$           F:\                             Default share

We then rebuilt our Recovery Storage Group, linked the original database back to the RSG on the f:\, placed our Database, Log and TEMP paths onto the f:\ and restored successfully!

Conclusion:

It would appear that when External USB Storage is used, Windows does not automatically create the Admin Shares for the USB drive.  As previously discussed, Admin Shares are created during system startup when the SERVER service fully initializes.  My "professional opinion" (I am after all an Exchange Specialist not a Windows specialist), is that most likely this occurs because the External Storage isn't fully initialized by the Disk Manager System when the SERVER service has completed startup.  This is why a "reboot" of the server wouldn't correct the issue (it would simply recreate the issue).  By restarting the SERVER service with the USB Drive fully initialized, you effectively force the shares to be re-evaluated and recreated.

So what is important to remember here is that if you ever have to perform a similar operation (restoring an Exchange Database to an External Drive), you need to ensure that the Admin Share for the Drive is configured properly prior to attempting the restore (as Exchange needs to directly map to the UNC path specified via the Admin Share). The Admin Share itself is not needed for a backup, but it should go without saying that no production Exchange Database should ever by physically located on a USB drive.  Otherwise, you and I will probably be working on an Exchange Performance related issue in the near to distant future (hello RPC pop-up).

Hope you find this interesting.

Happy Trails.

- Eric Norberg

Share this post :

Have you been playing with Exchange 2010? Do you have the urge to influence the shape of the upcoming Exchange 2010 MCP exams? The Microsoft Learning Certification team is looking for Subject Matter Experts to complete a simple exercise that will allow you to give us your opinion. Providing feedback is quick and painless and helps us to create a relevant exam based, in part, on your real world experience.

If you're interested, please email Sandi Resnick (snares AT microsoft DOT com) to get the details and the worksheet. We're creating two exams for Exchange (TS: Exchange 2010 Configuration and MCITP: Exchange 2010 Design/Deploy). Let Sandi know which exam you want to help out with, or both if you like.

Thanks in advance for your help!

- James Seymour

Share this post :

CodeTwo Exchange Rules 2007 enables easy and central defining of disclaimers added to all e-mails sent via Microsoft Exchange server. Use Exchange Rules to add different disclaimers depending on a message sender's address or domain to e-mails, to add personalized disclaimers containing sender's data available in the Active Directory database to e-mails. New features include the abiliy to insert images into disclaimers and insert the disclaimer directly under the response text in replies and forwarded messages. Free trial. Exchange 2007 only. Version 2.