Crossed OWA Sessions

OWA has become one of the most coveted features in any Exchange installation. The ability to safely and securely access email from any web browser is hugely convenient and loved by all. However, the backend behind OWA, while normally straightforward, can present some configuration challenges for large organizations. Seemingly innocuous settings on load balancers, can have a huge impact on the operation of OWA, which as a web service is performing a bit of magic in delivering mail from multiple mailbox servers to users.

One particularly problematic issue is when using a load balancing infrastructure in front of OWA to balance users across multiple CAS servers. Under certain circumstances, load balancers can actually cause OWA sessions to be crossed so that users log in and briefly see another user’s mailbox contents for a finite but definitely tangible amount of time. While the breach may not be long, it is real and in some cases the errant user will even bee able to navigate the mailbox of the other user.

The issue arises in the way many load balancers cache content to improve performance in website loads and will be more prominent with large deployments with thousands of users rather than small deployments where caching of content will be at a lower volume. The load balancer in these cases actually caches chunks and sections of users mailboxes and the links etc, contained therein are logged in on the CAS server itself, so the load balancer, in an attempt to speed up the load time, will deliver another users content to a session without regard to the fact that the content in question needed to be authenticated to be accessed.  Users will often not realize until it is too late, that they are in another users mailbox thanks to the malfunctioning load balancing infrastructure.  Its easy to see how this scenario is problematic.

I haven’t seen this specific scenario addressed clearly and in plain language with most load balancers, and is for that reason that this post exists. Many load balancers will have a list of optimal settings, but its easy to see how these can be overlooked by administrators, who are used to deploying load balancing infrastructure for other static content and don’t realize the potential for these crossed sessions in OWA.

The solution, and this applies to nearly any load balancer out there, is simple. Any settings related to caching content of sessions or optimization needs to be disabled unless the load balancer you are working on has specifically stated that it has integrated settings for OWA.  Ensure that no part of the OWA sessions are being cached on the device, and if you had it enabled and now need to disable, run whatever command is necessary to flush the cache after you are done.

Hope this post can help save someone hours of troubleshooting this issue as it is obscure.

Cheers.

Outlook 2010 Auto Complete Cache Corruption

A while back, I had written a fairly complete article on restoring the Outlook auto complete cache and working with .NK2 files.  Well, just when we thought that we could all return home in a victorious Napoleonic march, our enemy defeated, our pals down in Redmond reminded us all of Waterloo.

Outlook 2010, has completely done away with .NK2 files and has unfortunately replaced it with a system that seems far from bulletproof. My recent experiences, along with thousands of others on the net, suggests that the new system is rather prone to corruption and cannot be easily fixed.

But first, in a bid to dispel some of the misinformation out there, let me briefly go over how the new system works.

Users first using Outlook 2010, or syncing it to an active sync device are immediately going to notice a new set of contacts called suggested contacts that has been auto created upon opening a new mail account.  Inside, they will quickly notice that it is populated with email addresses of people whom they may not even remember having contact with. One soon realizes that this is a contact list that is being generated based on emails that have been sent out and incoming emails that do not have a proper contact in Outlook.  This idea, actually, seems to make a lot of sense as it is an easily backed up way of managing all of those contacts that were previously stored in the Auto complete cache’s .NK2 file.  It would be easy to come to the conclusion that this is the same thing as the .nk2 cache and this is what populates it. This, however, is wrong.  While it does store similar information, there is in fact a new .dat file that has taken the place of the venerable .NK2 file, but now has a GUID attached to it that is,ostensibly,l generated based on characteristics of the profile.  Apparently, this file can be transferred around just as was done with .NK2 files, but you would have to rename it with the exact same corresponding GUID for it to work. Definitely, not as slick as the simple renaming it to the name of the profile as in previous versions of Outlook. This won’t, however, be an issue if you have transitioned to Exchange 2010 as the file is actually stored on the Exchange server and synced back down to the client allowing for your auto complete cache to follow you as part of a new profile on a second computer.  Again, this is a great idea if it weren’t for the apparent instability of the file itself.

I have been able to find very little information on the causes of the issue, and nothing at all from Microsoft, but it appears that this file syncs itself back to the Exchange server upon closing Outlook. However, if for some reason, Outlook does not close cleanly, it appears to cut short the write of this data which subsequently corrupts the file. It is at this point extremely difficult to correct the situation. 

The file is located at %appdata%\Local\Microsoft\Outlook\RoamCache  and will be the file labeled Stream_Autocomplete_<GUID>.dat. As I stated, I have yet to come across anything that immediately fixes the issue in all cases, but here are a few things to try.

First, to verify that you are having the issue in question, you will a.) have an auto-complete cache that isn’t working and b.) a Stream_Autocomplete file that is 0 bytes in size and stays that way despite adding data to the autocomplete cache during a session. With this issue, your cache will work during the duration of Outlook being opened, but then when you close outlook and reopen it, all of the data is lost. 

Anyway, here are some methods that are known to work.

1. Delete the auto-cache data through the provided menu interface provided in Outlook.  To do this,

1. In Outlook 2010, Click the File menu and Select Options.

2. In the Outlook Options window Click the Mail tab.

3. Scroll down roughly halfway until you see Send messages. Uncheck the Use Auto-Complete List to suggest names when typing in the To, Cc, and Bcc lines box.

2. The second method is to manually delete the file mentioned above. Make sure and close outlook before you do this, as it will have a lock on the file and prevent you from doing so otherwise.  This method is really the same as what happens above, but some people have actually reported that the file did not delete as it was supposed to from the GUI.

3. Rename then whole Roam Cache folder and reopen outlook.

4. Disable cached exchange mode and try methods 1,2, or 3 above and repeat until the file starts to grow in size.

And please, if anyone else has any more information on the actual root cause of this, please comment below so we can all get through this.  There are many possible culprits, but I suspect that over time we will find that it is one particular add-on or program that is causing the vast majority of cases.

Cheers!

Working with the Auto-Complete Cache

“Fire!!” “Earthquake!!” “Tornado!!” – all of these desperate screams for help pale in comparison to the shriek heard from end users after getting a new mail profile. As the words “Hey, my contacts are gone!!” come bellowing down the hall, many techies simply feel like packing up, retiring and heading for a more rewarding career – perhaps that of a trash collector or road sweeper. Truly, the world has come to en end if [insert_name_annoying_user_here]  has lost his contacts!

Or, maybe not. While it is true that there are few things that both irk and are more readily noticed than a missing auto-complete cache, the file itself is fairly easily managed. With a little bit of extra work, you will be able to avoid many of the pitfalls that plague IT administrators after a profile switch and you will also be able to help end users clean up or reset their cache completely.

Before we get going, however, it is worth noting that the auto-complete cache is simply that – a cache. As such, it has very little offered by Microsoft in terms of editing it and in Microsoft’s eyes is more of a temporary data repository rather than some sort of proper database. Don’t expect to get stellar support from Microsoft should this become corrupt – the official line is that all addresses should be kept in your address book or outlook contacts. As a matter of principle, I must admit that I too agree with this advice.

Nevertheless, users expect their cache to be manageable so let’s dive in and look at what can be done with this.

The cache itself is implemented in the form of a file that has the .NK2 extension and is named after the name of your outlook profile. By default, this name is likely going to be “outlook.nk2”, but be aware of other names. In Vista and Windows 7, you will find this located at:

“C:\ Users\UserName\AppData\Roaming\Microsoft\Outlook”   

in Windows XP     

“C:\ Documents and Settings\UserName\Application Data\Microsoft\Outlook”      

In order to see this file, you will have to ensure that you have enabled the display of hidden files and folders.  (Learn how:Show Hidden Files)

So, once you have the directory open, it is likely that you will not see an auto-complete cache in that folder. Outlook only creates this after an email is sent and Outlook is closed and re-opened, so go ahead and send a test message to yourself, or whoever you see fit. This will cause an entry to be written to the .NK2 when Outlook is closed and reopened. After doing this, ensure that Outlook remains closed as having Outlook open will put a lock on this file and prevent you from renaming it.    

Now, take not of the name of the new .NK2 file. You will have to find the previous one from the old profile, move it to the new directory and rename it to the exact name of the new .NK2 file you have created. Go ahead and rename the new one to profilename.bak. Rename the old .Nk2 file to the name of the newly created one. Finally, reopen outlook and you should have all of the old auto-complete entries available.

Now, given this information, one of the easiest ways of creating a brand new .NK2 cache should the old one be corrupt, is to simply follow the procedure above and just rename the .NK2 file.

Another useful tidbit, is that individual entries in an an auto-complete cache can be deleted by simply hovering over the entry in the drop down and pressing the delete key while it is highlighted.

In Outlook 2010, Microsoft has also added a red X with which you may delete the entry by clicking on it. You can also get a brand new cache in Outlook 2010 by using the Outlook.exe /CleanAutoCompleteCache switch. Simply cut and paste this into the run box and you will end up with a brand new cache.

If you need more extensive editing capabilities of the .NK2 cache there are several free and relatively cheap editors out there that will allow the import of entries and correction in an easy to use editor.

So don’t worry about finding a pair of overalls just yet for your new career, there really are ways of dealing with the auto-complete cache.  I hope this little tutorial helps!

Cheers!

Outlook 2007 Inbox Renamed

We’ve been a little shy on updates lately folks. It has been a busy time of year, Paul is down in South America having the time of his life, and spring is fast approaching. Still none of this excuses our lack of writing and definitely doesn’t excuse the brevity of this post.

Nonetheless, this is all I have to offer for now.

A few days ago, one of my clients called me up and said that his Outlook 2007 inbox had suddenly been renamed to image001.jpg.  I have seen a lot of strange things, but this I really had to see.

As my log me in session connected and I focussed on the tiny text on my laptop screen, there it was. image001.jpg. This was indeed bizarre. I wasn’t really sure where to begin with this. The issue, in fact, doesn’t affect the end user in any way nor does it impede the normal flow of mail. But still, it is an annoyance. 

Historically, I have had great luck with the various switches that can be used to launch Outlook. So, I started digging in a bit, and sure enough I found a switch /resetfoldernames.

I was a little worried about running some random switch that I knew very little about, but again, past experience has been really good with these Outlook commands and away I went with it.  5 seconds later and the problem is gone.

So, for thos of you that have never used an outlook switch, here is the procedure. Open the run dialog box and enter the following command.

Outlook.exe /resetfoldernames

I always like to start this with outlook closed so that it opens one new instance of the program.

If anyone knows why this happens, I would love to hear about it. Please post your comments below.

Cheers!

Repairing an Exchange 2007 Database

As system administrators, there is nothing quite like the feeling you get when you are staring at a downed exchange server. Honestly, there is very little that would impact end users more than the loss of email. Unwittingly, I have even had a whole DC go down for several hours unnoticed just due to the fact that email and Internet was still working.  Losing exchange is a big deal.

All versions of exchange, despite their overwhelming complexity, have several tools available to help one repair and/or recover an exchange data store. Today’s post will mainly focus on Exchange Server 2007, but again, a lot of these tools are available with 2003 and will be available in the 2010 version.

Before we begin this, I think we are all working under the assumption that there is a recent and intact backup available. If there isn’t, I really must say – all bets are off. Backup! Backup! Backup…not doing so is playing with your career!

More often than not, the first tool you will use when dealing with Exchange database issues is the Database Recovery Management Tool. This tool can be found under the toolbox section of the exchange management console and will offer a lot of insight into why the database has failed. In worst case scenarios, it may even be able to recover the database to a working state.

Some of the functions of this tool are:

• to verify databse and transaction log files 
• to analyze log drive space
• to reset the log generation number
• to repair a database
• to show database-related event logs
• to merge or copy mailbox content
• to set the database can be overwritten by restore option.

One word of caution here though – the repair database option can and often does result in permanent data loss. Only use this option if you do not have a backed up database to restore from and this is your only option.

Another handy tool that can be used to work with your exchange databases is the Database Troubleshooter tool.

This is handy when you are faced with a database that will not mount, or there is some inconsistency in the log files due to them being excessively long or from a drive space issue.  The tool is pretty straightforward and will systematically test the database to identify issues stopping a successful connection and/or mounting.

The third tool that can be used is the venerable Exchange Server Database Utility often known as the ese utility.  The tool is located in the default Exchange install folder in the bin directory and can be used in many ways from the command line to repair or recover and exchange database. Again, another word of caution is needed here. The ese utility can result in data loss, so your first and best bet is to always try restoring from backup first if at all possible. In fact, the repair function in the (the /P switch) ese utility works by going through pages in the database that seem unrecoverable and simply discarding them to bring the database back into a consistent state. If that sounds a little scary too you, I am glad that you are fully comprehending the gravity of that switch. The /R switch will attempt to recover the database by rolling back or updating database files by replaying the transaction log. This, while time consuming, often works well in fixing the information store up.

One thing that should be mentioned at this time also – to use the eseutility, you have to dismount the database first.

Use the following command from the exchange management console to do this.

Dismount –Database –Identity “DBName\First Storage Group\Mailbox Database”  After running any of these commands, it is also advisable to use the /D switch to defragment the database as repair operations will often leave large areas of white space.

Finally, Exchage 2007 has a tool known as the integrity checker (isinteg.exe). This tool is effective in fixing relationships between databse tables such as the “public Folders” and mailbox tables. It runs at the logical schema level and because of this attempts to repair inconsistencies at the application level.   This tool is designed for disaster recovery and is often run after you have recovered a mailbox store using the eseutility or from backup.

Unfortunately, Exchange repair is never a fast or particularly fun task, but knowing and practice using the tools above will help alleviate the pressure of production server failures. Set up your own exchange environment, populate it with some garbage, and then practice breaking and recovering the database as many times as you can stomach. Try using a hex editor to cause corruption in the database and see how these tools work.  Trash it and see what you can do!

Good Luck.

NTBackup for Server 2008 with Exchange

As the exploding popularity of our “Running NtBackup under Windows 7” would suggest, the emergence of our latest and greatest operating systems has also left a few holes that Microsoft has yet to plug.  As our readers have seen from our first article, it is very easy to get NtBackup running under windows 7, but more and more people are also realizing that there is a significant need for this on their Server 2008 and Server 2008 R2 boxes that are running exchange.

Currently, Microsoft does not include any product that adequately backs up exchange server on these platforms. The biggest issue with all of the schemes is the fact that logs are not cleared and these servers end up with a lot of wasted space from these.

Honestly, I don’t know when Microsoft plans on addressing this issue officially, but given the screaming we have heard, I would hope that it is soon.

Anyway, to get NTBackup working for exchange in this scenario is almost identical to our Windows 7 tutorial with a couple of small exceptions.

The process goes like this:

Copying the following binaries from a server 2003 install (C:\Windows\System32\) to a new location such as (C:\Program Files\NTBackup\):

• ntbackup.exe
• ntmsapi.dll
• vssapi.dll

Then, install the removable storage manager in server 2008 which can be found under the features menu in server manager.

Finally, go into an older server and copy esebcli2.dll from your server (C:\Program Files\Exchsrvr\Bin) and put it into your new location which in our case would be (C:\Program Files\NTBackup\):

Then, change the esebcli2 reg key from (HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\BackupRestore\DLLPaths)

C:\Program Files\Micrsoft\Exchange Server\Bin\esebcli2.dll

to

C:\Program Files\NTBackup\esebcli2.dll.

Now go ahead and set up your exchange backup as a scheduled task just as you would have done in the past.

Please remember that this is more of a band-aid solution  and is totally unsupported by Microsoft. This will not work with tape drives, and it is very possible that future Microsoft service packs could break this functionality.  With that being said, it has been very solid in all of our testing thus far.

Cheers!

Managing Mail in Exchange 2007

Publicity isn’t everything. Or at least that’s what one of the greatest innovations ever to come out of the Microsoft camp has taught us.  PowerShell, by far one of the most powerful tools ever put into the computing public’s hands and the recipient of nothing but “stellar” press, seems to be nothing but a mere curiosity to most techs out there.  In fact, along with admitting that I myself rarely get a chance to jump in and use it, I do not know one single server administrator who regularly uses PowerShell in an any useful capacity whatsoever. It is a real shame!

But, this article isn’t on PowerShell itself if you will, rather it is just an introduction to one of the many uses of PowerShell which are wrapped into a tool called the Exchange Management Shell.  What a lot of people don’t realize is that the Exchange Management Shell itself is fully based on PowerShell and any of the methods called from within it, can also be accessed in almost exactly the same fashion from within PowerShell itself.

Obviously, this blog post cannot be exhaustive. But I will provide you with one very common and useful example of this utility and give you the links needed to go out and tailor this to your own unique Exchange environment. Literally, this tool opens up unlimited possibilities to managing mail and can be tailored to do anything you can dream of.

So, I know the one thing I face very often as an Exchange administrator is questions and requests about exporting mail. In the past, we might have used a tool like Exmerge to export these mailboxes – which actually worked well and was more than adequate – but there was always something that it couldn’t quite do that would have made your job easier.

Well, PowerShell and the Exchange Management Shell can now do exactly this.

So, let’s take a real life example. You want to export all of the mailboxes in exchange into some PST files that can be archived and kept for future use.

So, the simplest of commands that could be run would be as follows:

get-mailbox | Export-Mailbox -PSTFolderPath C:\PSTFiles\

 

This, as can be easily inferred from the command itself, simply exports everything as PST into the directory specified.  Now, to do more with this, simply add to the above parameters as necessary, for example you could run:

get-mailbox | Export-Mailbox –ContentKeywords “litigation” -PSTFolderPath C:\PSTFiles\

So now, we have refined the export a bit and it will only export those items that have the Keyword “litigation” somwhere within the body of the mesage.

The list of options and the ability to pipe output from one cmdlet into input in another is staggering. Mastering this tool, will definitely put you in the ranks of the top-paid excchange administrators.

Now, I think I have whetted your curiosity, so here are some links to allow you to start to master this tool.

http://technet.microsoft.com/en-us/library/aa998579.aspx – This is specific to the export command we looked at in our example.

http://technet.microsoft.com/en-us/library/aa998225.aspx – This is a general overview of all of the cmdlets available fpr the Exchange Management tools.

Now, hope you have fun with this. The more you get into it, the more fun it all becomes.

Cheers!!!

Windows Mobile Push Mail

Unfortunately, this has to be a bit of a rant.

Honestly, how would Microsoft ever expect compete with Blackberries, Iphones, or any other Mobile platform if they continue to put simple clean usable functionality behind some crappy unneeded security regime.

Last week, I finally bit the bullet and decided to get push mail running on my HTC Touch. Despite owning the phone for six months, I had never bothered to get it talking to exchange because the mere thought of dealing with all of the clunky menus was enough to get me on the bottle. But, I thought, "It can't be that bad?"

Well, after figuring out that it was actually through active sync that I configure this, and completely unrelated to "email settings", I entered in the pre-requisite information - you know - server name, email address, items to sync ya da ya da .... and, much to my surprise...within seconds actually, it started to sync.

That was easy......right? Well, it really was....until the damn thing stopped working about 12 hours later. Suddenly, active sync is now demanding that I install a personal certificate and refusing to sync. Again, this is not a problem, I can get the cert easily...after all, I work in an IT consultancy...this is a piece of cake.

Here's where the yelling, swearing, and finally, the frantic googling began. 2 certificates later, and a lot of error messages, it still wasn't working. I was at the end of my rope until finally, I find the answer. Hidden deep within the depths of Microsoft's Knowledge Base is an article describing the procedure required to make this work. It seems that all of the well-placed certificates and super strict security is just smoke and mirrors - Microsoft actually instructs users to have IIS and exchange server disregard client certificates to get this working. I won't detail the whole process in this blog as it is well documented in http://support.microsoft.com/kb/927467 but ...WOW .

Anyway, I guess the bigger point is, how many people would have just given up by this point? Should average Windows Mobile Owners have to contact their server admins to have client certificates disabled? Should they even have to know what a certificate is? I really like my Windows Mobile device, but the answer to those questions is obviously NO!.

Microsoft has hereforth been put on probation, let's hope Windows Mobile 7 is able to deliver right out of the box!