Colin’s IT, Security and Working Life blog

July 27, 2012

What your mother never told you about Exchange 2010 Migrations

Filed under: Techy — Tags: — chaplic @ 3:32 pm

Microsoft do a pretty good job of getting knowledge ‘out there’ to us techs; there’s formal documentation, blogs direct from the people that put it together, quick-start documentation – in just about any format you want. Best of all, it’s not hidden away behind a support contract login, It’s one Search away from locating it.

So, if you’re planning an Exchange 2010 migration, you’re probably familiar with the term ‘you had me at ehlo’ and various books with a blue/black cover.

But there’s no substitute for experience, and although no two migrations are ever the same, here’s my top list of my ‘surprises’ from an email migration running into the tens of thousands of mailboxes. You may never encounter them, nor may I again, but maybe, just maybe it’ll save you a 1AM conference call…

1) You really, really need to understand your user profile
Don’t rely on the Microsoft defaults provided with the Calculator. You have, I assume, an Exchange environment already, and  that you can go out there and measure. Once you have these stats, you might find that the idea of hosting 20,000 mailboxes on the old P3 laptop you’ve found in the corner of the office isn’t going to fly. Or more likely, you will find that your initially generous assumptions about deleted item retention and mailbox recovery might need to be trimmed a bit, and logfile disks and required IOPS bumped a little. Or a lot.

2) Firewalls need love, too.

Traditionally, a firewall would be put between the bad guys on the internet, and the internal network, and perhaps some partner organisations. However, in a diverse network arrangement, it’s quite common that there might be a firewall between your internal client machines and your CAS’. Your firewall guys will be wise to the fact that a ‘traditional’ outlook client connection uses MAPI based on RPC, in which we’ll look to use TCP/135 and high ports. So, bang the protocols and destination IP addresses in the firewall, and away we go?!

Not so.

Modern firewalls can determine exactly what is the nature of the RPC traffic and allow/deny access based on the specific nature of the protocol. So they can allow outlook MAPI traffic, but deny the pointing of a compmgmt.msc at your CAS machines. This is done by specifying the UUID of the MAPI communication protocol.

When your client machine initially connects on port 135 there’s a conversation with the server about the desired universally unique identifier your client is looking for. The firewall, being piggy-in-the-middle sees this communication then allows ongoing communication based on it not only ‘liking’ the UUID but also the destination and ports discussed in the connection with the RPC server

Firewalls being things that like order and predictability will then seek to statefully inspect these communications, and make sure everything is just so.

And herein lies some fun.

Your firewall might boast big numbers like “10GBit throughput” but that’s only half the story. Doing such analysis as described above is expensive in terms of firewall resources, and you may find you quickly run out of CPU capacity, and the default-size state table sizes aren’t big enough.  And whilst, we’re here, you might find that one packet in a million isn’t liked by the stateful inspection on the firewall.

3) If you’re migrating from Exchange 2003, you’re really migrating to Exchange 2007 too
I don’t mean you’re doing some kind of painful two step migration. Naturally, a lot of the literature about Exchange 2010 is comparing it to Exchange 2007.

That’s great if that’s your source platform is exchange 2007 but I bet many of you reading this are planning a migration away from Exchange 2003. During your preparations, you should read all the Exchange 2007 upgrade guidance too. Then you might figure things out like:

  • The users will notice a difference when their mailboxes are moved (and I don’t just mean they will get ‘your maibox is full’ messages less frequently. Outlook 2007  (and 2010 of course) are somewhat stifled by Exchange 2003 and doesn’t expose as many features to the users. This includes better free/busy information showing calendar entries (also known as “OMG EVERYONE CAN SEE MY CALENDAR” (you shared it out years ago!) ), modified  out of office and better room booking
  • You’d probably better check before you assure the secretary’s that they can access the boss’ mailboxes no problem no matter who is migrated and who isn’t
  • And – just for fun- let’s stop anyone editing a Distribution List that’s over 5000 entries. The big boss’ PA love this when they want to send out a business critical email (actually, I’m pretty sure that’s not in the 2007 guidance, but I thought Id throw that in there as a freebie for y’all)

4) Storage also needs love
Now, you’re a switched on chap/ lady (you’ve read this far!) so you know that Exchange 2010 is putting to bed the notion that a big, expensive SAN is not necessary and good old DAS is the way forward. That’s great, but it doesn’t always play well in large organisations who have certain ways of doing things and storage teams looking after spinning disk. Plus, with large re-seed times, it can sometimes make sense to avail the services of a SAN.

If your lovely Exchange 2010 databases with their low IO requirements are set to nestle on a SAN, don’t just assume that because you’re using an army of super-expensive disks, all will be well.  These disks connect through a fabric, and a storage controller, which all need to be up to the task of handling at least twice the load (or whatever your DR scenario is)

Sometimes, the more things change, the more they stay the same. Jetstress is still a critical tool in your arsenal whilst testing your change environment. Make sure you plan for it, and use it. It’s possibly a good idea at this moment to have a frank chat about jetstress, and day-to-day Exchange load on a SAN with your storage vendor, because you might find their interpretation of what’s required and what Microsoft produce out of the calculator (which you feed in to JetStress) might differ. Before you have that chat, have a look at the ESRP website, too. This provides paradigms of Storage designs that are certified to work in particular use cases. Chances are it might not fit your environment perfectly, but it provides a goo exemplar of what your design should achieve.

So, a few late nights then?

I’ve been involved with Exchange in one form or other since Exchange 4.0 and it’s probably my favourite Microsoft product. Whilst it is scalable and more robust than ever, the complexity has ratcheted up a few notches too and if nothing else I hope I’ve convinced you that you cannot be resourced, planned and prepared enough when if comes to an Exchange 2010 rollout and migration.

February 6, 2012

Considerations for Scheduling Exchange 2010 Migrations

Filed under: Techy — Tags: — chaplic @ 8:49 pm

So, you’ve designed your shiny new exchange environment, and you’re probably migrating from a creaking Exchange 2003 environment?

In theory, the outage for users takes as long as the size of their mailbox and connection speed. However, most organisations will want to schedule migration in batches probably out-of-hours, to allow for full system backups and operational tests.  It also makes user comms that much simpler.

Obviously it depends on your users work profiles, but if we assume standard office workers and a Monday to Friday working week, how many days to do migration? For the purposes of planning, I’d always include a spare day to allow a rescheduling of a failed migration. So, you could migrate 6 days-a-week, but what about the helpdesk hit when 3 times as many people get their migrated mailbox on a Monday morning?

How do you actually build a schedule for user moves? There are a few choices/ Decisions to be made

 

  • Per Server – start with the first user and first server and keep going till the last user on the last Server. Means you’ll get old servers shut down ASAP. However, the challenge here is how quickly can the old server spit out data? This is likely to be a choke point.

 

  • Per Business Unit – will likely mean a random split of users across all source exchange servers. However.. What if something goes drastically wrong – would you knock an entire department out at once? Depending how good your directory services are, just getting this information could be tricky

 

  • How Many people/ data – Given the speed of your connectivity and source servers (I really hope performance of your new exchange 2010 tin isn’t a concern!) your testing will suggest how much data you can transfer in a single migration window. However, if you could migrate tens of thousands of users in a night as they all had small mailboxes, what’s your appetite for helpdesk calls should even 0.1% of people have problems

Here’s an option that mixes the two approaches.

Firstly, don’t migrate on a per-business-unit approach. This makes scheduling a nightmare, but more to the point introduces the risk of an entire team being knocked out at once. So pick users seemingly at random then communicate with them individually

If you have 5 exchange servers, take one person off each Exchange server and increase your person counter, and also total up their mailbox size. Assuming you don’t reach your limit of number of people to migrate OR maximum data to transfer, then move to the next mailbox server and add the next user.

If you do reach a user or mailbox size limit, then zero your counters and declare a new batch.

Keep adding batches till you have a fixed set, of, say, two weeks s worth of migrations.

This approach should evenly spread the load across all source Exchange servers, and minimises the impact to an individual business area should a problem occur

The logic to actually achieve this is a bit complex. I resolved this issue for a client with a little bit of VBscript that throws all the values needed (mainly from AD)  into an SQL database, then a second bit of VBSCript which produces excel spreadsheets with the user details on them.

Decide how much advance warning you want to give users, 2 weeks from initial communication is probably enough.

14 days before you plan to migrate your first batch, email the users and let them know, inviting then to reschedule if inconvenient.

If you’re increasing users mailbox quotas, be sure to tell them this on the email, because for them it’s probably the biggest feature!

The next day, email your second batch of users, the next day your third batch, and so on.

Obviously, you will need follow-up reminder emails closer to the time. I’d recommend informing users you need at least 48 hours to cancel scheduled migrations.  OK, so we all know that’s not strictly true and important cancellations will still happen at the last minute but hopefully it will reduce the number of last-minute migrations.

Keep the messages clear, short and snappy, but direct users to an intranet site for further explanation and a wiki/ FAQ site.

 

 

 

 

October 28, 2009

Developing Custom IAG Application Optimisers

Filed under: Techy — chaplic @ 8:46 am
 

Microsofts Intelligent Application Gateway (IAG) includes a number of built-in application optimisers to secure and protect the applications you want to publish. It also features a high-performance search-and-replace engine which can filter web pages in-line and, coupled with URL rules, you can build your own complex application optimisations.

It’s easy to build custom filters with IAG.

It’s easy to build custom filters with IAG It’s easy and straightforward to create your own rules to meet your own business needs. This article explains how by delving into an example publishing Outlook Web Access (OWA) using labelling/metadata tags.

The Scenario

Like many organisations, Contoso.com makes use of metadata tags to assist with archiving, retention and security policies. These are implemented by the use of a label in the subject line of all emails.  Exchange transport rules add labels if users do not apply them.

Writing a labeled email

Due to the nature of Contoso.com’s business, they wish to prevent some emails being viewed from ‘untrusted’ workstations, for example, web cafes.

Contoso already uses IAG to provide access to OWA 2007. IAG already includes a number of powerful features that allows you to decide what a ‘trusted’ workstation is, for example, a registry key or anti-virus update level.

With Contoso, all IAG access will be from machines deemed untrusted, and therefore the following business rules need to apply:

  • Emails labelled [PERSONAL], [SOCIAL] or [LOWRISK] can be viewed
  • Emails labelled [FINANCIAL] or with no valid label cannot be viewed
The Process

This example takes you through the design, thought process and implementation of an application optimiser to satisfy the above business rules, and assumes a basic familiarity with IAG. The IAG example Virtual Machines available from the Microsoft website are a suitable candidate for following this example.

Step 1: Understand your application

In order to create the rule to enable this functionality, we need to understand how the application operates. The best way to do this is to browse a number of sessions using a network sniffer like WireShark, Netmon, or Fiddler to understand the HTML and related syntax produced.

Let’s take this example of an OWA page:

A labelled email through OWA

A full capture of the underlying HTML is available here; the edited highlights are below:

<!– Copyright (c) 2006 Microsoft Corporation. All rights reserved. –>
<!– OwaPage = ASP.forms_premium_readmessage_aspx –>
<html dir="ltr">
<head>
<title>[FINANCIAL] Takeover plans</title>
</head>
<body class="rdFrmBody">
<div id=divThm style=display:none _def=8.0.685.24/themes/base/>
</div>
<textarea id="txtBdy" class="w100 txtBdy" style="display:none">
&lt;div dir=&quot;ltr&quot;&gt;&lt;font face=&quot;Tahoma&quot; color=&quot;#000000&quot; size=&quot;2&quot;&gt;Let’s go buy litware inc&lt;/font&gt;&lt;/div&gt; &lt;/body&gt; &lt;/html&gt;
</textarea>
</div>
</body>
</html>

The screenshot earlier shows the HTML produced when a page is requested. What we are looking for is a repeatable, common pattern so we can then write a regular expression to define what is acceptable. Examining the HTML syntax above (and testing against a number of scenarios), we can see a pattern forming that would meet our needs:

  • The Subject text is included in the code preceded by <TITLE>
  • There is a consistent marker so show the last line that we’d want to match on: </html>

We also need to decide what we’re replacing the redacted text with. Ideally this should be as similar to the original page as possible to avoid script errors. For simplicity, in this example, the replacement text is to be:

<body class="frmBody">
<div id=divThm style=display:none _def=8.1.240.5/themes/base/>
</div>
The policy of your organisation does not permit access to this email from this location
</body>
</html>

Step 2: Define your regular expression

To do this, you need to understand how to construct regular expressions (also called regex or regexps), and there’s lots of guides available on the web.

To test your regexp, copy and paste your HTML grab into your favourite text editor that supports regular expressions to search for text.

In this case, we want to achieve the following:

Search for the body content of a webpage, and if it doesn’t have [PERSONAL], [SOCIAL] or [LOWRISK] at the start of the subject line, then redact the body text (replace the text with something else)

Referring back to Step 1, we note that we want to begin and end the search looking for:

<TITLE>something</html>

This would translate into a regexp as:

<TITLE>.*</html>

However, this isn’t good enough; it would match ALL pages and mean they would ALL be redacted.

What we have to do now is define the strings that poison the search expression – in other words, if these terms are in the search string then do not match the string.

This is a bit of a leap, so we need to go back to our business rules:

  • Emails labelled [PERSONAL], [SOCIAL] or [LOWRISK] can be viewed
  • Emails labelled [FINANCIAL] or with no valid label cannot be viewed

As it stands, it’s difficult to write a regexp to cover these two rules. However, they could be re-written to say:

  • Do not display any emails unless they are labelled [PERSONAL], [SOCIAL] or [LOWRISK]

This is semantically the same but much easier to write into a regexp as it is one rule. It also ‘fails safe’ because if a new label is used, the page will not be displayed by default.

We now need to write the ‘unless’ part of our regexp. For this, we’ll use the fantastically titled ‘negative forward lookahead’, combined with a wildcard.

The negative-forward lookahead is represented as a ?! and can be thought of as a ‘NOT’. It would look like this:

(?![PERSONAL]|[SOCIAL]|[LOWRISK]).*

Which reads as:

Zero or more of any characters apart from [PERSONAL] or [SOCIAL] or [LOWRISK] at the start.

So, putting it all together we get:

<title>(?![PERSONAL]|[SOCIAL]|[LOWRISK]).*</html>

Finally, although in IAG we can define what pages this search-and-replace will act upon, customise the search and replace syntax for the exact pages you intend to filter on and, if possible, minimise any mishaps.

If we examine the code, we can see the following:

<!– OwaPage = ASP.forms_premium_readmessage_aspx –>

So with a bit more experimentation, we can come up with:

.*readmessage.asp.*<title>(?![PERSONAL]|[SOCIAL]|[LOWRISK]).*</html>

Step 3: Integrate it with IAG

Now that we’ve built our regular expression, we need to build it into IAG.

On your IAG, load the Editor from the Whale Communications IAG\Additional Tools Menu, then the file

C:\Whale-Com\e-Gap\von\conf\SRATemplates\WhlFiltSecureRemote_HTTP.xml

(or WhlFiltSecureRemote_HTTPS.xml if that’s what your portal uses)

To get an example of what we are going to do, search for

<SEARCH encoding="base64">

The first thing you’ll notice is that the search string is garbled: this is encoded in Base64 as it makes life easier because you do not need to escape control characters.

To decipher the text, click your cursor at the start of the encoded text, hold shift then use the right arrow key to select text until the </SEARCH> tag, then click ‘From 64’, as shown on the screenshot below.

Don’t use the mouse to click and drag, because it often helpfully tries to select the end of the text – and fails!

IAG Editor

This then gives you an idea of the syntax required for our search-and-replace instruction:

<APPLICATION>
    <APPLICATION_TYPE>application name</APPLICATION_TYPE>
       <URL>
         <NAME>URL for the pages required</NAME> 
            <SEARCH mode="regexparam" encoding="base64">search regexp – base64 encoded
            </SEARCH>
            <REPLACE encoding="base64">Replacement text base 64 encoded </REPLACE>
      </URL>
</APPLICATION>

So, our example will be:

<APPLICATION>
    <APPLICATION_TYPE>owa2007</APPLICATION_TYPE>
       <URL>
         <NAME>.*</NAME> 
            <SEARCH mode="regexparam" encoding="base64">.*readmessage.asp.*<title>(?!\[PERSONAL\]|\[SOCIAL\]|\[LOWRISK\]).*</html> </SEARCH>
            <REPLACE encoding="base64"><body class="frmBody"> <div id=divThm style=display:none
_def=8.1.240.5/themes/base/></div> The policy of your organisation does not permit access to this email from this location </body></html> </REPLACE>
      </URL>
</APPLICATION>

Note:

  • The text above isn’t Base64 encoded yet to aid readability
  • Even though the text is going to be Base64 encoded, it is still necessary to escape the open and close square brackets
  • The <NAME>.*</NAME> defines what path/pagename the search string should match on; in this example, it will attempt to match against ALL OWA pages

Select a location for your search-and-replace instruction and nestle it between an existing
</APPLICATION> and <APPLICATION> tag.

IAG Editor

IAG is very, very unforgiving (and unhelpful) if you get any syntax wrong. One way to protect against this is to load the XML file in Internet Explorer and allow active content. This will highlight some syntactical errors.

IE Helping us to spot bugs

If we pass the IE test, then it’s time to activate the configuration. To do this, we need to activate the IAG configuration, and ensure the ‘Apply Changes made to external configuration settings’ is ticked

Activate the config

You may encounter one of the following errors:

  • IAG Admin Console Message Area reporting failure to save config
  • Invalid index when attempting to view the IAG portal

If so, it’s almost certainly to do with invalid syntax: are you sure you’ve Base64 encoded everything properly?

Now you have ironed out all the syntax errors, it’s time to test. Let’s look at a message that the business rules state we should not be able to see:

Our filtering rule being applied

Note the IE Script error message; this is due to our simplified replacement text. Now, let’s double-click on another prohibited email (the email with subject “Label-less”) and examine the results:

IAG Filtering rule in action

Finally, let’s confirm all is OK by attempting to read the email we should be able to access:

Access to an email

Success! Your information security policy can be complied with, and you can chalk up one more victory with the assistance of IAG.

Conclusion

Although the example above takes a number of design, development and test short-cuts, we can see it’s achievable to write your own application optimisers to meet your own business needs. Wordfish Ltd is a IT consultancy specialising in infrastructure design, novel solutions and web development. If you would like some help with your IAG application optimisers, I’d love to hear from you

August 7, 2009

Getting the hands dirty can be fun – and is vital.

Filed under: Techy — chaplic @ 4:29 pm

I can usually be found talking to clients drawing clouds on whiteboards and talking about IS4, Architecture Vision  or business requirements.

I’ve just picked up a task that’s somewhat different. A client has a pretty niffty IT infrastructure, built from the ground up a couple of years ago, and sports gigabit wide area links and robustly managed policies to control the desktop. Fundamentally, from an architectural point of view, it’s fit for purpose.

But it’s not ideal. The clients policy is to have the intranet load when people login, to ensure they always get the latest info. It’s a very dynamic website, so not great offline. Yet it will attempt to load when the laptop isn’t online. Roaming profiles are used well meaning no-one loses data from a faulty hard drive, but the login time to a new machine is arduous, and do we really need to keep crash recovery information in the profile?

All of these issues, and many more, are hardly business critical. But they do impact on all users, every  day, and take away any polish you might feel an infrastructure deserves.

So my task was to look at these and “make good”, without making wholesale changes to the environment.

Many changes were just the case of simple group policy tweaks, re-thinking the need for certain settings (do we have to force everyone to check their spelling when they send email)

Others called for a little bit of VBScripting. In the intranet case, instead of running IE direct on login, we check to see if we can connect to the website, and if so run IE, compose the website invisibly then BANG show it on the page.

OST files are heavily used in outlook. These are great. Not so if you don’t use the same PC every day, and your mailbox is 10GB. So I’m looking at a fix where a script does a WMI query to detect the machine type, and doesn’t use an OST if the machine is a desktop (laptop machines tend to be single user)

I’ve revelled in firing up procmon and wireshark to look at what’s under the surface and really tackling every small issue to hopefully end up with a pleasing result.

Whilst these are not things I typically play with every day, I think its vital that all techies can ‘deep dive’ into such areas when the need arises, and the past few days has reaffirmed this.

Create a free website or blog at WordPress.com.