Exchange 2010 Load Balancing with Netscaler 10.1

I felt like I really needed to write up my experiences on this topic. There is a lot of information out there about how to do this. All of it is pretty good but I could not find a solid, definitive guide on how to handle this from start to finish. Based off of a lot of reading and testing in my lab I came up with the best way (IMO) to do this and I felt like it might be a benefit to someone out there who wants to make this work end to end.

First some Important Notes:

  1. This endeavor is not to be taken lightly. If you have an existing email environment and are changing load balancers or adding a second CAS server to your email environment I highly recommend being able to test this in a lab prior to just jumping in and changing your production email environment around.
  2. For existing setups, this will create a small amount of downtime when you do the change over. It will also require most of the end point clients (outlook user) to have to “authenticate” again depending on how your current environment is setup. This usually means typing their domain\name and password in a prompt. There is a way around this by using a GPO to set the auth method up on the end clients but it’s impossible to make sure all of your end point clients will get the GPO before a user opens outlook. If they have left outlook open they are going to be in a disconnected state and they need to restart outlook.
  3. Schedule this and just plan for some planned downtime for client access to email.
  4. We are going to assume you want to load balance the following services: Outlook Anywhere (OA), Auto Discover, SMTP and Outlook Web Access (OWA). Active Sync can also be done the same way .
  5. We want to preserve the clients source IP because we are not using the Netscaler as our Gateway (for SMTP).
  6. Your results may vary, Exchange systems can be complicated. If you break your email system – insert standard disclaimer. I’m not responsible.
  7. You should have some basic understand of Exchange and how it uses SSL certs. You should also know how to install them on your Netscaler.

Overview

Here is the high level reference architecture for this guide:
Overview

Configure Netscaler Features

In order to make this work properly we are going to take advantage of several netscaler features that need to be enabled.

On your netscaler go to System -> Settings -> Configure Basic Features and make sure (at a minimum) that the following are checked:

  • SSL Offloading
  • HTTP Compression
  • Load Balancing
  • Content Switching
  • Rewrite

You may have other features enabled based on what you are using your netscaler for but for load balancing exchange services that this guide deals with, the above need to be enabled.

Once you have these basic features enabled you need to enable some advanced features on your netscaler. Go to System -> Settings -> Configure advanced features and select the following:

  • Responder

All things SSL

Now we need to import our SSL Certificate to the netscaler. I’m going to assuming you have an existing SSL Cert for exchange (else, what are you load balancing?) you’ll need to export it and import the cert to your netscaler. This SSL Cert needs to be a wildcard or a UCC cert with all the hostnames you are using for your environment.

The easiest way to do this is to open up your exchange powershell console and export the cert.

Note: Replace XXXXX with the thumbprint from the output of Get-ExchangeCertificate and make sure to take note of the password you use for the export in the dialog box that pops up.

Get-ExchangeCertificate
$cert = Export-ExchagneCertificate -Thumbprint XXXXX -BinaryEncoded:$true -Password (Get-Credential).Password
Set-Content -Path "C:\EXCERT.PFX" -Value $cert.FileData -Encoding Byte

EXCert

This will export the cert to the path you used to a PFX file. Now we need to Import the cert to the Netscaler.

On your Netcaler open Traffic Management -> SSL and click on the Import PKCS#12

Fill out the following fields:

Output File Name: Name this whatever you want the cert called on the netscaler
PKCS12 File: Click browse and select the exported PFX file you exported from Exchange.
Import Password: The password you used when you exported the PFX from exchange (in the dialog popup)
Encoding Format: Leave Blank

sslimport

This will upload the pfx and import the cert and key from it. In order to actually install the cert for use with our virtual LB servers you will need to drill into Traffic Management -> SSL -> Certificates and click the Install button.

Certificate-Key Pair Name: Fill out a name something like Exchange_CAS_CERT. Use something descriptive
Certificate File Name: Click “Browse” and select the the imported cert (my example used EX_CAS_CERT)
Key File Name: Click “Browse” and select the same file as the Certificate File Name.
Certificate Format: Select PEM (default)
Password: Enter the password you used when you exported the cert from Exchange.
Certificate Bundle: Leave unchecked (default)
Notify when Expires: Up to you.
Notification Period: If you checked Notify when expires – set how many days.

SSLINSTALL

Setting up Web Based Services – OWA, OA and Auto Discover

There are several web based services we are going to want to setup. Outlook Web Access (OWA), Outlook Anywhere (OA) and Auto Discover services all use HTTPS. Because we have a Netscaler that can content switch we don’t have to have an external and internal IP address for each one. We can create a two content switching virtual servers, look at the request data and then send them to the correct virtual load balancer on the netscaler. This only requires us to use one external and one internal IP address for all things HTTPS for exchange.

In our examples we will be using the following network setup:

External IP: 216.228.161.67
DMZ IP: 192.168.252.92
Internal IP: 172.16.1.169

DNS Setup:
Internal DNS server zone entries for autodiscover.bmctotalcare.com and webmail.bmctotalcare.com to 172.16.1.169
External DNS server zone entries for autodiscover.bmctotalcare.com and webmail.bmctotalcare.com to 216.228.161.67

Firewall Setup:
Firewall forwards HTTPS and HTTP traffic on 216.228.161.67 to the DMZ netscaler IP of the content switching server – 192.168.252.92

Create Servers

In your netscaler go to “Traffic Managment -> Load Balancing -> Servers” and click “Add”.

Enter your CAS server details
createserver

Click “Create” and repeat this process for each CAS server in your exchange farm.

Create a Custom Monitor

In order to monitor when a server is up we want to create a monitor so that the netscaler won’t send traffic to CAS servers that are down or having problems.

On the netscaler to go to “Traffic Managment -> Load Balancing -> Monitors” and click “Add”.

Fill out the name you want for the monitor and select the type as “HTTP-ECV”. The key to the custom monitor is under the Special Parameters tab. Use the following Send String to monitor if the OWA service is responding. If it’s not the load balancer will mark the server as offline and won’t send requests to that server.


GET /owa/auth/logon.aspx HTTP/1.1

Monitor

Create your Service Group

On your netscaler go to “Traffic Managment -> Load Balancing -> Service Groups” and click “Add”.

Name your group and select HTTP as the Protocol. We are going to be using HTTP for the communication from the Netscaler to the CAS servers. The client to the netscaler will still be via HTTPS. This is called SSL Offloading. The Netscaler will handle all the SSL but when it talks to the CAS servers it does not need the extra overhead.

Under the Members tab select Server Based, pick your server from the list, fill out the port field with 80 and click “Add”. Do this for each CAS server.

ServiceGroup

Under the Monitors tab you can select your custom monitor we made in the last step and click “Add”.

NOTE: If you add your monitor to the service group the CAS servers will probably show as down. This is because in most cases Exchange OWA setup out of the box requires HTTPS and our monitor is looking for a response via HTTP, which will fail. For now you can leave the monitor off if you wish, just remember to come back and add it to the service group after we change the CAS servers to serve HTTP requests.

Sgroup_Monitor

Under the Advanced tab make sure Compression is checked.

SGroup_Advanced

Create Load Balancer for OWA

Go to “Traffic Managment -> Load Balancing -> Virtual Servers” and click “Add”.

1. Enter a Name for your load balancer
2. Select SSL as the Protocol.
3. Uncheck the “Directly Addressable” check box. The content switching server will actually handle sending traffic to this load balancer.
4. On the Service Groups tab select the service group you created.
5. Under the Method and Persistence tab select Least Connection for the LB Method, CookieInsert for the persistence and SOURCEIP for the Backup Persistence. Also fill out your time-out in minutes for both persistence settings. This is the timeout for OWA.
6. On the SSL Settings tab select the SSL cert we imported and installed and click “Add”.

OWA_LB

owa_Method

OWA_SSL

Create the other Load Balancers for HTTPS services

These would include Outlook Anywhere (OA), Auto Discover, and ActiveSync. There are other services that use HTTPS as well but we don’t need different LB’s for each one because the content switching server will have a catch all for services like the Address Book, EWS, etc. below I’ve posted the settings I use for the method and persistence on each Load Balancer.

Outlook Anywhere – LB Method: Round Robin, Persistence: SourceIP, Backup persistence: None
Autodiscover – LB Method: Least Connection, Persistence: None, Backup persistence: None
ActiveSync – LB Method: Least Connection, Persistence: SourceIP, Backup persistence: None

Create a Load Balancer for each service you want just like we did for OWA using the Method and Persistence settings above.

Create Content Switching Server

The content switching server is the glue. It’s what all the endpoint clients are going to talk to. It will then read the incoming request and push it to the correct load balancer server we created above.

Go to “Traffic Managment -> Content Switching -> Actions” and click “Add” we are going to create an action for each Load Balancer we created.

Name your Action and point it to one of the Load balancers. Create an action for each Load Balancer you created.

actions

Now we need to create our Policies or rules so we know where to send each type of request. Go to “Traffic Managment -> Content Switching -> Policies” and click “Add” to create a new policy.

Name your Policy and select the Action you created in the previous step. Add in the correct expression based on the service. I’ve listed them below. Obviously, use your domain name and not mine.

OWA: HTTP.REQ.HOSTNAME.CONTAINS(“webmail.bmctotalcare.com”)
OA: HTTP.REQ.HOSTNAME.CONTAINS(“autodiscover.bmctotalcare.com”)&&HTTP.REQ.URL.PATH.TO_LOWER.STARTSWITH(“/rpc”)
Auto Discover: HTTP.REQ.HOSTNAME.CONTAINS(“autodiscover.bmctotalcare.com”)

You’ll notice the last one for Auto Discover is a catch-all. I don not load balance ActiveSync but if you do I think you would look for /Microsoft-Server-ActiveSync in the expression like we do for OA (I’m not 100% sure here).

CS_Policy

Now that we have our actions and our Policies we create the Content Switching Servers. We are going to create 4 in total. 2 for HTTPS (internal and external) and 2 for HTTP (internal and external). The HTTP ones are just to act as a redirect to HTTPS so everthing is using SSL on the client to netscaler end. Go to “Traffic Management -> Content Swithching -> Virtual Servers” and click “Add”.

Internal HTTPS CS Server

Give the content switching server a name. Use the IP address you reserved on your internal network for this. Select SSL as the protocol. Right click on the policy area and insert your policies. The Netscaler will look at the lowest priority number first so make sure you order your auto discover policy last (with the highest priority number).

CS_SERVER1

On the SSL Tab of the CS Server make sure to add your SSL Cert.

CS_SERVER2

External HTTPS CS Server

Do the same thing you just did for the Internal CS Server. Add a new CS Server and use your DMZ IP address your firewall forwards to. The rest of the settings are the same as above.

HTTP CS Servers (Internal & External)

The last step for the content switching servers is to create a responder so that will redirect all requests that come via HTTP to get pushed over to HTTPS. We will again create 2 CS servers, one for external and one for internal clients. You use the same IP addresses but this time set the Protocol to HTTP and leave the CSW policies blank. Click on the “Responder” icon, right click and insert a new policy for the responder.

Responder1

On the Responder policy Name it something like HTTP_TO_HTTPS and click “NEW” on the Action Line, name your Responder Action. Use Type of Redirect and the target express should look like this:

"https://" + HTTP.REQ.HOSTNAME.HTTP_URL_SAFE + HTTP.REQ.URL.PATH_AND_QUERY.HTTP_URL_SAFE

responderAction

Click OK and go back to the Configure Responder Policy Screen. Set the Undefined-Result Action to “RESET” and in the expression field use the following expression:

HTTP.REQ.IS_VALID

ResponderPolicy

Now setup the Internal HTTP CS server the same way using the internal IP address (same as the HTTPS Internal one). You don’t need to create the policy again you can right click and insert the same one you created.

Now any requests that come in to these IP’s via HTTP will get pushed over to HTTPS and all the communications with end clients will be over SSL to the netscaler.

You should now have 4 total Content Switching Servers using 2 total IP’s for all incoming HTTP and HTTPS traffic to Exchange CAS hosted services.

CS_Overview

Setup CAS Servers for SSL Offloading

By default Exchange CAS servers use HTTPS. HTTP is not even setup to respond. We need to fix this as we setup the netscaler to communicate with the Exchange CAS servers over HTTP (in our Service Group Setup). In order to make the changes, log into your CAS servers and open your IIS Admin console.

Open the following sites and set the SSL Settings like so:

  • Default Site
  • EWS
  • Exchange
  • OWA
  • ecp
  • Exchweb

IISConfig

The next step is to tell exchange you are offloading the SSL. Depending on what version of outlook you have there is different ways to do this. On the version I’m currently running (14.3 build 123.4) you can do this from the EMS (Exchange Management Console). Older versions you’ll need to use a registry entry. If you are using a build that supports this change from the EMS open it and navigate to the “Server Configuration -> Client Access” section. Right click on each CAS server and click the “Outlook Anywhere” tab and make sure it looks like this:

SSLOFFLOAD

Make sure NTLM is set and the “Allow secure channel (SSL) offloading” is checked.

If you do not see the SSL offloading checkbox you are on an older version of exchange 2010 and need to set it via a registry key per the image below (on each CAS server):

regedit

Now is a good time to test the setup – you should be able to pull up OWA from both inside and outside. Outlook should also now allow you to connect. If your outlook client prompts you for credentials it’s because the client is set to use “basic” authentication and not NTLM. You can use a GPO to force all of your domain clients to get set to NTLM by default or you can just delete the mail profile and relaunch outlook and autodiscover should work though the Netscaler just fine. The GPO works well for larger organizations but this is going to impact your help desk because even with the GPO some clients won’t get it prior to trying to launch outlook.

If you want to set the GPO you need to download and install the Office 2010 admin ADMX AD templates. The Policy you want to set is User based:

GPO

You can also test the HTTP to HTTPS redirect. Try pulling up OWA using HTTP and it should redirect for you to HTTPS.

Also – if you did not add the monitor to the service group, do so now so that traffic won’t get sent to servers that are offline.

IMAP, POP Setup

IMAP and POP3 mail needs to also be setup if you use these methods for client access. We don’t use a Content switching server as it’s a one to one. One port to the one service, unlike the various HTTPS services. To setup IMAP and POP3 you need to create 4 more load balancers (or just 2 if you only provide IMAP and POP3 service to internal clients). If you are providing external access to IMAP and POP3 you will also need to make sure your firewall is setup to forward those ports to the load balancer. You can use the same IP addresses as you did for the content switching servers but these LB’s will use a different Protocol and port. Use the same Service Group and SSL Cert as before. For Method and Persistence I use Least Connection and SOURCEIP.

Secure IMAP uses SSL_TCP on port 993. POP3 uses SSL_TCP and port 995 (for secure POP3). Non secure POP3 uses port 110 but that seems like a bad thing to do.

Here is an example of my IMAP Load balancer:

IMAP

SMTP

this is tricky to do properly. If you are doing any straight SMTP from clients (servers, printers, etc), there are some very interesting things we need to do to have SMTP flow though the netscaler properly.

First we need to create Load Balancer for Internal Clients. Do not create an external LB for this. This is for internal SMTP clients that need to use SMTP to send mail through exchange only. Outgoing email from exchange should flow out your smart hosts or edge devices. The load balancer can use the same internal IP we have been using. Set the Protocol to “ANY” and the Port to 25. Use the same service group as the other LB’s and set the LB method to Least Connection. You need to go to the Advanced Tab and set the Redirection Mode to “MAC BASED”

MACBASE

Before we go further I highly encourage you to read this Blog post by Stefan Holste. The rest of the steps in my guide are lifted right from his solution but if you want to know WHY we are doing this go read it. It’s well written and also shows how to set this up. You can find it on his blog article titled HowTo load balance while preserving a clients source IP but not using the NetScaler as your Gateway

Per the instruction on his article you need to add a local loopback adapter to your CAS servers. You can do this by opening your device manager right click on the server and select “Add Legacy Hardware”.

DevManager

Select “Install the hardware that I manually select from a list (Advanced), Select “Network Adapter”, Pick Microsoft as the Manufacturer.

loopback

Open up the Network connections panel right click the loopback adapter, uncheck everything except IPv4 and set the IP to the same IP you are using on the netscaler LB and set your netmask. You might also want to rename the adapter.

NICSETTINGS

Being that you are on windows 2008 or 2012 you need to set the some additional settings. Open a Command Prompt and use the following, changing the adapter names to whatever they are called on your system.


netsh interface ipv4 set interface "Your production network adapter name" weakhostreceive=enabled
netsh interface ipv4 set interface "Your loopback network adapter name" weakhostreceive=enabled
netsh interface ipv4 set interface "Your loopback network adapter name" weakhostsend=enabled

Again, for the WHY, go read Stefan’s Blog post HowTo load balance while preserving a clients source IP but not using the NetScaler as your Gateway.

Credits and Resources

As I stated in the onset of this post I was able to put this all together by reading a lot of various articles all over the internet and playing with settings until I got things working properly. Below is a list of the various source material that I used to gain a much greater understanding of how this all works.

NetScaler : Load Balancing Exchange 2010 by Chris Bradford
TMG Replacement for Exchange 2013 with NetScaler by Daniel Kuenzli
HowTo load balance while preserving a clients source IP but not using the NetScaler as your Gateway by Stefan Holste
Citrix NetScaler Deployment Guide for Microsoft Exchange 2010 by Citrix Systems

Special Thanks

Thanks to Randy Berteau, Vishnu Benkert and Eric Covert for helping build and test our environments.

Advertisements

One thought on “Exchange 2010 Load Balancing with Netscaler 10.1

  1. Pingback: Load Balancing Exchange 2010 with NetScaler: No SSL Offload | vEffort

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s