Update (CU2): With the release of Lync 2013 CU2 (July 2013) the upgrade process has changed from what is documented here. The new process in CU2 does not include the requirement to break the database mirror anymore. You do need to still ensure that the database servers are running on the Primary side before applying the Install-CsDatabase commands though. Please refer to the updated KB article here. There is a nice flow chart detailing the update process for front end servers here too.
There are now additional steps involved when running cumulative updates on Lync 2013 pools that have mirrored SQL databases. So I thought I would go into some detail about this new process and show how my Database Manager Tool can be used as part of this procedure. This post specifically covers the process documented for Lync 2013 CU1 (Feb 2013), however, it's likely the process will be similar for future cumulative updates. So this post should continue to function as a reference in the future, but be sure to always check the process documented for the specific CU you are installing before doing an upgrade.
The high level steps for this process are:
- Download and install the updates all of the Lync Servers.
- Run the Update Installer on all Lync Servers.
- Check that all of the databases are running on the Primary SQL server.
- Uninstall (remove) the database mirror for the Application database type.
- Install the schema updates on the Primary database.
- Recreate the database mirror for the Application Database type. This will recreate the Application databases back on the mirror server.
- Check the status of the database to see that they are all running correctly.
- (Optional) If your Monitoring backend databases are on a different server, update them now.
- Update the Central Management Database schema. (only if the database is on Lync 2013. If the CMD is on a Lync 2010 pool then you don’t do this)
- Enable the Mobility Service.
- Enable the UCWA service.
Step 1: Download Update
The first step in upgrading Lync is downloading the CU1 Update installer from the Microsoft site: http://www.microsoft.com/en-us/download/details.aspx?id=36820
Step 2: Run the update on all Lync Servers
Run the CU1 Update Installer executable on all of your Lync Servers (one at a time, as the tool will take the services offline when updating them). The Updater will show you that all of your Lync services need updating:
This will take the services offline and upgrade them. After they are upgraded the Update Installer will show them as being the latest version with a green tick:
If you run your Lync Servers in this state (ie. Without updating the database schema) you might notice some strange errors being logged in the Event Viewer, for example:
The above error states that “There was a problem communicating with the Group Pickup backend database” (Event ID 31055). This is because the CU1 release has added the new Group Pickup feature, which requires new schema additions in the Lync SQL databases for storing the data for this feature.
Step 3: Check the current state of the SQL databases
When you’re not using mirrored databases this process is relatively simple, and can be achieved running one or two commands. However, when you’re using database mirroring there is a requirement to remove the mirror for specific databases that need to have the schema updated, and then re-create the mirror databases after the update has been applied to the primary database.
To remove the database mirror you will first need to ensure that all of your databases are running on the Primary SQL server. Using the database Lync 2013 Mirror Manager Tool you can do this as shown:
Note: The Lync 2013 Mirror Manager can be downloaded from here.
If you have databases that are showing up in the Secondary column of the tool, then it means that they are running on the secondary SQL server and need to be moved back onto the primary SQL server (if you're running a witness server these database may have changed over to the secondary server without your knowledge). Move the databases back onto the primary simply by ticking all the check boxes in the Primary column and then click the Invokebutton. You will then be prompted in the Powershell window about changing the databases back to the Primary server. Type “y” as a response to these questions. After this is complete the tool will update and the databases should all appear in the Primary column.
Note: In the above example I haven’t checked the status of the Persistent Chat Databases. In the real world you should actually do this at this point. However, for the purposes of this example I haven’t done this to show what effect it has later when running the Database Install commands.
Step 4: Remove the Database Mirror for the 'Application' database
We now need to remove the mirrored database configuration for the 'Application' database type.
Note: In the future the need to remove the database mirror for other database types may become a requirement as well. So if you are coming to this post for a newer CU update, please refer to the specifics of the KB article relating to your specific CU release.
Run the following command:
Here is an example of the command running:
On the Primary SQL Server:
As can be seen above, the cpsdyn, rgsconfig, and rgsdyn databases (which make up the Application database type) have now had their mirror configuration removed (ie. There is no “Principal, Synchronized” state written next to them now).
An important part of the command that we just ran was the “-DropExistingDatabasesOnMirror” switch, which tells Lync to delete these databases from the mirror server. Unfortunately, this function doesn't seem to always work, and databases can be left on the mirror server. So now we need to check the SQL instance on the Mirror server to see if these were deleted.
On the Mirror SQL Server:
From the screenshot above we can see that on the secondary server, the rgsconfig database has been automatically deleted. The cpsdyn and rgsdyn databases, however, are still there but stuck in “Restoring…” state. These databases will get re-created later when the Mirror is reconfigured between the two SQL servers, so it’s safe to manually delete these databases:
Step 5: Install the new schema updates on the Primary Database
Run the following command to install the Schema updates:
NOTE: The error below can be caused by not having enough free disk space on the SQL servers. This is most likely to happen in a lab environment where you’ve only provisioned the minimum disk space for each machine. I had this happen with 15.3GB of disk space free, however, when I expanded the disk to have 29GB free this error went away.
The execution of this command will look like this:
As you can see in the above output, I had two failures in the upgrade process. These failures were on the ‘mgc’ and the ‘mgccomp’ database, which both happen to be Persistent Chat databases. The reason for this was that I didn’t check if they were running on the Primary SQL instance back in Step 3 (I did this for the example to show what happens when you don’t do this. Really you should have done this check back in Step 3). So now when I check the status of the Persistent Chat databases with the Database Mirror Manager I notice that they are both on the Secondary SQL instance:
To change these back over I just clicked on the Primary check boxes for both of these databases and then click the Invokebutton. Within the Powershell window you will be asked if you want to change the PersistentChat and PersistentChatCompliance databases over to Primary, which you answer “y” to:
Now the Persistent Chat databases are running back on the Primary SQL. So run the Install-CsDatabase command again to ensure that the Persistent Chat databases are also updated:
Now our Application database schema has been updated and is ready to run all the new features of CU1.
Step 6: Re-establish the Database Mirror
Run the following command:
Command output example:
Note: If you get an RPC error at this point like the example I've provided below, it’s likely caused by ports being blocked on the Windows Firewall of the SQL servers.
From the testing that I have done, I’ve found that the Ports that need to be open to avoid these errors are TCP 135, TCP 445 and TCP 49154-49155. This is in addition to the SQL Mirroring signalling ports that also need to be opened: TCP 5022 on the Primary and Secondary servers, and TCP 7022 on the Witness server (also the regular SQL ports UDP 1434, TCP 1433, and the Dynamic TCP port used when Named SQL database is used).
Step 7: Confirm that the databases are all back up and running
Check the state of the database mirror:
Note: You could also do this check with the Mirror Manager Tool. However, just in case there is some edge case where the mirror has failed to re-establish (with an error I haven’t seen before), it’s probably best to use the command in this circumstance.
From the above output you will see that rgsconfig, rgsdyn, and cpsdyn are back in Principal state on the Primary server.
Step 8 (Optional):If you have your Archiving and Monitoring databases on a different SQL backend than the main Lync databases are stored you need to run the install on it now:
Step 9: After we have finished updating all the Lync databases we need to now update the CMS database with the following command:
The output of the command will look something like this:
Step 10: To enable the Mobility service, run the following cmdlet:
Step 11: To enable the Unified Communications Web API (UCWA), you must run the Bootstrapper.exe tool again on all Lync Server 2013 servers on which the Web Components were installed and updated. The command to run the tool is as follows:
The command executes like this:
And we’re done!
The Wrap Up
It’s safe to say that updating a Lync server with a Mirrored Database is a bit more involved than doing a single database server. It’s one of those processes that if taken lightly might cause you some grief in the real world. Hopefully this post will help to make your life a little easier.