Geocortex Core and Clustering

The ability to cluster Geocortex Essentials was introduced to provide both fault tolerance and improved application performance. However, in the years since this was first implemented, new developments in virtualization and content delivery technology have presented better ways to provide a fast, robust web application. We no longer recommend using a cluster for new deployments, due to the management overhead and data risks associated with creating a cluster.

Geocortex Essentials 4.6 and later include an installation of Geocortex Core 3.0 or later, which can form a clustered Document Store. Geocortex Core is automatically installed or upgraded when the Essentials Post Installer is run.

If you have multiple installations of Geocortex Essentials that will form a cluster, ensure that each installation has the same instance name, or that the instances are all unnamed. This ensures the shared configuration matches across all Geocortex Essentials installations. You cannot form a cluster when instances use different names.

As of Geocortex Core 3.1.7, the default configuration for accepting connections has been updated to improve security. Specifically, the WatcherHost setting is now set to * instead of _site_ by default. This means that instead of allowing all requests from the local network, only requests from whitelisted IPs are allowed. For more information, see Protecting Geocortex Core from Unwanted Access.

Considerations When Clustering

When Geocortex Core is clustered across multiple servers, the data stored by Geocortex Essentials is distributed across all nodes participating in the cluster. Nodes are in constant communication to manage load across the cluster and to provide data updates to each other. The network connecting cluster nodes must be reliable and high-speed. Network interruptions may cause data integrity issues and unexpected behavior.

Creating a cluster of servers can improve performance if you are reaching the hardware limits of a single server. However, combining servers into a cluster will not reduce site load times or improve workflow performance. If site performance is a problem, we recommend investigating other factors (network speed, disk performance, ArcGIS Server response time, and so on) before creating a Geocortex Essentials cluster.

We recommend that you run Geocortex Core in a three-node cluster to maintain data integrity.

Prerequisites for Core Clustering

The following should be available in preparation for clustering your servers:

Cluster Instances of Core Across Multiple Servers

To cluster your instances of Core across multiple servers:

  1. Install Geocortex Essentials on your first server. We recommend installing to a fresh server that does not have any (Undefined variable: GE.Geocortex) software installed.

  2. Proceed through the installation process to install Geocortex Core on the first server.

    1. On the Configure Cluster Information page in the installation wizard, choose the I want to create a new cluster option.

      If you are upgrading from an older version of Core, a cluster name is already present. Take note of the cluster name. All of the instances of Core that you install will be clustered using this same cluster name.

    2. Enter a shared secret for the cluster.

      Other instances of Core use this shared secret to join the cluster.

    3. Finish the Core installation.

  3. Install Geocortex Essentials on your other servers using the Custom installation option and excluding Essentials Manager (previously known as REST Manager) from the installation.

    During each installation process, the Core installer runs.

    1. On the Configure Cluster Information page in the installation wizard, choose the I want to join an existing cluster option.

      If you need to specify a specific host name or port that the existing cluster runs on, select the Advanced checkbox to manually enter this information.

    2. From the drop-down menu, choose the existing cluster from the other machines running Geocortex Core on your network.

    3. Enter the shared secret for the cluster that you created during Step 2.

    4. Finish the Core installation.

If your Analytics hub and Essentials instance are installed on the same server, they share a single instance of Core and do not require additional Core configuration.

If you have a virtual machine (VM) as part of your cluster, you should always use the Geocortex Index Backup Utility to back up clustered data. While you may be able to take a snapshot of the VM, restoring from a VM snapshot may cause a cluster to lose data. If the first node to start is restored from a VM snapshot, this node will now be considered a master and any newer data written to other servers since the snapshot will be lost.

For information about backing up and restoring a clustered deployment, see Back Up a Clustered Installation.

Update Web.config to Specify the Sites and Temp Folders on the File Share

  1. Create 2 folders on the file share: one for Sites, and one for the Essentials Temp directory.

    • The identity you use to run the EssentialsAppPool4 application pool must be able to write to the Temp folder and read from the Sites folder.

    • The identity you use to run the EssentialsAdministrationAppPool4 application pool must be able to read and write to the Temp folder and the Sites folder.

  2. On the first server, edit the Web.config file for the REST application.

    The default location is:

    C:\Program Files (x86)\Latitude Geographics\Geocortex Essentials\Default\REST Elements\REST\Web.config

  3. Locate the key gcx.io.fileSystem.path and edit the value to point to the UNC path for your file share.

    For example, change the default from:

    <add key="gcx.io.fileSystem.path" value="C:\Program Files (x86)\Latitude Geographics\Geocortex Essentials\Default\REST Elements\Sites" />

    to:

    <add key="gcx.io.fileSystem.path" value="\\myfileshare\Essentials\Sites" />

    substituting the value with the actual share value.

  4. Add a line below the changed key and type the following new app setting for the path to your Temp folder.

    <add key="gcx.io.tempFileManager.commonTempFolder" value="\\myfileshare\Essentials\Temp" />

    substituting the value with the actual share value.

    The Temp folder, like the Sites folder, needs to be available from all nodes in the cluster.

  5. Save the Web.config file.

  6. Repeat the steps to update the key value and the setting to the Temp folder on the second and third servers.

  7. On the node where you installed Essentials Manager, make the same changes to the Web.config file located by default in:

    C:\Program Files (x86)\Latitude Geographics\Geocortex Essentials\Default\REST Elements\Manager\Web.config

  8. Save the updated file.

  9. Copy the contents from your local Sites folder to the Sites folder on the file share.

To validate your changes:

Set the URL for the Load Balancer using the Post Installation Configuration

You only need to complete the following steps on one server. Since the security store is shared across the cluster, the setting for one server applies to all of the servers in the cluster.

  1. Run the Post Installation Configuration on the node where you installed REST Manager.

  2. Click Configure Geocortex Essentials REST API in the navigation frame.

  3. Click the Hosts button.

  4. Add the URL that points to your load balancer.

  5. Click OK, then click Finish.

Additional Considerations in a Clustered-Server Implementation

This section includes the following topics:

 

Monitor the Health of a Cluster

The main component of a Geocortex Core cluster is the Document Store role.

To monitor the Document Store:

Geocortex Core will respond with a JSON object that represents the health of the cluster:

The request must be made from a trusted location, either locally from the server or from a whitelisted network address.

Clustered-Server Upgrade Considerations

When upgrading Essentials, complete the following to maintain the availability of your sites:

  1. Set the load balancer to point to a single node, but not the node on which Essentials Manager is installed.

  2. Upgrade Essentials on the other two nodes.

  3. Set the load balance to point to the 2 upgraded nodes, excluding the node not upgraded.

  4. Run Essentials Manager to update the sites and viewer to the new version (if applicable).

  5. Upgrade the remaining node to the new version of Essentials.

Remove a Node from a Geocortex Essentials Cluster

The Geocortex Core Document Store role manages all data for your cluster, and regularly checks that all of the nodes in the cluster are present and running.

If any node is unavailable from the cluster for five minutes or more:

When removing nodes from a cluster, remove them one by one to allow data to persist via this copy process.

To remove a node from a cluster:

  1. Stop the Geocortex Core service on the node you want to remove.

  2. Wait at least five minutes to ensure data integrity.

Restart the Geocortex Core Service

A cluster has the ability to perform a rolling restart, where servers are taken offline temporarily to install OS updates or perform similar maintenance. Use the following procedure to perform a restart of the Geocortex Core cluster.

To restart the Geocortex Core service:

  1. On your server, open the Windows Task Manager.

    You can open the Task Manager by pressing CTRL+ALT+DELETE and selecting Task Manager from the list of options that display.

  2. Select the Services tab.

  3. Locate the service named Geocortex Core and stop it:

    Right-click the Geocortex Core service and select Stop from the context menu.

  4. Restart the Geocortex Core service:

    Right-click the service and select Start from the context menu.

  5. Connect to another node in the cluster and monitor the cluster health until it returns a "green" status.

  6. Repeat these steps on all servers in the cluster.

Data in the Document Store

The following temporary data is shared across installations using the Document Store:

The following persistent data is shared across installations using the Document Store:

Loss of Connectivity in a Two-Server Cluster (Split-Brain Scenario)

If you have two Essentials instances clustered across two servers, loss of connectivity between the servers results in a 'split-brain' scenario, in which each server elects itself as master of the cluster and can write conflicting information to the database.

Solving this issue requires reconnecting and restarting the confused server, and optionally restoring your database from your last valid backup. A split-brain scenario always results in data loss, but restoring from backup is only necessary if the data loss included persistent data.

To reduce the risk of a split-brain scenario, we strongly recommend adding a third clustered server to your deployment. Alternatively, you can avoid a split-brain scenario by deploying your applications to a single server.

To set up a third clustered server, we recommend that you install Geocortex Essentials on a third server, following the instructions provided above. However, you do not need to maintain that third instance of Geocortex Essentials to have a third clustered server. Once you have run the Post-Installation Configuration and installed Geocortex Core, you can uninstall Geocortex Essentials from the system if you wish.

Geocortex Core Backwards Compatibility with Essentials 4.4 and 4.5

If you are running Essentials 4.6 or later, but need to maintain an instance of Essentials 4.4.x or 4.5.x, you can use the Geocortex Essentials Legacy Version Compatibility Tool to make Geocortex Essentials 4.4.x or 4.5.x compatible with Core 3.0. Without this tool, you cannot maintain your legacy instances of Essentials on the same server as, or as part of the same cluster of, Essentials versions that require Core 3.0.

You can find the tool on the VertiGIS Studio Support in the Essentials Downloads section.

The Geocortex Essentials Legacy Version Compatibility Tool does not work with versions of Geocortex Essentials older than 4.4.

To maintain a legacy version of Essentials while running an instance of Core 3.0:

  1. Download the Geocortex Essentials Legacy Version Compatibility Tool from the Geocortex Support Center.

    You can download the compatibility tool from the Essentials Downloads area of the Geocortex Support Center.

  2. Run the Geocortex Essentials Legacy Version Compatibility Tool executable.

    A list of your Essentials instances displays.

  3. Use the Apply Patch button next to each Essentials instance you want to patch.

  4. Confirm that you want to apply the patch by selecting Yes on the Apply Patch dialog box.

    The Essentials instance is patched.

  5. Exit the tool.