What is a Stretch MetroCluster

To understand Stretch MetroCluster you first need to understand how a HA pair operates. A Stretch MetroCluster is basically the next level up from a HA pair, or a HA Mirrored pair to be more specific.

Standard HA Pair

In a HA pair, which is a very common Netapp controller deployment, the cluster can handle the outage of a physical link or the entire controller and still provide access to the underlying storage without impacting data access for end users. Each controller in the HA pair shares the same set of disks or own its own distinct set of disks but either way in the event of a controller failure all reads/writes are sent to the remaining controller that can still access the failed controllers disks. There is a HA interconnect between the controllers that’s used for both keep-alive monitoring and mirroring of NVRAM. The HA pair provides fault tolerance and allows non-disruptive upgrades as a takeover and giveback can be performed for planned migration of read/writes to second controller in the HA pair.

Next up is a Mirrored HA pair. Mirrored HA pairs maintain two copies of all data in the form of plexes. These are continually updated synchronously using SyncMirror and provides protection in the event of disk failures. You can also set the mirroring to be asynchronous if there is a need for that. The major drawback to Mirrored HA pairs is that it does not provide failover to the partner node in the event of a controller failure. The Mirrored HA controller pair need to be within the 5 metre SAS limit.  This is where stretch MetroCluster comes in.

Stretch MetroCluster configurations provide data mirroring and the additional ability to initiate a failover if an entire data center becomes unavailable. They are typically an on-campus solution where there are two data centers on either end of the campus. The Stretch MetroCluster configuration employs SyncMirror to build a system that can continue to serve data even after complete loss of one of the data center.

Like mirrored HA pairs, stretch MetroCluster configurations contain two complete copies of the specified data volumes or file systems that you indicated as being mirrored volumes or file systems in your HA pair. These copies are called plexes and are continually and synchronously updated every time data is written to the disks. Plexes are physically separated from each other across different groupings of disks or array LUNs.

Unlike Mirrored HA pairs, stretch MetroCluster configurations provide the capability to force a failover when an entire node (including the controllers and storage) is destroyed or unavailable. A MetroCluster configuration provides the cf forcetakeover -d command, giving you a single command to initiate a failover. If a disaster occurs at one of the node locations and destroys your data there, your data not only survives on the other node, but can be served by that node while you address the issue or rebuild the configuration.

Requirements for a HA pair:

  • Both nodes must have the same system model and be running the same Data ONTAP software and system firmware versions.
  • The number of disks or array LUNs must not exceed the maximum configuration capacity.
  • FC, SATA, and SAS storage are supported in standard HA pairs.
  • FC disks cannot be mixed on the same loop as SATA or SAS disks.
  • One node can have only one type of storage and the partner node can have a different type, if needed.
  • Multipath HA is required on all HA pairs except for some FAS2040 and FAS22xx system configurations, which use single-path HA and lack the redundant standby connections. N.B – this is not a requirement for MetroCluster
  • Mailbox disks or array LUNs on the root volume
  • Two disks are required if the root volume is on a disk shelf.
  • One array LUN is required if the root volume is on a storage array.

Requirements for Stretch MetroCluster

The requirements for Stretch MetroCluster configurations include those for a standard HA pair and those for a mirrored HA pair. In addition, the following requirements apply:

  • Your storage system must meet all the compatibility requirements for FibreBridge 6500N bridges in the MetroCluster Compatibility Matrix on the NetApp Support Site at support.netapp.com.
  • SAS, SATA, and Fibre Channel storage is supported on stretch MetroCluster configurations, but both plexes of the same aggregate must use the same type of storage.
  • Stretch MetroCluster configuration using SAS disk shelves is supported up to 5 meters. For Stretch MetroCluster having distance greater than five meters require the use of the FibreBridge 6500N bridges. Each stack of SAS disk shelves (up to 10 shelves in a stack) requires two FibreBridges.
  • Stacks of SAS disk shelves can be added to a MetroCluster configuration that has DS14mk2 or DS14mk4 disk shelves.
  • A stack of SAS disk shelves can contain shelves of SAS disk drives and shelves of SATA disk drives, but each SAS disk shelf can only contain SAS or SATA disk drives; you cannot mix SAS and SATA disk drives in the same disk shelf.
  • For stretch MetroCluster configurations using SAS disk shelves, each stack requires two Fibre Channel ports on each controller
  • The following licenses must be enabled on both nodes:
    • cf
    • syncmirror_local
    • cf_remote

Failover requirements & the Mailbox

In order to understand how failover works and is handled it’s important to know what the mailbox is and what it does. When you are running a failover you will see calls to the mailbox and in the case of MetroClusters this is extremely important to ensure that data is consistent and no data is lost during the failover. The mailbox disks sit on the root volume and maintain consistency between the node pairs. The mailbox continually checks whether the other node is running or whether a takeover has been performed. If for whatever reason your mailbox becomes corrupted you’re in deep trouble. I haven’t heard of it happening but I can imagine the impact. The mailbox also stores configuration information that is not specific to any particular node, which allows it to know what’s going on on the other nodes in the cluster.

In order for failover to work, outside of the requirements of the mailbox, the nodes must be attached to the same network and the NICs must be configured correctly. This may seem quite obvious but it’s worth reiterating as the important of correct cabling in a MetroCluster cannot be enforced heavily enough. Licensing is one of the other aspects that needs to be in place. Both nodes will need to have licenses such as SyncMirror, CIFS and NFS. If the takeover node does not have a license that was being used by the partner node to serve data, your HA pair loses functionality after a takeover.


To implement the stretch MetroCluster configuration, you must install an FC-VI adapter in each controller to provide the HA interconnect between the systems. When the FC-VI adapter is installed in the system, the internal InfiniBand interconnect is automatically disabled.

Leave a Reply

Your email address will not be published. Required fields are marked *