Applies to: BLAZE

Summary

BLAZE supports multi-server clustering, which allows up to 50 servers to operate as a single unified system. This article covers how the cluster architecture works, how configuration data stays in sync across servers, and what to consider when scaling the cluster by adding or removing servers.

Cluster Architecture

BLAZE clustering allows multiple servers to operate as a single unified system, providing redundancy and load distribution for connected cameras and devices.

A BLAZE cluster uses a Master/Standby architecture:

• Master Server — The designated coordinator of the cluster. This server manages failover decisions and maintains system-wide configuration.
   
• Standby Servers — Servers that join the cluster and share device management responsibilities. Each standby server maintains a local copy of the cluster database.

When the current Master becomes unavailable, the system automatically promotes another standby server as the new Master. The candidate selected is the standby server with the oldest registration time in the cluster.

• All servers in the cluster must be on the same Class C subnet (the first three octets of the IP address must match, e.g., 192.168.1.x).
   
• All servers must be able to communicate with each other over the network.
   
• All servers must be running the same version.

Servers must share the same Class C subnet (e.g. 192.168.1.x). The system compares the first three octets of each server's IP address before allowing a new server to join.

How Servers Stay Connected

Each server sends a heartbeat signal every second to all other servers in the cluster. This heartbeat carries the server's current status.

Other servers monitor these heartbeats to determine whether a peer is healthy. If no heartbeat is received from a server for 10 seconds, and the connection is confirmed lost, the cluster treats that server as failed and initiates failover.

The 10-second startup grace period is applied when a server first connects. The timeout clock does not start until the first heartbeat is received, which prevents false failover alerts during initialization. Servers automatically reconnect and resume normal operation once the connection is restored. No manual intervention is required for reconnection.

Server status in the cluster:

Data Synchronization

When a configuration change is made on any server in the cluster (for example, adding a camera or updating a channel setting), that change is automatically propagated to all other cluster members.

Configuration Changes

• Items synchronized: Licenses; channel, and alarm status (e.g., channel add, channel/alarm setting changes).
   
• Items NOT synchronized: Events and audit logs; recorded files (video, audio, metadata, best shot images, etc.).
   
• If a server goes offline, any pending sync changes for that server are discarded. When the server comes back online and reconnects, it will receive a full database sync to catch up.
   
• If a server is running a different version, configuration sync to that server is automatically paused until the version mismatch is resolved.

Failover

Failover is triggered when a server is detected as failed through one of two conditions:

• Network failure — No heartbeat is received for 10 seconds and the connection is confirmed dropped. For a network failure, the cluster initiates failback automatically once the connection is restored. 
   
• Storage failure — The server detects a storage (HDD/SSD) failure and reports it in its heartbeat. Other servers receive this notification and immediately initiate failover for the affected server's cameras, even if the network connection is still active. For a storage failure, failback does not occur automatically — the administrator must manually trigger failback from the Desktop Client after confirming the storage issue has been resolved.

If no server has enough available channel capacity to take over a camera, that camera remains in a Pending state until capacity becomes available.

Failback

When a failed server comes back online:

1. The recovering server signals that it is ready to reclaim its cameras.
    
2. Each server currently hosting the recovered server's cameras releases them.
    
3. The cameras are returned to the original server and resume normal operation.
    
4. Data synchronization to the recovered server restarts automatically.

Storage Failure and Failback

If a server experiences a storage failure, it reports this in its heartbeat and other servers initiate failover for its cameras. Once the storage issue is resolved, the affected server must manually confirm recovery before the cluster initiates failback and returns its cameras. This ensures the server is fully operational before cameras are returned.

Scaling the Cluster

A single cluster supports up to 50 servers, with each server managing up to 256 camera channels, for a potential of up to 12,800 channels across the cluster.

Adding a Server

• The new server must be on the same Class C subnet as the existing cluster and running the same version.
   
• When a server is added, the system automatically creates a database backup, merges licenses, and pushes the full cluster database to the new server. Ensure sufficient disk space is available before starting.
   
• Existing cluster operations are not interrupted during the process. Cameras and recording continue normally while the new server is being initialized.
   

• If the process fails at any step, the operation is cancelled and changes are rolled back. 

  IMPORTANT: If you add a server that was previously in use to the cluster, the 
  system will overwrite the target server's database with the master's database 
  during synchronization. Licenses are merged before this step, but other local 
  settings on the target server (e.g., devices, channels, recording settings) 
  will be replaced by the master's configuration.

Removing a Server

• Before removing a server, verify that cameras assigned to it have been manually reassigned or that remaining servers have enough channel capacity to absorb them.
   
• If a server is removed while still running, it automatically detects that it no longer belongs to the cluster and stop communicating with other servers.

Version Compatibility

All servers in a cluster must run the same version. The major, minor, and patch version numbers must all match.

If a version mismatch is detected, the mismatched server is moved to Isolated status and configuration sync is paused. The server will automatically rejoin normal operation once all servers are on the same version.

It is not recommended to add a server with mixed versions in a cluster. Update all servers to the same version as part of any upgrade procedure.

Known Considerations

• Same Class C subnet is required. The first three octets of each server's IP address must match (e.g. 192.168.1.x). Clustering across different subnets is not supported.
   
• Sync delay under heavy load. Configuration changes are synced one at a time per server. A large number of changes in quick succession may take a moment to propagate to all cluster members.
   
• Isolated server falls out of date. A server in Isolated status will not receive configuration updates. Resolve version mismatches promptly.
   
• No manual Master selection. The Master server is promoted automatically based on registration order. There is no option to manually designate a Master server.