This article applies to versions 6.7.0, 6.7.1 and 7.0 of BoKS Manager.

Description

When a Server Agent needs to contact a Replica, a load balancing algorithm is used to distribute Server Agent connections between Replicas in the BoKS domain.

The load balancing in BoKS 6.7 works as follows: The Server Agent sends UDP probe messages to Replicas in the BoKS domain as a UDP broadcast and/or as UDP unicasts to servers listed in the bcastaddr file. Replicas with a request queue longer than BRIDGE_QUEUE_LOW will delay the reply to the probe message and if the request queue is longer than BRIDGE_QUEUE_HIGH on a Replica it will not reply at all. The Server Agent will select the Replica that responds first. The connection will then be kept open until an idle timeout occurs.

The above algorithm has several problems:

I. The algorithm assumes that connections from Server Agents are relatively short lived otherwise it is not possible to adapt to varying load within a reasonable time. Connections are normally only closed when they have been idle for more than 30 seconds so the assumption about short lived connections is not true in a BoKS domain with busy Server Agents (connections that have been open for a week or more have been observed at customer sites).

II. Before the first threshold level is reached the actual Replica load only indirectly affects the selection process by assuming that a higher loaded Replica will respond more slowy to the probe message. Effects like network latency etc. will also affect the selection process but are beyond the scope of the algorithm. If connections are unevenly distributed when load is low this can turn into a problem when load increases as connections can stay open for a very long time.


Resolution / Workaround

To resolve these problems, install HFBM-0191 (BoKS Manager 6.7) or HFBM-0192 (BoKS Manager 7.0), available for download from the HelpSystems Community Portal.

This hotfix adds a time limit to Server Agents' connections to the Replicas to make it possible to react faster to varying Replica load. The default idle timeout is also reduced from 30 seconds to 10 seconds.

Replicas receiving the probe message will make an estimate of how long an arriving request needs to wait in queue before being processed based on the average queue time for the last 10 messages multiplied by the current queue length. The estimated queue time is then included in the probe message reply. The Server Agent sending out probe messages will wait for multiple probe replies to be received and then make a weighted random selection with weights inversely proportional to the estimated queue time received from each Replica. This way the load balancing algorithm will be in effect even at low load levels.

To take full advantage of the hotfix it should be installed on Master/Replicas and on Server Agents. Server Agents that don't have the hotfix installed will use the old load balancing algorithm.

New configuration variables
---------------------------

BRIDGE_SERVC_R_MAX_TIME and BRIDGE_SERVC_S_MAX_TIME
These parameters are used to force the servc-receive-bridge on Replicas and servc-send-bridge on Server Agents respectively to close down connections after a maximum connection time. The reason for having a time limit at both ends of the connection is that closing down from the Sever Agent side is preferred but requires all Server Agents to have this hotfix installed. The time limit on the Replica side is intended to handle Server Agents that don't have the hotfix installed. The Server Agent side time limit should be set lower than the Replica side limit so that the Sever Agent limit is triggered first if the hotfix is installed.
Default values:
BRIDGE_SERVC_R_MAX_TIME - 600 seconds.
BRIDGE_SERVC_S_MAX_TIME - 300 seconds.


BRIDGE_SERVC_S_MAX_RESP and BRIDGE_SERVC_S_MIN_WAIT
These variables control how long the Server Agent will wait for replies to the UDP probe messages sent out to Replicas.


BRIDGE_SERVC_S_MAX_RESP
If broadcasting is not disabled, see bcastaddr(4B), by default the waiting is not limited by number of responses received, but BRIDGE_SERVC_S_MAX_RESP can be used to set an explicit limit. If broadcasting is disabled the response limit used is either the number of servers in primary server section of the bcastaddr file or value of BRIDGE_SERVC_S_MAX_RESP, whichever is lowest. Setting BRIDGE_SERVC_S_MAX_RESP to <= 0 disables this variable.
Default is disabled.


BRIDGE_SERVC_S_MIN_WAIT
By default the time limit for waiting is set to double the time it tok for the first response to arrive, but if BRIDGE_SERVC_S_MIN_WAIT is set, the minimum wait time will always be at least the time specified by BRIDGE_SERVC_S_MIN_WAIT. Setting BRIDGE_SERVC_S_MIN_WAIT <= 0 disables this variable.

Value is set in msec and default is 100 msec.


BRIDGE_SERVC_S_BCASTSTAT_INTERVAL
Statistics about the servc-send-bridge Replica selection is logged to $BOKS_var/monitoring/bridge_servc_s_bcast.stat
Default log interval is 3600 seconds. A zero value disables logging.


Changed configuration variables
-------------------------------

BRIDGE_IDLE_TIMEOUT_
The default value for this parameter is changed from 30 seconds to 10 seconds.


BRIDGE_QUEUE_DELAY, BRIDGE_QUEUE_HIGH and BRIDGE_QUEUE_LOW
These variables only affect load balancing operation of Server Agents that don't have this hotfix installed and are thus using the old load balancing algorithm.


Changed CLI command
-------------------
The showmaster command has been updated with new options to list multiple Replicas when invoked with the -s option, see showmaster(1B).


Multihomed Replicas and bcastaddr file
--------------------------------------
A Server Agent probing for Replicas will identify replying Replicas based on reply message source IP address. If a multihomed Replica generates multiple replies with different source addresses the Replica's weight will be multiplied by the number of replies with different source addresses. To get a correct weight for each Replica when doing the weighted random selection it is important that only one IP address of multihomed Replicas is entered in the bcastaddr file and broadcasting should be disabled.


Still have questions? We can help. Submit a case to Technical Support.

Last Modified On: May 25, 2018