In a cluster, a private interconnect is used by cluster nodes to monitor each node’s status and communicate with each other. When two or more nodes fail to ping or connect to each other via this private interconnect, the cluster gets partitioned into two or more smaller sub-clusters each of which cannot talk to others over the interconnect. Oblivious of the existence of other cluster fragments, each sub-cluster continues to operate independently of the others. This is called “Split Brain”. In such a scenario, the integrity of the cluster and its data might be compromised due to uncoordinated writes to shared data by independently operating nodes. Hence, the split-brain must be resolved to protect the integrity of the cluster and its data.
Explanation-1:
It will arise when two or more instances attempt to control a cluster database. In a two-node environment, one instance attempts to manage updates simultaneously while another attempts to manage updates. In the Oracle RAC environment, all the instances/servers communicate with each other using high-speed interconnects on the private network. This private network interface or interconnect is redundant and is only used for inter-instance Oracle data block transfers. Now talking about the split-brain concept concerning Oracle RAC systems, it occurs when the instance members in an RAC fail to ping/connect to each other via this private interconnect, but the servers are all physically up and running and the database instance on each of these servers is also running. These individual nodes are running fine and can conceptually accept user connections and work independently. Due to lack of communication, the instance thinks that the other instance that it is not able to connect is down and it needs to do something about the situation. The problem is if we leave these instances running, the sane block might get read, and updated in these individual instances and there would be a data integrity issue, as the blocks changed in one instance, will not be locked and could be over-written by another instance. Oracle has efficiently implemented checks for the split-brain syndrome. When there is no network heart bit then Split brain is done using Disk (vote disk) information.
Explanation-2:
Oracle has efficiently implemented checks for the split-brain syndrome. In RAC if any node becomes inactive, or if other nodes are unable to ping/connect to a node in the RAC, then the node that first detects that one of the nodes is not accessible, will evict that node from the RAC group. e.g. there are 4 nodes in a RAC instance, and node 3 becomes unavailable, and node 1 tries to connect to node 3 and finds it not responding, then node 1 will evict node 3 out of the RAC groups and will leave only Node1, Node2 & Node4 in the RAC group to continue functioning. The split brain concepts can become more complicated in large RAC setups. For example, there are 10 RAC nodes in a cluster. And say 4 nodes are not able to communicate with the other 6. So there are 2 groups formed in this 10-node RAC cluster ( one group of 4 nodes and the other of 6 nodes). Now the nodes will quickly try to affirm their membership by locking the control file, and then the node that locks the control file will try to check the votes of the other nodes. The group with the most number of active nodes gets the preference and the others are evicted. Moreover, I have seen this node eviction issue with only 1 node getting evicted and the rest function fine, so I cannot really testify that that shows it works by experience, but this is the theory behind it. When we see that the node is evicted, usually Oracle rac will reboot that node and try to do a cluster reconfiguration to include back the evicted node. You will see Oracle error: ORA-29740 when there is a node eviction in RAC. There are many reasons for a node eviction like heartbeat not being received by the control, unable to communicate with the Clusterware, etc.
How does the Oracle Grid Infrastructure Clusterware resolve a 'Split Brain' situation?
Voting disk is used by Oracle Cluster Synchronization Services Daemon (ocssd) on each node, to mark its own attendance and also to record the nodes it can communicate with. In a split-brain situation,
a voting disk is used to determine which node(s) will survive and which node(s) will be evicted.
Which Process evicts nodes from the RAC Cluster?
- CSSD Monitors and evicts nodes.
- Monitors nodes using 2 communication Channels :
- Private Interconnect <=> Network Heartbeat
- Voting Disk Based communication <=> Disk Heartbeat
- Evicts (forcibly removes nodes from a cluster) nodes dependent on heartbeat feedback(failures).
Now, who will decide which node will survive and which node will face fencing ??
The answer is Voting Disk.
In a split-brain situation, a voting disk will be used to determine which node(s) survive and which node(s) will be evicted.
Please feel free to ask. thank you 🙂
Toufique Khan
Excellent 👍 explained.
ReplyDeleteThanks
Delete