How to Handle Database Failures? Don't Panic, Follow These Steps (Part 1)

Published on 2024-07-30

When a database fails, how do you perform emergency handling? This article sorts out various anomalies that may occur during the use of the GBase 8a database system, aiming to help you respond to unexpected failures of the GBase 8a database and provide basic problem-solving ideas. This issue will cover three categories: hardware failures, operating system failures, and abnormal resource usage.

1. Hardware-Level Failures

1.1 Hardware-Level Failure on a GBase 8a Data Node

Symptom Description
A single GBase 8a node goes down or hangs.

Symptom Analysis
The host where the GBase 8a data node resides crashes due to hardware issues such as a power module failure or motherboard fault; in addition, local disk failures, external service network interruptions, or all-channel connection interruptions caused by RAID card failures prevent the system from providing normal external services.

Emergency Handling Procedure
The GBase 8a cluster allows a single machine to leave the cluster and continue running in a degraded state for a short period, but it cannot run for an extended time. It is necessary to stop business as soon as possible and repair the faulty hardware. Recommended handling methods:
1) The operations department contacts the open platform to confirm the issue and performs subsequent handling;
2) The open platform notifies the equipment maintenance vendor for on-site support to repair the faulty hardware; (10 minutes)
3) The operations department stops jobs on the faulty cluster. (Depending on the size of the tasks at the time, usually between 1 and 4 hours)
4) The hardware vendor repairs the faulty machine. (4–8 hours)
5) GBase on-site support starts the database service, checks the data synchronization status, and repairs any anomalies. (30 minutes)
6) The operations department restarts cluster jobs.

1.2 Hardware-Level Failure on a Loading Machine

Symptom Description
A loading machine goes down or hangs.

Symptom Analysis
The host where the loading machine resides crashes due to hardware issues such as a power module failure or motherboard fault; in addition, local disk failures, external service network interruptions, or all-channel connection interruptions caused by RAID card failures prevent the system from providing normal external services.

Emergency Handling Procedure
The current big data platform application architecture implements high availability for loading machines. If any one or more (but not all) of the 8 loading machines fail, application interruption will not occur. Recommended handling methods:
1) The operations department contacts the open platform to confirm the issue and performs subsequent handling;
2) The open platform notifies the equipment maintenance vendor for on-site support to repair the faulty hardware; (10 minutes)
3) The hardware vendor repairs the faulty machine. (4–8 hours)
4) GBase on-site support or the operations department administrator starts the loading and application services on the loading machine. (30 minutes)

2. Operating System-Level Failures

2.1 Operating System-Level Failure on a GBase 8a Data Node

2.1.1 Operating System Corruption

Symptom Description
The operating system of a single node is corrupted.

Symptom Analysis
A RAID card failure or operating system fault on a GBase 8a data node renders the system unable to provide services, requiring a reinstallation of the operating system.

Emergency Handling Procedure
The GBase 8a cluster allows a single machine to leave the cluster and run in a degraded state for a short period, but it cannot run for an extended time. It is necessary to stop business as soon as possible and repair the faulty machine. A pre-prepared standby machine can be added to the GBase 8a cluster to reduce the time for reinstalling the operating system and shorten the repair process. Recommended handling methods:
1) The operations department contacts the open platform to confirm the issue and performs subsequent handling;
2) Set up a standby machine and prepare it to join the cluster. (10 minutes)
3) The operations department stops jobs on the faulty cluster. (Depending on the size of the tasks at the time, usually between 1 and 4 hours)
4) GBase on-site support stops the faulty machine, configures the IP of the standby machine, and performs GBase 8a data synchronization; (Depending on the data volume, usually between 12 and 24 hours)
5) GBase on-site support starts the GBase 8a cluster. (20 minutes)
6) The operations department starts cluster jobs.

2.1.2 File System Failure

Symptom Description
A file system or logical volume failure occurs.

Symptom Analysis
Damage to local disks or storage disks leads to a file system or logical volume failure, as well as full disk space, resulting in abnormal read/write operations of application data on the disk.

Emergency Handling Procedure
Local disk failure causes abnormal system I/O read/write, making normal external service impossible. Recommended recording methods:
1) The operations department contacts the open platform to confirm the issue and performs subsequent handling;
2) The open platform notifies the hardware maintenance vendor to check hardware logs and locate the issue;
3) Attempt to log in to the system and check system logs and disk read/write conditions;
4) Generally, local hard drives use RAID5, and the probability of this scenario occurring is low; hardware failure is more likely;
5) The hardware vendor replaces the faulty disk;
6) If files are lost, use backup files for recovery. If GBase 8a database files are damaged, use the GBase 8a synchronization function for repair.
7) GBase on-site support starts the service and observes whether the issue is resolved.

2.2 Operating System-Level Failure on a Loading Machine

2.2.1 Operating System Corruption

Symptom Description
The operating system of a single node is corrupted.

Symptom Analysis
A RAID card failure or operating system fault on a GBase 8a data node renders the system unable to provide services, requiring a reinstallation of the operating system.

Emergency Handling Procedure
The current big data platform application architecture implements high availability for loading machines. If any one or more (but not all) of the 8 loading machines fail, application interruption will not occur. Recommended handling methods:
1) The operations department contacts the open platform to confirm the issue and performs subsequent handling;
2) The open platform reinstalls the operating system. (1 hour)
3) The open platform configures the IP, deploys the GBase 8a loading service, client, application services, etc. (1 hour)
4) GBase on-site support or the operations department administrator starts the services on this loading machine.

2.2.2 File System Failure

Symptom Description
A file system or logical volume failure occurs.

Emergency Handling Procedure
The big data platform application architecture implements high availability for loading machines. If any one or more (but not all) of the 8 loading machines fail, application interruption will not occur. Recommended recording methods:
1) The operations department contacts the open platform to confirm the issue and performs subsequent handling;
2) The open platform notifies the hardware maintenance vendor to check hardware logs and locate the issue;
3) Attempt to log in to the system and check system logs and disk read/write conditions;
4) Generally, local hard drives use RAID5, and the probability of this scenario occurring is low; hardware failure is more likely;
5) The hardware vendor replaces the faulty disk;
6) GBase on-site support or the operations department administrator starts the service and observes whether the issue is resolved.

3. Abnormal Resource Usage

3.1 Increased Swap Usage

Symptom Description
Swap usage increases significantly on a large number of nodes in the cluster.

Symptom Analysis
A GBase 8a software anomaly or abnormal SQL causes GBase 8a memory overflow, with memory usage continuously increasing. If not addressed promptly, this can cause the Swap space to become full and the system to crash abnormally.

Emergency Handling Procedure
This type of anomaly is mostly caused by a GBase 8a software issue or abnormal SQL, requiring notification to the application side to assist in troubleshooting the root cause.
1) The operations department contacts the open platform to assist in troubleshooting and notifies GBase on-site support to assist in troubleshooting;
2) The operations department and GBase on-site support analyze the abnormal running SQL in the system.
3) The operations department stops the SQL causing the problem.
4) The open platform cleans up the operating system memory to reduce Swap usage.
5) GBase on-site support assists the development team in optimizing the abnormal SQL.
6) The operations department avoids running untested SQL in the production environment.

3.2 Increased CPU Usage

Symptom Description
CPU usage increases significantly on a large number of nodes in the cluster, and I/O approaches saturation.

Symptom Analysis
Most of the CPU time is spent on context switching; the concurrent processing in GBase 8a is too high, and there are several extremely long-running tasks (not completed within 2 hours).

Emergency Handling Procedure
This type of anomaly is mostly caused by excessive concurrency in business scheduling, which leads to a decrease in the overall speed of task processing.
1) The operations department contacts the open platform to assist in troubleshooting and notifies GBase on-site support to assist in troubleshooting;
2) The operations department and GBase on-site support analyze the number of concurrently running tasks in the system.
3) If the concurrency is too high, the operations department reduces the concurrency level. If there is an exceptionally long-running SQL, discuss whether it needs to be killed first to avoid dragging down overall performance.
4) GBase on-site support assists the development team in optimizing the abnormal SQL and avoids running untested SQL in the production environment.
5) The operations department avoids manually triggering jobs outside the unified scheduling system.

3.3 Abnormally Busy Disk I/O

Symptom Description
I/O on a single node or multiple nodes in the cluster is abnormally busy.

Symptom Analysis
The DISKBUSY of one or more nodes in the cluster is significantly higher than that of other nodes (for example, DISKBUSY remains above 80% for more than 12 hours).

Emergency Handling Procedure
This type of anomaly is mostly caused by hard disk failures, disk backplane failures, or RAID card failures, leading to a decrease in the overall speed of task processing.
1) The operations department contacts the open platform to confirm the issue and performs subsequent handling;
2) The hardware vendor collects hardware operation logs and analyzes them.
3) If the concurrency is too high, the operations department reduces the concurrency level. If there is an exceptionally long-running SQL, discuss whether it needs to be killed first to avoid dragging down overall performance.
4) After the vendor analyzes the logs and determines the faulty hardware, replace the hardware. If replacing the hardware requires stopping the operating system, the cluster service must be stopped.
5) GBase on-site support restores the service.

3.4 Disk Space Full or Exceeding Threshold

Symptom Description
Disk space of one or more nodes in the cluster is full or usage exceeds 80%.

Symptom Analysis
Disk space usage of one or more nodes in the cluster exceeds 80%. Because the GBase 8a cluster data nodes must retain 20%–30% as temporary space, after the total disk space is full, some SQL queries will report errors, and GBase 8a service processes may even crash.

Emergency Handling Procedure
Generally, a sharp increase in disk usage in a short period is caused by Cartesian product SQL or a bug in the GBase 8a execution plan.
1) The operations department analyzes the usage of GBase 8a temporary space.
2) The operations department analyzes the running SQL to determine which SQL caused the issue.
3) Kill the SQL and observe whether the space is freed.
4) If it is a Cartesian product, feed back to the development department for handling; if it is a GBase execution plan issue, feed back to the database vendor and request a short-term solution and a long-term fix plan.
5) Restore the service.