What to Do When a Database Fails? Don't Panic — Follow These Steps (Part 2)
When a database fails, how do you handle it? This article sorts out various anomalies that may occur in the GBase 8a database system, aiming to help you respond to unexpected GBase 8a failures and provide basic troubleshooting ideas. Continuing from the previous part, this issue focuses on database service anomalies, data loss, and other exception types.
4. Database Service Anomalies
4.1 GBase Cluster Service Process Crash
Description
The five cluster node services (gclusterd, gbased, gcware, gcrecover, gc_sync_server) crash abnormally.
Analysis
The five cluster node services (gclusterd, gbased, gcware, gcrecover, gc_sync_server) crash abnormally.
Emergency Response Procedure
This type of anomaly is mostly caused by a specific SQL statement or scenario triggering a GBase bug. The application team needs to be notified to assist in troubleshooting the cause.
1) Notify the Open Platform and the GBase vendor to assist in troubleshooting.
2) The operations department analyzes the abnormal SQL running in the system.
3) The operations department stops the SQL causing the problem.
4) The GBase vendor analyzes the problem scenario, provides a short-term workaround, and sets a subsequent fix timeline.
4.2 GBase Cluster Service Fails to Start
Description
The cluster node services gclusterd, gbased, gcware, gcrecover, and gc_sync_server cannot start.
Analysis
The inability to start cluster services gclusterd, gbased, gcware, gcrecover, and gc_sync_server is usually caused by a bug in the GBase 8a cluster product.
Emergency Response Procedure
Usually caused by a GBase 8a cluster product bug.
1) The operations department notifies the Open Platform and the GBase vendor to assist in troubleshooting.
2) The operations department and the GBase vendor analyze the logs and runtime scenarios.
3) The GBase vendor analyzes the scenario, provides a short-term workaround, and sets a fix timeline.
5. Data Loss
5.1 Multi-Node Failure Leading to Cluster Data Loss
Description
Multi-node failure, cluster data loss.
Analysis
In extreme cases, multi-node failures of the GBase 8a database cause cluster data loss that cannot be repaired.
Emergency Response Procedure
Use backup data for recovery.
1) Notify the Open Platform and the GBase vendor to assist in troubleshooting;
2) The operations department stops running tasks. (10 minutes)
3) The GBase vendor stops the database service;
4) The GBase vendor restores the latest backup data from the backup media; (recovery time varies greatly depending on data volume, typically between 12 and 24 hours)
5) The GBase vendor starts the service and verifies cluster data consistency; (30 minutes)
6) The operations department restores the service and notifies the operations team to start tasks.
6. Other Anomalies
6.1 Data Inconsistency Errors
Description
Data inconsistency alerts occur on cluster nodes.
Analysis
When a node experiences a brief network interruption, data inconsistency may occur. Usually, data will be automatically synchronized after the network recovers. If the data remains inconsistent for a long time, manual synchronization is required.
Emergency Response Procedure
Once the network is restored, the data-inconsistent node should recover automatically. Check the network status. If data is still not synchronized one hour after network recovery, consider manual synchronization.
1) The operations department notifies the Open Platform and the GBase vendor to assist in troubleshooting.
2) For temporary table loading alerts, you may wait 10 minutes; if the cluster automatically synchronizes successfully, the issue ends normally. Otherwise, on-site GBase support is needed to determine whether to stop the cluster service and running tasks, and then perform steps 3-6. (Depending on the task size, typically 1 to 4 hours)
3) GBase 8a stops the database service (20 minutes)
4) The GBase vendor analyzes the inconsistent tables and performs manual synchronization. (Depending on table size and the number of inconsistent tables, typically 2 to 8 hours)
5) The GBase vendor starts the database service and checks data consistency (30 minutes)
6) GBase notifies the operations department that the system is restored and tasks can be started.
6.2 Data Errors
Description
Incorrect result set from a SQL statement execution.
Analysis
Caused by a bug in the GBase 8a database execution plan, leading to incorrect result sets.
Emergency Response Procedure
If such issues are found, the application team must cooperate to analyze the scope of impact on the current system and consider subsequent remediation.
1) The operations department notifies the Open Platform and the GBase vendor to assist in troubleshooting;
2) The GBase vendor analyzes and locates the problem, providing a detailed cause explanation, fix plan, and workaround;
3) The application department analyzes the impact scope based on the vendor's explanation and investigates the affected areas;
4) The operations department and the GBase vendor correct the erroneous data and modify programs to avoid the issue.
5) The GBase vendor provides a fixed version.
6.3 Execution Errors
Description
A SQL statement execution error occurs.
Analysis
Caused by a GBase 8a database bug, leading to execution errors.
Emergency Response Procedure
If such issues are found, the vendor must be instructed to analyze the bug cause and provide a resolution deadline.
1) The operations department notifies the Open Platform and the GBase vendor to assist in troubleshooting;
2) The GBase vendor analyzes and provides a workaround.
3) The application department implements the workaround based on the vendor's explanation;
4) The GBase vendor provides a fixed version.
6.4 High Database Node Load Due to Excessive Concurrency
High concurrency can be identified by the following:
(1) Average system CPU usage exceeds 90%.
(2) Disk I/O is near saturation.
(3) Using show processlist reveals high concurrency and 1–3 excessively long tasks (running for about or over 1 hour).
Solutions:
1) Reduce the concurrency of the scheduling system.
2) Adjust the concurrency order of long and short jobs so that they run evenly, avoiding concentration of long jobs.