GBase 8a
运维管理
文章

业务高峰时gcware重启

发表于2024-12-30 17:57:0477次浏览3个评论

问题现象:业务并发高峰时间段(数据库并发为220左右,该节点的会话数为70左右)监控脚本执行到管理节点时报错,查看业务进程corosync异常,提示有 /usr/lib/systemd/ssytemd-coredump 44873 0 0 11 160343103 corosync 收集dump信息,一会显示gcmmonit,gcmonit,gclusterd,corosync,gcrecover进程全部重启。

问题分析:查看system.log日志中有如下报错信息,其他日志中没有看到明显的报错信息,从日志来看是由于crm 回去句柄失败导致的。

计算机生成了可选文字:
201023
201023
201023
201023
201023
201023
201023
13:35
=2
201023
13:35
=2
201023
13:35
=2
201023
13:36
=2
201023
13:36
=2
201023
13:36
=2
201023
13:36
=2
201023
13:36
=2
201023
13:36
=2
201023
13:36
=2
201023
13:36
=2
201023
13:36
=2
201023
13:36
=2
201023
13:36
=2
201023
13:37
201023
13:37
201023
13:37
201023
13:37
201023
13:38
201023
13:38
201023
13:38
201023
13:38
201023
13:38
201023
201023
201023
13:39
201023
13:39
201023
13:39
201023
13:39
201023
13:39
201023
13:39
201023
13:39
201023
13:40
201023
13:40
201023
13:40
201023
13:40
201023
13:40
13:35:35
13:35:35
13:35:36
13:35:37
13:35:37
13:35:37
:37
:37
:43
:16
:16
:16
:17
:17
:17
:18
:18
:18
:18
:07
:07
:07
:07
:47
:47
:47
:47
:47
13:39:28
13:39:28
:30
巧4
巧4
巧7
巧7
巧8
巧9
:02
:07
:07
:07
[Note)gcwCrmHand1eInThreadGet()failed,err0r=2
[Note)gcwCrmHand1eInThreadGet()failed,err0r=2
CERROR)(GSA一030T一0003)Can'tgetCrmhand1et0check
CERROR)(GSA一030T一0003)Can'tgetCrmhand1et0check
CERROR)(GSA一030T一0003)Can'tgetCrmhand1et0check
CERROR)(GSA一030T一0003)Can'tgetCrmhand1et0check
C1uSter
C1uSter
C1uSter
C1uSter
StatuS
StatuS
StatuS
StatuS
having
having
having
having
err0r:
err0r:
err0r:
err0r:
GC_AIS_ERR_LIBRARY
GC_AIS_ERR_LIBRARY
GC_AIS_ERR_LIBRARY
GC_AIS_ERR_LIBRARY
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
gcwCrmHand1elnThreadGet()
sess10n247427c0nnect10n
sess10n247404c0nnect10n
sess10n247381c0nnect10n
sess10n247365c0nnect10n
sess10n247427c0nnect10n
sess10n247425c0nnect10n
sess10n247404c0nnect10n
sess10n247381c0nnect10n
sess10n247365c0nnect10n
failed,
failed,
failed,
failed,
failed,
failed,
failed,
failed,
failed,
failed,
failed,
failed,
failed,
failed,
br0ken
br0ken
br0ken
br0ken
br0ken
br0ken
br0ken
br0ken
br0ken
err0r
err0r
err0r
err0r
err0r
err0r
err0r
err0r
err0r
err0r
err0r
err0r
err0r
err0r
CERROR)L0cktab1esfailed:
(GSA一02L0一0002)Failedt010ck:[mdmprd.m一pm一Ite一flow一cell一而580D5F90一8287一4199一80S7一E6FBD44B5BFA]GC_AIS_ERR_LIBRARY
CERROR)mdmprd.m_pm_lte_flowcell_mi
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
[Note)
:01[Note)
[Note)
[Note)
[Note)
[Note)
/opt/gcluster/server/bin/gclusterd:N0rma1shutdown
EventSchedu1er:Ki1ngtheschedu1erthread,threadid2
EventSchedu1er:Waitingfortheschedu1erthreadt0reply
gcwCrmHand1eInThreadGet()failed,err0r=53
gcwCrmHand1eInThreadGet()failed,err0r=53
gcwCrmHand1eInThreadGet()failed,err0r=53
gcwCrmHand1eInThreadGet()failed,err0r=53
gcwCrmHand1eInThreadGet()failed,err0r=53
gcwCrmHand1eInThreadGet()failed,err0r=53
EventSchedu1er:Stopped
EventSchedu1er:Ki1ngtheschedu1erthread,threadid1
EventSchedu1er:Waitingfortheschedu1erthreadt0reply

/var/log/corosync.log日志中报错信息如下

计算机生成了可选文字:
C0r0SynCLLCK
qlm_grant_lockgrantShare]0Ck
node'
Oct
1一ebdb一0ca9一3ca4一a73400390fea-dynamic_hw_app—pool,max:35,Cur:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
13:35
,C0nn:0×3
,name:
Oct2313:35:54C0r0SynCCLCK
]EXECrequest:saLckResourceun10ckAsyncleaveglobal_10ck_count=751350×3S617101788340385378992168
Oct2313:35:55C0r0SynCCLCK
]EXECrequest:saLckResourceLockAsyncenterglobal_10ck_count=751350×314b8b0
Oct2313:35:55C0r0SynCCLCK
]qlm—grant_lockgrantshare10ck
nodeid:17S88101,C0nn:0×314b8b0,10ckid:196424860S0070200S7,time:20201023133555,pid:42770,name:685bf6乙
一ebdb一0ca9一3ca4一a73400390fea-dynamic_default一p001,max:0,Cur:1
Oct2313:35:55C0r0SynCCLCK
Oct2313:35:55C0r0SynCCLCK
Oct2313:35:55C0r0SynCCLCK
a73400390fea-dynamic_default_pool
EXECrequest:saLckResourceLockAsyncleaveglobal_10ck_count=761350×314b8b0
EXECrequest:saLckResourceun10ckAsyncenterglobal_10ck_count=761350×314b8b0196424860S0070200S7
qlm_unlock:nodeid:17S88101,C0nn:0×314b8b0,10ckid:196424860S0070200S7,time:20201023133555,pid:42770,mode:1,name:685bf621一ebdb一0ca9一3ca4.
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
Oct
23
23
23
23
23
23
23
23
23
23
23
23
23
23
23
23
23
23
23
23
23
23
23
,max:0,cur:1
EXECrequest:
:56
:56
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
C0r0SynC
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
CLCK
L18
L18
L18
L18
L18
L18
L18
L18
L18
L18
L18
L18
L18
L18
L18
L18
L18
L18
L18
L18
request:
request:
request:
request:
request:
request:
request:
request:
request:
request:
request:
request:
request:
request:
request:
request:
request:
request:
request:
request:
EXECrequest:
saLckResourceunlockAsync
gCLCkRecoverMasterNodeGet
gCLCkRecoverMasterNodeGet
gcLckRecoverRegistInfoGet
gcLckRecoverRegiStInfoGet
gcLckRecoverRegiStInfoGet
gcLckRecoverRegistInfoGet
gcLckRecoverRegistInfoGet
gcLckRecoverRegiStInfoGet
gcLckRecoverRegiStInfoGet
gcLckRecoverRegistInfoGet
gcLckRecoverRegistInfoGet
gcLckRecoverRegiStInfoGet
gcLckRecoverRegiStInfoGet
gcLckRecoverRegistInfoGet
gcLckRecoverRegistInfoGet
gcLckRecoverRegiStInfoGet
gcLckRecoverRegiStInfoGet
gcLckRecoverRegistInfoGet
gcLckRecoverRegistInfoGet
gcLckRecoverRegiStInfoGet
saLckResourceunlockAsync
leaveglobal_10ck_count=751350×314b8b0196424860S0070200S7
CENTER)
masternodeid
520904581CLEAVE)
CENTER)
Info
CLEAVE)
CENTER)
Info
CLEAVE)
CENTER)
Info
CLEAVE)
CENTER)
Info
CLEAVE)
CENTER)
Info
CLEAVE)
CENTER)
Info
CLEAVE)
enterglobal_10ck_count=751350×30111407450854366828822544
qlm_unlock:nodeid:18S360261,C0nn:0×3011140,10ckid:7450854366828822544,time:20201023133112,pid:4246,mode:1,name:685bf621一ebdb一0ca9一3ca4.
a73400390fea-dynamic—wy—pool,max:40,Cur:40
Oct2313:35:56C0r0SynCCLCK
]qlm—grant_lockgrantshare10ck
nodeid:353132421,C0nn:0×409a290,10ckid:385S410SS040192S40,time:20201023133518,pid:10483,name:685bf6乙
一ebdb一0ca9一3ca4一a73400390fea-dynamic—wy—pool,max:40,Cur:40
Oct2313:35:56C0r0SynCCLCK
]EXECrequest:saLckResourceun10ckAsyncleaveglobal_10ck_count=751340×30111407450854366828822544
Oct2313:35:56C0r0SynCCLCK
]EXECrequest:saLckResourceun10ckAsyncenterglobal_10ck_count=751340×409a290385541055040192540
Oct2313:35:56C0r0SynCCLCK
]qlm_unlock:n0deid:3S3132421,C0nn:0×409a290,10ckid:385541055040192540,tin:20201023133S18,pid:10483,mode:,name:685bf621一ebdb一0ca9一3ca4.
a73400390fea-dynamic—wy—pool,max:40,Cur:40
Oct2313:35:56C0r0SynCCLCK
]qlm—grant_lockgrantshare10ck
nodeid:17S88101,C0nn:0×28Sb0f0,10ckid:633384706951348292,time:20201023133519,pid:42770,name:68Sbf621.
ebdb一0ca9一3ca4一a73400390fea-dynamic—wy—pool,max:40,Cur:40
Oct2313:35:56C0r0SynCCLCK
]EXECrequest:saLckResourceun10ckAsyncleaveglobal_10ck_count=751330×409a290385541055040192540
Oct2313:35:56C0r0SynCCLCK
]EXECrequest:saLckResourceLockAsyncenterglobal_10ck_count=751330×409a290

解决方法:根据system.log和corosync.log中的日志记录,定位为创建线程失败导致进程重启的,检查参数gcluster_local_max_conn_pool=50,为系统默认参数,随着业务量增长,连接数逐渐增加,一旦超限处于保护机制进程容易重启,将该参数调大。设置为80.

Gbase>set global gcluster_local_max_conn_pool=80;

后期业务高峰时进程还是会重启,后来查看系统参数ulimit -a发现其中的max user processes有问题,该节点与其他节点配置不一致,后来手动修改该参数为1030175后恢复正常,业务高峰时未再重启。

评论

登录后才可以发表评论
用户头像
levvel发表于 6个月前
讲得很清楚!
用户头像
levvel发表于 2个月前
太有深度了!
xinyiedc!发表于 2个月前
向大佬学习