Summary

S2 - https://harmonyone.pagerduty.com/incidents/Q1WXHGZB39IFPC 11 min down S1 - https://harmonyone.pagerduty.com/incidents/Q039HJKM2PHRMH 47 min down

Both happened at the same time and at epoch change. Both shard would eventually recovers without doing anything special and just waiting for validator to catchup their beacon shard DB

During S1 troubleshooting internal validator was at beacon chain block was at 24739505 and next epoch block was at 24739840. Network voting power was at 59% explaining why we lost consensus. Once epoch block hit the epoch block and we realized consensus was back, voting power was at 98%. On top of the internal node, there was definitely external node also having the same issue.

Root Cause Analysis - 5 Why’s

S1 and S2 consensus loss

Why did S1 and S2 lose consensus

Voting power was below the required threshold of 66.66%

Why shard voting power went below 66.66% ?

Internal and external validators node beacon chain DB wasn’t synced / lagged behind to the new epoch block where all the new staking information reside allowing the node to properly sync

Why validator node was lagging behind ?

network was spammed by arbitrage bot using the entire 80M gas and slowing down the beacon chain block insertion

Action item

Communication

[ ] Remind External validator to put in place proper monitoring/alerting to monitor the beacon shard chain DB, they can use watchdog that has telegram/PD alerting

Network spam

[x] temporarily measure to limit the gas to 20m ‣ and pushed all the leader already
[x] arbitrage bot documented here ‣ and implemented https://github.com/harmony-one/harmony/pull/4118

Improve sync

[ ] ‣ has been created for validator node to sync last epoch block. Investigate/test if a node can start that process if it was lagging behind at epoch change to minimize the shard consensus downtime
[ ] Investigate the slow block sync issue https://github.com/harmony-one/harmony/issues/4121

Monitoring

[ ] beacon out of sync are more important when a validator node is impacted. Change the watchdog alert severity of the explorer node from high to medium so validator beacon out of sync are better seen. Watchdog issue : ‣