This morning, core devs at Agoric OpCo received several pages from our monitoring service around 10:36am PST that indicated several validator nodes were not voting on new blocks. Around the same time, several validators in the peer support Discord channel also began reporting issues with blocktimes and having issues connecting to peers. During this time, the chain did not come to a full halt.
After about 5 minutes, the chain began to recover and the number of nodes participating in voting increased. We anticipate that the chain operators will continue to recover, and will continue to dig into the root cause for the increase in blocktime.
Though it is early in the investigation, we do have a few clues as to what is likely happening here. The timing of this issue coincides with the end of the voting period for Proposal # 17, which added an additional PSM contract to mainnet to for Axelar DAI. Though the proposal was tested on a local chain and investigated on Emerynet, we will be keeping a close eye on any opportunities to improve testing and qualification of changes before they are introduced to mainnet via governance.
Here is the Github issue we are working from to diagnose where the issue comes from: Issue #6625: investigating mainnet slowdown around block 7709170
If you have any questions, please feel free to engage with us here or in Discord. We will be sharing updates here and in the Github issue as more information becomes available.