# Troubleshooting Validator Node Restart Issues

**URL:** <https://community.agoric.com/t/troubleshooting-validator-node-restart-issues/114>\
**Category:** Validators\
**Created:** [November 19, 2022, 12:17am UTC](https://community.agoric.com/t/troubleshooting-validator-node-restart-issues/114 "2022-11-19T00:17:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jessysaurusrex](https://sea1.discourse-cdn.com/flex019/user_avatar/community.agoric.com/jessysaurusrex/32/34_2.png) [@jessysaurusrex](https://community.agoric.com/u/jessysaurusrex)\
**Post date:** [November 19, 2022, 12:17am UTC](https://community.agoric.com/t/troubleshooting-validator-node-restart-issues/114/1 "2022-11-19T00:17:19Z")

</div>

This morning, the two node operators reported issues with restarting nodes in Discord. Since the initial discussion came up, Agoric core devs have been busy digging into potential causes for this issue, and though the impact to the network appears small (and impacted nodes are back online), we will continue investigating the issue and diagnose the root cause.

For now, the Agoric team recommends that validators who encounter restart issues restore their nodes from a snapshot provider on the network like Polkachu or kjnodes.

If you are a validator that runs into this problem or any other sort of consensus failure, please send us a copy of your node’s state to troubleshoot this issue (e.g. via s3, Google Drive, Dropbox).

- To share your state, please run the following command _after_ the node shuts down, but _before_ you delete the state (e.g. to replace it with a snapshot) and then send us the tarball that it creates:  
` tar czf ~/agoric-swingset-state.tar.gz ~/.agoric/data/ag-cosmos-chain-state`

- If you happen to collect slogfiles (i.e. the SLOGFILE= environment variable is set), please compress the output and send us those to us as well.

Finally, whether you have problems or not, we’re interested in how long your validator was running before the restart. One of the avenues that we are investigating is the the possibility that runtime affects the probability of the bug being triggered, so please let us know the last time you restarted your node by adding a comment to this thread.

The Agoric team is working on the issue [in Github](https://github.com/Agoric/agoric-sdk/issues/6588), and will update this discussion and continue to engage with validators in Discord as we continue our bug hunt. 🪲

---

<div class="post-metadata">

**Author:** ![Colinka](https://sea1.discourse-cdn.com/flex019/user_avatar/community.agoric.com/colinka/32/159_2.png) [@Colinka](https://community.agoric.com/u/Colinka)\
**Post date:** [November 19, 2022, 9:03am UTC](https://community.agoric.com/t/troubleshooting-validator-node-restart-issues/114/2 "2022-11-19T09:03:12Z")

</div>

Hello, I investigated my sentry nodes, they all worked without rebooting since the last upgrade on block 7179263. I rebooted my nodes one by one, there were no problems.

---

<div class="post-metadata">

**Author:** ![Craci\_BwareLabs](https://sea1.discourse-cdn.com/flex019/user_avatar/community.agoric.com/craci_bwarelabs/32/62_2.png) [@Craci\_BwareLabs](https://community.agoric.com/u/Craci_BwareLabs)\
**Post date:** [November 21, 2022, 12:45pm UTC](https://community.agoric.com/t/troubleshooting-validator-node-restart-issues/114/3 "2022-11-21T12:45:49Z")

</div>

Ours is working fine as well.

---

<div class="post-metadata">

**Author:** ![dckc](https://sea1.discourse-cdn.com/flex019/user_avatar/community.agoric.com/dckc/32/46_2.png) [@dckc](https://community.agoric.com/u/dckc)\
**Post date:** [December 19, 2022, 7:52pm UTC](https://community.agoric.com/t/troubleshooting-validator-node-restart-issues/114/4 "2022-12-19T19:52:40Z")

</div>

As noted in [#6588](https://github.com/Agoric/agoric-sdk/issues/6588), we have identified the root cause and provided a fix that has been verified by a few affected validators.

For any validators who are affected by this going forward, we prepared a release including the fix:

- [Release pismoB · Agoric/agoric-sdk](https://github.com/Agoric/agoric-sdk/releases/tag/pismoB) `08ca9d4`
