This article gives notice and details of a patched security vulnerability on the Agoric network. The agoric-upgrade-11 release has patched this issue. The Agoric network is no longer vulnerable.
On June 16th, 2023, Agoric OpCo engineers discovered a high severity, externally-exploitable issue that could halt the Agoric mainnet chain running agoric-upgrade-10. While serious, this issue could not be exploited to alter token balances, or otherwise compromise the integrity of the chain. Until patched as part of the agoric-upgrade-11, this issue was treated as a security incident with active monitoring for attempted exploitation.
Vulnerability Details
The vstorage module of the Agoric chain implements a key-value store on block chain storage. Vstorage presents a write-only interface to contracts running in the JavaScript VM. gRPC API endpoints provide read-only access to external clients like dApps.
Validation in vstorage enforces constraints on the type and format of keys. However, a bug in vstorage key validation unintentionally allowed violations of the constraints. As a result, the JavaScript VM was able to write invalid keys to vstorage. Some invalid keys could cause the node to panic, while others would result in a divergent execution for nodes that were restored from a state sync snapshot.
While contracts have never been able to directly write invalid keys to vstorage, agoric-upgrade-10 introduced a new usage for vstorage, saving a copy of the state of the JavaScript VM to support state sync export for node operators. During export, the JavaScript VM wrote invalid keys to vstorage, which resulted in a panic for nodes restoring from state-sync. It would have been possible for a malicious transaction to cause all validator nodes to panic, resulting in a chain halt.
Ongoing Improvements
Agoric OpCo identified several improvements to find similar issues before they reach production.
In addition to existing unit, integration tests, and live testing on testnets, we’re working on augmenting testing with better coverage for critical integration points for the platform. Some of the testing improvements we’ve been working on include integration with SwingSet (off-chain), the chain upgrade itself, and the chain state.
Our existing testing framework supports quickly spinning up a local version of the chain, applying desired state to it, and interrogating its state as part of end-to-end tests. This has significantly improved our capability to test on-chain scenarios, helping to identify and fix issues before testnet release, adding substantial testing coverage and confidence to release candidates. Through this incident, we’ve found local chain state needs to better reflect mainnet, as a closer simulation would have uncovered behaviors we missed.
To overcome the limitations of synthetic state, and to ensure better coverage of how changes affect mainnet, we’re working towards being able to test against a local mainnet fork where we can add additional upgrades and tests. In combination with other tests, this will allow us to ensure, for example, that any changes in development, such as new features or contracts, will successfully apply to mainnet.
Next Steps
To reiterate, there is no action needed from validators at this time, as agoric-upgrade-11 patched this issue.
Stay tuned for the next scheduled chain upgrade! In addition to features and improvements to support partners as part of the Mainnet-2 milestone, the next upgrade will restore state sync functionality on mainnet.