Unbork Emerynet: Cleanup Auctioneers, PriceAuthorities, and QuoteNotifiers

[toliaqat]

What is the Problem Being Solved?

Emerynet currently suffers from severe performance degradation due to multiple generations of auctioneer vats running simultaneously. This has led to the accumulation of thousands of ephemeral observers and QuoteNotifiers (QNs). Each additional price submission triggers massive computations and delays, causing push-price operations to run for hours. The problem is aggravated by old auctioneers continually waking up on timers, creating new observer chains and further inflating the backlog of QNs. This situation makes Emerynet slow, unresponsive, and difficult to use for testing and validation.

Description of the Design

The proposed solution involves a multi-step cleanup and reset process:

(Optional) Halt Auctioneer Timers (for Gen4/Gen5)

Attempt to stop timers associated with problematic auctioneers (e.g., v661 and v665) to prevent further growth of observers/QNs. If this isn't possible via existing governance tools, move directly to terminating these vats.

Terminate Old Auctioneer Vats

Use governance actions (coreProposals) to kill all old auctioneers (including gen1 through gen5) on Emerynet. Removing these vats will cascade a cleanup of their associated observers, QNs, and references in scaledPriceAuthorities and priceFeeds over time.

Submit Prices for All Denoms

Push updated prices for every denom to trigger the release and garbage collection of QN references in scaledPriceAuthorities (sPA) and priceFeeds (pf). While each price submission will take hours initially, this step eventually clears the backlog of QNs and observers.

Create a New, Clean Auctioneer

After full cleanup, instantiate a fresh auctioneer vat that references updated priceAuthorities without legacy baggage. This ensures a stable and clean testing environment going forward.

Monitor and Verify Stability

Observe Emerynet after the cleanup to ensure:

• Regular auction cycles proceed without excessive overhead.
• Price pushes complete in minutes, not hours.
• No runaway growth of QNs or observers reoccurs.

Security Considerations

• Terminating vats and pushing prices are governed actions.
• Ensure termination and re-creation of vats follow proper governance proposals to maintain chain security and trust.
• Verify that no active purses or valuable resources remain locked in terminated vats.

Scaling Considerations

• The cleanup process itself may be slow and computationally expensive, involving many hours of block time.
• Once complete, the load and resource usage should return to manageable levels, enabling Emerynet to scale normally again.
• This operation provides insights into future upgrades and how to handle large-scale cleanup to avoid performance bottlenecks.

Test Plan

• Before Cleanup:
  Measure the current backlog and record how long price submissions take.
• During Cleanup:
  • Terminate old auctioneer vats and push prices.
  • Continuously monitor logs and system metrics to ensure processes are proceeding.
• After Cleanup:
  • Confirm that QNs and observers have been dramatically reduced.
  • Push a new price and confirm it completes within minutes.
  • Run a series of standard Emerynet tests to ensure normal operation.

Upgrade Considerations

• Future upgrades should include mechanisms to cleanly stop or replace auctioneers and related infrastructures without leaving behind accumulations of QNs and observers.
• Consider revisiting the design of scaledPriceAuthorities and priceFeeds to avoid complex observer chains.
• Document lessons learned to ensure mainnet upgrades (like U18/U19) won’t encounter similar issues.

Sub-Tasks:

• Investigate Timer Controls for Auctioneers

  • Determine whether it's possible to halt timers in gen4+gen5 auctioneers. If not, proceed to termination.

• Terminate Old Auctioneer Vats

  • Identify and terminate old auctioneers (v39, v353, v367, v661, v665).
  • Use a coreProposal to kill these vats and confirm dropImports/retireImports behavior in kernel logs.

• Submit Prices for All Denoms

  • Push updated prices for each managed denom to trigger cleanup.
  • Monitor the process; expect hours of block time but ensure ultimate QN reduction.

• Confirm QN and Observer Cleanup

  • Verify that the number of QNs/observers in sPA and pf vats decreases significantly.
  • Ensure no new QNs accumulate during or after the process.

• Create a New Auctioneer

  • After full cleanup, instantiate a new, clean auctioneer via a coreProposal.
  • Confirm that it references a stable, simplified priceAuthority setup.

• Monitor Post-Cleanup Stability

• Run test pushes of prices.

  • Validate normal operation and confirm that no runaway QN growth occurs again.

[warner]

@siarhei-agoric : for reference, #9483 (comment) has a lot of notes about how to find the necessary objects to terminate the old vats. And @gibson042 probably has even more ideas.