Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the billing service UCAN stream handler and use BlobRegistry to directly insert deltas into the diff table #304

Open
BravoNatalie opened this issue Jan 24, 2025 · 2 comments
Assignees
Labels
refactor refactor

Comments

@BravoNatalie
Copy link

BravoNatalie commented Jan 24, 2025

This issue aligns with RFC#39 (removal of the UCAN stream) and aims to implement a mid-term solution to improve space usage tracking in the billing sub-system.

Objectives:

  1. Insert space usage deltas directly into the DynamoDB diff table from the BlobRegistry at the time of item additions or removals.
  2. Remove the UCAN stream handler from the billing service.

Benefits:

  • Improved Visibility: Tasks are performed immediately at the point they are needed, rather than delayed, offering clearer oversight on what needs to be done and when.
  • Explicit Coupling: Makes the connection between upload-api and billing sub-systems explicit.
  • Early Failure Detection: Surfaces failures earlier in the process, reducing debugging complexity.
  • System Consistency: Ensures consistency across the system by maintaining atomicity in operations.
  • Fixing Double Diff Entries: Resolves issues like double entries in the diff table if a blob already exists.

Implementation Details:

  • repository: w3infra
  • Update BlobRegistry.register and BlobRegistry.deregister to insert the space usage delta into the space-diff table when blob/accept and blob/remove are executed, respectively.
  • Ensure atomic updates to maintain data consistency across the system.
  • The ucan-stream is currently configured in stacks/billing-stack.js.
@travis
Copy link
Member

travis commented Jan 24, 2025

this looks right to me! I just want to make sure this work should be done in w3infra rather than upload-service-infra - @alanshaw does that sound right to you given the current state of The Merge?

@alanshaw
Copy link
Member

Yes

BravoNatalie added a commit to storacha/upload-service that referenced this issue Jan 29, 2025
The new blob/remove flow introduced in
storacha/w3infra#458 removes the ucan-stream
(storacha/project-tracking#304) and updates
the space-diff table directly from the blob-registry.

To properly create a space-diff entry in deregister, the invocation CID
is required. However, unlike register, it is not currently passed as a
parameter. This update ensures the necessary data is available for
consistent and accurate tracking of space usage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
refactor refactor
Projects
Status: In Progress
Development

No branches or pull requests

3 participants