slog sender with block chunking

[mhofman]

What is the Problem Being Solved?

Currently we have 2 kinds of slog senders: write to a single file or stream events to some cloud service (generally through open telemetry).

The latter has been somewhat unreliable and not suitable for long term archiving. However a single slog file is terrible when we need to look up the detail of some execution.

Cosmic-swingset has an intrinsic chunk limit: block boundaries. We should use this to create a series of slog files.

Description of the Design

A new slog sender that understand higher level boundaries (kernel init start/finish, block lifecycle) and creates different slog files for each. Some of that logic already exists for the contextual slog sender (see #10269) and the otel trace slog sender. We'd likely want to compress the output (making sure to flush as appropriate), and possibly direct cloud storage upload as well (to avoid relying on sync scripts).

Suggestion of naming for slog files:

• slogfile_init_${startTimestamp}.jsonl
• slogfile_block_${blockHeight}_${startTimestamp}.jsonl (we need to differentiate the slog files from 2 different instances, either same machine or another, and startTimestamp is likely a good enough differentiator)

NB: While the after block stats event comes after the "forceFlush` of the sender, that event should be part of the current block slog chunk file.

Security Considerations

None

Scaling Considerations

None

Test Plan

We do not currently have a good story for integration testing of slog senders

Upgrade Considerations

Slog senders are usually part of chain software but do not affect consensus and can be enabled by env configuration.