[WIP] Registry manifest and Schema diff #400
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
# Conflicts: # crates/weaver_semconv/src/group.rs
lquerel
changed the title
# Conflicts: # .clippy.toml # Cargo.toml # crates/weaver_semconv_gen/src/lib.rs # src/registry/search.rs # src/registry/stats.rs # src/registry/update_markdown.rs
| @@ -39,7 +39,7 @@ Brief: {{ resource.brief }} | |||
| - Sampling relevant: {{ attribute.sampling_relevant }} | |||
| {%- endif %} | |||
| {%- if attribute.deprecated %} | |||
| - Deprecated: {{ attribute.deprecated }} | |||
| - Deprecated: {{ attribute.deprecated.note }} | |||
There was a problem hiding this comment.
Is/will this be a breaking change?
There was a problem hiding this comment.
Yes, it is a breaking change at the rendering level because, without the .note, Jinja renders the new JSON of the deprecated field instead of the text that now corresponds to the note. I couldn’t find another way to handle this at the rendering level. However, at the semconv parsing level, it is not a breaking change since both formats remain supported.
There was a problem hiding this comment.
Related comment: https://github.com/open-telemetry/weaver/pull/400/files#r1868713601 ( I thought we'd only keep the note on the attribute without introducing one on deprecated).
But if we do have deprecated.note, it'd still optional thing that does not need to describe the action.
I think codegen would need to generate something like
/*
* ...
* @deprecated renamed to {@link
* io.opentelemetry.semconv.FooAttributes#FOO_BAR}.
* Here's an additional note.
*/from
deprecated:
action: renamed
new_name: foo.bar
note: Here's an additional note.so maybe we should eventually provide some helper filters to render those.
In the scope of this PR, if someone uses {{ attr.deprecated }}, I wonder if jinja can call to_string on the original deprecated object to return a custom string? We'd format it similarly to "Renamed to foo.bar", but someone who wants to use new structured info would be able to obtain it with attr.deprecated.action, etc
WDYT?
| .map(|group| match group_type { | ||
| GroupType::AttributeGroup | ||
| | GroupType::Event | ||
| | GroupType::Span |
There was a problem hiding this comment.
Are we going to require span names now?
There was a problem hiding this comment.
Good question, we still have this thorn in our side with spans. How do you see things from the perspective of semantic conventions?
There was a problem hiding this comment.
we can start relying on group.id across all signals now - open-telemetry/semantic-conventions#1512
I'll keep making a slow progress on spans to improve it further - open-telemetry/semantic-conventions#1513, but it should not be a blocker
There was a problem hiding this comment.
Okay, now that we have a form of uniqueness for group IDs within the same registry, I’m fine with it. We’ll need to scope these IDs by registry once we add multi-registry support.
| } | ||
|
|
||
| // Attributes in the registry | ||
| self.diff_attributes(baseline_schema, &mut changes); |
There was a problem hiding this comment.
I have two concerns here:
- overall attribute renames rely on the registry concept. We can only look at renames/added/removed from attribute groups there. We should probably avoid looking at deprecated on a per-signal basis in that instance.
- On a per-signal basis, we should be tracking added/removed attributes. (We can ignore changed, as those should be caught in overall "registry" renames.
Example:
- There are two attributes
error.typeandhttp.status_code. - There is an event
http.my.count. It currently useserror.type. - Example1:
http.my.countevent is updated to usehttp.status_codeinstead, using some kind of "ref"/"deprecated" combo. - Example2:
http.my.countupdateserror.typefromopt-intorecommended.
Perhaps we don't want to handle this yet. Curious on @lmolkova's thoughts.
There was a problem hiding this comment.
Concerns 1 and 2 make sense to me. I can remove rename tracking for everything except the attributes in the registry.
Regarding the examples, could you describe more precisely what you’d like to see in the diff?
There was a problem hiding this comment.
I think this PR detects renames only (for attributes, metrics, etc) and does not analyze what's changed within a group.
I can see cases when we want to go deeper, but they are beyond changelog/schema transformation automation. E.g. we do it in compatibility policy checks.
I wonder if good old weaver.yaml + JQ + jinja with access to baseline and new semconv at the same time can cover all possible complex diff scenarios one would want to do.
There was a problem hiding this comment.
I think at some point we'll want diffs on what to include in a metric or span. As Lidumila calls out we can defer that until later .
…l-weaver into generate-otel-schema
…ed on the message
…cated::Renamed::renamed_to
# Conflicts: # crates/weaver_codegen_test/build.rs
I have a question regarding the format of the new Old approach (still supported for compatibility reasons): or or With this, we can handle simple attribute renaming scenarios, as well as merge scenarios (e.g., A and B are renamed to C; Weaver will detect this automatically). However, we currently have no way to represent a split (e.g., A is renamed to B and C). So with the current implementation, the semconv author will need to set We could make this explicit in the format of the deprecated field and in the diff output. This would allow for migration documentation that more accurately reflects the desired changes. However, it still wouldn’t enable automatic downgrades in the schema processor for the split scenario (at least without logic taking into account some additional context). Question: Adding such an advanced definition for the deprecated field isn’t particularly complicated, so I don’t mind including it. What do you think? Are there other types of deprecations you’d like to codify? |
Note: This scope of this PR has been reduced to focus only focus on the schema diff feature. Github issues have been created to track the features that have been postponed #482, #483.
This PR implements the command
registry diff, see the following example:In this example, the diff is displayed in markdown format. The following formats are supported: json, yaml, markdown, ansi, ansi_stats
Tasks:
registry-manifest.yaml.Notes: Some people would like to generate database migration scripts based on the diff between two registries. This reinforces the need to decouple the registry diff from how the diff is used (e.g., generation of OTEL schema, migration guides (documentation), db migration script, etc.).
Notes:
weaver_otel_schemais not essential for this PR; it was initially included as part of the preparations for theregistry schema-updatecommand. We have decided to implement this command in a future PR. However, for simplicity, I prefer to keep the preparation code in place instead of removing it. Same thing forall_changesinweaver_version.List of modifications to apply to the semantic conventions repository after the release of the Weaver containing the current PR:
registry-manifest.yamlfile with the version of the next release.Closes: #186