Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support to process tag #1806

Merged
merged 29 commits into from
Jan 22, 2025
Merged

feat: support to process tag #1806

merged 29 commits into from
Jan 22, 2025

Conversation

Abingcbc
Copy link
Collaborator

@Abingcbc Abingcbc commented Oct 14, 2024

支持重命名和修改Tag

配置

  • Input级别配置(Reader实现):控制插件级tag pathinode、容器元信息、ExternalEnvTag、ExternalK8sLableTag
  • Global级别配置(内置插件实现):控制Pipeline级tag File Tag、ENV Tag、sourcehostnameuser_defined_id

内部数据结构定义调整

  • metadata:仅用于存储处理和发送阶段需要使用的信息
  • tag:用于保存所有输出字段

@Abingcbc Abingcbc changed the title feat: support to rename and delete tag feat: support to process tag Oct 16, 2024
core/common/ParamExtractor.h Outdated Show resolved Hide resolved
core/common/TagConstants.h Outdated Show resolved Hide resolved
core/file_server/FileTagOptions.h Show resolved Hide resolved
core/file_server/FileTagOptions.cpp Show resolved Hide resolved
core/file_server/FileTagOptions.cpp Outdated Show resolved Hide resolved
core/pipeline/Pipeline.cpp Outdated Show resolved Hide resolved
Copy link
Collaborator

@henryzhx8 henryzhx8 Oct 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. 整体实现需要重新看一下文档
  2. 这个插件应该是pipeline级别的了
  3. C++和Go串联的情况下,应该全部由C++这边来完成处理,Go应该啥都不做,只有Aggregator的Topic是个例外

core/runner/ProcessorRunner.cpp Outdated Show resolved Hide resolved
pkg/helper/docker_center.go Outdated Show resolved Hide resolved
pluginmanager/plugin_runner_helper.go Outdated Show resolved Hide resolved
@henryzhx8 henryzhx8 added enhancement Feature enhancement core Core feature labels Oct 22, 2024
@henryzhx8 henryzhx8 added this to the v2.2 milestone Oct 22, 2024
@Abingcbc Abingcbc force-pushed the process_tag branch 3 times, most recently from 062c0d2 to 05d5f38 Compare January 10, 2025 04:58
core/file_server/FileTagOptions.h Outdated Show resolved Hide resolved
core/file_server/FileTagOptions.h Outdated Show resolved Hide resolved
core/file_server/FileTagOptions.h Outdated Show resolved Hide resolved
core/common/ParamExtractor.cpp Outdated Show resolved Hide resolved
core/common/ParamExtractor.cpp Outdated Show resolved Hide resolved
core/plugin/processor/inner/ProcessorTagNative.cpp Outdated Show resolved Hide resolved
core/plugin/processor/inner/ProcessorTagNative.cpp Outdated Show resolved Hide resolved
pluginmanager/plugin_runner_v2.go Outdated Show resolved Hide resolved
pluginmanager/processor_tag_enterprise.go Outdated Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

现有C++和go的tag行为不一致,需要保持一致,而不是继续遵从现状,详见设计文档。

core/common/ParamExtractor.cpp Outdated Show resolved Hide resolved
core/file_server/FileTagOptions.cpp Outdated Show resolved Hide resolved
core/common/ParamExtractor.cpp Show resolved Hide resolved
core/common/ParamExtractor.h Outdated Show resolved Hide resolved
core/file_server/reader/LogFileReader.cpp Outdated Show resolved Hide resolved
core/plugin/processor/inner/ProcessorTagNative.h Outdated Show resolved Hide resolved
core/plugin/processor/inner/ProcessorTagNative.cpp Outdated Show resolved Hide resolved
core/plugin/processor/inner/ProcessorTagNative.cpp Outdated Show resolved Hide resolved
pluginmanager/logstore_config.go Show resolved Hide resolved
pluginmanager/plugin_runner_v1.go Outdated Show resolved Hide resolved
core/constants/TagConstants.cpp Outdated Show resolved Hide resolved
core/common/ParamExtractor.cpp Outdated Show resolved Hide resolved
core/common/ParamExtractor.cpp Outdated Show resolved Hide resolved
core/pipeline/Pipeline.cpp Outdated Show resolved Hide resolved
docs/cn/plugins/flusher/extended/loki.md Outdated Show resolved Hide resolved
core/constants/TagConstants.cpp Outdated Show resolved Hide resolved
core/models/PipelineEventGroup.cpp Show resolved Hide resolved
)

const (
hostNameDefaultTagKey = "host_name"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里部分商业开源?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里只有开源的,商业版在另一个文件,单独提交到内部

core/collection_pipeline/CollectionPipeline.h Outdated Show resolved Hide resolved
core/collection_pipeline/CollectionPipeline.cpp Outdated Show resolved Hide resolved
core/common/ParamExtractor.cpp Outdated Show resolved Hide resolved
core/file_server/FileTagOptions.cpp Outdated Show resolved Hide resolved
core/common/ParamExtractor.cpp Outdated Show resolved Hide resolved
core/constants/TagConstants.h Outdated Show resolved Hide resolved
core/constants/TagConstants.h Outdated Show resolved Hide resolved
Comment on lines 109 to 115
std::string key = tags[i - 1].asString();
std::string value = tags[i].asString();
// 老版本或者容器元信息
if (isOldCheckpoint && containerNameTag.find(key) != containerNameTag.end()) {
containerInfo.AddMetadata(key, value);
} else {
containerInfo.mTags.emplace_back(tag);
containerInfo.mTags.emplace_back(key, value);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这段逻辑和qiuwencheng确认下,感觉有点怪

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

之前代码判断写错了

core/file_server/FileTagOptions.cpp Outdated Show resolved Hide resolved
core/constants/TagConstants.cpp Outdated Show resolved Hide resolved
docs/cn/configuration/collection-config.md Outdated Show resolved Hide resolved
docs/cn/plugins/input/native/input-file.md Outdated Show resolved Hide resolved
core/constants/TagConstants.h Outdated Show resolved Hide resolved
@Abingcbc Abingcbc merged commit 03ccb95 into alibaba:main Jan 22, 2025
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core feature enhancement Feature enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants