feat: support host monitor #1890

Abingcbc · 2024-11-18T02:08:43Z

打通进程元信息采集链路

TODO：

支持更多字段
可观测指标

core/common/timer/HostMonitorTimerEvent.cpp

core/models/PipelineEventGroup.h

core/runner/sink/http/HttpSink.cpp

core/host_monitor/collector/MockCollector.cpp

core/host_monitor/collector/CollectorManager.cpp

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

core/constants/EntityConstants.cpp

core/host_monitor/SystemInformationTools.cpp

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

core/host_monitor/collector/ProcessCollector.cpp

core/common/FileSystemUtil.h

core/common/StringTools.cpp

core/common/timer/HostMonitorTimerEvent.cpp

core/plugin/input/InputHostMeta.h

core/plugin/input/InputHostMeta.cpp

core/host_monitor/HostMonitorInputRunner.cpp

core/common/timer/Timer.h

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

core/runner/sink/http/HttpSink.cpp

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

core/host_monitor/collector/ProcessCollector.h

core/host_monitor/collector/ProcessCollector.cpp

core/plugin/processor/inner/ProcessorHostMetaNative.cpp

core/host_monitor/collector/ProcessCollector.cpp

messixukejia · 2024-11-30T11:20:51Z

core/host_monitor/collector/ProcessCollector.cpp

+
+    int readCount = 0;
+    WalkAllProcess(PROCESS_DIR, [&](const std::string& dirName) {
+        if (++readCount > mProcessSilentCount) {


个数控制是不是也在控制缓存里的？不应该只是控制去直接操作系统交互的吗

缓存是为了计算增量的CPU使用率，前一次和后一次的top n进程可能不一样，所以需要保存所有的

core/common/MachineInfoUtil.cpp

core/common/timer/Timer.h

core/plugin/input/InputHostMeta.cpp

core/host_monitor/collector/ProcessEntityCollector.cpp

messixukejia · 2024-12-29T03:11:15Z

core/plugin/input/InputHostMeta.cpp

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "InputHostMeta.h"


core/plugin/input/InputHostMeta.cpp

core/host_monitor/HostMonitorInputRunner.cpp

messixukejia · 2024-12-29T07:04:11Z

core/host_monitor/HostMonitorInputRunner.h

+    ThreadPool mThreadPool;
+
+    mutable std::shared_mutex mRegisteredCollectorMapMutex;
+    std::unordered_map<std::string, bool> mRegisteredCollectorMap;


std::unordered_map<std::string, bool> mRegisteredCollectorMap; std::unordered_map<std::string, std::shared_ptr<BaseCollector>> mCollectorInstanceMap;

这两个变量有点不合理。
包括
1、HostMonitorInputRunner::GetCollector的使用上。应该是自闭环合法性
2、锁的关系等
3、RegisterCollector 操作的变量是 mCollectorInstanceMap，但是另一个没操作的变量却叫reg

可能的优化
1、保持现有逻辑。一组功能不用过多发散变量，以上改成一个map。value用结构体表示。整体实现结构优化下。
2、UpdateCollector时再注册ProcessEntityCollector实例，这样能省一个变量。

core/host_monitor/collector/ProcessEntityCollector.cpp

core/common/timer/Timer.cpp

messixukejia · 2024-12-30T08:56:13Z

core/host_monitor/collector/ProcessEntityCollector.cpp

+
+const std::string ProcessEntityCollector::sName = "process_entity";
+
+ProcessEntityCollector::ProcessEntityCollector() : mProcessSilentCount(INT32_FLAG(process_collect_silent_count)) {


任务执行异步
超时影响
不同任务类型见例如Prometheus、host entity、processo不影响

线程池放入不会堵塞，runner拿到任务后，可以立刻放入线程池

在一个任务执行结束后，再放入下一个任务。一个任务超时，会计算从上一次执行时间开始，增加n个周期，到达下一个未来可执行的时间点。
nextExecTime = execTime + n * interval

messixukejia · 2025-01-07T01:24:21Z

core/common/MachineInfoUtil.cpp

 #else
    ECSMeta ecsMeta;
    if (FetchECSMeta(ecsMeta)) {
        UpdateECSMetaAndHostid(ecsMeta);
    }
 #endif
+    updateHostId();


要调整翊韬先把之前的设计补出来，说明下原因

messixukejia · 2025-01-07T01:24:59Z

core/common/MachineInfoUtil.h

    void DumpECSMeta();
+#ifdef __ENTERPRISE__


不要用这么怪的方式

messixukejia · 2025-01-07T01:31:59Z

docs/cn/plugins/input/native/input-host-meta.md

+```yaml
+enable: true
+inputs:
+  - Type: input_host_meta


与其他保持一致，补充完整

messixukejia · 2025-01-07T01:45:11Z

core/host_monitor/collector/ProcessEntityCollector.cpp

+
+        event->SetContent(DEFAULT_CONTENT_KEY_FIRST_OBSERVED_TIME, processCreateTime);
+        event->SetContent(DEFAULT_CONTENT_KEY_LAST_OBSERVED_TIME, std::to_string(logtime));
+        auto interval = group.GetMetadata(EventGroupMetaKey::HOST_MONITOR_COLLECT_INTERVAL);


1、采集配置中的间隔、host_monitor_default_interval 看上去并不是完全独立的参数。

2、凡是外部的参数都要有合法性保护，这里的也就是要有最小值。

messixukejia · 2025-01-07T01:48:46Z

core/constants/EntityConstants.cpp

+// process entity
+const std::string DEFAULT_CONTENT_VALUE_ENTITY_TYPE_ECS_PROCESS = "acs.ecs.process";
+const std::string DEFAULT_CONTENT_VALUE_ENTITY_TYPE_HOST_PROCESS = "infra.host.process";
+const std::string DEFAULT_CONTENT_KEY_PROCESS_PID = "pid";


字段是如何跟安全保持一致的？

messixukejia · 2025-01-07T01:49:17Z

core/host_monitor/HostMonitorInputRunner.cpp

+
+namespace logtail {
+
+HostMonitorInputRunner::HostMonitorInputRunner() : mThreadPool(ThreadPool(3)) {


3 的合理性？

messixukejia · 2025-01-07T02:01:52Z

core/models/PipelineEventGroup.h

@@ -61,6 +61,8 @@ enum class EventGroupMetaKey {
    PROMETHEUS_STREAM_ID,
    PROMETHEUS_STREAM_TOTAL,

+    HOST_MONITOR_COLLECT_INTERVAL,


使用的地方都属于input控制范围，可以通过HostMonitorInputRunner::UpdateCollector来传递

{"process_entity"},

messixukejia · 2025-01-07T02:10:55Z

core/host_monitor/HostMonitorInputRunner.cpp

+    RegisterCollector<ProcessEntityCollector>();
+}
+
+void HostMonitorInputRunner::UpdateCollector(const std::vector<std::string>& newCollectors,


保持变量含义的统一性，这里应该是collectornames。

messixukejia · 2025-01-07T02:16:14Z

core/host_monitor/collector/ProcessEntityCollector.cpp

+                                             std::string& entityType,
+                                             std::string& hostEntityID,
+                                             std::string& hostEntityType) {
+    ECSMeta metaObj = HostIdentifier::Instance()->GetECSMeta();


合库前需要：GetECSMeta的使用还需要完善下设计及实现

messixukejia · 2025-01-07T02:31:44Z

core/app_config/AppConfig.cpp

@@ -165,6 +165,8 @@ DEFINE_FLAG_STRING(loong_collector_operator_service, "loong collector operator s
 DEFINE_FLAG_INT32(loong_collector_operator_service_port, "loong collector operator service port", 8888);
 DEFINE_FLAG_INT32(loong_collector_k8s_meta_service_port, "loong collector operator service port", 9000);
 DEFINE_FLAG_STRING(_pod_name_, "agent pod name", "");
+DEFINE_FLAG_INT32(host_monitor_default_interval, "default interval for host monitor", 60);


调整的入口单一，没必要的入口去掉

messixukejia · 2025-01-07T02:36:24Z

core/host_monitor/HostMonitorInputRunner.cpp

+
+namespace logtail {
+
+HostMonitorInputRunner::HostMonitorInputRunner() : mThreadPool(ThreadPool(3)) {


messixukejia · 2025-01-07T02:37:35Z

core/models/PipelineEventGroup.h

@@ -61,6 +61,8 @@ enum class EventGroupMetaKey {
    PROMETHEUS_STREAM_ID,
    PROMETHEUS_STREAM_TOTAL,

+    HOST_MONITOR_COLLECT_INTERVAL,


{"process_entity"},

Abingcbc force-pushed the host_monitor branch 6 times, most recently from 6150e52 to bfdd9c2 Compare November 19, 2024 06:27

messixukejia requested changes Nov 25, 2024

View reviewed changes

henryzhx8 requested changes Nov 26, 2024

View reviewed changes

feat: support host monitor

3e12438

Abingcbc force-pushed the host_monitor branch from 4110e3d to 3e12438 Compare November 27, 2024 06:22

Abingcbc added 2 commits November 29, 2024 00:01

fix

8f9f8ee

Merge remote-tracking branch 'alibaba/main' into host_monitor

0455a6c

messixukejia requested changes Nov 30, 2024

View reviewed changes

messixukejia requested changes Dec 4, 2024

View reviewed changes

core/common/MachineInfoUtil.cpp Outdated Show resolved Hide resolved

core/common/MachineInfoUtil.cpp Outdated Show resolved Hide resolved

core/common/timer/Timer.h Show resolved Hide resolved

Abingcbc added 7 commits December 27, 2024 00:34

fix

5435d03

Merge remote-tracking branch 'alibaba/main' into host_monitor

6f8d8d4

fix

f7b71ab

fix

cc1f455

fix

0693bb1

fix

77f1384

fix

719fb39

messixukejia reviewed Dec 29, 2024

View reviewed changes

catdogpandas reviewed Dec 30, 2024

View reviewed changes

core/common/timer/Timer.cpp Outdated Show resolved Hide resolved

Abingcbc added 2 commits January 5, 2025 22:03

fix

84332c9

Merge remote-tracking branch 'alibaba/main' into host_monitor

3fd57b3

messixukejia reviewed Jan 6, 2025

View reviewed changes

Abingcbc added 3 commits January 6, 2025 09:59

fix

f2e1885

lint

3b27c1b

fix

2cb4b7b

Abingcbc force-pushed the host_monitor branch from 8fdce38 to 2cb4b7b Compare January 6, 2025 07:45

fix

c4614a8

messixukejia requested changes Jan 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support host monitor #1890

feat: support host monitor #1890

Abingcbc commented Nov 18, 2024

messixukejia Nov 30, 2024

Abingcbc Dec 27, 2024

messixukejia Dec 29, 2024

messixukejia Dec 29, 2024

messixukejia Dec 30, 2024

messixukejia Dec 30, 2024

Abingcbc Jan 6, 2025 •

edited

Loading

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025

messixukejia Jan 7, 2025


		const std::string ProcessEntityCollector::sName = "process_entity";

		ProcessEntityCollector::ProcessEntityCollector() : mProcessSilentCount(INT32_FLAG(process_collect_silent_count)) {


		namespace logtail {

		HostMonitorInputRunner::HostMonitorInputRunner() : mThreadPool(ThreadPool(3)) {

feat: support host monitor #1890

Are you sure you want to change the base?

feat: support host monitor #1890

Conversation

Abingcbc commented Nov 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Abingcbc Jan 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Abingcbc Jan 6, 2025 •

edited

Loading