Extend InfluxDB2 Helm Chart for Aggregated Buckets and Task Creation to Support Grafana Performance Optimization

[davidpepm]

Extend InfluxDB2 Helm Chart for Aggregated Buckets and Task Creation to Support Grafana Performance Optimization

---

Summary

The current InfluxDB2 Helm chart does not support automatic creation of aggregated buckets or scheduling Flux tasks. This feature is critical for optimizing Grafana dashboards that rely on aggregated data for improved performance. To address this, the chart should be enhanced to automatically create aggregated buckets and define tasks to populate them with preprocessed data.

---

Problem Statement

Grafana dashboards built on top of the default bucket experience performance issues when querying large datasets over extended time periods. To improve performance:

1. Aggregated data needs to be precomputed and stored in dedicated buckets (default_agg_topic, default_agg_stats, default_agg_kpi).
2. Recurring Flux tasks must process data and populate these buckets efficiently.

Currently, these steps require manual configuration after deploying the InfluxDB instance, increasing setup time and complexity. This lack of automation hinders scalability and can lead to errors in production environments.

---

Proposed Solution

Enhance the InfluxDB2 Helm chart to:

1. Create Aggregated Buckets:

   • Define aggregated buckets in values.yaml for storing preprocessed data.
   • Automatically create these buckets via a Kubernetes Job.

2. Create and Schedule Flux Tasks:

   • Include a ConfigMap to define the Flux script (task.flux) for data aggregation.
   • Add a Kubernetes Job to create tasks based on the script and link them to the aggregated buckets.

3. Support for Performance Optimization:

   • Ensure the solution is designed to work seamlessly with Grafana dashboards, enabling them to query aggregated data efficiently and avoid timeouts or performance bottlenecks.

---

Key Requirements

1. New Fields in values.yaml:

   • createBucketsAndTask.enabled: Boolean flag to toggle the feature.
   • createBucketsAndTask.buckets: List of aggregated bucket names to create.
   • createBucketsAndTask.task: Task configuration, including name, schedule, and Flux script.

2. New Resources:

   • ConfigMap: Stores the Flux script to define the aggregation logic.
   • Kubernetes Job: Executes the logic to create buckets and tasks.

3. Conditional Deployment:

   • Ensure resources are deployed only when createBucketsAndTask.enabled is set to true.

---

Acceptance Criteria

• The Helm chart creates specified aggregated buckets during deployment.
• A Flux task is scheduled and populates the buckets with aggregated data.
• Grafana dashboards are able to query the aggregated data seamlessly, improving performance.
• The feature is optional and backward-compatible with existing deployments.

---

Steps to Implement

1. Modify values.yaml:

   • Add fields to configure bucket and task creation.

2. Add Templates:

   • Create a ConfigMap template for storing the Flux script (task-flux-config.yaml).
   • Define a Job template for bucket and task creation (create-buckets-and-task-job.yaml).

3. Update templates/_helpers.tpl:

   • Add helper functions to conditionally deploy the new resources.

4. Update Chart.yaml:

   • Include metadata for the new resources.

5. Update templates/deployment.yaml:

   • Ensure the new Job is deployed conditionally.

---

Benefits

1. Optimized Performance:

   • Preaggregated data significantly reduces query time in Grafana dashboards.
   • Avoids timeouts and improves reliability for long-range queries.

2. Automation:

   • Streamlines deployment by automating bucket and task creation.
   • Eliminates manual intervention, reducing setup errors.

3. Scalability:

   • Enables easy scaling of data aggregation workflows for production environments.

---

Risks and Mitigations

• Risk: The task creation Job may execute before InfluxDB is fully initialized.
  • Mitigation: Add a wait condition or retry mechanism in the Job.
• Risk: Misconfiguration of values.yaml may cause incorrect setup.
  • Mitigation: Add validation checks or default values to handle common errors.

---

Additional Context

This enhancement is critical to support Grafana dashboards that query aggregated data to improve performance, particularly for large datasets spanning months or years.

• Primary Use Case: Ensure Grafana dashboards querying aggregated buckets perform optimally without timing out.
• References:
  • [InfluxDB2 Helm Chart Documentation](https://github.com/influxdata/helm-charts/tree/master/charts/influxdb2)
  • [Flux Task Creation Documentation](https://docs.influxdata.com/influxdb/cloud/process-data/create-task/)
  • [Helm Job Documentation](https://helm.sh/docs/topics/charts_hooks/#post-install-hooks)

---

This issue aligns with performance optimization requirements and will significantly enhance the usability and scalability of the InfluxDB-Grafana ecosystem.

[davidpepm]

This issue blocks #62.