Improving troubleshooting for SigNoz

[pranay01]

We have created a repo with tools on how to easily debug your SigNoz installations - https://github.com/SigNoz/troubleshoot

Would love any any inputs on it in this discussion thread

[chrisbake-apl]

Getting some unexpected results.. I'm trying to auto instrument a PHP application

(base) ➜ apl-app git:(signoz-telemetry) ✗ ./troubleshoot checkEndpoint --endpoint=xxx.xxx.xxx.xxx:4317 2024-04-15T21:26:17.985-0400 INFO workspace/main.go:28 STARTING! 2024-04-15T21:26:17.985-0400 INFO checkEndpoint/checkEndpoint.go:41 checking reachability of SigNoz endpoint 2024-04-15T21:26:18.278-0400 INFO workspace/main.go:46 Successfully sent sample data to signoz ...

`2024-04-16T01:29:07.212Z info exporterhelper/retry_sender.go:177 Exporting failed. Will retry the request after interval.

{"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "error": "clickhouse: dateTime overflow. timestamp must be between 1900-01-01 00:00:00 and 2262-04-11 23:47:16", "interval": "27.279984461s"}

2024-04-16T01:29:34.512Z error clickhousetracesexporter/writer.go:385 Could not append span to batch: {"kind": "exporter", "data_type": "traces", "name": "clickhousetraces", "span": {"traceId": "", "spanId": "", "parentSpanId": "", "name": "TestingSpan", "durationNano": 0, "startTimeUnixNano": 11651379494838206464, "serviceName": "", "kind": 0, "statusCode": 0, "externalHttpMethod": "", "httpUrl": "", "httpMethod": "", "httpHost": "", "httpRoute": "", "httpCode": "", "msgSystem": "", "msgOperation": "", "externalHttpUrl": "", "component": "", "dbSystem": "", "dbName": "", "dbOperation": "", "peerService": "", "gRPCCode": "", "gRPCMethod": "", "rpcSystem": "", "rpcService": "", "rpcMethod": "", "responseStatusCode": "", "hasError": false, "errorID": "", "errorGroupID": "", "errorEvent": {"name": "", "timeUnixNano": 0, "isError": false, "attributeMap": "map[]"}, "traceModel": {"traceId": "", "spanId": "", "name": "TestingSpan", "durationNano": 0, "startTimeUnixNano": 11651379494838206464, "serviceName": "", "kind": 0, "statusCode": 0, "hasError": false, "references": [{"traceId": "", "spanId": "", "refType": "CHILD_OF"}], "tagMap": "map[]", "event": "[]"}, "event": "[]", "tagMap": "map[]"}, "error": "clickhouse: dateTime overflow. timestamp must be between 1900-01-01 00:00:00 and 2262-04-11 23:47:16"}

github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).writeModelBatch
/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:385
github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*SpanWriter).WriteBatchOfSpans
/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/writer.go:412
github.com/SigNoz/signoz-otel-collector/exporter/clickhousetracesexporter.(*storage).pushTraceData
/home/runner/work/signoz-otel-collector/signoz-otel-collector/exporter/clickhousetracesexporter/clickhouse_exporter.go:439
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesRequest).Export
/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:60
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:41
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/retry_sender.go:138
go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/traces.go:177
go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).start.func1
/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/queue_sender.go:126
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Start.func1
/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/exporterhelper/internal/bounded_memory_queue.go:52`

The timestamp sent is in nanoseconds, but the clickhouse receiver complains that it doesn't match a valid timestamp.

Signoz is running on a Docker VM running on Ubuntu 20.04.3 LTS

otel config yaml:

root@ip-172-31-45-99:/opt/signoz-docker/deploy/docker/clickhouse-setup# cat otel-collector-config.yaml
receivers:
  syslog:
    udp:
      listen_address: "0.0.0.0:54527"
    protocol: rfc3164
    location: UTC
    operators:
      - type: move
        from: attributes.message
        to: body
  tcplog/docker:
    listen_address: "0.0.0.0:2255"
    operators:
      - type: regex_parser
        regex: '^<([0-9]+)>[0-9]+ (?P<timestamp>[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\.[0-9]+)?([zZ]|([\+-])([01]\d|2[0-3]):?([0-5]\d)?)?) (?P<container_id>\S+) (?P<container_name>\S+) [0-9]+ - -( (?P<body>.*))?'
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      - type: move
        from: attributes["body"]
        to: body
      - type: remove
        field: attributes.timestamp
        # please remove names from below if you want to collect logs from them
      - type: filter
        id: signoz_logs_filter
        expr: 'attributes.container_name matches "^signoz-(logspout|frontend|alertmanager|query-service|otel-collector|otel-collector-metrics|clickhouse|zookeeper)"'
      - type: filter
        id: hotroad
        expr: 'attributes.container_name matches "^(hotrod|load-hotrod|superset|grafana|apl-admin-nginx|apl-admin|maildev_maildev|apl-heartbeat|bookstack|bookstack_db|grafana_prometheus|grafana_blackbox-exporter|graphite)"'
  opencensus:
    endpoint: 0.0.0.0:55678
  otlp/spanmetrics:
    protocols:
      grpc:
        endpoint: localhost:12345
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  jaeger:
    protocols:
      grpc:
        endpoint: 0.0.0.0:14250
      thrift_http:
        endpoint: 0.0.0.0:14268
      # thrift_compact:
      #   endpoint: 0.0.0.0:6831
      # thrift_binary:
      #   endpoint: 0.0.0.0:6832
  hostmetrics:
    collection_interval: 30s
    scrapers:
      cpu: {}
      load: {}
      memory: {}
      disk: {}
      filesystem: {}
      network: {}
  prometheus:
    config:
      global:
        scrape_interval: 60s
      scrape_configs:
        # otel-collector internal metrics
        - job_name: otel-collector
          static_configs:
          - targets:
              - localhost:8888
            labels:
              job_name: otel-collector


processors:
  batch:
    send_batch_size: 10000
    send_batch_max_size: 11000
    timeout: 10s
  signozspanmetrics/prometheus:
    metrics_exporter: prometheus
    latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s ]
    dimensions_cache_size: 100000
    dimensions:
      - name: service.namespace
        default: default
      - name: deployment.environment
        default: default
      # This is added to ensure the uniqueness of the timeseries
      # Otherwise, identical timeseries produced by multiple replicas of
      # collectors result in incorrect APM metrics
      - name: 'signoz.collector.id'
  # memory_limiter:
  #   # 80% of maximum memory up to 2G
  #   limit_mib: 1500
  #   # 25% of limit up to 2G
  #   spike_limit_mib: 512
  #   check_interval: 5s
  #
  #   # 50% of the maximum memory
  #   limit_percentage: 50
  #   # 20% of max memory usage spike expected
  #   spike_limit_percentage: 20
  # queued_retry:
  #   num_workers: 4
  #   queue_size: 100
  #   retry_on_failure: true
  resourcedetection:
    # Using OTEL_RESOURCE_ATTRIBUTES envvar, env detector adds custom labels.
    detectors: [env, system] # include ec2 for AWS, gcp for GCP and azure for Azure.
    timeout: 2s

extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  zpages:
    endpoint: 0.0.0.0:55679
  pprof:
    endpoint: 0.0.0.0:1777

exporters:
  clickhousetraces:
    datasource: tcp://clickhouse:9000/?database=signoz_traces
    docker_multi_node_cluster: ${DOCKER_MULTI_NODE_CLUSTER}
    low_cardinal_exception_grouping: ${LOW_CARDINAL_EXCEPTION_GROUPING}
  clickhousemetricswrite:
    endpoint: tcp://clickhouse:9000/?database=signoz_metrics
    resource_to_telemetry_conversion:
      enabled: true
  clickhousemetricswrite/prometheus:
    endpoint: tcp://clickhouse:9000/?database=signoz_metrics
  prometheus:
    endpoint: 0.0.0.0:8889
  # logging: {}

  clickhouselogsexporter:
    dsn: tcp://clickhouse:9000/
    docker_multi_node_cluster: ${DOCKER_MULTI_NODE_CLUSTER}
    timeout: 5s
    sending_queue:
      queue_size: 100
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888
  extensions:
    - health_check
    - zpages
    - pprof
  pipelines:
    traces:
      receivers: [jaeger, otlp]
      processors: [signozspanmetrics/prometheus, batch]
      exporters: [clickhousetraces]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [clickhousemetricswrite]
    metrics/generic:
      receivers: [hostmetrics]
      processors: [resourcedetection, batch]
      exporters: [clickhousemetricswrite]
    metrics/prometheus:
      receivers: [prometheus]
      processors: [batch]
      exporters: [clickhousemetricswrite/prometheus]
    metrics/spanmetrics:
      receivers: [otlp/spanmetrics]
      exporters: [prometheus]
    logs:
      receivers: [otlp, syslog]
      processors: [batch]
      exporters: [clickhouselogsexporter]

[nitishfy]

This is what I found from logs - clickhouse: dateTime overflow. timestamp must be between 1900-01-01 00:00:00 and 2262-04-11 23:47:16

[nitishfy]

Furthermore, did you get a chance to go through https://signoz.io/blog/opentelemetry-php/?