How can I make Lightstep work for \basic_phoenix_ecto

[cpppd]

I uncommented lines for Lightstep, followed the instruction (opentelemetry-erlang-contrib/examples/basic_phoenix_ecto/README.md), to run the project, but I still cannot see any service on Lightstep UI. The changes are as follows, what do I miss to make it work for Lightstep? Thank you for the help.

(I tried open-telemetry/opentelemetry-demo and it works, all services show on lightstep, but I cannot find out the key difference to make open-telemetry/opentelemetry-erlang-contrib work for lightstep)

I check the otlp container, and find error message like this

error   exporterhelper/queued_retry.go:401      
Exporting failed. The error is not retryable. Dropping data.     
{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "Permanent error: rpc error: code = InvalidArgument desc = Invalid runtime in Report (missing or empty service name)", "dropped_items": 1}

otel    | go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
otel    |   go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:401
otel    | go.opentelemetry.io/collector/exporter/exporterhelper.(*tracesExporterWithObservability).send
otel    |   go.opentelemetry.io/[email protected]/exporter/exporterhelper/traces.go:137
otel    | go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
otel    |   go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:205
otel    | go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
otel    |   go.opentelemetry.io/[email protected]/exporter/exporterhelper/internal/bounded_memory_queue.go:61

On jaeger UI, the Service dropdown list always shows the service OTLPResourceNoServiceName which is expected to be demo.

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
      http:
processors:
  batch:
    send_batch_size: 1024
    timeout: 5s
exporters:
  otlp:
    # Uncomment below for using Honeycomb
    # endpoint: "api.honeycomb.io:443"
    # headers:
    #   "x-honeycomb-team": "<YOUR_ACCESS_TOKEN>"
    #   "x-honeycomb-dataset": "<YOUR_DATASET_NAME>"
    # Uncomment below for using Lightstep
    endpoint: ingest.lightstep.com:443
    headers:
      "lightstep-access-token": "my_token_ewfzj......"
  zipkin:
    endpoint: "http://zipkin:9411/api/v2/spans"
    tls:
      insecure: true
  jaeger:
    endpoint: "jaeger:14250"
    tls:
      insecure: true
extensions:
  zpages:
    endpoint: :55679
service:
  extensions: [zpages]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      # exporters: [jaeger, zipkin]
      # Add 'otlp' if you're exporting to Honeycomb
      # or Lightstep
      exporters: [zipkin, jaeger, otlp]

runtime.exs

# config/runtime.exs is executed for all environments, including
# during releases. It is executed after compilation and before the
# system starts, so it is typically used to load production configuration
# and secrets from environment variables or elsewhere. Do not define
# any compile-time configuration in here, as it won't be applied.
# The block below contains prod specific runtime configuration.

# Start the phoenix server if environment is set and running in a release
if System.get_env("PHX_SERVER") && System.get_env("RELEASE_NAME") do
  config :demo, DemoWeb.Endpoint, server: true
end

if config_env() == :prod do
  database_url =
    System.get_env("DATABASE_URL") ||
      raise """
      environment variable DATABASE_URL is missing.
      For example: ecto://USER:PASS@HOST/DATABASE
      """

  maybe_ipv6 = if System.get_env("ECTO_IPV6"), do: [:inet6], else: []

  config :demo, Demo.Repo,
    # ssl: true,
    url: database_url,
    pool_size: String.to_integer(System.get_env("POOL_SIZE") || "10"),
    socket_options: maybe_ipv6

  # The secret key base is used to sign/encrypt cookies and other secrets.
  # A default value is used in config/dev.exs and config/test.exs but you
  # want to use a different value for prod and you most likely don't want
  # to check this value into version control, so we use an environment
  # variable instead.
  secret_key_base =
    System.get_env("SECRET_KEY_BASE") ||
      raise """
      environment variable SECRET_KEY_BASE is missing.
      You can generate one by calling: mix phx.gen.secret
      """

  host = System.get_env("PHX_HOST") || "example.com"
  port = String.to_integer(System.get_env("PORT") || "4000")

  config :demo, DemoWeb.Endpoint,
    url: [host: host, port: 443],
    http: [
      # Enable IPv6 and bind on all interfaces.
      # Set it to  {0, 0, 0, 0, 0, 0, 0, 1} for local network only access.
      # See the documentation on https://hexdocs.pm/plug_cowboy/Plug.Cowboy.html
      # for details about using IPv6 vs IPv4 and loopback vs public addresses.
      ip: {0, 0, 0, 0, 0, 0, 0, 0},
      port: port
    ],
    secret_key_base: secret_key_base

  config :opentelemetry, :processors,
    otel_batch_processor: %{
      # Using `otel` here since we are starting through docker-compose where
      # otel refer to the hostname of the OpenCollector,
      #
      # If you are running it locally, kindly change it to the correct
      # hostname such as `localhost`, `0.0.0.0` and etc.
      exporter: {:opentelemetry_exporter, %{endpoints: ["http://otel:4318"]}}

      # Example configuration to directly export to Honeycomb.io
      # exporter: {
      #   :opentelemetry_exporter, %{
      #     endpoints: ["https://api.honeycomb.io:443"],
      #     headers: [
      #       {"x-honeycomb-team", System.get_env("HONEYCOMB_TEAM")},
      #       {"x-honeycomb-dataset", System.get_env("HONEYCOMB_DATASET") || "demo"}
      #     ]
      #   }
      # }

      # Example configuration to directly export to Lightstep.com
      #
      # Since Lightstep has a different OTLP trace endpoint, we are going need
      # to configure it in a separate configuration.
      # exporter: {
      #   :opentelemetry_exporter, %{}
      # }
    }

  # Example configuration for Lightstep.com, for more refers to:
  # https://github.com/open-telemetry/opentelemetry-erlang/tree/main/apps/opentelemetry_exporter#application-environment
  config :opentelemetry_exporter,
    # You can also configure the compression type for exporting traces.
    oltp_traces_compression: :gzip,
    otlp_traces_endpoint: "https://ingest.lightstep.com:443/traces/otlp/v0.9",
    otlp_headers: [
      {"lightstep-access-token", System.get_env("LIGHTSTEP_ACCESS_TOKEN")}
    ]

  # ## Using releases
  #
  # If you are doing OTP releases, you need to instruct Phoenix
  # to start each relevant endpoint:
  #
  #     config :demo, DemoWeb.Endpoint, server: true
  #
  # Then you can assemble a release by calling `mix release`.
  # See `mix help release` for more information.

  # ## Configuring the mailer
  #
  # In production you need to configure the mailer to use a different adapter.
  # Also, you may need to configure the Swoosh API client of your choice if you
  # are not using SMTP. Here is an example of the configuration:
  #
  #     config :demo, Demo.Mailer,
  #       adapter: Swoosh.Adapters.Mailgun,
  #       api_key: System.get_env("MAILGUN_API_KEY"),
  #       domain: System.get_env("MAILGUN_DOMAIN")
  #
  # For this example you need include a HTTP client required by Swoosh API client.
  # Swoosh supports Hackney and Finch out of the box:
  #
  #     config :swoosh, :api_client, Swoosh.ApiClient.Hackney
  #
  # See https://hexdocs.pm/swoosh/Swoosh.html#module-installation for details.
end

.env

DATABASE_URL=ecto://postgres:postgres@postgres/demo_dev
SECRET_KEY_BASE=J2yvvaGbOmvsFVOnzgdWzGJtCbgSOC7aFhWziMgEFnT1T2CiWE0a67FZw8i87yRS
PHX_SERVER=true
PHX_HOST=localhost
OTEL_SERVICE_NAME=demo

# If you are exporting to the following 3rd party service,
# do remember to update the following:
# HONEYCOMB_TEAM=your_honeycomb_team
# HONEYCOMB_DATASET=your_honeycomb_dataset
LIGHTSTEP_ACCESS_TOKEN=my_token_ewfzj......

[tsloughter]

I see you have zipkin and jaeger still in the collector's pipeline. Are you seeing traces in those services? Just making sure it is the export to lightstep that is at issue and no a more general issue of traces from the Phoenix app.

[cpppd]

Yes, I've seen traces in zipkin and jaeger (but on jaeger UI Service dropdown menu shows OTLPResourceNoServiceName ), and the Phoenix app works well. But the lightstep UI shows nothing.

[cpppd]

I check the otlp container, and find error message like this.

error   exporterhelper/queued_retry.go:401      
Exporting failed. The error is not retryable. Dropping data.     
{"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "Permanent error: rpc error: code = InvalidArgument desc = Invalid runtime in Report (missing or empty service name)", "dropped_items": 1}

I assume the error message is for lightstep.

On jaeger UI, the Service dropdown list always shows the service OTLPResourceNoServiceName which is expected to be demo. Except for this issue, it looks fine.

[bryannaegele]

There's a mixed configuration happening here. If you're using the collector, nothing needs to be changed in your app config files, just the update to the collector.

[cpppd]

Thank you for pointing out the issue. However, I discard changed in runtime.exs to use collector, lightstep remains empty. I comment out the service of otel in docker-compose.yaml and comment in the following codes in runtime.ex to try not use the collector, but nonthing changes in lightstep.

config :opentelemetry_exporter,
    # You can also configure the compression type for exporting traces.
    oltp_traces_compression: :gzip,
    otlp_traces_endpoint: "https://ingest.lightstep.com:443/traces/otlp/v0.9",
    otlp_headers: [
      {"lightstep-access-token", System.get_env("LIGHTSTEP_ACCESS_TOKEN")}
    ]