Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jikkou - kafka config #517

Open
rubinaprusty opened this issue Jan 21, 2025 · 0 comments
Open

Jikkou - kafka config #517

rubinaprusty opened this issue Jan 21, 2025 · 0 comments
Labels
question Further information is requested

Comments

@rubinaprusty
Copy link

Hi Team,

I am trying to run Jikkou within docker image - streamthoughts/jikkou-api-server:main in mac and it is running fine. Now I would like to automate kafka topic creation by passing a config yml file -
My kafka cluster is running locally using localhost:9092.
I am using below yml file to run the topic to be created in kafka cluster -

apiVersion: io.streamthoughts.jikkou/v1beta1
kind: ResourceList
resources:
  # Kafka connection configuration
  - apiVersion: jikkou.io/v1beta1
    kind: KafkaConfiguration
    metadata:
      name: kafka-cluster
    spec:
      bootstrap.servers: "localhost:9092"
      security.protocol: PLAINTEXT
      
  # Define a Kafka topic
  - apiVersion: kafka.jikkou.io/v1beta1
    kind: KafkaTopic
    metadata:
      name: test-topic-20012025
    spec:
      partitions: 3
      replicas: 2
      config:
        cleanup.policy: "delete"
        retention.ms: "604800000"  # 7 days
        max.message.bytes: "1048576"

My docker file -

# Use the base image for Jikkou API Server
FROM streamthoughts/jikkou-api-server:main
COPY kafka-topics.yml /config/kafka-topics.yml
# Default command to apply Kafka topic configurations using Jikkou
CMD ["jikkou", "apply", "--files", "/config/kafka-topics.yml"]

Now I am able to run Jikkou on my machine but could not able to connect to the kafka cluster.

below is my admin config -

AdminClientConfig values: 
        auto.include.jmx.reporter = true
        bootstrap.controllers = []
        bootstrap.servers = [localhost:9092]
        client.dns.lookup = use_all_dns_ips
        client.id = 
        connections.max.idle.ms = 300000
        default.api.timeout.ms = 60000
        enable.metrics.push = true
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 30000
        retries = 2147483647
        retry.backoff.max.ms = 1000
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.connect.timeout.ms = null
        sasl.login.read.timeout.ms = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.login.retry.backoff.max.ms = 10000
        sasl.login.retry.backoff.ms = 100
        sasl.mechanism = GSSAPI
        sasl.oauthbearer.clock.skew.seconds = 30
        sasl.oauthbearer.expected.audience = null
        sasl.oauthbearer.expected.issuer = null
        sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
        sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
        sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
        sasl.oauthbearer.jwks.endpoint.url = null
        sasl.oauthbearer.scope.claim.name = scope
        sasl.oauthbearer.sub.claim.name = sub
        sasl.oauthbearer.token.endpoint.url = null
        security.protocol = PLAINTEXT
        security.providers = null
        send.buffer.bytes = 131072
        socket.connection.setup.timeout.max.ms = 30000
        socket.connection.setup.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = https
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.3
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.certificates = null
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS

10:27:02.791 [scheduled-executor-thread-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka version: 3.7.0
10:27:02.792 [scheduled-executor-thread-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka commitId: 2ae524ed625438c5
10:27:02.792 [scheduled-executor-thread-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1737455222791
10:27:02.794 [scheduled-executor-thread-1] INFO  i.s.j.k.internals.KafkaBrokersReady - Checking for Kafka to be ready. Expected broker(s): 1
10:27:32.799 [kafka-admin-client-thread | adminclient-52] INFO  o.a.k.c.a.i.AdminMetadataManager - [AdminClient clientId=adminclient-52] Metadata update failed
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: fetchMetadata
10:28:02.800 [kafka-admin-client-thread | adminclient-52] INFO  o.a.k.c.a.i.AdminMetadataManager - [AdminClient clientId=adminclient-52] Metadata update failed

I am able to create the same topic using cli command -

Please help me here what I am doing wrong - I am guessing my configuration is wrong or provide me a sample config and docker file.

@rubinaprusty rubinaprusty added the question Further information is requested label Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant