add docker-compose command to developer docs for running integration tests

[mjperrone]

@alexdemeo gave this to me to help me run the integration tests locally when implementing #414

I assume he grabbed the docker config from here. I grabbed the RECAP_URLS value from here .

Admittedly I did not run all the integration tests as that was too long (just RECAP_URLS='["postgresql://postgres:password@localhost:5432/testdb"]' pdm run pytest tests/integration/clients/test_postgresql.py but this should be closer to describing how to all the int tests.

[criccomini]

Nice! This looks good, but I think we should include the other services as well. I ran the integration-tests section through ChatGPT and got:

version: '3.8'

services:
  postgres:
    image: postgres:14
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_DB: testdb
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD", "pg_isready"]
      interval: 10s
      timeout: 5s
      retries: 5

  mysql:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: testdb
      MYSQL_USER: mysql
      MYSQL_PASSWORD: password
    ports:
      - "3306:3306"
    healthcheck:
      test: ["CMD", "mysqladmin", "ping", "--silent"]
      interval: 10s
      timeout: 5s
      retries: 5

  bigquery:
    image: ghcr.io/criccomini/bigquery-emulator:0.4.3-envvar
    environment:
      BIGQUERY_EMULATOR_PROJECT: test_project
      BIGQUERY_EMULATOR_DATASET: test_dataset
    ports:
      - "9050:9050"

  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
    healthcheck:
      test: ["CMD", "echo", "ruok", "|", "nc", "localhost", "2181"]
      interval: 10s
      timeout: 5s
      retries: 5

  kafka:
    image: confluentinc/cp-kafka:latest
    ports:
      - "9092:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    depends_on:
      - zookeeper
    healthcheck:
      test: ["CMD", "kafka-topics", "--bootstrap-server", "localhost:9092", "--list"]
      interval: 10s
      timeout: 5s
      retries: 5

  schema-registry:
    image: confluentinc/cp-schema-registry:latest
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_LISTENERS: http://schema-registry:8081
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092
    depends_on:
      - kafka
      - zookeeper
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://schema-registry:8081/"]
      interval: 10s
      timeout: 5s
      retries: 5

  hive-metastore:
    image: ghcr.io/criccomini/hive-metastore-standalone:latest
    ports:
      - "9083:9083"

Would you mind testing this out? I don't have docker on my computer. We might want to change 3.8 to 3.9--not sure.

Might have to drop the ZK health check, too test: ["CMD", "echo", "ruok", "|", "nc", "localhost", "2181"] . Not sure if compose is going to like the vertical pipe.

[mjperrone]

Would you mind testing this out?

It's valid config and up resulted in containers running seemingly successfully.

Then running the int tests:

tests/integration/test_cli.py ........s.                                                                                                  [ 25%]
tests/integration/clients/test_bigquery.py .x...xx.                                                                                       [ 45%]
tests/integration/clients/test_confluent_registry.py ....                                                                                 [ 55%]
tests/integration/clients/test_fs.py ...                                                                                                  [ 62%]
tests/integration/clients/test_hive_metastore.py EEE                                                                                      [ 70%]
tests/integration/clients/test_mysql.py ..                                                                                                [ 75%]
tests/integration/clients/test_postgresql.py ...                                                                                          [ 82%]
tests/integration/server/test_gateway.py .....s.                                                                                          [100%]
...
                                      message='TSocket read 0 bytes')
E           thrift.transport.TTransport.TTransportException: TSocket read 0 bytes

.venv/lib/python3.10/site-packages/thrift/transport/TSocket.py:166: TTransportException

Looks like that's happening here.

I don't have docker on my computer.

Are you only running the integration tests through github actions? 😅

[criccomini]

Are you only running the integration tests through github actions? 😅

Hah, nah. I just run the infra locally. Most of it is just JVMs, so easy enough (and I already had PG/MySQL via brew). 😈

But I've caved and installed Docker. I just ran the integration tests using the docker-compose.yml I posted. It worked! I suspect your issue is because you're having a port conflict on 9083 or some CA cert issue maybe (that's what popped up with cursory Google). Given that mine is passing just fine, want to update the PR with my YML and I'll merge it?

EDIT: Updated the PR myself. Merging!