Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install postgres in Dockerfile for psycopg2 to build correctly #205

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

enricorotundo
Copy link
Collaborator

Fixes #204.
Read this msg on Discord for context.

@enricorotundo enricorotundo changed the title fix: install postgres in Dockerfile Install postgres in Dockerfile for psycopg2 to build correctly Feb 13, 2025
Copy link
Contributor

@moarshy moarshy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this on my ec2 and its failing (testing Dockerfile-node). Will take a deeper look and update.

node-app       |   File "/app/node/storage/db/init_db.py", line 55, in <module>
node-app       |     init_db()
node-app       |   File "/app/node/storage/db/init_db.py", line 23, in init_db
node-app       |     engine = create_engine(LOCAL_DB_URL)
node-app       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
node-app       |   File "<string>", line 2, in create_engine
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
node-app       |     return fn(*args, **kwargs)  # type: ignore[no-any-return]
node-app       |            ^^^^^^^^^^^^^^^^^^^
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/engine/create.py", line 602, in create_engine
node-app       |     dbapi = dbapi_meth(**dbapi_args)
node-app       |             ^^^^^^^^^^^^^^^^^^^^^^^^
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 696, in import_dbapi
node-app       |     import psycopg2
node-app       |   File "/app/.venv/lib/python3.12/site-packages/psycopg2/__init__.py", line 51, in <module>
node-app       |     from psycopg2._psycopg import (                     # noqa
node-app       | ImportError: libpq.so.5: cannot open shared object file: No such file or directory
node-app       | + poetry run python -m node.storage.db.init_db
node-app       | Traceback (most recent call last):
node-app       |   File "<frozen runpy>", line 198, in _run_module_as_main
node-app       |   File "<frozen runpy>", line 88, in _run_code
node-app       |   File "/app/node/storage/db/init_db.py", line 55, in <module>
node-app       |     init_db()
node-app       |   File "/app/node/storage/db/init_db.py", line 23, in init_db
node-app       |     engine = create_engine(LOCAL_DB_URL)
node-app       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
node-app       |   File "<string>", line 2, in create_engine
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
node-app       |     return fn(*args, **kwargs)  # type: ignore[no-any-return]
node-app       |            ^^^^^^^^^^^^^^^^^^^
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/engine/create.py", line 602, in create_engine
node-app       |     dbapi = dbapi_meth(**dbapi_args)
node-app       |             ^^^^^^^^^^^^^^^^^^^^^^^^
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 696, in import_dbapi
node-app       |     import psycopg2
node-app       |   File "/app/.venv/lib/python3.12/site-packages/psycopg2/__init__.py", line 51, in <module>
node-app       |     from psycopg2._psycopg import (                     # noqa
node-app       | ImportError: libpq.so.5: cannot open shared object file: No such file or directory

@@ -15,11 +15,16 @@ RUN apt-get install curl gcc curl git -y

# add conda install to path; use base environment
ENV PATH="/opt/conda/bin:${PATH}"
RUN conda create -n node python=3.12
RUN conda create -y -n node python=3.12
RUN echo "source activate node" > /root/.bashrc
ENV PATH="/opt/conda/envs/node/bin:$PATH"
RUN echo
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe a typo

RUN echo "source activate node" > /root/.bashrc
ENV PATH="/opt/conda/envs/node/bin:$PATH"
RUN echo

# install postgres (required for poetry to build psycopg2 from source)
RUN conda install -y conda-forge::postgresql=17.2

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like LDFLAGS and CPPFLAGS are missing?

@enricorotundo
Copy link
Collaborator Author

@moarshy Can you verify if pg_config binary is in PATH?

@moarshy
Copy link
Contributor

moarshy commented Feb 14, 2025

@moarshy Can you verify if pg_config binary is in PATH?

RUN apt-get install gcc curl git libpq-dev -y # Added libpq-dev here

this is the only change we would need to get it working

@K-Mistele
Copy link
Collaborator

I tested this on my ec2 and its failing (testing Dockerfile-node). Will take a deeper look and update.

node-app       |   File "/app/node/storage/db/init_db.py", line 55, in <module>
node-app       |     init_db()
node-app       |   File "/app/node/storage/db/init_db.py", line 23, in init_db
node-app       |     engine = create_engine(LOCAL_DB_URL)
node-app       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
node-app       |   File "<string>", line 2, in create_engine
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
node-app       |     return fn(*args, **kwargs)  # type: ignore[no-any-return]
node-app       |            ^^^^^^^^^^^^^^^^^^^
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/engine/create.py", line 602, in create_engine
node-app       |     dbapi = dbapi_meth(**dbapi_args)
node-app       |             ^^^^^^^^^^^^^^^^^^^^^^^^
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 696, in import_dbapi
node-app       |     import psycopg2
node-app       |   File "/app/.venv/lib/python3.12/site-packages/psycopg2/__init__.py", line 51, in <module>
node-app       |     from psycopg2._psycopg import (                     # noqa
node-app       | ImportError: libpq.so.5: cannot open shared object file: No such file or directory
node-app       | + poetry run python -m node.storage.db.init_db
node-app       | Traceback (most recent call last):
node-app       |   File "<frozen runpy>", line 198, in _run_module_as_main
node-app       |   File "<frozen runpy>", line 88, in _run_code
node-app       |   File "/app/node/storage/db/init_db.py", line 55, in <module>
node-app       |     init_db()
node-app       |   File "/app/node/storage/db/init_db.py", line 23, in init_db
node-app       |     engine = create_engine(LOCAL_DB_URL)
node-app       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
node-app       |   File "<string>", line 2, in create_engine
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
node-app       |     return fn(*args, **kwargs)  # type: ignore[no-any-return]
node-app       |            ^^^^^^^^^^^^^^^^^^^
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/engine/create.py", line 602, in create_engine
node-app       |     dbapi = dbapi_meth(**dbapi_args)
node-app       |             ^^^^^^^^^^^^^^^^^^^^^^^^
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 696, in import_dbapi
node-app       |     import psycopg2
node-app       |   File "/app/.venv/lib/python3.12/site-packages/psycopg2/__init__.py", line 51, in <module>
node-app       |     from psycopg2._psycopg import (                     # noqa
node-app       | ImportError: libpq.so.5: cannot open shared object file: No such file or directory

which EC2 AMI are you using? Amazon Linux boxes are notorious for having missing C libraries compared to other distros like debian and ubuntu.

@moarshy
Copy link
Contributor

moarshy commented Feb 15, 2025

I tested this on my ec2 and its failing (testing Dockerfile-node). Will take a deeper look and update.

node-app       |   File "/app/node/storage/db/init_db.py", line 55, in <module>
node-app       |     init_db()
node-app       |   File "/app/node/storage/db/init_db.py", line 23, in init_db
node-app       |     engine = create_engine(LOCAL_DB_URL)
node-app       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
node-app       |   File "<string>", line 2, in create_engine
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
node-app       |     return fn(*args, **kwargs)  # type: ignore[no-any-return]
node-app       |            ^^^^^^^^^^^^^^^^^^^
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/engine/create.py", line 602, in create_engine
node-app       |     dbapi = dbapi_meth(**dbapi_args)
node-app       |             ^^^^^^^^^^^^^^^^^^^^^^^^
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 696, in import_dbapi
node-app       |     import psycopg2
node-app       |   File "/app/.venv/lib/python3.12/site-packages/psycopg2/__init__.py", line 51, in <module>
node-app       |     from psycopg2._psycopg import (                     # noqa
node-app       | ImportError: libpq.so.5: cannot open shared object file: No such file or directory
node-app       | + poetry run python -m node.storage.db.init_db
node-app       | Traceback (most recent call last):
node-app       |   File "<frozen runpy>", line 198, in _run_module_as_main
node-app       |   File "<frozen runpy>", line 88, in _run_code
node-app       |   File "/app/node/storage/db/init_db.py", line 55, in <module>
node-app       |     init_db()
node-app       |   File "/app/node/storage/db/init_db.py", line 23, in init_db
node-app       |     engine = create_engine(LOCAL_DB_URL)
node-app       |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
node-app       |   File "<string>", line 2, in create_engine
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
node-app       |     return fn(*args, **kwargs)  # type: ignore[no-any-return]
node-app       |            ^^^^^^^^^^^^^^^^^^^
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/engine/create.py", line 602, in create_engine
node-app       |     dbapi = dbapi_meth(**dbapi_args)
node-app       |             ^^^^^^^^^^^^^^^^^^^^^^^^
node-app       |   File "/app/.venv/lib/python3.12/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 696, in import_dbapi
node-app       |     import psycopg2
node-app       |   File "/app/.venv/lib/python3.12/site-packages/psycopg2/__init__.py", line 51, in <module>
node-app       |     from psycopg2._psycopg import (                     # noqa
node-app       | ImportError: libpq.so.5: cannot open shared object file: No such file or directory

which EC2 AMI are you using? Amazon Linux boxes are notorious for having missing C libraries compared to other distros like debian and ubuntu.

I am on ubuntu. Does it matter though? since its failing in the docker? Anyways, it was just missing a single lib libpq-dev. Got it to work by installing the missing library in the docker.
RUN apt-get install gcc curl git libpq-dev -y # Added libpq-dev here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

Docker build is broken: Error: pg_config executable not found.
3 participants