Using Prefect Cloud with proxies

[zzstoatzz]

Note

This was originally posted elsewhere in Oct. 2022, but should generally apply as of this discussion's creation. feel free to comment here with any proposed updates

Proxies intermediate network requests between a server and a client. They are typically found in the locked down networks of large enterprises, and are useful in a variety of privacy and security contexts.

In order to communicate with Prefect Cloud, the Prefect client library makes HTTPS requests. These requests are made using the httpx and aiohttp libraries, both of which respect accepted proxy environment variables. Because of this, the Prefect client works with proxies. Simply set the HTTPS_PROXY and SSL_CERT_FILE environment variables as appropriate and things should “just work.”

Demonstration

In order to test that this was true, we set up two EC2 instances, both of which were running Ubuntu 22.04 LTS. The first instance was connected to the internet in a public subnet. The second instance was placed in a private subnet with no assigned IP address and thus had no internet connection.

On the first instance, we downloaded and set up mitmproxy,

wget https://snapshots.mitmproxy.org/8.1.1/mitmproxy-8.1.1-linux.tar.gz

tar xvf mitmproxy-8.1.1-linux.tar.gz

./mitmproxy
With mitmproxy running, on the second instance we installed Prefect and then logged into cloud,

mkdir ~/.mitmproxy
scp user@proxy-server:~/mitmproxy/* ~/.mitmproxy/

export HTTPS_PROXY=proxy-server:port
export SSL_CERT_FILE=~/.mitmproxy/mitmproxy-ca-cert.pem
export REQUESTS_CA_BUNDLE=~/.mitmproxy/mitmproxy-ca-cert.pem

curl https://bootstrap.pypa.io/get-pip.py > get-pip.py
python3 get-pip.py
pip3 install prefect

prefect version
prefect cloud login

[jakekaplan]

Prefect communicates with the server (self hosted or cloud) over regular http requests and websockets. As of this PR, Prefect supports connecting over an http proxy for websocket communication as well.

To enable communication over a proxy, set HTTP_PROXY/HTTPS_PROXY and prefect will respect the setting for both http requests and websockets.

Note: Your proxy will likely need explicit configuration updates to allow websocket connections. See here for an example with squid.