fleetctl --endpoint http://x.x.x.x:4001 journal -f never closes ssh connection

[andymotta]

Hello FleetUI Contributors,

First, thank you for your hard work and excellent product. Our developers really appreciate fleet-ui.

We noticed that journal -f ssh connections are not closed upon closing of the websocket. This leaves a number of ssh tunnels connected to the requested host, and eventually exhausts all available incoming ssh connections on said host.

Every time we bring up the journal these tunnels are created, and never terminated. If we look at the same journal 10 times we create 10 forever connections.
Inside container:
10932 13 root S 11820 1% 0% fleetctl --endpoint http://172.x.x.x:4001 journal -f [email protected]
12750 13 root S 11820 1% 0% fleetctl --endpoint http://172.x.x.x:4001 journal -f [email protected]

After a while (~104 requests to the same host), we are unable to ssh to the requested host at all:
➜ ~ ssh [email protected]
ssh_exchange_identification: Connection closed by remote host
➜ git ssh [email protected]
ssh_exchange_identification: read: Connection reset by peer

After bringing the socket-activated sshd service back up, we see this in the service journal: Jun 03 23:29:46 ip-172-x-x-x.us-west-2.compute.internal systemd[1]: sshd.socket: Too many incoming connections (64)

Is this because of the --follow nature of the connection?

[djmaze]

I think this is a bug. The connections should be terminated as soon as the log overlay is closed.

[hauptmedia]

I ran into this problem today and I'm now unable to login via ssh on some nodes :(

[andymotta]

Hey Julian,
When that happens we've been restarting the fleet-ui container which releases the connections, has been the easiest workaround so far.

[hauptmedia]

This works as workaround. Thank you for the tip!