PyNUTClient needs telnetlib which is deprecated

[jimklimov]

Raised in #2181 discussion - after Python-3.13 we will need another implementation for the layer of text-based interaction over TCP in Python.

[aquette]

side notes: while playing with Home Assistant NUT addon, I'm thinking about improving a bit pyNUT... :D
scan, device dump esp., possibly the cmd cookies (not sure it supports...)
cheers @jimklimov ;)

[Mausy5043]

Although I currently have no time to work on this (again), I'd like to point anyone who does to this repo. I've deprecated it a month or so ago. But the implementation of pexpect to replace telnetlib was completed in that repo. Hope this helps.

[opoplawski]

We just started development on Fedora 41 which will ship with Python 3.13. So it would be nice to see this addressed relatively soon. The above comment seems promising.

Also FWIW:
According to https://docs.python.org/3.13/whatsnew/3.13.html:
PEP 594: Remove the telnetlib module, deprecated in Python 3.11: use the projects telnetlib3 or Exscript instead.
(Contributed by Victor Stinner in gh-104773.)

[jimklimov]

Took some time to investigate telnetlib3, but it seems too great a paradigm change.

With original PyNUT code, we open a client connection to the NUT server once, and in different methods post this or that request and parse replies.

With telnetlib3+asyncio, it seems that each interaction must be a completely (pre-)defined I/O loop, so a separate connection, login, query, response, exit for each bit of work. Lots of avoidable extra server stress too, I guess. Also it seems to need separate annotations for the methods, so it is not as easy as "if mode==1 ... elif mode==3..."

Libs like pexpect seem to require actually calling a telnet program (or equivalent like netcat) and tap into its stdin/stdout, much like original expect. This limits portability (not all OSes have these clients pre-installed) and performance.

[jimklimov]

I won't be able to dedicate much time to it, so PRs are welcome from someone who knows modern Python (and the other alternative libs) better :)

Ideally, with dual-targeted code (so it runs on old systems too), but a separate same-API module like PyNUT3 is also an option if not avoidable... (make check-NIT can take care of ensuring API compatibility in either case)

[jimklimov]

It seems that one recommended approach per https://discuss.python.org/t/pep-594-take-2-removing-dead-batteries-from-the-standard-library/13508/59 is to just copy telnetlib.py from a distro which shipped one, and use as part of the consuming project that is hard to update otherwise. At least, this can be our first step in the area (until someone comes up with a more modern port).

Although examples at one of the recommended replacements, https://github.com/knipknap/exscript/ / https://exscript.readthedocs.io/en/latest/ seem to be conceptually similar to what PyNUT worked like.

[AlaBouali]

hello guys, please try my alternative to telnetlib:

https://github.com/AlaBouali/xtelnet

This is an easy to use telnet module to interact with a remote system smoothly over this protocol! It is a very minimalistic alterative to "telnetlib". xtelnet is a powerful and user-friendly Python library designed for managing Telnet sessions with ease and efficiency. With its intuitive interface and robust functionality, xtelnet simplifies the process of interacting with Telnet servers, offering a range of features for seamless communication. xtelnet offers a comprehensive solution for Telnet communication, providing developers with the tools they need to effectively manage Telnet sessions and interact with remote systems. Whether you're a seasoned developer or new to Telnet protocols, xtelnet empowers you to achieve your goals efficiently and reliably.

Why should I use xtelnet?
Easy to use and stable
Simple Authentication mechanism
Compatible with almost all servers when it comes to authentication and executing the commands
Available Command line tool
Thread-safe: if the session is shared among threads to execute commands, the commands will be executed one by one
Supports running multiple sessions concurrently
Can connect simultaneously and run in parallel the same command on: single or some or all connected hosts
Allow reconnect after closing the connection
Allow escape ANSI characters
Grab banners
Available "ping" function to use if you want to keep the connection open
Supports SOCKS 4 / 5 proxies
Supports SSL
Supports sending JSON data

[opoplawski]

Just a heads up that Python 3.13b2 has now landed in Fedora Rawhide so NUT is failing to build.

[jimklimov]

@opoplawski : thanks. WDYT : would making a private copy of the telnetlib module from original recent Python distribution be a decent stop-gap solution (try/except to import locally if not available globally)? Their migration docs seem to suggest as much...

[opoplawski]

Yes, it would be acceptable. Though I fear that it might become the permanent solution...

[jimklimov]

Might... maybe a better one would be to just go socket programming (same as done in C) - we do not need the full power of telnet/ssh here...

[jimklimov]

With PR #2501 merged, we now have at least that, a stashed copy. Not perfect, so keeping this ticket open to track a desire for simpler code to work with that socket layer, but good enough to keep on walking.

@opoplawski : can you please check if this passes for builds on systems with "too new" a version of Python?

[opoplawski]

So, just applying that PR to 2.8.2 I get:

+ ./configure --build=x86_64-redhat-linux --host=x86_64-redhat-linux --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --runstatedir=/run --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-all --without-powerman --with-libltdl --without-wrap --with-cgi --with-python=/usr/bin/python3 --with-python3=/usr/bin/python3 --without-python2 --datadir=/usr/share/nut --with-user=nut --with-group=dialout --with-statepath=/run/nut --with-pidpath=/run/nut --with-altpidpath=/run/nut --sysconfdir=/etc/ups --with-cgipath=/var/www/nut-cgi-bin --with-drvpath=/usr/sbin --without-gpio --with-systemdsystemunitdir=/usr/lib/systemd/system --with-systemdshutdowndir=/lib/systemd/system-shutdown --with-pkgconfig-dir=/usr/lib64/pkgconfig --disable-static --with-udev-dir=/usr/lib/udev --libdir=/usr/lib64
checking for CONFIG_FLAGS... --build=x86_64-redhat-linux --host=x86_64-redhat-linux --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --runstatedir=/run --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-all --without-powerman --with-libltdl --without-wrap --with-cgi --with-python=/usr/bin/python3 --with-python3=/usr/bin/python3 --without-python2 --datadir=/usr/share/nut --with-user=nut --with-group=dialout --with-statepath=/run/nut --with-pidpath=/run/nut --with-altpidpath=/run/nut --sysconfdir=/etc/ups --with-cgipath=/var/www/nut-cgi-bin --with-drvpath=/usr/sbin --without-gpio --with-systemdsystemunitdir=/usr/lib/systemd/system --with-systemdshutdowndir=/lib/systemd/system-shutdown --with-pkgconfig-dir=/usr/lib64/pkgconfig --disable-static --with-udev-dir=/usr/lib/udev --libdir=/usr/lib64
...
checking if we can and should install NUT-Monitor desktop application... no: Missing some or all of these Python3 modules: 're,glob,codecs,PyQt5.uic,configparser' and some or all of these Python2 modules: 're,glob,codecs,gtk,gtk.glade,gobject,ConfigParser'
checking if we can and should install PyNUT module (note for warnings from python 3.11 and beyond: we have a fallback nut_telnetlib module just in case)... Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import telnetlib
ModuleNotFoundError: No module named 'telnetlib'
./configure: line 33905: cd: script/python/module: No such file or directory
configure: error: Prerequisites for PyNUT not found, can't install as required

Another comment would be to call out that presumably that file is under the Python license rather than the GPL.

If I try to build from git master I get an autoreconf failure:

configure.ac:5144: error: required file 'scripts/augeas/nutupsconf.aug.in' not found
configure.ac:5144: error: required file 'scripts/devd/nut-usb.conf.in' not found
configure.ac:5036: error: required file 'scripts/systemd/nut-common-tmpfiles.conf.in' not found
configure.ac:5144: error: required file 'scripts/udev/nut-usbups.rules.in' not found

This is with autoconf 2.72.

[jimklimov]

Thanks for testing, oddly this passed here so will check what happens. Probably I've not chopped enough pieces from the local python (some pyc binaries could remain?..)

Looking at the errors, I see some of my mistakes:

• script*s*/...
• and probably should prepend $srcdir equivalent for out-of-tree builds just in case

...and presumably yours for the build from git master: did you ./autogen.sh first?

[jimklimov]

Comments should have got addressed by #2505

[opoplawski]

That looks good to me. A release at this point would be very helpful for us. Thanks!

[jimklimov]

Thanks for the check-up! I have some fixes in progress to wrap up, but will try to not delay too much about a release then.

[Scroker]

Is it helpful if I start working on a Python library migration from telnetlib to asyncio? I am developing my own custom connection library for this application https://gitlab.com/mkswap/UPSMonitor, but if possible I would prefer to work on the official one.

[jimklimov]

If that ultimately works, why not.

So far the libs I looked at were quite aimed at proper (perhaps even interactive-ish) telnet/ssh communication sessions, with all the handling for terminal emulation, sending keypresses, etc.

What NUT bindings need is much smaller in scale, and can be seen in C and now C++ client lib implementations - open a socket, send/receive bytes in one true encoding, parse the replies into language types (dict/list/str/int/... here).

So overheads of switching to alternative libraries, balanced vs. the little time I have to spare and benefit of going from one thing that works to another that also works as much, proved to be a blocker for me - path of least resistance, ain't broken don't fix, bang for buck, and all that.

One other wish is for new NUT releases to keep working on older systems - some may have no Python3 builds packaged for them. Currently UI client code is split by tech (QT vs GTK) and language syntax nuances, but the library source works for both python 2 and 3. Ideally a rewrite would keep that capability (at least at import PyNUTClient level, even if that pulls one or the other implementation), as well as API to minimize surprises for other consumers - if it is suitable.

Can always start another version too though, but would cause headaches to adapt any downstreams...

[cgarz]

+1 about switching to simpler socket instead of telnet.

Why did PyNUTClient ever use a telnetlib at all? Even the nut docs themselves say:

upsd doesn’t speak telnet and will probably misunderstand your first request due to the extra junk in the buffer.
( https://networkupstools.org/docs/developer-guide.chunked/net-protocol.html )

I've been doing this myself in a small simple logger I threw together to log some values from my ups and haven't noticed any issues. Only been running a week though.

[jimklimov]

From my reading of sources - a legacy of historic choice. Just some layer to send and get strings, with no need for specifically telnet bells and whistles (terminals, escapes, etc.), without socket programming involved on the side of PyNUT(Client) developer. Also it was in core Python, so no extra deps needed either.

As PoCs tend to go, there are no temporary solutions - and so we're stuck with that "good-enough" for life.

PRs are still welcome to replace with some thinner layer that does the job.

[cgarz]

PRs are still welcome to replace with some thinner layer that does the job.

Would it be possible to just switch all the self.__srv_handler write to send in scripts/python/module/PyNUT.py.in?
From a quick glance I don't see much else that would be needed. Maybe a read_until function.

I'll try and remember to give this a test tomorrow.

[jimklimov]

No idea OTOH and no time to research in short-term future. Thanks for giving it a look :)

[cgarz]

Didn't have a chance to work on it yesterday but it seems the change really was as minimal as I guessed. At least based on my limited testing with no issues so far anyway. I have put it in PR #2792.

[jimklimov]

Fixed by PR #2792 thanks to @cgarz

[jimklimov]

FWIW, current state of the module should be uploaded to https://test.pypi.org/project/pynutclient/2.8.2.2132/ now, so can be tested as a standalone module.