Switch to PEP 420 (Implicit namespace packages) #194
Comments
icemac
added
enhancement
|
This looks like a ton of work... but if we could really get rid of setuptools it would be worth it. Setuptools is nothing but trouble at this point. |
|
I left two comments in the setuptools discussion, with some numbers. Let's see if we can get a timeline for when setuptools will drop support. Zope and Plone can only move to native namespaces in a major release. Since Plone 6 was only released a few months ago, I expect it will take two to three years before we have Plone 7, which would need a Zope 6. We may need to speed that up. But I hope we can let this rest for another year. If we know that for example setuptools 70 is going to drop support, I suppose we could add |
|
+1 to start the process of modernization here. Some remarks:
|
|
The idea of getting rid of setuptools was not based on any knowledge of replacements, but on some comments in that setuptools discussion on GitHub. It may not be possible. Where I come from is endless issues with the Zope packages that use C-code in conjunction with macOS and the different builds that exist for it (x86_64, aarch64 and universal2). I described that at #181. |
|
I am not aware of an alternative to |
|
Request for clarification: is it possible to mix and match PEP-420 namespace packages with pkg_resources namespace packages from the same namespace in the same virtualenv? Are we talking about a flag day here, with major version bumps and coordinated releases for every zope.* package at the same time? (And similar efforts for zc.*, z3c.*, grok.*, collective.* namespaces)? |
|
Marius Gedminas wrote at 2023-3-8 23:31 -0800:
Request for clarification: is it possible to mix and match PEP-420 namespace packages with pkg_resources namespace packages from the same namespace in the same virtualenv?
As I read PEP-420 it does not provide a transistion path with
`pkg_resources` but it does for `pkgutil.extend_path` -- with
the limitation that in this case, `__path__` recomputation is not guaranteed
(maybe: will not happen).
It might be possible that `pkg_resources.declare_namespace` has
been extended in modern `setuptools` versions to handle
implicit namespace packages correctly.
However, I am not sure: implicit namespace packages make `__path__` read-only
which may make the task for `declare_namespace` much more difficult.
|
|
Marius Gedminas wrote at 2023-3-8 23:31 -0800:
Request for clarification: is it possible to mix and match PEP-420 namespace packages with pkg_resources namespace packages from the same namespace in the same virtualenv?
I have had a second thought.
When I understand PEP-420 correctly, then it modifies the import of
package *p* as follows:
* it looks for *p* on the current `__path__`
* if it finds a directory *p* with `__init__`, then *p*
is instantiated as a "normal" package
* only if all directories for *p* lack a `__init__`, it is
instantiated as a namespace package.
This would imply: either all directories for a namespace package
*p* lack an `__init__` or the existing `__init__`s must emulate
the namespace package treatment.
Accoding to PEP-420, `pkgutil.extend_path` can be expected to
partially emulate the namespace package treatment (with the
exception of support for subsequent `__path__` changes).
Modern `setuptools` versions might have a `pkg_resources.declare_namespace`
which does the same.
|
I did not read the PEPs in detail nor the code about it, but about a year ago I just tried it and it just did not work. Ago then, I found some source-on-the-internet stating exact: it does not mix (but currently I can not find it again). |
|
That would be the Python Packaging User Guide, which says:
It also gives one specific reason for using Do we need that for some reason? |
"It is very unlikely that the values of zip_safe will affect modern deployments that use pip for installing packages. Moreover, new users of setuptools should not attempt to create egg files using the deprecated build_egg command. Therefore, this flag is considered obsolete." https://setuptools.pypa.io/en/latest/deprecated/zip_safe.html |
|
for the I think the packaging tutorial docs may be incorrect as there's a test for pkg_resources and pkgutil namespaces in one virtual environment https://github.com/pypa/sample-namespace-packages/blob/df7530eeb8fa0cb7dbb8ecb28363e8e36bfa2f45/noxfile.py#L79-L88 which shows a staged migration is possible |
|
There's now work to test staged migration of namespace packages in pypa/sample-namespace-packages#22 referring to the table it looks like as long as you don't mind about not supporting |
|
Let's see if I understand it correctly. So "staged migration" means: migrate some packages in a namespace to pep420 while some others still use pkg_resources? For Zope/Plone in that table the lines near the bottom with "cross_pkg_resources_pep420" are the relevant ones. I don't see a difference in behavior for Python versions, so that is good. That is better than I had hoped for! It does make it a bit tricky in the transition period. Say you want to start with the |
|
Heads up: |
|
Btw., I wrote I small blog about the mix of old and new style packages in one namespace problem last year, while confronted with this problem while working on a different project: https://yenzenz.com/blog/2023/mixed_python_pep420/ |
|
Here's a vote for |
|
Nicholas Pilon wrote at 2024-6-8 09:18 -0700:
...
Are there other documented strange behaviors? If it’s just mixed editable installs it seems like the right approach is just upgrade and maybe have minimum versions in requirements on cross dependencies?
I have not yet worked with implicit namespace packages.
But I have seen serious problems with (explicit) namespace packages
when some subpackages have been installed via `pip`
and others via `zc.buildout`: strange import errors occured.
I failed to understand the problem details but I assume that
the path extension logic cannot cope with this situation.
The implicit namespace PEP speaks of a fullback for subpackages
not yet using implicit namespaces **BUT** warns against its use:
Normally, the `__path__` of an implicit namespace package
is automatically updated when the parent path changes.
However, the fallback changes `__path__` itself and this disables
future automatic updates of it. I fear that thereafter adding additional
implicit namespace subpackages to the parant path no longer
updates `__path__` and import errors are to be expected.
|
|
That’s actually exactly what I ran into trying to tinker together a simple solution for one zope subpackage yesterday so I think I have some idea about what’s going on; I’m hoping that starting with zc.buildout I can bypass that. |
|
Nicholas Pilon wrote at 2024-6-8 11:45 -0700:
That’s actually exactly what I ran into trying to tinker together a simple solution for one zope subpackage yesterday so I think I have some idea about what’s going on; I’m hoping that starting with zc.buildout I can bypass that.
I am less optimistic that `zc.buildout` alone can fully support
mixed namespace packages -- unless it transforms explicit
namespace packages to implicit ones on installation.
We will still have the problem of packages partly installed via `zc.buildout`
and partly externally (via the OS, via `pip`) -- but this problem
already exists in the "all explicit" world.
|
|
Does anyone already have a timeline in mind for when to starting work on moving all repos to native namespaces? With my Plone hat on, where we obviously need to do the same, I am inclined to say it is too early now. But somewhere between autumn 2025 and spring 2026 would sound good. That would be in time for a Plone 7 release in the second half of 2026. But this timeline involves looking into a crystal ball, and I am not sure mine is working. ;-) |
|
@mauritsvanrees After resolving the issue in our company that I am tired of the many deprecation warnings and want to do something about it. As this change is backwards in-compatible we might need to backport necessary code changes to the previous versions for a while. (with "we == the people who need specific backports") I think this approach would make it possible to get progress now for "early" adopters and the majority still can stay on the previous releases. |
|
Would it make sense to come up with a plan or a priority-based list of package names? I'm not afraid of the work, that's just simple drudgery. I am afraid if these updates are not coordinated well and all of a sudden things start breaking (like Zope) because some dependencies from the same namespace are converted and some are not. It feels like once you start on a namespace all of its packages should be completed in very quick succession. Also, we need to brace ourselves when complaints come flooding in from package maintainers that use e.g. |
|
Doing one namespace completely in short succession seems best. But as far as I know, the only real problem is still when you want to use a checkout or editable install of both a native and non-native package within the same namespace. For packages that have been released, it does not seem to matter. So it is more a problem for people developing on a namespace, than on integrators who just install a final package. But that is still hear-say, so that may need to be checked, both in pip and buildout. Since Plone also uses the |
This matches my experience - but note that it's going to be a problem for CI suites because |
This could be the reason why I saw this problem with |
|
@mauritsvanrees Yes we should do one namespace at a time. Especially the shared ones like |
You're right, I have run into this so many times and always wondered why. On all buildouts that still have some |
I'm game. As @npilon mentioned, we may have broken CI runs, but if we stick with just the minimum needed change set for each package and push/publish even if CI fails we should be able to get it done reasonably quickly. The C-based packages may present a special problem because the PyPI publishing requires the package to build and the tests to succeed. |
|
I'm also interested, and would like to help as well, though I'm a bit unreliable as of late due to personal/family circumstances... |
|
My understanding is that the standard python whl files, as you would upload to PyPI are fully backwards compatible with the legacy pkg_resources build. Could you default to pkg_resources mode and support py3+ namespace packages by using an environment variable to choose the mode. Then in the near future build all wheels with the new mode opted in, and in the long future default to opted in, and in the far future remove the compat pkg_resources mode? |
|
I have never heard of anyone making that package configuration depend on an environment variable. How would that even look like in practice? And why would that be in any way advantageous? |
|
It would mean anyone relying on building in editable mode from sdist or git would be unaffected |
|
The developers here have enough control over the namespaces to convert them quickly enough. There is no need for extra complication. We cannot prevent all problems, that is true, but IMHO the extra effort for such a selective switch is not worth the effort. The effort isn't clear, anyway, because you didn't answer my question how that would look like in practice (code!). |
|
Ah I didn't understand your question, I assumed you meant how it would appear from a user perspective |
|
Also looking at the migration table, you wouldn't need an environment variable at all - you can just check the current type of namespace package in use and use that, and for wheels on PyPI use pep420 |
|
This is all hand waving. From what I know the change involves removing an |
|
so in your setup.py you'd have something like (untested psuedo code): import importlib.resources
import setuptools.commands.build_py
import pathlib
import os
NAMESPACE = "zope"
def _check():
if os.environ.get("ZOPE_USE_PEP_420", ""):
return False
default_init = importlib.resources.files(NAMESPACE) / "__init__.py"
try:
if b"pkg_resources" in default_init.read_bytes():
return True
except FileNotFoundError:
return False
raise RuntimeError(
"pkgutil ns package in use"
) # probably possible to handle this too
USE_PKG_RESOURCES = _check()
def maybe_create_init_py():
if USE_PKG_RESOURCES:
(pathlib.Path(NAMESPACE) / "__init__.py").write_bytes(
b"__import__('pkg_resources').declare_namespace(__name__)"
)
class BuildPy(setuptools.commands.build_py.build_py):
def run(self):
# Base class code
if self.py_modules:
self.build_modules()
if self.packages:
self.build_packages()
self.build_package_data()
# Our modification!
maybe_create_init_py()
# Remaining base class code
self.byte_compile(self.get_outputs(include_bytecode=0))
extra_kwargs = (
{"cmdclass": {"build_py": BuildPy}, "namespace_packages": NAMESPACE}
if USE_PKG_RESOURCES
else {}
)
setuptools.setup(
...,
packages=setuptools.find_namespace_packages(...),
**extra_kwargs,
)then you'd set ZOPE_USE_PEP_420 in your github actions to build your whl that gets uploaded to pypi |
|
As an alternative we could set maximum version numbers for the dependencies to the current major release, cut a release. Afterwards the migration is done including requiring the next major version of the dependencies. code example for REQUIRES = [
'setuptools',
'zope.interface >= 5.0.0, <8',
'zope.event < 6',
]
TESTS_REQUIRE = [
'zope.i18nmessageid < 8',
'zope.testing < 6',
'zope.testrunner < 7',
]
setup(
name='zope.schema',
version='7.1.dev0',
...
namespace_packages=['zope', ],
install_requires=REQUIRES,
...
extras_require={
'docs': [
'Sphinx',
'repoze.sphinx.autointerface',
],
'test': TESTS_REQUIRE,
},
)These changes then get released as a feature release. after migration to PEP-420: REQUIRES = [
'setuptools',
'zope.interface >= 8',
'zope.event >= 6',
]
TESTS_REQUIRE = [
'zope.i18nmessageid >= 8',
'zope.testing >= 6',
'zope.testrunner >= 7',
]
setup(
name='zope.schema',
version='8.0.dev0',
...
install_requires=REQUIRES,
...
extras_require={
'docs': [
'Sphinx',
'repoze.sphinx.autointerface',
],
'test': TESTS_REQUIRE,
},
)This would be more effort but allows to do the migration even without merging after broken tests runs (besides circular dependencies). |
|
Setting the max versions could be started at any time as it does not break anything. Maybe it could even be (semi-)automated. |
|
With all due respect, that is a whole bunch of code required for exactly what gain? Like I said, except for |
Currently we use pkg_resources-style namespace packages. Using
pkg_resourcesis deprecated (see https://setuptools.pypa.io/en/latest/pkg_resources.html), it seems not yet clear when it is removed from setuptools.The current future are PEP 420: Implicit namespace packages.
This requires:
__init__.pyfiles existing to create a namespacenamespace_packagesdeclaration fromsetup.pysetuptools? (Maybe we could even switch topyproject.toml.)zopefoundationhosts the following namespaces:See the discussion in pypa/setuptools#3434 for some more details.
Suggestion:
zopefoundation(grokcore?, z3c?, zope.app?)zc.buildout), adapt the other namespaces.zopefoundationbut using the same namespaces also have to switch to PEP 420.Cc-ing: @dataflake @mauritsvanrees @gforcada @mgedmin @jensens
See also https://packaging.python.org/en/latest/guides/packaging-namespace-packages/
The text was updated successfully, but these errors were encountered: