PyYaml-6.0.3 seems incompatible with Python 3.12

[AmritaChoudhary]

Starting today after the release of PyYaml-6.0.3, our CI/CD jobs have started to fail pointing to incompatibility between two. It worked fine after locking down PyYaml to 6.0.2. One of the error we received was,

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/rcp/scripts/build_gha/__main__.py", line 17, in <module>
    from .lib import Locality, Options
  File "/rcp/scripts/build_gha/lib.py", line 85, in <module>
    import yaml
  File "/usr/local/lib/python3.12/site-packages/yaml/__init__.py", line 13, in <module>
    from .cyaml import *
  File "/usr/local/lib/python3.12/site-packages/yaml/cyaml.py", line 7, in <module>
    from yaml._yaml import CParser, CEmitter
  File "yaml/_yaml.pyx", line 1, in init yaml._yaml
TypeError: Metaclasses with custom tp_new are not supported.

[nitzmahone]

PyYAML is heavily tested against CPython 3.8-3.14. At first blush, this looks like either a broken local build against an ancient version of Cython or a broken Python path that's picking up a mismatched version of PyYAML's C extension from another place.

Are you using a packaged PyYAML wheel from PyPI, one you built yourself, or something packaged by a distro?

[danielhedgecock]

This may be unrelated, but I ran into an issue trying to upload the Py312-manylinux version of PyYAML 6.0.3 (PyYAML-6.0.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl and the aarch wheel) to Packagecloud. I tried downloading from the github release page and using pip download

I was getting the following error:

repository: Python package parsing error. If this is a Python egg, ensure it was created with python [setup.py](http://setup.py/) bdist_egg.

Version 6.0.2 uploaded just fine.

[AmritaChoudhary]

@nitzmahone We are using a packaged PyYAML wheel from PyPI, following is from the logs which shows Cython as well as PyYAML version downloaded -

Collecting Cython>=0.29.24 (from -r scripts/build_gha/requirements.txt (line 1))
  Downloading cython-3.1.4-cp312-cp312-musllinux_1_2_x86_64.whl (3.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 39.0 MB/s eta 0:00:00
Collecting boto3>=1.26.151 (from -r scripts/build_gha/requirements.txt (line 2))
  Downloading boto3-1.40.39-py3-none-any.whl (139 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.3/139.3 kB 17.4 MB/s eta 0:00:00
Collecting PyYAML>=6.0 (from -r scripts/build_gha/requirements.txt (line 5))
  Downloading pyyaml-6.0.3-cp312-cp312-musllinux_1_2_x86_64.whl (790 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 

[nitzmahone]

@danielhedgecock If I had to guess, Packagecloud is upset about the package name casing on the filenames not being normalized- the mixed-case filenames you mentioned don't actually match what we uploaded to PyPI (except for 3.8- a whole other mess). PyYAML is much older than package name canonicalization, and canonicalizing the package name now on our end is risky. We might try to fix it for 7.0, but I'm worried about how many bugs we're going to find in other random ecosystem tools if we do.

Anyway, I'm guessing whatever download process you used munged those filenames- try lowercasing the package name in the filenames and see if it's happy with that- it's definitely a fully-standard package generated by cibuildwheel (not an egg), and both twine and PyPI are happy with its structure.

[danielhedgecock]

@danielhedgecock If I had to guess, Packagecloud is upset about the package name casing on the filenames not being normalized- the mixed-case filenames you mentioned don't actually match what we uploaded to PyPI (except for 3.8- a whole other mess). PyYAML is much older than package name canonicalization, and canonicalizing the package name now on our end is risky. We might try to fix it for 7.0, but I'm worried about how many bugs we're going to find in other random ecosystem tools if we do.

Anyway, I'm guessing whatever download process you used munged those filenames- try lowercasing the package name in the filenames and see if it's happy with that- it's definitely a fully-standard package generated by cibuildwheel (not an egg), and both twine and PyPI are happy with its structure.

Thank you for your response! That makes sense. I will say though that Packagecloud does not have an issue with version 6.0.2, despite the mixed-case filename.

One difference between version 6.0.2 and 6.0.3 is that 6.0.3 also includes manylinux_2_28 in its build whereas 6.0.2 does not.
PyYAML-6.0.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl uploads to packagecloud just fine
pyyaml-6.0.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl does not.

[nitzmahone]

Packagecloud does not have an issue with version 6.0.2

At the end of the day, Packagecloud is rejecting wheels built by the latest versions of standard Python packaging tools and that have passed numerous validity checks. PyYAML 6.0.3 is literally the exact code from 6.0.2 with one code comment line added, but packaged by the newest versions of cibuildwheel, setuptools, and Cython. I have a hard time believing that whatever is causing them to reject the 6.0.3 wheels is unique to PyYAML, and will likely end up being fixed soon as other packages hit the same issues.

[nitzmahone]

@AmritaChoudhary "PyYAML 6.0.3 doesn't work in my CI" doesn't provide enough information to help diagnose whatever the issue is, but I'd start by inspecting sys.path and sys.meta_path in the interpreter process that's failing to ensure you aren't picking up mismatched extensions from different builds, a build for the wrong Python version, or a broken local build that snuck in as a side-effect of a different package install. The paths you've shown from your output imply a customized Python environment- it's very easy to accidentally pick up the wrong extension with wildly differing results.

[danielhedgecock]

Packagecloud does not have an issue with version 6.0.2

At the end of the day, Packagecloud is rejecting wheels built by the latest versions of standard Python packaging tools and that have passed numerous validity checks. PyYAML 6.0.3 is literally the exact code from 6.0.2 with one code comment line added, but packaged by the newest versions of cibuildwheel, setuptools, and Cython. I have a hard time believing that whatever is causing them to reject the 6.0.3 wheels is unique to PyYAML, and will likely end up being fixed soon as other packages hit the same issues.

That's fair. There were two other libraries (out of about 40) that Packagecloud also rejected, so it's entirely possible it's something on their end that needs to be fixed. I just wanted to mention it here for your awareness.

Thanks again for your responsiveness!