Support Pyodide 0.28 and bump to Python >=3.13 #57

agriyakhetarpal · 2025-03-14T07:08:13Z

Closes #47
Closes #45

rth · 2025-03-14T07:30:02Z

Thanks @agriyakhetarpal for taking over the maintenance!

agriyakhetarpal · 2025-03-14T12:22:17Z

Pyodide 0.26.4 and 0.27.3 fail with a LookupError claiming an unsupported cp437 encoding after being stripped. I've traced this in Pyodide to the fact that the stripped stdlib also excludes the various encoding implementations from python_stdlib/encodings/, and CPython raises this error when it can't find them (as zipfile tries to decode with either a specified encoding or cp437 – and the latter is undefined).

Tracing this further, I see that Pyodide 0.24.1 kept the cp437 encoding implementation in encodings/, but Pyodide 0.25.1, 0.26.4, and 0.27.3 do not, and all the other encodings are lost upon stripping/bundling – just UTF-8 is present.

I have to note that 0.25.1 and 0.27.3 fail with different errors, so it is also likely that changes in the order of the imports, etc., are causing things to fail. Importing encodings before zip file will not help here, as the encoding itself is lost. We'll need a way to avoid stripping out this specific encoding from this folder. I think the way to do this would be to dive deeper into what discovery.js is doing.

rth · 2025-03-14T13:21:37Z

Regarding encodings I don't remember the exact details, but indeed the overall idea was that they were taking space so utf8 + a few specific ones should be enough to start Python. But maybe indeed the list of minimal required encodings changed.

so it is also likely that changes in the order of the imports, etc., are causing things to fail
I think the way to do this would be to dive deeper into what discovery.js is doing.

Yeah sorry part of it might also be relying on internal emscripten APIs so if those parts (in discovery.js) changed that would explain the breakage.

agriyakhetarpal · 2025-03-14T17:08:50Z

Thanks! I fixed the LookupError, now one more error remains, so we're almost there!

There is no proper error message in the CI logs, but I am able to reproduce the failure locally, which does have a stack trace:

pyodide.ffi.JsException: Error: Didn't expect to load any more file_packager files!

Tap to expand full error message

PythonError: Traceback (most recent call last):
  File "/home/pyodide/pyodide_pack_loader.py", line 10, in setup
    await _module.API.loadDynlib(path, bool(is_shared))
pyodide.ffi.JsException: Error: Didn't expect to load any more file_packager files!

    at new_error (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:10009)
    at wasm://wasm/0268a37a:wasm-function[308]:0x16e384
    at wasm://wasm/0268a37a:wasm-function[507]:0x17855f
    at wasm://wasm/0268a37a:wasm-function[4586]:0x328642
    at wasm://wasm/0268a37a:wasm-function[1151]:0x1c44db
    at wasm://wasm/0268a37a:wasm-function[3622]:0x2ca38d
    at wasm://wasm/0268a37a:wasm-function[2184]:0x20c2d8
    at wasm://wasm/0268a37a:wasm-function[1159]:0x1c4bc2
    at wasm://wasm/0268a37a:wasm-function[1162]:0x1c4ed1
    at wasm://wasm/0268a37a:wasm-function[1163]:0x1c4f4f
    at wasm://wasm/0268a37a:wasm-function[3428]:0x2a0d61
    at wasm://wasm/0268a37a:wasm-function[3429]:0x2a72bf
    at wasm://wasm/0268a37a:wasm-function[1165]:0x1c508f
    at wasm://wasm/0268a37a:wasm-function[1160]:0x1c4cf8
    at wasm://wasm/0268a37a:wasm-function[494]:0x177bbf
    at callPyObjectKwargs (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:62939)
    at Module.callPyObjectMaybePromising (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:64121)
    at Immediate.wrapper (/Users/agriyakhetarpal/Desktop/pyodide-pack/node_modules/pyodide/pyodide.asm.js:10:27499)
    at process.processImmediate (node:internal/timers:511:21) {
  type: 'JsException',
  __error_address: 15818000
}

Node.js v23.6.0
FAILED

============================================================================== FAILURES ==============================================================================
_______________________________________________________________________ test_all[scikit-learn] _______________________________________________________________________

example_dir = PosixPath('/Users/agriyakhetarpal/Desktop/pyodide-pack/examples/scikit-learn')
tmp_path = PosixPath('/private/var/folders/b3/2bq1m1_50bs4c7305j8vxcqr0000gn/T/pytest-of-agriyakhetarpal/pytest-230/test_all_scikit_learn_0')

    @pytest.mark.parametrize("example_dir", gen_all_examples())
    def test_all(example_dir, tmp_path):
        stdout = StringIO()
        os.chdir(tmp_path)
        # Assumes there is a node_modules in the current directory
        (tmp_path / "node_modules").symlink_to(
            BASE_DIR / "node_modules", target_is_directory=True
        )
        with redirect_stdout(stdout):
>           cli.main(
                example_path=example_dir / "app.py",
                config_path=None,
                verbose=False,
                include_paths=None,
                write_debug_map=True,
            )

/Users/agriyakhetarpal/Desktop/pyodide-pack/examples/test_examples.py:30: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/Users/agriyakhetarpal/Desktop/pyodide-pack/pyodide_pack/cli.py:248: in main
    runner.run()
/Users/agriyakhetarpal/Desktop/pyodide-pack/pyodide_pack/runners/node.py:39: in run
    subprocess.run(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

I'm not sure how to debug that so far as it's quite cryptic, so any suggestions would be welcome in case you're familiar with it. :)

rth · 2025-03-14T17:25:56Z

I think I did encounter it the past, but I'm not 100% sure/remember the cause.

It's raised here but in this case the error is misleading, because we are not trying to load file_packager (legacy pre-wheel) files.

Probably something is not quite right with the sequence in which dynamic libraries are loaded (like if something changed in pyodide in that respect). If you don't find it, ask Hood :)

rth · 2025-03-14T17:34:21Z

Maybe related to pyodide/pyodide#4726, see @hoodmane's comments in the code there. It looks like _module.API.loadDynlib would be calling locateFile after we are expecting it should no longer happen.

I don't know maybe monkeypatching locateFile back to the original value would be enough to make it work?

agriyakhetarpal

Here is a self-review: I'm not fully sure whether my changes are correct, but I'm open to suggestions as this is the first time I'm going through our dynamic library loading procedure and trying to understand it better.

Either way, this is ready for review and the tests are passing, marking it as such!

.github/workflows/main.yml

agriyakhetarpal · 2025-03-15T14:33:15Z

examples/netcdf4/app.py

I added this test because we haven't had any other tests like it so far. Here, we have both a shared library (libhdf5) and a static library (libnetcdf), a package that depends on both of them (h5py), and eventually netCDF4 – i.e., we weren't testing the integration with any static libraries, just shared ones (SciPy + OpenBLAS). I can drop it if you suggest I do so.

pyodide_pack/ast_rewrite.py

agriyakhetarpal · 2025-03-15T14:41:43Z

pyodide_pack/loader/pyodide_pack_loader.py

I'd like to ask if it would be okay if I could address missing coverage in a further PR? The change doesn't do much; it just fails more gracefully by printing a warning when loading a dynlib. What I understand here that this loader is an elementary check and that we don't load the libs in the order they are supposed to be loaded in – we just run over what comes first in the list. It could be the case that I might have made a mistake somewhere.

With this, the netcdf4 test does display this locally:

Warning: Failed to load /lib/python3.12/site-packages/h5py/_conv.cpython-312-wasm32-emscripten.so: Error: Didn't expect to load any more file_packager files! Warning: Failed to load /lib/python3.12/site-packages/h5py/_errors.cpython-312-wasm32-emscripten.so: Error: Didn't expect to load any more file_packager files!

but at the same time, the code in examples/netcdf4/app.py executes and runs completely fine till the end, which I don't understand.

agriyakhetarpal · 2025-03-15T14:42:53Z

pyodide_pack/tests/test_ast_rewrite.py



-@settings(deadline=300)
+@settings(deadline=None)


I can't get this test to pass within 500ms :)

rth

Thanks a lot @agriyakhetarpal . Changes wise LGTM, but I haven't followed latest pyodide development.

rth · 2025-03-15T15:58:37Z

pyodide_pack/runtime_detection.py

+            # Always mark shared libraries as shared=True to ensure they're loaded
+            # globally, for libraries that need these global symbols
+            if extension == ".so":
+                dll.shared = True


Are you sure about this? Is far as I remember cython generated objects are also .so and they don't need to be managed as global symbol.

I think the idea was more to look whether the shared_library tag in packages https://github.com/pyodide/pyodide/blob/main/packages/openblas/meta.yaml#L6 and in which case load it.

But yeah, in any case this part is suboptimal so feel free to merge as is, and investigate this separately (if you are interested in this topic). Because updating to the latest Pyodide version is probably more important.

agriyakhetarpal · 2025-03-15T17:22:25Z

🤔 I tried the netcdf4 example I added with Pyodide 0.26.4 and it seems to pass without needing to ignore unloadable dynlibs, which makes me feel that there is something that I'm missing in the equation. I'll mark it as a draft for now till I investigate more about why it fails with 0.27.3.

Thanks for the review in the meantime!

agriyakhetarpal · 2025-03-15T19:19:51Z

Some more investigation for this particular example (netcdf4) reveals that:

the number of accessed modules is getting reduced with Pyodide 0.27.3 in comparison to 0.26.4; i.e., the latter accesses 5 modules instead of 15 for the former. Modules from numpy.fft and numpy.random do not show up in the bundle .sos list for 0.27.3.
the order of the accessed modules is incorrect – libhdf5_hl.so in h5py.libs/ shows up at the end of the list, rather than before other h5py-specific modules that rely on it. This disarrangement is likely what is causing file_packager errors when loading, as dependent dynlibs will fail to load until their parent dynlibs are loaded.

`bundle-so-list.txt` from Pyodide 0.26.4

Tap to expand

/lib/python3.12/site-packages/numpy/core/_multiarray_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_multiarray_umath.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_operand_flag_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_rational_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_simd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_struct_ufunc_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/core/_umath_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/fft/_pocketfft_internal.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/_umath_linalg.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/lapack_lite.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_bounded_integers.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_common.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_mt19937.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_pcg64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_philox.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_sfc64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/bit_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/mtrand.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/cftime/_cftime.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py.libs/libhdf5.so,True
/lib/python3.12/site-packages/h5py.libs/libhdf5_hl.so,True
/lib/python3.12/site-packages/h5py/_conv.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_errors.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_objects.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_proxy.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_selector.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/defs.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5a.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ac.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5d.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ds.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5f.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5fd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5g.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5i.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5l.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5o.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5p.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5pl.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5r.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5s.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5t.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5z.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/utils.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/netCDF4/_netCDF4.cpython-312-wasm32-emscripten.so,False

`bundle-so-list.txt` with Pyodide 0.27.3

Tap to expand

/lib/python3.12/site-packages/numpy/_core/_multiarray_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_multiarray_umath.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_operand_flag_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_rational_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_simd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_struct_ufunc_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/_core/_umath_tests.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/fft/_pocketfft_umath.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/_umath_linalg.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/linalg/lapack_lite.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_bounded_integers.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_common.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_mt19937.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_pcg64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_philox.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/_sfc64.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/bit_generator.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/numpy/random/mtrand.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/cftime/_cftime.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_conv.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_errors.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_objects.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_proxy.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/_selector.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/defs.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5a.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ac.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5d.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5ds.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5f.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5fd.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5g.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5i.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5l.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5o.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5p.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5pl.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5r.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5s.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5t.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/h5z.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py/utils.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/h5py.libs/libhdf5.so,False
/lib/python3.12/site-packages/h5py.libs/libhdf5_hl.so,False
/lib/python3.12/site-packages/netCDF4/_netCDF4.cpython-312-wasm32-emscripten.so,False
/lib/python3.12/site-packages/netcdf4.libs/libhdf5.so,False
/lib/python3.12/site-packages/netcdf4.libs/libhdf5_hl.so,False

Unfortunately, installing my debug build of Pyodide does not work here for getting extra logging, as I keep encountering function signature mismatches in the installation stage (with micropip) itself...

ryanking13 · 2025-03-18T12:57:38Z

Tracing this further, I see that Pyodide 0.24.1 kept the cp437 encoding implementation in encodings/, but Pyodide 0.25.1, 0.26.4, and 0.27.3 do not, and all the other encodings are lost upon stripping/bundling – just UTF-8 is present.

I don't exactly remember, but there might be some changes how CPython deepfreezes encodings module in Python 3.11 and 3.12. I think there are some room to investigate, but for now your change looks fine to me.

The only relevant changes in this area that made it to the Pyodide 0.27 alphas were through pyodide/pyodide#4876 and pyodide/pyodide#4871. I don't think we changed anything after them up to the arrival of 0.27 stable. My hunch is that these PRs will provide the necessary information for us to proceed here.

Right. I changed the library to be loaded locally if possible. If pyodide-pack is relying on whether the library is loaded locally or globally, I guess it should be changed.

agriyakhetarpal · 2025-03-19T00:26:33Z

Tracing this further, I see that Pyodide 0.24.1 kept the cp437 encoding implementation in encodings/, but Pyodide 0.25.1, 0.26.4, and 0.27.3 do not, and all the other encodings are lost upon stripping/bundling – just UTF-8 is present.

I don't exactly remember, but there might be some changes how CPython deepfreezes encodings module in Python 3.11 and 3.12. I think there are some room to investigate, but for now your change looks fine to me.

Sounds good, I'll leave that for exploration in another PR!

The only relevant changes in this area that made it to the Pyodide 0.27 alphas were through pyodide/pyodide#4876 and pyodide/pyodide#4871. I don't think we changed anything after them up to the arrival of 0.27 stable. My hunch is that these PRs will provide the necessary information for us to proceed here.

Right. I changed the library to be loaded locally if possible. If pyodide-pack is relying on whether the library is loaded locally or globally, I guess it should be changed.

Yes, I've adjusted for it. I haven't found a solution to how the netCDF4 test can be fixed. It's good that I added such a test, though, as the rest of the tests were passing and we would have merged with broken functionality. The netCDF4 test passes on 0.24.1, so the failure with 0.27.1 is indeed related.

Do we have a guarantee that the libraries in mypackage.libs/ will always be loaded first, and then the ones insidemypackage/ and its subfolders? If yes, I can tweak the order of how the libraries are added to the bundled .sos list and we should be good to go. Otherwise, I think it needs more investigation. I didn't find any changes we made upstream that might have changed the order of the library detection/import – just that the libraries are imported locally, which I addressed. Maybe @hoodmane would know about it.

ryanking13 · 2025-03-20T08:41:41Z

Do we have a guarantee that the libraries in mypackage.libs/ will always be loaded first, and then the ones insidemypackage/ and its subfolders?

Yes. After we land RPATH in emscripten, the order of the import should be:

LD_LIBRARY_PATH (which will be something like /usr/lib:/lib)
paths in the RPATH field.

we can adjust the order in RPATH, so I think we have options to choose the load order.

agriyakhetarpal · 2025-03-20T10:02:18Z

Sounds good! Though, I don't think we should wait for RPATH support to unblock this PR, as that could likely be after Pyodide 0.28 (which will then break compatibility with pyodide-pack again).

ryanking13 · 2025-03-21T10:19:02Z

Sounds good! Though, I don't think we should wait for RPATH support to unblock this PR, as that could likely be after Pyodide 0.28 (which will then break compatibility with pyodide-pack again).

Sure, this PR can go without waiting for the RPATH support.

hoodmane · 2025-03-21T10:32:10Z

that could likely be after Pyodide 0.28

I don't want to release Pyodide 0.28 without the RPATH support. The Emscripten 4.0.5 linker supports RPATH already. Once we update to Emscripten >= 4.0.5 we can patch the runtime to use it by backporting some version of emscripten-core/emscripten#23872.

agriyakhetarpal · 2025-05-31T22:25:00Z

Now that we have support for RPATH in the nightlies, I think we can revisit this. I can start testing with Pyodide 0.28.0a3 here; I'll need to read up a bit on how the RPATH support works.

agriyakhetarpal · 2025-06-01T01:04:16Z

With Pyodide 0.28, the first hurdle is to get stdlib stripping working. We can see that

┏━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ No ┃ Package ┃ All files ┃ .so libs ┃   Size (MB) ┃ Reduction ┃
┡━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│  0 │ stdlib  │   575 → 0 │          │ 2.34 → 0.00 │   100.0 % │
└────┴─────────┴───────────┴──────────┴─────────────┴───────────┘

so all of the modules are lost, and the Python interpreter fails to start.

agriyakhetarpal added 7 commits March 14, 2025 12:36

Bump to Node 22

f15d0e9

Test against Pyodide 0.27.3

db199bb

Fix typo

3940244

Update CHANGELOG for pyodide#57

3ce8480

Note compatibility with Pyodide 0.27 and later

ccac8f9

Use Python 3.12 everywhere

eacee75

Downgrade to Pyodide 0.26.4

7c83198

agriyakhetarpal added 6 commits March 14, 2025 13:42

Drop check for number of paths

72a091f

Go back to Node 20

dc0cfc2

Fix RTD builds

9fe3724

Fix coverage path

4b88204

Drop deprecated ast.Str

8b6c1ce

Upgrade to py312 for pyupgrade

6e35b9f

agriyakhetarpal changed the title ~~Support Pyodide 0.27~~ Support Pyodide 0.27 and bump to Python >=3.12 Mar 14, 2025

agriyakhetarpal added 6 commits March 14, 2025 13:58

Use -n auto for parallel testing

9b29a0e

Don't depend on tomli for Python <3.11

b0e99f3

Update to Python 3.12 everywhere

5551c85

Bump up Hypothesis deadline

6481503

Bisect; go down further to Pyodide 0.25.1

e16d2fe

Bump up test suite verbosity

490c3f1

agriyakhetarpal added 4 commits March 14, 2025 22:08

Import encodings.cp437

8462777

Update runtime module elimination docs

4ec4bdf

Bump to Pyodide 0.27.3 again

af67dcc

Bump to latest Node.js LTS again

1cd3e48

agriyakhetarpal added 4 commits March 15, 2025 19:48

Add a more complex test

8795efc

Handle duplicate shared libraries gracefully?

ed4bde3

Merge branch 'main' into update/pyodide-0.27

4b57e0f

Mark compatibility with Python 3.12 in README

3cc8c03

agriyakhetarpal commented Mar 15, 2025

View reviewed changes

agriyakhetarpal marked this pull request as ready for review March 15, 2025 14:45

agriyakhetarpal requested review from rth and ryanking13 March 15, 2025 14:45

rth approved these changes Mar 15, 2025

View reviewed changes

agriyakhetarpal marked this pull request as draft March 15, 2025 17:23

agriyakhetarpal added 3 commits March 16, 2025 01:08

Fix coverage directory

f70fa86

Revert graceful checking for dynlib duplication

7c6c3e2

Drop shared=True markers

31a3f52

agriyakhetarpal force-pushed the update/pyodide-0.27 branch from 4e3717d to 31a3f52 Compare March 15, 2025 19:38

agriyakhetarpal added 3 commits June 1, 2025 04:26

Install Pyodide 0.28.0a3

16b6787

Fix typo for npm Pyodide package

1d44dce

"alpha-3", not "a3"

f7911c5

Bump to Python 3.13 everywhere

cf78f2f

agriyakhetarpal changed the title ~~Support Pyodide 0.27 and bump to Python >=3.12~~ Support Pyodide 0.28 and bump to Python >=3.13 Jun 1, 2025

agriyakhetarpal mentioned this pull request Jun 24, 2025

Optimize the size of minimal pyodide build pyodide/pyodide#646

Open

6 tasks



		@settings(deadline=300)
		@settings(deadline=None)

Support Pyodide 0.28 and bump to Python >=3.13 #57

Are you sure you want to change the base?

Support Pyodide 0.28 and bump to Python >=3.13 #57

Uh oh!

Conversation

agriyakhetarpal commented Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rth commented Mar 14, 2025

Uh oh!

agriyakhetarpal commented Mar 14, 2025

Uh oh!

rth commented Mar 14, 2025

Uh oh!

agriyakhetarpal commented Mar 14, 2025

Uh oh!

rth commented Mar 14, 2025

Uh oh!

rth commented Mar 14, 2025

Uh oh!

agriyakhetarpal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

agriyakhetarpal Mar 15, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

agriyakhetarpal Mar 15, 2025

Choose a reason for hiding this comment

Uh oh!

agriyakhetarpal Mar 15, 2025

Choose a reason for hiding this comment

Uh oh!

rth left a comment

Choose a reason for hiding this comment

Uh oh!

rth Mar 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agriyakhetarpal commented Mar 15, 2025

Uh oh!

agriyakhetarpal commented Mar 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

bundle-so-list.txt from Pyodide 0.26.4

bundle-so-list.txt with Pyodide 0.27.3

Uh oh!

ryanking13 commented Mar 18, 2025

Uh oh!

agriyakhetarpal commented Mar 19, 2025

Uh oh!

ryanking13 commented Mar 20, 2025

Uh oh!

agriyakhetarpal commented Mar 20, 2025

Uh oh!

ryanking13 commented Mar 21, 2025

Uh oh!

hoodmane commented Mar 21, 2025

Uh oh!

agriyakhetarpal commented May 31, 2025

Uh oh!

agriyakhetarpal commented Jun 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

agriyakhetarpal commented Mar 14, 2025 •

edited

Loading

rth Mar 15, 2025 •

edited

Loading

agriyakhetarpal commented Mar 15, 2025 •

edited

Loading

`bundle-so-list.txt` from Pyodide 0.26.4

`bundle-so-list.txt` with Pyodide 0.27.3