GitHub Actions: Test project exporting on CI

[Calinou]

This allows finding issues in headless project export early on, including when exporting for a dedicated server.

We also use this opportunity to check whether the audiovisual output between the project being run from its files and the exported PCK matches (it should always be a perfect match, assuming the same GPU is used for both runs). This can be used to catch audiovisual discrepancies, which could indicate a bug in the export process. This also tests Movie Maker mode functionality while we're at it.

The test project is currently part of the main repository, so I've attempted to minimize the file size while making use of as many Godot features as possible. We could move this test project to a separate repository, but this would increase the complexity of the setup and make it harder for contributors to test this project locally (as well as further improve it).

This export test is performed on the Linux Editor w/ Mono job, which will allow for testing C# (not implemented in this PR yet).

The development of this PR has already uncovered several bugs so far:

• Crash when importing a 3D scene with UV2 unwrap enabled in a double-precision development build #109097
• Shader baker still exports a shader cache for dedicated server exports #109098
• QOA assertion failure on import when running with sanitizers enabled #109099
• Baked VoxelGIData and LightmapGIData is still exported in dedicated server exports #109141
• Exporting a project from headless mode prints an error when VoxelGIData is present in the project #109145
• Per-instance shader parameter values are not correctly exported when the export is performed in --headless mode #109177
• CSV translation causes errors to be printed on export, even though it's functional (no issue open yet?)
  • I can't reproduce this in a new project, such as test_csv_export_error.zip.

Sounds that were

Preview

The project features:

• LightmapGI setup, as well as most 3D rendering features enabled.
• Custom text and visual shaders (the black and white spheres), with parameters set to non-default values. Global and per-instance uniforms are also used in the text shader. The default parameters are red, while the overrides are green. If the project is run/exported correctly, you should only see green spheres, not red ones.
• All texture compression types (with and without mipmaps) as shown in the top-left corners. Size Limit is also used here, with a size of 29×29 to ensure we have valid behavior for texture sizes that aren't a multiple of 4.
• 3D scene import in all supported formats (the apples).
• Translation and font imports (standard, LCD subpixel, MSDF, bitmap).
• GDScript, C# and GDExtension C++ using godot-cpp.
  • C# work moved to this branch: https://github.com/Calinou/godot/tree/ci-test-project-export-csharp
  • GDExtension work moved to this ZIP: addons.zip[^1]
  • Actions code to build the extension:

    - name: Build test GDExtension
      shell: sh
      run: |
        cd misc/test_project/addons/test_extension/
        git clone --depth=1 https://github.com/godotengine/godot-cpp.git src/godot-cpp/
        scons target=template_debug
        scons target=template_release
        cd ../../../../

• OBJ scene/mesh import
  • Moved to this branch due to errors being printed on export on CI (not on my machine): https://github.com/Calinou/godot/tree/ci-test-project-export-obj-voxelgi
• VoxelGI export
  • Moved to this branch due to errors being printed on export on CI (not on my machine): https://github.com/Calinou/godot/tree/ci-test-project-export-obj-voxelgi
  • Strip VoxelGIData and LightmapGIData using placeholders for dedicated server exports #112744 fixes the issue, so we can readd VoxelGI to the test project once that PR is merged.
• Audio playback
  • Removed due to leaked instances on exit, causing CI to fail.
  • Sound files to be readded when the issue is fixed: sounds.zip

TODO

• Add DDS and KTX texture loading.
• Add more resource types. Perhaps use an EditorScript to generate all of them, which might uncover some more bugs in the process.

For a future PR (due to the complexity):

• Test exporting to Android with both Gradle and APK workflows. This will require waiting on the Android build to finish, then making this CI job download the artifacts and use them for export. It would also require setting up the Android SDK before exporting.
• Test exporting to Windows and web platforms. Not as involved as Android, but it'll still require changing the CI setup to add dependencies to other platforms' workflows.
• Test exporting to macOS and iOS. This requires using a macOS host, which may not be technically feasible due to Metal rendering driver fails to run on Apple's software Metal implementation (paravirtual device) #101773.

[Repiteo]

This is fantastic for testing purposes, and already coming along really nicely; great job!

Though, the test project itself shouldn't be in the repo. It's not something to worry about during this testing phase, but we should do something similar to the regression test project where it's downloaded from a remote location

[mihe]

Anything that can be done to help move this one forward? I'd love to see this merged in some form for 4.6, since it would've helped catch one or two of the regressions that I ran into during the last beta.

[Calinou]

Anything that can be done to help move this one forward? I'd love to see this merged in some form for 4.6, since it would've helped catch one or two of the regressions that I ran into during the last beta.

I'll take a look at it again next week. Once I get CI to pass, we can look into creating a new repository in the @godotengine organization then update this PR to remove the test project from it (and make it clone the other repository instead).

There's already https://github.com/godotengine/godot-tests, but we don't seem to be using the tests from that repository anywhere currently.

The alternative is to modify https://github.com/godotengine/regression-test-project to include two projects side-by-side (the existing project + the new one).

What do you think?

[akien-mga]

There's already https://github.com/godotengine/godot-tests, but we don't seem to be using the tests from that repository anywhere currently.

I guess using that repository makes sense. Eventually we might want to run more of this kind of integration tests which could be hosted in that repo. The existing tests may or may not be obsolete but you can likely replace the project_export one with your new project.

[Calinou]

I'm getting a CI failure due to sanitizers on the regression test project (not the project added by this PR). How come it doesn't occur on master? I just rebased my PR on top of current master.

[Calinou]

I've removed C# for now as it was too difficult to get working. We can look into it again in a future PR, especially since the way C# is integrated in Godot may change in the near future.

I've also switched localization to gettext, as CSV constantly caused errors on export (even on my own machine in non-headless export). This is presumably due to its one-to-many nature, where importing a single CSV file creates multiple *.translation files. In contrast, gettext has a one-to-one mapping between source files and resulting translation files (the PO file is the translation file).

Also, the GDExtension build takes a very long time (10+ minutes). Is there a way to speed it up? Maybe we should commit a precompiled Linux library in the test project when I open a PR to godot-tests.

[mihe]

Also, the GDExtension build takes a very long time (10+ minutes). Is there a way to speed it up?

I sped up the build times of godot-cpp in Godot Jolt a lot by using CMake's built-in "unity builds", i.e. what Godot refers to as SCU builds. I feel like it would be fairly straight-forward to add support for that to godot-cpp's SCons setup. You really only need it for the generated *.cpp files, which make up the majority of the build times, and which would be trivial to ensure that they adhere to the various SCU quirks.

I'd be curious to hear what you think of that, @dsnopek.

EDIT: Maybe caching the build artifacts in GitHub Actions is an option as well?

[Calinou]

I've removed VoxelGI and OBJ mesh/scenes from the test project for now, since they caused (benign) errors on headless export I couldn't fix. I have a fix in the works for VoxelGI, which will also make PCKs smaller by introducing a PlaceholderVoxelGI resource, but it'll be in a separate PR.

check_ci_log.py was modified to be optionally more strict, according to the GODOT_CHECK_CI_LOG_ALL_ERRORS=1 environment variable.

I'm still running into issues with GDExtension though. Running the project directly works, but running the exported PCK results in the GDExtension not being found.

[Calinou]

CI is passing now, so I've opened a PR on godot-tests with the test project:

• Add test project export for CI godot-tests#35

Once the above PR is merged, I'll change this PR to download the test project from godot-tests instead of embedding it here.

[mihe]

@Calinou Any plans on resuming the GDExtension part of this if/when this is merged, or did you find yourself stuck on the dynamic library loading issue you were having?

[Calinou]

@Calinou Any plans on resuming the GDExtension part of this if/when this is merged, or did you find yourself stuck on the dynamic library loading issue you were having?

Yes, I'll look into it once this is merged. I've tried to use --export-release "Linux" instead of --export-pack, but it didn't seem to work (possibly because I accidentally left the output path as /tmp/test_project.zip instead of /tmp/test_project.x86_64, and this seemed to create a zipped export with missing GDExtension libraries).

I've reverted the PR to use --export-pack for now, but ideally, I would expect --export-pack to copy GDExtension libraries next to the PCK/ZIP just like --export-release (since it knows the platform it's exporting for).

[akien-mga]

Once the above PR is merged, I'll change this PR to download the test project from godot-tests instead of embedding it here.

Just a reminder that this still needs to be done before merging.