Does ant support Linux Wayland?

[mazdabat]

Hi, I got ant project built on EndeavourOs. However, it crashed when I started up the simple test. I found EndeavourOs is using Wayland.

This is my test command:
./bin/linux/debug/ant ./test/simple/main.lua

Here is the trace:

[2024-06-22 08:53:34.52][INFO ]( root ) startup 1
[2024-06-22 08:53:34.52][INFO ]( io ) startup 2
[2024-06-22 08:53:34.52][INFO ]( ant.engine|timer ) startup 3
[2024-06-22 08:53:34.52][INFO ]( ant.engine|logger ) startup 4
[2024-06-22 08:53:34.57][INFO ]( /main.lua ) startup 5
[2024-06-22 08:53:34.57][INFO ]( ant.hwi|bgfx ) startup 7
[2024-06-22 08:53:34.57][INFO ]( ant.rmlui|rmlui ) startup 8
[2024-06-22 08:53:34.57][INFO ]( ant.window|window ) startup 6
[2024-06-22 08:53:34.59][INFO ]( ant.resource_manager|resource ) startup 9
[2024-06-22 08:53:34.79][INFO ]( io )(.../git_repo/ant/pkg/ant.compile_resource/tool_exe_path.lua:69) Use tools path:
[2024-06-22 08:53:34.79][INFO ]( ant.engine|subprocess ) startup 10
[2024-06-22 08:53:34.88][DEBUG]( ant.window|window )(/home/chao/git_repo/ant/pkg/ant.world/main.lua:738) world initializing
[2024-06-22 08:53:34.94][INFO ]( ant.rmlui|rmlui ) quit.
[2024-06-22 08:53:34.94][ERROR]( ant.window|window ) attempt to index a nil value (field '?')
stack traceback:
        ( service:2 )
        ...t_repo/ant/pkg/ant.compile_resource/material/toolset.lua:55: in upvalue 'get_shader_option'
        ...t_repo/ant/pkg/ant.compile_resource/material/toolset.lua:62: in upvalue 'gen_commands'
        ...t_repo/ant/pkg/ant.compile_resource/material/toolset.lua:109: in field 'compile'
        ...t_repo/ant/pkg/ant.compile_resource/material/compile.lua:573: in function <...t_repo/ant/pkg/ant.compile_resource/material/compile.lua:572>
        [C]: in function 'xpcall'
        ( service:2 )
        ...t_repo/ant/pkg/ant.compile_resource/material/compile.lua:601: in upvalue 'compile'
        ...t_repo/ant/pkg/ant.compile_resource/material/convert.lua:16: in field '?'
        /home/chao/git_repo/ant/pkg/ant.compile_resource/main.lua:58: in field 'compile_file'
        engine/console/io.lua:155: in upvalue 'COMPILE'
        engine/console/io.lua:63: in upvalue 'getresource'
        engine/console/io.lua:84: in upvalue 'getfile'
        engine/console/io.lua:93: in local 's'
        ( service:9 )
        [string "local name = ......"]:7: in upvalue 'call'
        [string "local name = ......"]:15: in field 'read'
        /home/chao/git_repo/ant/pkg/ant.io/main.lua:7: in upvalue 'readall'
        /home/chao/git_repo/ant/pkg/ant.io/main.lua:12: in field 'readall'
        /home/chao/git_repo/ant/pkg/ant.serialize/main.lua:29: in field 'load'
        ...it_repo/ant/pkg/ant.resource_manager/thread/material.lua:223: in upvalue 'material_create'
        ...it_repo/ant/pkg/ant.resource_manager/thread/material.lua:248: in local 's'
        ( service:8 )
        [C]: in function 'error'
        /home/chao/git_repo/ant/pkg/ant.rmlui/core/init_shader.lua:26: in function </home/chao/git_repo/ant/pkg/ant.rmlui/core/init_shader.lua:25>
        [C]: in function 'xpcall'
        ( service:8 )
        [C]: in function 'error'
        /home/chao/git_repo/ant/pkg/ant.rmlui/core/init_shader.lua:37: in local 'func'
        /home/chao/git_repo/ant/engine/packagemanager.lua:54: in global 'require'
        /home/chao/git_repo/ant/pkg/ant.rmlui/core/startup.lua:76: in local 'func'
        /home/chao/git_repo/ant/engine/packagemanager.lua:54: in global 'require'
        /home/chao/git_repo/ant/pkg/ant.rmlui/service/rmlui.lua:1: in local 'func'
        [C]: in function 'xpcall'
        ( service:8 )
        /engine/firmware/ltask_service.lua:816: in local 's'
        ( service:1 )
        engine/firmware/ltask_root.lua:148: in upvalue 'f'
        ( service:1 )
        /engine/firmware/ltask_service.lua:530: in field 'multi_wait'
        engine/firmware/ltask_root.lua:156: in upvalue 'spawn_unique'
        engine/firmware/ltask_root.lua:207: in local 's'
        ( service:1 )
        /engine/firmware/ltask_service.lua:530: in field 'multi_wait'
        engine/firmware/ltask_root.lua:203: in local 's'
        ( service:6 )
        /engine/firmware/ltask_service.lua:598: in field 'queryservice'
        /home/chao/git_repo/ant/pkg/ant.world/inputmgr.lua:13: in upvalue 'f'
[2024-06-22 08:53:34.94][INFO ]( /main.lua ) quit.
[2024-06-22 08:53:34.96][ERROR]( root ) Root fatal:     attempt to index a nil value (field '?')
stack traceback:
        ( service:2 )
        ...t_repo/ant/pkg/ant.compile_resource/material/toolset.lua:55: in upvalue 'get_shader_option'
        ...t_repo/ant/pkg/ant.compile_resource/material/toolset.lua:62: in upvalue 'gen_commands'
        ...t_repo/ant/pkg/ant.compile_resource/material/toolset.lua:109: in field 'compile'
        ...t_repo/ant/pkg/ant.compile_resource/material/compile.lua:573: in function <...t_repo/ant/pkg/ant.compile_resource/material/compile.lua:572>
        [C]: in function 'xpcall'
        ( service:2 )
        ...t_repo/ant/pkg/ant.compile_resource/material/compile.lua:601: in upvalue 'compile'
        ...t_repo/ant/pkg/ant.compile_resource/material/convert.lua:16: in field '?'
        /home/chao/git_repo/ant/pkg/ant.compile_resource/main.lua:58: in field 'compile_file'
        engine/console/io.lua:155: in upvalue 'COMPILE'
        engine/console/io.lua:63: in upvalue 'getresource'
        engine/console/io.lua:84: in upvalue 'getfile'
        engine/console/io.lua:93: in local 's'
        ( service:9 )
        [string "local name = ......"]:7: in upvalue 'call'
        [string "local name = ......"]:15: in field 'read'
        /home/chao/git_repo/ant/pkg/ant.io/main.lua:7: in upvalue 'readall'
        /home/chao/git_repo/ant/pkg/ant.io/main.lua:12: in field 'readall'
        /home/chao/git_repo/ant/pkg/ant.serialize/main.lua:29: in field 'load'
        ...it_repo/ant/pkg/ant.resource_manager/thread/material.lua:223: in upvalue 'material_create'
        ...it_repo/ant/pkg/ant.resource_manager/thread/material.lua:248: in local 's'
        ( service:8 )
        [C]: in function 'error'
        /home/chao/git_repo/ant/pkg/ant.rmlui/core/init_shader.lua:26: in function </home/chao/git_repo/ant/pkg/ant.rmlui/core/init_shader.lua:25>
        [C]: in function 'xpcall'
        ( service:8 )
        [C]: in function 'error'
        /home/chao/git_repo/ant/pkg/ant.rmlui/core/init_shader.lua:37: in local 'func'
        /home/chao/git_repo/ant/engine/packagemanager.lua:54: in global 'require'
        /home/chao/git_repo/ant/pkg/ant.rmlui/core/startup.lua:76: in local 'func'
        /home/chao/git_repo/ant/engine/packagemanager.lua:54: in global 'require'
        /home/chao/git_repo/ant/pkg/ant.rmlui/service/rmlui.lua:1: in local 'func'
        [C]: in function 'xpcall'
        ( service:8 )
        /engine/firmware/ltask_service.lua:816: in local 's'
        ( service:1 )
        engine/firmware/ltask_root.lua:148: in upvalue 'f'
        ( service:1 )
        /engine/firmware/ltask_service.lua:530: in field 'multi_wait'
        engine/firmware/ltask_root.lua:156: in upvalue 'spawn_unique'
        engine/firmware/ltask_root.lua:207: in local 's'
        ( service:5 )
        [C]: in function 'error'
        /home/chao/git_repo/ant/pkg/ant.window/main.lua:21: in function </home/chao/git_repo/ant/pkg/ant.window/main.lua:20>
        [C]: in function 'xpcall'
        ( service:5 )
        [C]: in function 'error'
        /home/chao/git_repo/ant/pkg/ant.window/main.lua:33: in field 'start'
        /home/chao/git_repo/ant/test/simple/main.lua:1: in local 'func'
        [C]: in function 'xpcall'
        ( service:5 )
        /engine/firmware/ltask_service.lua:816: in local 's'
        ( service:1 )
        [C]: in function 'error'
        engine/firmware/ltask_root.lua:126: in upvalue 'spawn'
        engine/firmware/ltask_root.lua:217: in field 'spawn_service'
        engine/firmware/ltask_root.lua:273: in local 'bootstrap'
        engine/firmware/ltask_root.lua:279: in local 'func'
        [C]: in function 'xpcall'
        ( service:1 )
        /engine/firmware/ltask_service.lua:816: in local 's'

[cloudwu]

About wayland, See #136

https://github.com/ejoy/ant/blob/master/pkg/ant.compile_resource/material/toolset.lua#L55

But this issue seems to be caused by not finding the correct renderer, for linux it (renderer) should be vulkan.

I suggest that you can dump the setting for this function (
https://github.com/ejoy/ant/blob/master/pkg/ant.compile_resource/material/compile.lua#L556 ) and find the reason.

[mazdabat]

Thanks for the prompt reply. I played around with vulkan. Now, I can see a blank dialog but crashed after a while with these info:

[2024-06-23 19:45:44.50][INFO ]( root ) startup 1
[2024-06-23 19:45:44.50][INFO ]( io ) startup 2
[2024-06-23 19:45:44.50][INFO ]( ant.engine|timer ) startup 3
[2024-06-23 19:45:44.55][INFO ]( ant.engine|logger ) startup 4
3rd/bgfx/src/renderer_vk.cpp(3848): ASSERT texture.m_currentImageLayout == texture.m_sampledLayout -> Mismatching image layout. Texture currently used as a framebuffer attachment?
Fatal error at 3rd/bgfx/src/renderer_vk.cpp(3848): [DebugCheck] Mismatching image layout. Texture currently used as a framebuffer attachment?
Aborted (core dumped)

Any clue?