WindowServer OOM happened when launch Installer Process

[DCMMC]

After launchctl load /var/jb/usr/macOS/LaunchDaemons, I recevied many IOHID errors and finally could see a black cursor in the left upper corner with a whole of gray background. WindowServer OOM happened after seconds and the screen black again.

<Notice> 01:54:32 Installer Progress[4210]: Progress app is loading…
<Notice> 01:54:32 Installer Progress[4210]: Progress app is running with no progress set
<Notice> 01:54:32 Installer Progress[4210]: Progress app is running…
<Error>  01:54:35 WindowServer_arm64[4213]: Timeout waiting for getProperty: backlight-calibration-parameters, _timeoutCount: 1
<Error>  01:54:35 WindowServer_arm64[4213]: Got back calibration from DCP of unknown type (null)!!
<Notice> 01:54:35 WindowServer_arm64[4213]: <private>:-[CBAPEndpoint copyProperty:] called for key: backlight-calibration-parameters
<Notice> 01:54:35 WindowServer_arm64[4213]: capabilities = {
<Notice> 01:54:35 WindowServer_arm64[4213]: Framebuffer 0x126809000, DisplayID: 1, brightness callback enabled: 1
<Notice> 01:54:35 WindowServer_arm64[4213]: DisplayID: 1 got brightness capabilities {
<Notice> 01:54:35 WindowServer_arm64[4213]: Display 1 posting brightness notification
<Notice> 01:54:38 runningboardd[3723]: [osservice<com.apple.backupd>:4160] termination reported by launchd (0, 0, 0)
<Error>  01:54:40 WindowServer_arm64[4213]: Timeout waiting for getProperty: backlight-calibration-parameters, _timeoutCount: 2
<Error>  01:54:40 WindowServer_arm64[4213]: Got back calibration from DCP of unknown type (null)!!
<Notice> 01:54:40 WindowServer_arm64[4213]: capabilities = {
<Notice> 01:55:01 WindowServer_arm64[4213]: Display 1 setting display headroom hint to 1
<Notice> 01:55:07 kernel_task[0]: WindowServer_arm64[4213] Corpse allowed 1 of 5
<Notice> 01:55:07 kernel_task[0]: EXC_RESOURCE -> WindowServer_arm64[4213] exceeded mem limit: InactiveSoft 5120 MB (non-fatal)
<Notice> 01:55:07 ReportCrash[3768]: Formulating simulated 309 report for corpse[4213] WindowServer_arm64

Full log is here. I’d really appreciate it if anyone could help me.

[DCMMC]

I also received:
<Notice> 15:09:45 kernel_task[0]: 20277.920 memorystatus: killing_top_process pid 5431 [MTLSimDriverHost] (vm-compressor-space-shortage 10) 147073KB - memorystatus_available_pages: 132553 compressor_size:7946

[DCMMC]

lldb:

(lldb) process connect connect://192.168.5.7:3335
Process 7612 stopped
* thread #1, stop reason = signal SIGSTOP
    frame #0: 0x0000000102908a40 dyld`_dyld_start
dyld`_dyld_start:
->  0x102908a40 <+0>:  mov    x0, sp
    0x102908a44 <+4>:  and    sp, x0, #0xfffffffffffffff0
    0x102908a48 <+8>:  mov    x29, #0x0 ; =0
    0x102908a4c <+12>: mov    x30, #0x0 ; =0
Target 0: (WindowServer_arm64) stopped.
(lldb) bt
* thread #1, stop reason = signal SIGSTOP
  * frame #0: 0x0000000102908a40 dyld`_dyld_start
(lldb) c
Process 7612 resuming
(lldb)
error: Process is running.  Use 'process interrupt' to pause execution.
warning: libobjc.A.dylib is being read from process memory. This indicates that LLDB could not find the on-disk shared cache for this device. This will likely reduce debugging performance.

Process 7612 stopped
* thread #1, name = 'ws_main_thread', queue = 'com.apple.main-thread', stop reason = EXC_RESOURCE (RESOURCE_TYPE_MEMORY: high watermark memory limit exceeded) (limit=5120 MB)
    frame #0: 0x0000000193ff3f14 libsystem_kernel.dylib`mach_msg2_trap + 8
libsystem_kernel.dylib`mach_msg2_trap:
->  0x193ff3f14 <+8>: ret

libsystem_kernel.dylib`macx_swapon:
    0x193ff3f18 <+0>: mov    x16, #-0x30 ; =-48
    0x193ff3f1c <+4>: svc    #0x80
    0x193ff3f20 <+8>: ret
Target 0: (WindowServer_arm64) stopped.
(lldb) bt
* thread #1, name = 'ws_main_thread', queue = 'com.apple.main-thread', stop reason = EXC_RESOURCE (RESOURCE_TYPE_MEMORY: high watermark memory limit exceeded) (limit=5120 MB)
  * frame #0: 0x0000000193ff3f14 libsystem_kernel.dylib`mach_msg2_trap + 8
    frame #1: 0x0000000194006240 libsystem_kernel.dylib`mach_msg2_internal + 80
    frame #2: 0x00000001971bc71c IOKit`io_connect_method + 516
    frame #3: 0x00000001971bc4ec IOKit`IOConnectCallMethod + 236
    frame #4: 0x000000019d4607b8 IOSurface`IOSurfaceClientCreateChild + 268
    frame #5: 0x000000019d45cbfc IOSurface`-[IOSurface initWithProperties:] + 104
    frame #6: 0x000000019d45cb78 IOSurface`IOSurfaceCreate + 52
    frame #7: 0x000000019b6b90f4 QuartzCore`CA::SurfaceUtil::CAIOSurfaceCreate(unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, unsigned long long, CA::SurfaceUtil::SurfaceAlignment, __CFString const*) + 16196
    frame #8: 0x000000019b81ecac QuartzCore`CA::WindowServer::IOSurface::allocate_iosurface(CA::WindowServer::Display*, unsigned int, unsigned int, unsigned int, unsigned int, CA::WindowServer::SurfaceColorPrimaries, unsigned int, unsigned int, unsigned long long, __CFString const*) + 196
    frame #9: 0x000000019b8bfe40 QuartzCore`CA::WindowServer::IOMFBDisplay::create_surface(unsigned int, unsigned int, unsigned int, unsigned int, CA::WindowServer::SurfaceColorPrimaries, unsigned int, unsigned long long, __CFString const*) + 60
    frame #10: 0x000000019b756b3c QuartzCore`CA::WindowServer::Display::allocate_surface(unsigned int, unsigned int, unsigned int, unsigned int, CA::WindowServer::SurfaceColorPrimaries, unsigned int, unsigned long long, __CFString const*, bool) + 556
    frame #11: 0x000000019b8cec90 QuartzCore`CA::WindowServer::IOMFBDisplay::current_page_surface(bool, bool, bool, bool, unsigned long long, bool) + 848
    frame #12: 0x000000019b85ee0c QuartzCore`-[CAWindowServerDisplay currentSurface:protection:needsClear:readOnly:vsyncOff:] + 116
    frame #13: 0x0000000198e94abc SkyLight`WS::Displays::SLCADisplay::begin_update(WS::Displays::Update&) + 3044
    frame #14: 0x0000000198fb0ba4 SkyLight`WS::Updater::UpdateDisplays(CGXConnection*, bool) + 13148
    frame #15: 0x0000000198fa3264 SkyLight`update_display_callback(void*, double) + 1368
    frame #16: 0x000000019901714c SkyLight`run_timer_pass + 504
    frame #17: 0x0000000199017258 SkyLight`non_coalesced_timer_handler + 24
    frame #18: 0x0000000199055f60 SkyLight`post_port_data + 280
    frame #19: 0x0000000199055bb8 SkyLight`CGXRunOneServicesPass + 1404
    frame #20: 0x00000001990566c4 SkyLight`server_loop + 140
    frame #21: 0x0000000199056504 SkyLight`SLXServer + 1332
    frame #22: 0x00000001027f33bc WindowServer_arm64`_mh_execute_header + 13244
    frame #23: 0x0000000193cdbf28 dyld`start + 2236

[khanhduytran0]

Can you try unsetting CA_DISABLE_SWAP_ICC environment variable?

[DCMMC]

After unsetting CA_DISABLE_SWAP_ICC, oom issue disappeared but still got just a black mouse in the screen without any other content. Tried to run a vnc server but got signal SIGABRT. Seems that XPC_ERROR_CONNECTION_INVALID is the main reason of abort.

vnc server output:

2025-09-14 11:39:07.132 OSXvnc-server[51621:1780122] Detected HiDPI Display with scaling factor of 2.000000
2025-09-14 11:39:07.132 OSXvnc-server[51621:1780122] OS_LOG: Requesting ScreenCapture access
2025-09-14 11:53:32.560 OSXvnc-server[51621:1780122] There was an error getting screen shot

oslog (full log):

<Notice> 02:39:07 OSXvnc-server[51621]: OS_LOG: SEND: %d/%d %{public}s to %{public}s: request: %{public}s
<Error>  02:39:07 OSXvnc-server[51621]: send_message_with_reply(): XPC_ERROR_CONNECTION_INVALID for message 0x1139043b0
<Error>  02:39:07 OSXvnc-server[51621]: TCCAccessRequest_block_invoke: Connection invalid
<Notice> 02:39:07 OSXvnc-server[51621]: OS_LOG: Set a breakpoint at %s to catch errors/faults as they are logged.
<Notice> 02:39:07 OSXvnc-server[51621]: Set a breakpoint at SLSLogBreak to catch errors/faults as they are logged.
<Error>  02:39:07 OSXvnc-server[51621]: 0 is not a valid connection ID
<Error>  02:39:07 WindowServer_arm64[51623]: MTLCompiler: Compiler encountered XPC_ERROR_CONNECTION_INVALID: <private> (is the OS shutting down?)
(null)02:39:07 WindowServer_arm64[51623]: MTLCompiler: failed to ping the compiler service 10 times over 0.061375 ms
<Error>  02:39:07 WindowServer_arm64[51623]: <private>
<Notice> 02:39:07 WindowServer_arm64[51623]: OS_LOG: Hit the server for a process handle %llx that resolved to: %{public}@
<Notice> 02:39:07 WindowServer_arm64[51623]: Hit the server for a process handle 12a75f6000009003 that resolved to: [(null):36867]
<Notice> 02:44:26 launchservicesd[51387]: OS_LOG: SESSION: DestroyNonExistentSessions(), checking for sessions which can be reaped.
<Notice> 02:44:26 launchservicesd[51387]: OS_LOG: LSNotification::DestroyNotificationsForNonExistentSessions()
<Notice> 02:44:26 launchservicesd[51387]: OS_LOG: LSNotification::DestroyNotificationsForNonExistentSessions(), finished
<Notice> 02:44:26 launchservicesd[51387]: OS_LOG: launchservicesd running with an acceptable number of mach_port_ts.  No need to write a bug for this log; see <rdar://problem/29169308>. ports=%ld(send=%ld,recv=%ld,sendonce=%ld,set=%ld,dead=%ld,label=%ld) max=%ld
<Notice> 02:49:46 launchservicesd[51387]: OS_LOG: SESSION: DestroyNonExistentSessions(), checking for sessions which can be reaped.
<Notice> 02:49:46 launchservicesd[51387]: OS_LOG: LSNotification::DestroyNotificationsForNonExistentSessions()
<Notice> 02:49:46 launchservicesd[51387]: OS_LOG: LSNotification::DestroyNotificationsForNonExistentSessions(), finished
<Notice> 02:49:46 launchservicesd[51387]: OS_LOG: launchservicesd running with an acceptable number of mach_port_ts.  No need to write a bug for this log; see <rdar://problem/29169308>. ports=%ld(send=%ld,recv=%ld,sendonce=%ld,set=%ld,dead=%ld,label=%ld) max=%ld
<Notice> 02:53:32 kernel_task[0]: WindowServer_arm64[51623] Corpse allowed 1 of 5
<Notice> 02:53:32 launchservicesd[51387]: OS_LOG: CLIENT: %p/%d Received XPC_ERROR on connection from our client, so invalidating it and our connection.
<Error>  02:53:32 OSXvnc-server[51621]:
<Notice> 02:53:32 OSXvnc-server[51621]: There was an error getting screen shot
<Notice> 02:53:32 OSXvnc-server[51621]: OS_LOG: Entering exit handler.
<Notice> 02:53:32 OSXvnc-server[51621]: Entering exit handler.
<Notice> 02:53:32 ReportCrash[29830]: ASI found [SkyLight] (sensitive) 'MTLRenderPipelineState creation failed for device Apple iOS simulator (100000635).
This typically indicates a Metal compiler failure.
Error from Metal: Error Domain=CompilerError Code=1 "Compiler encountered XPC_ERROR_CONNECTION_INVALID (is the OS shutting down?)" UserInfo={NSLocalizedDescription=Compiler encountered XPC_ERROR_CONNECTION_INVALID (is the OS shutting down?)}

backtraces:

(lldb) bt
* thread #1, name = 'ws_main_thread', queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * frame #0: 0x0000000199858724 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x000000019988fc28 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x000000019979dae8 libsystem_c.dylib`abort + 180
    frame #3: 0x000000019e73d56c SkyLight`MetalShader::CopyPipelineState(MetalContext*, bool, bool) + 2288
    frame #4: 0x000000019e76c3d4 SkyLight`MetalCompositeSimpleColorLayer + 136
    frame #5: 0x000000019e8a2458 SkyLight`MetalCompositeLayers + 56
    frame #6: 0x000000019e8a2058 SkyLight`CompositorMetal::composite(WSCompositeSourceLayer*, WSCompositeDestination*, bool, bool) + 348
    frame #7: 0x000000019e8a1ed8 SkyLight`CompositorMetal::CompositeLayersToDestination(WSCompositeSourceLayer*, WSCompositeDestination*, unsigned long long) + 276
    frame #8: 0x000000019e823cc0 SkyLight`WSRedrawWindowsToDestinationAndRegion + 2032
    frame #9: 0x000000019e886c8c SkyLight`CaptureSurfaceMetal::Populate(CGXWindowSubArray, CGSRegionObject*, CGSRegionObject**, CGXRedrawOption, WSWindowTagBitMask, WSWindowTagBitMask, bool) + 260
    frame #10: 0x000000019e7e18dc SkyLight`CaptureSurfacePopulateWithArray + 96
    frame #11: 0x000000019e6316b8 SkyLight`invocation function for block in WS::Capture::create_iosurface_for_window_list(CGXConnection*, unsigned int*, unsigned int, unsigned int, unsigned int, CGRect const&, CGColorSpace*, bool, bool, bool*, int, WSMatrix, CGSize, float vector[4], bool, WS::Capture::SurfacePool*, double*, CGRect*) + 608
    frame #12: 0x000000019e63104c SkyLight`WS::Capture::create_iosurface_for_window_list(CGXConnection*, unsigned int*, unsigned int, unsigned int, unsigned int, CGRect const&, CGColorSpace*, bool, bool, bool*, int, WSMatrix, CGSize, float vector[4], bool, WS::Capture::SurfacePool*, double*, CGRect*) + 3400
    frame #13: 0x000000019e8f3edc SkyLight`_XHWCaptureDesktop + 772
    frame #14: 0x000000019e67f400 SkyLight`invocation function for block in connectionHandler(unsigned int, mach_msg_header_t*, void*) + 300
    frame #15: 0x000000019e67f288 SkyLight`invocation function for block in connectionHandler(unsigned int, mach_msg_header_t*, void*) + 112
    frame #16: 0x000000019e84eb3c SkyLight`CGXHandleMessage + 156
    frame #17: 0x000000019e67ef1c SkyLight`connectionHandler(unsigned int, mach_msg_header_t*, void*) + 256
    frame #18: 0x000000019e8b1f60 SkyLight`post_port_data + 280
    frame #19: 0x000000019e8b1bb8 SkyLight`CGXRunOneServicesPass + 1404
    frame #20: 0x000000019e8b26c4 SkyLight`server_loop + 140
    frame #21: 0x000000019e8b2504 SkyLight`SLXServer + 1332
    frame #22: 0x000000010293b3bc WindowServer_arm64`_mh_execute_header + 13244
    frame #23: 0x0000000199537f28 dyld`start + 2236

[DCMMC]

Should I do something to make the process in chroot environment find the MTLCompilerService in iOS side?

[DCMMC]

Finally I fixed this issue by replacing xpc_add_bundle to _xpc_bootstrap_services back when trying to register MTLCompilerService.xpc, lol

@@ -63,11 +63,7 @@ void loadImageCallback(const struct mach_header* header, intptr_t vmaddr_slide)
            *swapEnd = 0x52808d83; // mov    w3, #0x46c
        });
    } else if(!strncmp(info.dli_fname, libxpcPath, strlen(libxpcPath))) {
        // register MTLCompilerService.xpc
        xpc_object_t dict = (xpc_object_t)xpc_dictionary_create(NULL, NULL, 0);
        xpc_dictionary_set_uint64(dict, "/System/Library/Frameworks/Metal.framework/Metal", 2);
        void(*_xpc_bootstrap_services)(xpc_object_t) = MSFindSymbol((MSImageRef)header, "__xpc_bootstrap_services");
        _xpc_bootstrap_services(dict);
        // xpc_add_bundle("/System/Library/Frameworks/Metal.framework/XPCServices/MTLCompilerService.xpc", 2);
    }
}

[DCMMC]

[DCMMC]

thank you for all your work. it's awesome

[khanhduytran0]

Finally I fixed this issue by replacing xpc_add_bundle to _xpc_bootstrap_services back when trying to register MTLCompilerService.xpc, lol

Oh 😭, I guess I never encountered this cause I never deleted all cached shaders to force a shader recompile again. Anyways congrats and thank you for finding it out!