XCTool crash when running Kiwi test suite

[BobReid]

Kiwi test suite does not run in XCTool due to a crash in the test name parsing.

Test name when run in XCode 7 is:

 -[JSTDocumentSpec JSTDocument_CreatesAnEmptyInstanceWithDefaultConstructor]

XCTool reports

Test name seems to be malformed: -[JSTDocumentSpec (null)]

Environment:
XCTool: 0.2.6
XCode: 7
Kiwi: 2.4

Full crash report:

CRASH REPORT: Joist_2015-09-24-132221_distiller-build-2.crash

Process:               Joist [5749]
Path:                  /Users/USER/Library/Developer/CoreSimulator/Devices/53600017-30F1-428A-A16A-25C6CEDBD849/data/Containers/Bundle/Application/FC9EB987-ACB4-4013-ACFD-F3250F77A81C/Joist.app/Joist
Identifier:            Joist
Version:               2.3.9 (2.3.9.220-jw/xcode7_build_fix)
Code Type:             X86-64 (Native)
Parent Process:        launchd_sim [5666]
Responsible:           launchd_sim [5666]
User ID:               502

Date/Time:             2015-09-24 13:22:17.475 -0700
OS Version:            Mac OS X 10.10.5 (14F27)
Report Version:        11
Anonymous UUID:        753DDB54-A115-A5AA-BF28-5453F6681B0C


Time Awake Since Boot: 2000 seconds

Crashed Thread:        0  Dispatch queue: xctool.events

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000

Application Specific Information:
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: '*** Assertion failure in void ParseClassAndMethodFromTestName(NSString **, NSString **, NSString *), /tmp/xctool20150924-468-106dw9i/Common/ParseTestName.m:38: Test name seems to be malformed: -[JSTDocumentSpec (null)]'
terminating with uncaught exception of type NSException
abort() called
CoreSimulator 179 - Device: iPhone 6 - Runtime: iOS 9.0 (13A340) - DeviceType: iPhone 6

Application Specific Backtrace 1:
0   CoreFoundation                      0x000000010ed7ff65 __exceptionPreprocess + 165
1   libobjc.A.dylib                     0x000000010e6fedeb objc_exception_throw + 48
2   CoreFoundation                      0x000000010ed7fe9d +[NSException raise:format:] + 205
3   otest-shim-ios.dylib                0x000000010a7cd08a -[XCToolAssertionHandler handleFailureInFunction:file:lineNumber:description:] + 246
4   otest-shim-ios.dylib                0x000000010a7cf768 ParseClassAndMethodFromTestName + 702
5   otest-shim-ios.dylib                0x000000010a7ced08 __XCToolLog_testCaseDidStart_block_invoke + 61
6   libdispatch.dylib                   0x000000010fee44a7 _dispatch_client_callout + 8
7   libdispatch.dylib                   0x000000010feccfd0 _dispatch_barrier_sync_f_invoke + 99
8   otest-shim-ios.dylib                0x000000010a7ce8d8 XCTestLog_testCaseDidStart + 120
9   XCTest                              0x000000011b93d6b4 -[XCTestObservationCenter _testCaseDidStart:] + 529
10  XCTest                              0x000000011b92c6b4 -[XCTestCaseRun start] + 85
11  XCTest                              0x000000011b931116 -[XCTestCase performTest:] + 142
12  otest-shim-ios.dylib                0x000000010a7cf3ec XCPerformTestWithSuppressedExpectedAssertionFailures + 505
13  XCTest                              0x000000011b92ef09 -[XCTestSuite performTest:] + 377
14  XCTest                              0x000000011b92ef09 -[XCTestSuite performTest:] + 377
15  XCTest                              0x000000011b93c91b -[XCTestObservationCenter _observeTestExecutionForBlock:] + 615
16  XCTest                              0x000000011b9619e9 _XCTestMain + 1052
17  CoreFoundation                      0x000000010ecac08c __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__ + 12
18  CoreFoundation                      0x000000010eca1d15 __CFRunLoopDoBlocks + 341
19  CoreFoundation                      0x000000010eca1472 __CFRunLoopRun + 850
20  CoreFoundation                      0x000000010eca0e98 CFRunLoopRunSpecific + 488
21  GraphicsServices                    0x0000000110f7aad2 GSEventRunModal + 161
22  UIKit                               0x000000010ce78676 UIApplicationMain + 171
23  Joist                               0x000000010939327f main + 111
24  libdyld.dylib                       0x000000010ff1492d start + 1
25  ???                                 0x0000000000000005 0x0 + 5

Thread 0 Crashed:: Dispatch queue: xctool.events
0   libsystem_kernel.dylib          0x0000000110260286 __pthread_kill + 10
1   libsystem_c.dylib               0x000000010ffc0a44 abort + 129
2   otest-shim-ios.dylib            0x000000010a7cd4d0 __abort + 104
3   libc++abi.dylib                 0x000000010f7df051 abort_message + 257
4   libc++abi.dylib                 0x000000010f804ac9 default_terminate_handler() + 267
5   libobjc.A.dylib                 0x000000010e6ff046 _objc_terminate() + 103
6   libc++abi.dylib                 0x000000010f80226e std::__terminate(void (*)()) + 8
7   libc++abi.dylib                 0x000000010f8022e3 std::terminate() + 51
8   libobjc.A.dylib                 0x000000010e6fefc5 objc_terminate + 9
9   libdispatch.dylib               0x000000010fee44bb _dispatch_client_callout + 28
10  libdispatch.dylib               0x000000010feccfd0 _dispatch_barrier_sync_f_invoke + 99
11  otest-shim-ios.dylib            0x000000010a7ce8d8 XCTestLog_testCaseDidStart + 120
12  com.apple.dt.XCTest             0x000000011b93d6b4 -[XCTestObservationCenter _testCaseDidStart:] + 529
13  com.apple.dt.XCTest             0x000000011b92c6b4 -[XCTestCaseRun start] + 85
14  com.apple.dt.XCTest             0x000000011b931116 -[XCTestCase performTest:] + 142
15  otest-shim-ios.dylib            0x000000010a7cf3ec XCPerformTestWithSuppressedExpectedAssertionFailures + 505
16  com.apple.dt.XCTest             0x000000011b92ef09 -[XCTestSuite performTest:] + 377
17  com.apple.dt.XCTest             0x000000011b92ef09 -[XCTestSuite performTest:] + 377
18  com.apple.dt.XCTest             0x000000011b93c91b -[XCTestObservationCenter _observeTestExecutionForBlock:] + 615
19  com.apple.dt.XCTest             0x000000011b9619e9 _XCTestMain + 1052
20  com.apple.CoreFoundation        0x000000010ecac08c __CFRUNLOOP_IS_CALLING_OUT_TO_A_BLOCK__ + 12
21  com.apple.CoreFoundation        0x000000010eca1d15 __CFRunLoopDoBlocks + 341
22  com.apple.CoreFoundation        0x000000010eca1472 __CFRunLoopRun + 850
23  com.apple.CoreFoundation        0x000000010eca0e98 CFRunLoopRunSpecific + 488
24  com.apple.GraphicsServices      0x0000000110f7aad2 GSEventRunModal + 161
25  com.apple.UIKit                 0x000000010ce78676 UIApplicationMain + 171
26  ???                             0x000000010939327f 0 + 4449710719
27  libdyld.dylib                   0x000000010ff1492d start + 1

Thread 1:: Dispatch queue: com.apple.libdispatch-manager
0   libsystem_kernel.dylib          0x0000000110261232 kevent64 + 10
1   libdispatch.dylib               0x000000010fed465a _dispatch_mgr_thread + 54

Thread 2:
0   libsystem_kernel.dylib          0x000000011026094a __workq_kernreturn + 10
1   libsystem_pthread.dylib         0x00000001102253dd start_wqthread + 13

Thread 3:
0   libsystem_kernel.dylib          0x000000011026094a __workq_kernreturn + 10
1   libsystem_pthread.dylib         0x00000001102253dd start_wqthread + 13

Thread 4:
0   libsystem_kernel.dylib          0x000000011026094a __workq_kernreturn + 10
1   libsystem_pthread.dylib         0x00000001102253dd start_wqthread + 13

Thread 5:
0   libsystem_kernel.dylib          0x000000011026094a __workq_kernreturn + 10
1   libsystem_pthread.dylib         0x00000001102253dd start_wqthread + 13

Thread 6:
0   libsystem_kernel.dylib          0x000000011026094a __workq_kernreturn + 10
1   libsystem_pthread.dylib         0x00000001102253dd start_wqthread + 13

Thread 7:
0   libsystem_kernel.dylib          0x000000011026094a __workq_kernreturn + 10
1   libsystem_pthread.dylib         0x00000001102253dd start_wqthread + 13

Thread 8:
0   libsystem_kernel.dylib          0x000000011026094a __workq_kernreturn + 10
1   libsystem_pthread.dylib         0x00000001102253dd start_wqthread + 13

Thread 9:
0   libsystem_kernel.dylib          0x000000011026048a __semwait_signal + 10
1   libc++.1.dylib                  0x000000010a826100 std::__1::this_thread::sleep_for(std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l> > const&) + 75
2   com.apple.JavaScriptCore        0x0000000116f4e5db bmalloc::Heap::scavenge(std::__1::unique_lock<bmalloc::StaticMutex>&, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000l> >) + 155
3   com.apple.JavaScriptCore        0x0000000116f4e514 bmalloc::Heap::concurrentScavenge() + 68
4   com.apple.JavaScriptCore        0x0000000116f509ba bmalloc::AsyncTask<bmalloc::Heap, void (bmalloc::Heap::*)()>::entryPoint() + 90
5   com.apple.JavaScriptCore        0x0000000116f50959 bmalloc::AsyncTask<bmalloc::Heap, void (bmalloc::Heap::*)()>::pthreadEntryPoint(void*) + 9
6   libsystem_pthread.dylib         0x000000011022805a _pthread_body + 131
7   libsystem_pthread.dylib         0x0000000110227fd7 _pthread_start + 176
8   libsystem_pthread.dylib         0x00000001102253ed thread_start + 13

Thread 10:: WebThread
0   libsystem_kernel.dylib          0x0000000110260166 __psynch_mutexwait + 10
1   com.apple.WebCore               0x0000000115cea8a4 _WebTryThreadLock(bool) + 100
2   com.apple.WebCore               0x0000000115cec32e WebRunLoopLock(__CFRunLoopObserver*, unsigned long, void*) + 30
3   com.apple.CoreFoundation        0x000000010ecab9d7 __CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__ + 23
4   com.apple.CoreFoundation        0x000000010ecab947 __CFRunLoopDoObservers + 391
5   com.apple.CoreFoundation        0x000000010eca1720 __CFRunLoopRun + 1536
6   com.apple.CoreFoundation        0x000000010eca0e98 CFRunLoopRunSpecific + 488
7   com.apple.WebCore               0x0000000115cec305 RunWebThread(void*) + 469
8   libsystem_pthread.dylib         0x000000011022805a _pthread_body + 131
9   libsystem_pthread.dylib         0x0000000110227fd7 _pthread_start + 176
10  libsystem_pthread.dylib         0x00000001102253ed thread_start + 13

Thread 11:: AFNetworking
0   libsystem_kernel.dylib          0x000000011025b4de mach_msg_trap + 10
1   libsystem_kernel.dylib          0x000000011025a64f mach_msg + 55
2   com.apple.CoreFoundation        0x000000010eca21d4 __CFRunLoopServiceMachPort + 212
3   com.apple.CoreFoundation        0x000000010eca162f __CFRunLoopRun + 1295
4   com.apple.CoreFoundation        0x000000010eca0e98 CFRunLoopRunSpecific + 488
5   com.apple.Foundation            0x000000010c001891 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 267
6   com.apple.Foundation            0x000000010c094221 -[NSRunLoop(NSRunLoop) run] + 74
7   ???                             0x000000010960d2c8 0 + 4452307656
8   com.apple.Foundation            0x000000010c09f36b __NSThread__start__ + 1198
9   libsystem_pthread.dylib         0x000000011022805a _pthread_body + 131
10  libsystem_pthread.dylib         0x0000000110227fd7 _pthread_start + 176
11  libsystem_pthread.dylib         0x00000001102253ed thread_start + 13

Thread 12:: com.apple.NSURLConnectionLoader
0   libsystem_kernel.dylib          0x000000011025b4de mach_msg_trap + 10
1   libsystem_kernel.dylib          0x000000011025a64f mach_msg + 55
2   com.apple.CoreFoundation        0x000000010eca21d4 __CFRunLoopServiceMachPort + 212
3   com.apple.CoreFoundation        0x000000010eca162f __CFRunLoopRun + 1295
4   com.apple.CoreFoundation        0x000000010eca0e98 CFRunLoopRunSpecific + 488
5   com.apple.CFNetwork             0x000000010b44a084 +[NSURLConnection(Loader) _resourceLoadLoop:] + 412
6   com.apple.Foundation            0x000000010c09f36b __NSThread__start__ + 1198
7   libsystem_pthread.dylib         0x000000011022805a _pthread_body + 131
8   libsystem_pthread.dylib         0x0000000110227fd7 _pthread_start + 176
9   libsystem_pthread.dylib         0x00000001102253ed thread_start + 13

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000000  rbx: 0x0000000000000006  rcx: 0x00007fff5697c8d8  rdx: 0x0000000000000000
  rdi: 0x000000000000070b  rsi: 0x0000000000000006  rbp: 0x00007fff5697c900  rsp: 0x00007fff5697c8d8
   r8: 0x0000000000000000   r9: 0x0000000000000cd0  r10: 0x0000000008000000  r11: 0x0000000000000206
  r12: 0x00007fff5697ca80  r13: 0x00007ffc81e765e0  r14: 0x000000011022e300  r15: 0x00007fff5697c960
  rip: 0x0000000110260286  rfl: 0x0000000000000206  cr2: 0x000000011022a2ec

Logical CPU:     0
Error Code:      0x02000148
Trap Number:     133

[Rypac]

+1 Also experiencing the same issue when running an entire suite of Kiwi tests.

Running a single spec however works fine.

[ExtremeMan]

I've fixed similar bug in other testing frameworks: gtest and Specta. Also I've seen Quick framework fixed it on their side. Did you open same issue on Kiwi github repo? I will appreciate if they fix it on their end rather then I will try to fix it in xctool.

[ExtremeMan]

Links with more info: #572, specta/specta#184, Quick/Quick#388, mattstevens/xcode-googletest#3.

[Rypac]

Great, thanks for the info.

Looks like there is already a pending fix for Kiwi.
kiwi-bdd/Kiwi#648

[ExtremeMan]

@Rypac, can you check if that PR fixes xctool compatibility as well, please?

[Rypac]

@ExtremeMan Will do.

[Rypac]

@ExtremeMan Yeah the PR fixes the issue for me. Thanks for your help!

[BobReid]

@ExtremeMan The comments on the quick bug seem to indicate they think its an issue on xctool side but they merged it anyway to expedite the fix.

Not sure where the most appropriate fix should be made.

[ExtremeMan]

@BobReid, I know. I don't agree that it is an xctool bug. Closing this issue since there is a PR that fixes it in Kiwi.

[ExtremeMan]

@Rypac, thanks for confirming! I appreciate that.

[BobReid]

Cool, I'll test the fix as well

[PaulTaykalo]

@BobReid @ExtremeMan As far as I see - it's not a xctool bug. I've described what's going on there - kiwi-bdd/Kiwi#648.
Xcode also isn't happy with Kiwi tests
Probably the only "bug" is that xctool crashes, when Xcode simply showing incorrect test name as (null) or not showing anything at all