ExitException when calling System.exit(0) from Method#invoke

[pcworld]

Try to run the following class e.g. on final2.subtests.ValidCommandLineArgumentsTest#orderArgumentTest:

package reflectiontest;

import java.lang.reflect.InvocationTargetException;

public class Test {

    public static void main(String[] args) {
        new Test();
    }

    public Test() {
        try {
            getClass().getMethod("test").invoke(this);
        } catch (IllegalAccessException e) {
            e.printStackTrace();
        } catch (IllegalArgumentException e) {
            e.printStackTrace();
        } catch (InvocationTargetException e) {
            e.printStackTrace();
        } catch (NoSuchMethodException e) {
            e.printStackTrace();
        } catch (SecurityException e) {
            e.printStackTrace();
        }
        // if using the following instead of invoke, everything works fine:
        // test();
    }

    public void test() {
        System.exit(0);
    }

}

leads to the following exception (stacktrace printed on System.out):

java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at reflectiontest.Test.<init>(Test.java:13)
    at reflectiontest.Test.main(Test.java:8)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at test.TestObject.run(TestObject.java:651)
    at test.TestObject.runStaticVoid(TestObject.java:468)
    at test.TestObject.runStaticVoid(TestObject.java:481)
    at test.InteractiveConsoleTest.noOutputTest(InteractiveConsoleTest.java:450)
    at test.InteractiveConsoleTest.noOutputTest(InteractiveConsoleTest.java:433)
    at final2.subtests.ValidCommandLineArgumentsTest.orderArgumentTest(ValidCommandLineArgumentsTest.java:44)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: test.framework.ExitException: The method called System.exit(0)
    at test.framework.NoExitSecurityManager.checkExit(NoExitSecurityManager.java:33)
    at java.lang.Runtime.exit(Runtime.java:107)
    at java.lang.System.exit(System.java:962)
    at reflectiontest.Test.test(Test.java:30)
    ... 26 more
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at reflectiontest.Test.<init>(Test.java:13)
    at reflectiontest.Test.main(Test.java:8)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at test.TestObject.run(TestObject.java:651)
    at test.TestObject.runStaticVoid(TestObject.java:468)
    at test.TestObject.runStaticVoid(TestObject.java:481)
    at test.InteractiveConsoleTest.noOutputTest(InteractiveConsoleTest.java:450)
    at test.InteractiveConsoleTest.noOutputTest(InteractiveConsoleTest.java:433)
    at final2.subtests.ValidCommandLineArgumentsTest.orderArgumentTest(ValidCommandLineArgumentsTest.java:45)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: test.framework.ExitException: The method called System.exit(0)
    at test.framework.NoExitSecurityManager.checkExit(NoExitSecurityManager.java:33)
    at java.lang.Runtime.exit(Runtime.java:107)
    at java.lang.System.exit(System.java:962)
    at reflectiontest.Test.test(Test.java:30)
    ... 26 more

even though ValidCommandLineArgumentsTest calls setAllowedSystemExitStatus(SystemExitStatus.WITH_0) in its constructor.
When calling System.exit(0) directly without Method#invoke, the test succeeds without exceptions (as expected).

[pcworld]

The issue is that TestObject relies on an InvocationTargetException to test for the program's exit code. If the tested program catches this exception itself, this can lead to unexpected behavior in the user's program.

[pcworld]

A workaround is to add the following code whenever you catch InvocationTargetException in your program (you might need to adjust the exit code):

if (e.getCause().getClass().getCanonicalName().equals("test.framework.ExitException")) {
    System.exit(0);
}

If the test framework is in your class path, you could also use something like (untested):

if (e.getCause() instanceof test.framework.ExitException) {
    System.exit(((test.framework.ExitException) e.getCause()).getStatus());
}

[jGleitz]

I'd like to explain what's actually happening here, if someone is interested.

• JUnit does not support testing methods that call System#exit. If tested code does exit the VM through System#exit, this leaves the test in some kind of hanging state.
• Therefore, we use an SecurityManager that throws an Exception if the code tries to call System#exit. As long as nobody catches this exception, it is wrapped in an InvocationTargetException by Java Reflection.
• This exception usually causes the code to terminate just like System#exit would – as long as nobody is catching ExitException or InvocationTargetException.
• TestObject than catches the InvocationTargetException and fails the test if the system exit status was not appropriate.

[jGleitz]

In the moment I do not see how to avoid this bug. We need to make sure that a tested class calling System#exit terminates on the spot if it does so, but does not actually get to call System#exit! If it was a bad exit code, we could just fail the test, but what should we do if it was an allowed one? I don't know how to gracefully abort a test without failing it.

Therefore, we rely on that exception. As long as no one uses reflection in his code, this is not really a problem. But I see that it's not the best way to do this. System Rules by Stefan Birkner seem to offer a solutions. Unfortunately, Eclipse does not ship his library per default. And I don't quite want to include libraries written by others in this repository.

Does anybody else see a better solution than the current one?

[jGleitz]

But anyway, what the heck are you trying to achieve that you found this bug, @pcworld?

[pcworld]

System Rules by Stefan Birkner seem to offer a solutions.

I skimmed over their code and it seems that their architecture is similar to ours, only that you can pass assertions to their exit security manager which are then tested after a program exit. I.e. this would be easy to add to our implementation.
The hard part is that if the exit code is valid, we would have to pass all the logic for testing the program's output to the security manager (by defining an interface and implementing it for the test methods, e.g. #sessionTest), which would require some refactoring, as currently the tests work with the output of the TestObject#run* methods, but currently do not pass their assertions as an argument.
Correct me if I'm wrong about how the system-rules framework works.

But anyway, what the heck are you trying to achieve that you found this bug, @pcworld?

I wrote a class that creates a simple CLI using annotations and reflections (similar to cliche), and in final2 my #quit and #move methods have to be able to call System.exit.

[jGleitz]

and it seems that their architecture is similar to ours

Well, that doesn't help then.

The hard part is that if the exit code is valid, we would have to pass all the logic for testing the program's output to the security manager

I don't have any problems with refactoring. But: If the tested class calls System#exit with a correct value and produces correct output, the test will pass. If we don't throw an exception then, the tested class will continue executing while the programmer relied on the code being terminated. Therefore, we'd quite likely have failed test for a completely correct run. Correct me if I'm wrong, but it would not improve a thing.

I wrote a class that creates a simple CLI

Never thought about doing such a thing, but that's just beautiful! I don't like using reflection for the final tasks, but if I will ever have to do an CLI again, I remember this way of doing it!

[pcworld]

Oh, I got that wrong then. Does system-rules provide a solution to our problem at all?

Some "hacky" solution that might work: Do Method#invoke in a new thread, and as soon as the security manager catches a System.exit call, use the "evil" Thread#stop method to kill the thread.
Doesn't work that easily if the tested program uses more than one thread, though I'm not sure if our current way works with multi-threading.

On the other hand, I'm the only one who's complained about this bug so far, and I can live with the mentioned workaround, so fixing this might not be worth the effort.

[jGleitz]

If we're talking about hacky solutions here, I found this thread about an (almost) uncatchable ChuckNorrisException (worth a read for anybody interested in the guts of Java, imho).

Some of the solutions suggested there might work for rethrowing the SystemExitException.

[jGleitz]

I want to point out, that the issue here is not exactly that we are using an exception. Because nobody will catch it in “normal” code. But it gets wrapped in a Java-API-Exception, which one might catch. And that's a problem.

[pcworld]

(almost) uncatchable ChuckNorrisException

Nice, though even if this surpasses Method#invoke, we might not be able to catch it either.

But it gets wrapped in a Java-API-Exception, which one might catch

Something else that might work:
Replace Java's reflection invoke methods (using a classloader or so and somehow bypassing Java's security which might try to prevent that), and let them only rethrow the original invoke method's InvocationTargetException if e.getCause() is not an instance of ExitException, otherwise just rethrow the ExitException.

[jGleitz]

Replace Java's reflection invoke methods

As described in #99, I unfortunately did not find any way to replace (mock) a class in java.* yet.

[pcworld]

This is not related to our exit test mechanism, but was another side effect of using reflections in my application in combination with the test framework. When testing #155's MoveTest#testConsi(s)tensy, I kept getting the following exception on the 16th call of my move method (via Method#invoke):

java.lang.IllegalAccessError: class sun.reflect.GeneratedMethodAccessor1 cannot access its superclass sun.reflect.MethodAccessorImpl
    at sun.misc.Unsafe.defineClass(Native Method)
    at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
    at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395)
    at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:77)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:46)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at edu.kit.informatik.cli.CLI.processInputLine(CLI.java:96)
    at edu.kit.informatik.cli.CLI.commandLoop(CLI.java:127)
    at edu.kit.informatik.final2.AntMain.<init>(AntMain.java:89)
    at edu.kit.informatik.final2.AntMain.main(AntMain.java:360)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at test.TestObject.run(TestObject.java:657)
    at test.TestObject.runStaticVoid(TestObject.java:474)
    at test.TestObject.runStaticVoid(TestObject.java:487)
    at test.InteractiveConsoleTest.sessionTest(InteractiveConsoleTest.java:610)
    at test.InteractiveConsoleTest.sessionTest(InteractiveConsoleTest.java:648)
    at final2.subtests.MoveTest.testConsitensy(MoveTest.java:173)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

Looking at OpenJDK's sun.reflect.ReflectionFactory, there is an inflationThreshold property set to 15 (GPLv2):

    // "Inflation" mechanism. Loading bytecodes to implement
    // Method.invoke() and Constructor.newInstance() currently costs
    // 3-4x more than an invocation via native code for the first
    // invocation (though subsequent invocations have been benchmarked
    // to be over 20x faster). Unfortunately this cost increases
    // startup time for certain applications that use reflection
    // intensively (but only once per class) to bootstrap themselves.
    // To avoid this penalty we reuse the existing JVM entry points
    // for the first few invocations of Methods and Constructors and
    // then switch to the bytecode-based implementations.
    //
    // Package-private to be accessible to NativeMethodAccessorImpl
    // and NativeConstructorAccessorImpl
    private static boolean noInflation        = false;
    private static int     inflationThreshold = 15;

When passing -Dsun.reflect.noInflation=true to the JVM's arguments, all my Method#invoke calls fail. So sun.reflect.MethodAccessorGenerator seems broken.
This appears to be a bug in OpenJDK, or possibly Ubuntu's variant of it. Hopefully I'll get around to creating a simple reproduction and submitting it to Ubuntu/Debian/OpenJDK.
A workaround is to pass -Dsun.reflect.inflationThreshold=2147483647 (taken from this blog post) to the JVM's arguments.
This doesn't happen when I run the same test manually without making use of our test framework. Maybe the bug has something to do with nested calls of Method#invoke.