Don't let VMThread acquire Heap_lock (#300)

The Heap_lock performs safepoint check. Mutators may hold such locks and
enter safepoint. If the VMThread attempts to acquire this lock in a
VMOperation while some mutators are still holding this lock, the
VMThread will have to wait forever because mutators will not resume
until the VMOperation ends. Later versions of OpenJDK forbid the
VMThread from acquiring any lock that performs safepoint check.

We notify the ReferenceHandler thread when resuming mutators. But the
current code does this notification in a VMOperation, and will cause the
deadlock. When this happens, the process will freeze when a GC is
triggered. This is likely to happen when the heap size is small and GC
is triggered often.

We move the notification code to `mmtk_resume_mutators` which is
executed on a GC worker thread *after* mutators have been resumed. This
solves the deadlock issue.

Author: wks

openjdk/mmtkUpcalls.cpp

@@ -87,6 +87,15 @@ static void mmtk_resume_mutators(void *tls) {
     MutexLockerEx locker(MMTkHeap::heap()->gc_lock(), Mutex::_no_safepoint_check_flag);
     MMTkHeap::heap()->gc_lock()->notify_all();
   }
+
+  log_debug(gc)("Notifying mutators blocking on Heap_lock for reference pending list...");
+  // Note: That's the ReferenceHandler thread.
+  {
+    MutexLockerEx x(Heap_lock, Mutex::_no_safepoint_check_flag);
+    if (Universe::has_reference_pending_list()) {
+      Heap_lock->notify_all();
+    }
+  }
 }
 
 static const int GC_THREAD_KIND_WORKER = 1;

openjdk/mmtkVMCompanionThread.cpp

@@ -166,11 +166,4 @@ void MMTkVMCompanionThread::do_mmtk_stw_operation() {
     log_trace(gc)("do_mmtk_stw_operation: Reached _thread_resumed state. Notifying...");
     _lock->notify_all();
   }
-
-  {
-    MutexLockerEx x(Heap_lock, Mutex::_no_safepoint_check_flag);
-    if (Universe::has_reference_pending_list()) {
-      Heap_lock->notify_all();
-    }
-  }
 }