Skip to content

Disable deadlock detection for GC thread shutdown - #17885

Merged
gacholio merged 1 commit into
eclipse-openj9:masterfrom
tajila:criu_3
Aug 4, 2023
Merged

Disable deadlock detection for GC thread shutdown#17885
gacholio merged 1 commit into
eclipse-openj9:masterfrom
tajila:criu_3

Conversation

@tajila

@tajila tajila commented Jul 28, 2023

Copy link
Copy Markdown
Contributor

Disable deadlock detection for GC thread shutdown

GC releases threads in a multi-thread fashion. Threads will need to remove themselves from the threadgroup which requires a lock. Disable deadlock detection temporarily while this happens.

@tajila
tajila force-pushed the criu_3 branch 2 times, most recently from 0a46df9 to 211cf0e Compare July 28, 2023 20:35
@tajila
tajila requested a review from JasonFengJ9 July 28, 2023 20:36
Comment thread runtime/criusupport/criusupport.cpp Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While running j9gc_prepare_for_checkpoint(currentThread), the non-checkpoint thread(s) such as wait/sleep/park (threadhelp.cpp:timeCompensationHelper) in single-threaded mode has a small window to proceed and will stop eventually due to J9_PUBLIC_FLAGS_HALT_THREAD_FOR_CHECKPOINT set for those threads.
There is no risk of JVMCheckpointException since the hooks are already finished.
The only side-effect I see is that this short run might affect the application hook results if it assumes all non-checkpoint threads are paused, do we accept such a small restriction?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ya, I was thinking about this, perhaps we need two flags

@tajila
tajila force-pushed the criu_3 branch 4 times, most recently from 36fc62d to 6d3ebd2 Compare July 31, 2023 16:16
@tajila

tajila commented Jul 31, 2023

Copy link
Copy Markdown
Contributor Author

jenkins test sanity xlinux jdk17

@tajila

tajila commented Aug 1, 2023

Copy link
Copy Markdown
Contributor Author

@JasonFengJ9

Copy link
Copy Markdown
Member

@tajila Just launched a 2000x grinder with JDK17 x86-64_linux - https://openj9-jenkins.osuosl.org/job/Grinder/2637/

The intermittent inet issue was reproducible at JDK21 zLinux

For the best comparison, please provide a JDK21 zLinux SDK.

@JasonFengJ9

Copy link
Copy Markdown
Member

The intermittent CRIUSupport.clearInetAddressCache() issue was reproduced - https://openj9-jenkins.osuosl.org/job/Grinder_iteration_8/20/tapResults/
javacore.20230731.235505.2189.0001.txt

3XMTHREADINFO      "main" J9VMThread:0x0000000000017000, omrthread_t:0x00007F038001C3D0, java/lang/Thread:0x0000000687651A40, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000006876582C0)
3XMTHREADINFO1            (native thread ID:0x88E, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00041020)
3XMTHREADINFO2            (native stack address range from:0x00007F03879F5000, to:0x00007F0387A76000, size:0x81000)
3XMCPUTIME               CPU usage total: 0.111071548 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=24856 (0x6118)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/ref/ReferenceQueue.poll(ReferenceQueue.java:78(Compiled Code))
4XESTACKTRACE                at java/lang/invoke/MethodType$ConcurrentWeakInternSet.expungeStaleElements(MethodType.java:1406(Compiled Code))
4XESTACKTRACE                at java/lang/invoke/MethodType$ConcurrentWeakInternSet.get(MethodType.java:1366(Compiled Code))
4XESTACKTRACE                at java/lang/invoke/MethodType.makeImpl(MethodType.java:341(Compiled Code))
4XESTACKTRACE                at java/lang/invoke/MethodType.fromDescriptor(MethodType.java:1167)
4XESTACKTRACE                at java/lang/invoke/MethodType.fromMethodDescriptorString(MethodType.java:1143)
4XESTACKTRACE                at java/lang/invoke/MethodTypeHelper.vmResolveFromMethodDescriptorString(MethodTypeHelper.java:259)
4XESTACKTRACE                at java/lang/invoke/MethodHandleResolver.getCPMethodTypeAt(Native Method)
4XESTACKTRACE                at java/lang/invoke/MethodHandleResolver.getAdditionalBsmArg(MethodHandleResolver.java:389)
4XESTACKTRACE                at java/lang/invoke/MethodHandleResolver.resolveInvokeDynamic(MethodHandleResolver.java:188)
4XESTACKTRACE                at org/eclipse/openj9/criu/CRIUSupport.clearInetAddressCache(CRIUSupport.java:627)
4XESTACKTRACE                at org/eclipse/openj9/criu/CRIUSupport$$Lambda$13/0x0000000000000000.run(Bytecode PC:0)
4XESTACKTRACE                at org/eclipse/openj9/criu/J9InternalCheckpointHookAPI$J9InternalCheckpointHook.runHook(J9InternalCheckpointHookAPI.java:116)
4XESTACKTRACE                at org/eclipse/openj9/criu/J9InternalCheckpointHookAPI.runHooks(J9InternalCheckpointHookAPI.java:80)
4XESTACKTRACE                at org/eclipse/openj9/criu/J9InternalCheckpointHookAPI.runPostRestoreHooks(J9InternalCheckpointHookAPI.java:97)
4XESTACKTRACE                at org/eclipse/openj9/criu/CRIUSupport.checkpointJVMImpl(Native Method)
4XESTACKTRACE                at org/eclipse/openj9/criu/CRIUSupport.checkpointJVM(CRIUSupport.java:680)
4XESTACKTRACE                at org/openj9/criu/CRIUTestUtils.checkPointJVM(CRIUTestUtils.java:77)
4XESTACKTRACE                at org/openj9/criu/OptionsFileTest.dumpOptionsTestRequireDynamic(OptionsFileTest.java:295)
4XESTACKTRACE                at org/openj9/criu/OptionsFileTest.main(OptionsFileTest.java:69)
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK               protectedBacktrace+0x12 (0x00007F0387941162 [libj9prt29.so+0x26162])
4XENATIVESTACK               omrsig_protect+0x2b1 (0x00007F0387945881 [libj9prt29.so+0x2a881])
4XENATIVESTACK               omrintrospect_backtrace_thread_raw+0xbb (0x00007F038794164B [libj9prt29.so+0x2664b])
4XENATIVESTACK               omrsig_protect+0x2b1 (0x00007F0387945881 [libj9prt29.so+0x2a881])
4XENATIVESTACK               omrintrospect_backtrace_thread+0x7a (0x00007F0387940FDA [libj9prt29.so+0x25fda])
4XENATIVESTACK               setup_native_thread+0x1e3 (0x00007F0387941FB3 [libj9prt29.so+0x26fb3])
4XENATIVESTACK               omrintrospect_threads_startDo_with_signal+0x427 (0x00007F0387943107 [libj9prt29.so+0x28107])
4XENATIVESTACK               omrsig_protect+0x2b1 (0x00007F0387945881 [libj9prt29.so+0x2a881])
4XENATIVESTACK               _ZN18JavaCoreDumpWriter28writeThreadsWithNativeStacksEv+0x420 (0x00007F0385C8E020 [libj9dmp29.so+0x1a020])
4XENATIVESTACK               protectedWriteThreadsWithNativeStacks+0xd (0x00007F0385C8E8DD [libj9dmp29.so+0x1a8dd])
4XENATIVESTACK               omrsig_protect+0x2b1 (0x00007F0387945881 [libj9prt29.so+0x2a881])
4XENATIVESTACK               _ZN18JavaCoreDumpWriter18writeThreadSectionEv+0x151 (0x00007F0385C8ACE1 [libj9dmp29.so+0x16ce1])
4XENATIVESTACK               protectedWriteSection+0x1d (0x00007F0385C85ADD [libj9dmp29.so+0x11add])
4XENATIVESTACK               omrsig_protect+0x2b1 (0x00007F0387945881 [libj9prt29.so+0x2a881])
4XENATIVESTACK               _ZN18JavaCoreDumpWriterC2EPKcP16J9RASdumpContextP14J9RASdumpAgent+0x3f5 (0x00007F0385C86FE5 [libj9dmp29.so+0x12fe5])
4XENATIVESTACK               runJavadump+0x1c (0x00007F0385C9125C [libj9dmp29.so+0x1d25c])
4XENATIVESTACK               doJavaDump+0x3f (0x00007F0385C7916F [libj9dmp29.so+0x516f])
4XENATIVESTACK               protectedDumpFunction+0x15 (0x00007F0385C787C5 [libj9dmp29.so+0x47c5])
4XENATIVESTACK               omrsig_protect+0x2b1 (0x00007F0387945881 [libj9prt29.so+0x2a881])
4XENATIVESTACK               runDumpFunction+0x60 (0x00007F0385C7BF50 [libj9dmp29.so+0x7f50])
4XENATIVESTACK               runDumpAgent+0x15d (0x00007F0385C7C0CD [libj9dmp29.so+0x80cd])
4XENATIVESTACK               createAndRunOneOffDumpAgent+0x184 (0x00007F0385C7C794 [libj9dmp29.so+0x8794])
4XENATIVESTACK               triggerOneOffDump+0xb5 (0x00007F0385C933F5 [libj9dmp29.so+0x1f3f5])
4XENATIVESTACK               setCRIUSingleThreadModeJVMCRIUException+0x7d (0x00007F03862DD05D [libj9vm29.so+0x3a05d])
4XENATIVESTACK               old_slow_jitMonitorEntry+0x1ca (0x00007F038594802A [libj9jit29.so+0x9ef02a])
4XENATIVESTACK                (0x00007F038595719F [libj9jit29.so+0x9fe19f])

3XMTHREADINFO      "Finalizer thread" J9VMThread:0x000000000018DC00, omrthread_t:0x00007F0380298478, java/lang/Thread:0x00000007FFEDB078, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1D, isDaemon:true)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000006876582C0)
3XMTHREADINFO1            (native thread ID:0x8AD, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x00007F035DC2A000, to:0x00007F035DCAA000, size:0x80000)
3XMCPUTIME               CPU usage total: 0.000534989 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/util/Hashtable.remove(Hashtable.java:508)
5XESTACKTRACE                   (entered lock: java/util/Hashtable@0x00000006876830D8, entry count: 2)
4XESTACKTRACE                at java/lang/ClassLoader$ClassNameLockRef.run(ClassLoader.java:140)
5XESTACKTRACE                   (entered lock: java/util/Hashtable@0x00000006876830D8, entry count: 1)
4XESTACKTRACE                at java/lang/ref/ReferenceQueue.enqueue(ReferenceQueue.java:175)
4XESTACKTRACE                at java/lang/ref/Reference.enqueueImpl(Reference.java:160)
5XESTACKTRACE                   (entered lock: java/lang/ClassLoader$ClassNameLockRef@0x00000007FFF114A0, entry count: 1)
3XMTHREADINFO3           Native callstack:
4XENATIVESTACK                (0x00007F0387941162 [libj9prt29.so+0x26162])
4XENATIVESTACK                (0x00007F0387945881 [libj9prt29.so+0x2a881])
4XENATIVESTACK                (0x00007F038794164B [libj9prt29.so+0x2664b])
4XENATIVESTACK                (0x00007F0387945881 [libj9prt29.so+0x2a881])
4XENATIVESTACK                (0x00007F0387940FDA [libj9prt29.so+0x25fda])
4XENATIVESTACK                (0x00007F038794253C [libj9prt29.so+0x2753c])
4XENATIVESTACK                (0x00007F038745F980 [libpthread.so.0+0x12980])
4XENATIVESTACK               pthread_cond_wait+0x241 (0x00007F038745AAD1 [libpthread.so.0+0xdad1])
4XENATIVESTACK                (0x00007F0387909CA5 [libj9thr29.so+0x7ca5])
4XENATIVESTACK                (0x00007F038633137C [libj9vm29.so+0x8e37c])
4XENATIVESTACK                (0x00007F03862BFEFC [libj9vm29.so+0x1cefc])
4XENATIVESTACK                (0x00007F03859432FF [libj9jit29.so+0x9ea2ff])
4XENATIVESTACK                (0x00007F0385955FE6 [libj9jit29.so+0x9fcfe6])

@tajila

tajila commented Aug 1, 2023

Copy link
Copy Markdown
Contributor Author

Okay, ill try something else

@tajila
tajila force-pushed the criu_3 branch 2 times, most recently from 5f1d68f to b392275 Compare August 1, 2023 14:09
@tajila

tajila commented Aug 1, 2023

Copy link
Copy Markdown
Contributor Author

jenkins test sanity xlinux jdk17

@tajila

tajila commented Aug 1, 2023

Copy link
Copy Markdown
Contributor Author

jenkins test sanity xlinux jdk17

@tajila

tajila commented Aug 1, 2023

Copy link
Copy Markdown
Contributor Author

@JasonFengJ9

JasonFengJ9 commented Aug 1, 2023

Copy link
Copy Markdown
Member

@tajila Just launched a grinder - https://openj9-jenkins.osuosl.org/job/Grinder/2641/

@JasonFengJ9

Copy link
Copy Markdown
Member

@tajila The 2000x grinder #17885 (comment) passed.

@tajila

tajila commented Aug 2, 2023

Copy link
Copy Markdown
Contributor Author

okay thats good

Comment thread runtime/criusupport/criusupport.cpp Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not an issue, should this be before vm->extendedRuntimeFlags2 &= ~J9_EXTENDED_RUNTIME2_CRIU_SINGLE_THREAD_MODE;?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be combined with the line below (for compactness, not correctness).

Comment thread runtime/oti/j9nonbuilder.h Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this an intended change?

@JasonFengJ9 JasonFengJ9 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tajila
tajila requested a review from gacholio August 3, 2023 14:06
@tajila
tajila force-pushed the criu_3 branch 2 times, most recently from 004ff04 to 2ebab0a Compare August 3, 2023 18:33
GC releases threads in a multi-thread fashion. Threads will need to
remove themselves from the threadgroup which requires a lock. Disable
deadlock detection temporarily while this happens.

Also, delay shutdown of finalizer thread until after java hooks have run.

Signed-off-by: Tobi Ajila <tobi_ajila@ca.ibm.com>
@tajila

tajila commented Aug 3, 2023

Copy link
Copy Markdown
Contributor Author

jenkins test sanity alinux64 jdk17

@gacholio

gacholio commented Aug 3, 2023

Copy link
Copy Markdown
Contributor

One of the testlists failed to launch properly (GIT appears very flaky today).

@gacholio

gacholio commented Aug 3, 2023

Copy link
Copy Markdown
Contributor

The restart link in https://openj9-jenkins.osuosl.org/job/Pipeline_Build_Test_JDK17_aarch64_linux/1531/console just redirects back to console.

@pshipton

pshipton commented Aug 3, 2023

Copy link
Copy Markdown
Member

You need to be logged in to jenkins, there is a login link top right. I restarted it but it failed again.

@pshipton

pshipton commented Aug 3, 2023

Copy link
Copy Markdown
Member

It keeps failing, I don't know how to fix it. I have noticed jenkins is being really slow to respond today, maybe it needs a reboot. I've gotten a few gateway timeout errors.

I've started a grinder to run the test.
https://openj9-jenkins.osuosl.org/view/Test/job/Grinder/2645/ - passed

@gacholio

gacholio commented Aug 4, 2023

Copy link
Copy Markdown
Contributor

jenkins test sanity alinux64 jdk17

@gacholio
gacholio merged commit 0c265f6 into eclipse-openj9:master Aug 4, 2023
* @return true if the Reference was successfully
* enqueued, false otherwise.
*/
/* [IF CRIU_SUPPORT] */

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be /*[IF CRIU_SUPPORT]*/, otherwise caused JDK8 compilation failure

/tmp/bld_55732/pConfig/pConfig_SIDECAR18-SE/src/java/lang/ref/Reference.java:119: error: cannot find symbol
	@NotCheckpointSafe
	 ^
  symbol:   class NotCheckpointSafe

FYI @TobiAjila

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't see this before merge. @tajila If you don't fix this in the next few hours, I'll revert this for the long weekend.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JasonFengJ9 I dont understand the comment, /*[IF CRIU_SUPPORT]*/, was added

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And CRIU_SUPPORT is disabled for JDK8

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh is it the spacing?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants