DaaLoadTest, MathLoadTest timeout on amac

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_aarch64_mac_Personal_testList_1/53 - mac11-aarch64-3
DaaLoadTest_daa3_special_5m_2 -Xgcpolicy:optthruput -Xjit:count=0,optlevel=hot,gcOnResolve,rtResolve -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_special.system_aarch64_mac_Personal_testList_1/53/system_test_output.tar.gz

DLT 03:02:54.268 - Completed 93.7%. Number of tests started=18448 (+1325)
STF 03:03:00.355 - Heartbeat: Process DLT is still running
STF 03:03:01.367 - **FAILED** Process DLT has timed out

---

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_aarch64_mac_Personal_testList_4/53 - mac11-aarch64-1
DaaLoadTest_daa1_special_5m_8 -Xgcpolicy:gencon -Xshareclasses -Xjit -Xnocompressedrefs

12:59:50  DLT 02:59:49.755 - Completed 13.4%. Number of tests started=4919 (+2498)
13:00:02  STF 03:00:00.149 - **FAILED** Process DLT has timed out

---

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_aarch64_mac_Personal_testList_2/53
DaaLoadTest_all_special_5m_1 -Xgcpolicy:optthruput -Xjit:count=0 -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_special.system_aarch64_mac_Personal_testList_2 - mac11-aarch64-4/53/system_test_output.tar.gz

DLT 03:01:00.758 - Completed 93.7%. Number of tests started=37557 (+3314)
STF 03:01:10.237 - Heartbeat: Process DLT is still running
STF 03:01:11.256 - **FAILED** Process DLT has timed out

[pshipton]

Only the first one has a javacore, and there is only one. I don't know that it's actually hung.

3XMTHREADINFO      "load-0" J9VMThread:0x00000001309DF500, omrthread_t:0x0000000128068D58, java/lang/Thread:0x00000002804B0EC0, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x19, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000002800364E0)
3XMTHREADINFO1            (native thread ID:0xC11B64F, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x000000016DC2C000, to:0x000000016DC6F000, size:0x43000)
3XMCPUTIME               CPU usage total: 213.629549000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=348128 (0x54FE0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/util/Random.next(Random.java:207(Compiled Code))
4XESTACKTRACE                at java/util/Random.nextInt(Random.java:323(Compiled Code))
4XESTACKTRACE                at net/openj9/test/arithmetics/TestArithmeticOperations.testMultiplyPackedDecimalRandom(TestArithmeticOperations.java:12199(Compiled Code))

3XMTHREADINFO      "load-1" J9VMThread:0x0000000130A5DB00, omrthread_t:0x0000000128069260, java/lang/Thread:0x00000002804B1400, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1A, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000002800364E0)
3XMTHREADINFO1            (native thread ID:0xC11B650, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x000000016DC78000, to:0x000000016DCBB000, size:0x43000)
3XMCPUTIME               CPU usage total: 213.480619000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=208856 (0x32FD8)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/ThreadLocal.get(ThreadLocal.java:165(Compiled Code))
4XESTACKTRACE                at com/ibm/dataaccess/PackedDecimal.computeValue(PackedDecimal.java:551(Compiled Code))
4XESTACKTRACE                at com/ibm/dataaccess/PackedDecimal.addPackedDecimal_(PackedDecimal.java:309(Compiled Code))
4XESTACKTRACE                at com/ibm/dataaccess/PackedDecimal.addPackedDecimal(PackedDecimal.java:294(Compiled Code))
4XESTACKTRACE                at net/openj9/test/arithmetics/TestArithmeticOperations.testAddPackedDecimalLeadingZeros(TestArithmeticOperations.java:11978(Compiled Code))

3XMTHREADINFO      "load-2" J9VMThread:0x0000000130A65500, omrthread_t:0x0000000130A62850, java/lang/Thread:0x00000002804B1758, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1B, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000002800364E0)
3XMTHREADINFO1            (native thread ID:0xC11B651, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x000000016DCC4000, to:0x000000016DD07000, size:0x43000)
3XMCPUTIME               CPU usage total: 213.167593000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=368552 (0x59FA8)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/ThreadLocal.get(ThreadLocal.java:163(Compiled Code))
4XESTACKTRACE                at com/ibm/dataaccess/PackedDecimal.copyRemainingDigits(PackedDecimal.java:200(Compiled Code))
4XESTACKTRACE                at com/ibm/dataaccess/PackedDecimal.computeDifference(PackedDecimal.java:544(Compiled Code))
4XESTACKTRACE                at com/ibm/dataaccess/PackedDecimal.computeValue(PackedDecimal.java:594(Compiled Code))
4XESTACKTRACE                at com/ibm/dataaccess/PackedDecimal.addPackedDecimal_(PackedDecimal.java:309(Compiled Code))
4XESTACKTRACE                at com/ibm/dataaccess/PackedDecimal.addPackedDecimal(PackedDecimal.java:294(Compiled Code))
4XESTACKTRACE                at net/openj9/test/arithmetics/TestArithmeticOperations.testAddPackedDecimalPossibleNegativeNumbers(TestArithmeticOperations.java:12029(Compiled Code))

3XMTHREADINFO      "load-3" J9VMThread:0x0000000130A6CF00, omrthread_t:0x0000000130A62D58, java/lang/Thread:0x00000002804B1AB0, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1C, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000002800364E0)
3XMTHREADINFO1            (native thread ID:0xC11B652, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x000000016DD10000, to:0x000000016DD53000, size:0x43000)
3XMCPUTIME               CPU usage total: 212.786753000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=507808 (0x7BFA0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/ThreadLocal.get(ThreadLocal.java:165(Compiled Code))
4XESTACKTRACE                at com/ibm/dataaccess/PackedDecimal.computeSum(PackedDecimal.java:418(Compiled Code))
4XESTACKTRACE                at com/ibm/dataaccess/PackedDecimal.computeValue(PackedDecimal.java:560(Compiled Code))
4XESTACKTRACE                at com/ibm/dataaccess/PackedDecimal.addPackedDecimal_(PackedDecimal.java:309(Compiled Code))
4XESTACKTRACE                at com/ibm/dataaccess/PackedDecimal.addPackedDecimal(PackedDecimal.java:294(Compiled Code))
4XESTACKTRACE                at net/openj9/test/arithmetics/TestArithmeticOperations.testAddPackedDecimalRandom(TestArithmeticOperations.java:11933(Compiled Code))

3XMTHREADINFO      "load-4" J9VMThread:0x0000000130A74900, omrthread_t:0x0000000130A63260, java/lang/Thread:0x00000002804B1E70, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1D, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000002800364E0)
3XMTHREADINFO1            (native thread ID:0xC11B653, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x000000016DD5C000, to:0x000000016DD9F000, size:0x43000)
3XMCPUTIME               CPU usage total: 212.905389000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=368600 (0x59FD8)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/lang/Class.getConstructors(Class.java:735(Compiled Code))
4XESTACKTRACE                at org/junit/runners/model/TestClass.getOnlyConstructor(TestClass.java:203(Compiled Code))
4XESTACKTRACE                at org/junit/runners/BlockJUnit4ClassRunner.createTest(BlockJUnit4ClassRunner.java:217(Compiled Code))

3XMTHREADINFO      "load-5" J9VMThread:0x0000000130A7C300, omrthread_t:0x0000000130A79650, java/lang/Thread:0x00000002804B21C8, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1E, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x00000002800364E0)
3XMTHREADINFO1            (native thread ID:0xC11B654, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x000000016DDA8000, to:0x000000016DDEB000, size:0x43000)
3XMCPUTIME               CPU usage total: 213.669797000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=196576 (0x2FFE0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/util/Random.next(Random.java:209(Compiled Code))
4XESTACKTRACE                at java/util/Random.nextInt(Random.java:323(Compiled Code))
4XESTACKTRACE                at net/openj9/test/arithmetics/TestArithmeticOperations.testAddPackedDecimalLeadingZeros(TestArithmeticOperations.java:11973(Compiled Code))

[pshipton]

@knn-k fyi

[pshipton]

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_special.system_aarch64_mac_Personal_testList_4/10 - mac11-aarch64-2
MathLoadTest_bigdecimal_special_5m_7 -Xgcpolicy:optavgpause -Xjit:count=0,optlevel=warm,gcOnResolve,rtResolve -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_special.system_aarch64_mac_Personal_testList_4/10/system_test_output.tar.gz

MLT 03:00:06.028 - Completed 93.5%. Number of tests started=14459 (+1058)
STF 03:00:21.397 - Heartbeat: Process MLT is still running
STF 03:00:22.412 - **FAILED** Process MLT has timed out

---

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_special.system_aarch64_mac_Personal_testList_0/10/
DaaLoadTest_daa2_special_5m_24
-Xgcpolicy:optthruput -Xdebug -Xrunjdwp:transport=dt_socket,address=8888,server=y,onthrow=no.pkg.foo,launch=echo -Xjit:enableOSR,enableOSROnGuardFailure,count=1,disableAsyncCompilation

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_special.system_aarch64_mac_Personal_testList_0/10/system_test_output.tar.gz

DLT 03:04:17.746 - Completed 93.9%. Number of tests started=14089 (+1107)
STF 03:04:25.499 - Heartbeat: Process DLT is still running
STF 03:04:26.008 - **FAILED** Process DLT has timed out

---

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_special.system_aarch64_mac_Personal_testList_2/10/ - mac11-aarch64-5
DaaLoadTest_all_special_5m_1 -Xgcpolicy:optthruput -Xjit:count=0 -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_special.system_aarch64_mac_Personal_testList_2/10/system_test_output.tar.gz

DLT 03:02:32.424 - Completed 93.7%. Number of tests started=37149 (+3717)
STF 03:02:41.019 - Heartbeat: Process DLT is still running
STF 03:02:42.038 - **FAILED** Process DLT has timed out

---

https://openj9-jenkins.osuosl.org/job/Test_openjdk19_j9_special.system_aarch64_mac_Personal_testList_1/10/ - mac11-aarch64-7
DaaLoadTest_daa1_special_5m_9 -Xaggressive -Xgcpolicy:gencon -Xjit -Xnocompressedrefs

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk19_j9_special.system_aarch64_mac_Personal_testList_1/10/system_test_output.tar.gz

DLT 02:59:40.934 - Completed 73.4%. Number of tests started=41824 (+4401)
STF 03:00:00.016 - **FAILED** Process DLT has timed out

Only this last one has javacore files, there are two. Only the load-1 thread is in the same state between dumps.

3XMTHREADINFO      "load-1" J9VMThread:0x0000000125826900, omrthread_t:0x0000000129163358, java/lang/Thread:0x00000002804170B0, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1D, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x0000000280077EA8)
3XMTHREADINFO1            (native thread ID:0xB7074DE, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x00000001703B4000, to:0x00000001703F7000, size:0x43000)
3XMCPUTIME               CPU usage total: 220.037444000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=17651104 (0x10D55A0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/math/MutableBigInteger.divideMagnitude(MutableBigInteger.java:1678(Compiled Code))
4XESTACKTRACE                at java/math/MutableBigInteger.divideKnuth(MutableBigInteger.java:1232(Compiled Code))
4XESTACKTRACE                at java/math/MutableBigInteger.divide(MutableBigInteger.java:1158(Compiled Code))
4XESTACKTRACE                at java/math/MutableBigInteger.divide(MutableBigInteger.java:1152(Compiled Code))
4XESTACKTRACE                at java/math/BigInteger.smallToString(BigInteger.java:4152(Compiled Code))
4XESTACKTRACE                at java/math/BigInteger.toString(BigInteger.java:4205(Compiled Code))
4XESTACKTRACE                at java/math/BigInteger.toString(BigInteger.java:4223(Compiled Code))
4XESTACKTRACE                at java/math/BigInteger.toString(BigInteger.java:4100(Compiled Code))
4XESTACKTRACE                at java/math/BigInteger.toString(BigInteger.java:4274(Compiled Code))
4XESTACKTRACE                at net/openj9/test/Utils.toPrecision(Utils.java:489(Compiled Code))
4XESTACKTRACE                at net/openj9/test/arithmetics/TestPerformance.getComparisionReferenceValue(TestPerformance.java:375(Compiled Code))
4XESTACKTRACE                at net/openj9/test/arithmetics/TestPerformance.testLE(TestPerformance.java:231(Compiled Code))
4XESTACKTRACE                at net/openj9/test/arithmetics/TestPerformance.testIntrinsicPerformance(TestPerformance.java:89(Compiled Code))

3XMTHREADINFO      "load-1" J9VMThread:0x0000000125826900, omrthread_t:0x0000000129163358, java/lang/Thread:0x0000000280417098, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1D, isDaemon:false)
3XMJAVALTHRCCL            jdk/internal/loader/ClassLoaders$AppClassLoader(0x0000000280077E90)
3XMTHREADINFO1            (native thread ID:0xB7074DE, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x00000001703B4000, to:0x00000001703F7000, size:0x43000)
3XMCPUTIME               CPU usage total: 254.263769000 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=41896664 (0x27F4AD8)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/math/MutableBigInteger.divideMagnitude(MutableBigInteger.java:1678(Compiled Code))
4XESTACKTRACE                at java/math/MutableBigInteger.divideKnuth(MutableBigInteger.java:1232(Compiled Code))
4XESTACKTRACE                at java/math/MutableBigInteger.divide(MutableBigInteger.java:1158(Compiled Code))
4XESTACKTRACE                at java/math/MutableBigInteger.divide(MutableBigInteger.java:1152(Compiled Code))
4XESTACKTRACE                at java/math/BigInteger.smallToString(BigInteger.java:4152(Compiled Code))
4XESTACKTRACE                at java/math/BigInteger.toString(BigInteger.java:4205(Compiled Code))
4XESTACKTRACE                at java/math/BigInteger.toString(BigInteger.java:4223(Compiled Code))
4XESTACKTRACE                at java/math/BigInteger.toString(BigInteger.java:4100(Compiled Code))
4XESTACKTRACE                at java/math/BigInteger.toString(BigInteger.java:4274(Compiled Code))
4XESTACKTRACE                at net/openj9/test/Utils.toPrecision(Utils.java:489(Compiled Code))
4XESTACKTRACE                at net/openj9/test/arithmetics/TestPerformance.getReferenceValues(TestPerformance.java:432(Compiled Code))
4XESTACKTRACE                at net/openj9/test/arithmetics/TestPerformance.testRem(TestPerformance.java:190(Compiled Code))
4XESTACKTRACE                at net/openj9/test/arithmetics/TestPerformance.testIntrinsicPerformance(TestPerformance.java:80(Compiled Code))

[JasonFengJ9]

JDK11 aarch64_mac 0.38 m2 build(macaarch64rt2)

openjdk version "11.0.19" 2023-04-18
IBM Semeru Runtime Open Edition 11.0.19.0-m2 (build 11.0.19+6)
Eclipse OpenJ9 VM 11.0.19.0-m2 (build v0.38.0-release-353bdb06f, JRE 11 Mac OS X aarch64-64-Bit 20230329_294 (JIT enabled, AOT enabled)
OpenJ9   - 353bdb06f
OMR      - 855813495
JCL      - b04e256c1c based on jdk-11.0.19+6)

[2023-03-30T15:02:49.804Z] variation: Mode104
[2023-03-30T15:02:49.804Z] JVM_OPTIONS:  -Xgcpolicy:optthruput -Xjit:count=0,optlevel=hot,gcOnResolve,rtResolve -Xnocompressedrefs 

[2023-03-30T15:24:38.693Z] STF 11:07:43.217 - Heartbeat: Process MLT is still running
[2023-03-30T15:24:38.693Z] MLT 11:07:49.125 - Completed 88.9%. Number of tests started=2756482 (+260834)
[2023-03-30T15:24:38.693Z] STF 11:24:13.970 - **FAILED** Process MLT has timed out
[2023-03-30T15:24:38.693Z] STF 11:24:13.977 - Collecting dumps for: MLT

[2023-03-30T15:24:45.194Z] MathLoadTest_autosimd_special_5m_2_FAILED

[2023-03-30T17:15:40.143Z] variation: Mode122
[2023-03-30T17:15:40.143Z] JVM_OPTIONS:  -Xgcpolicy:optavgpause -Xjit:count=0,optlevel=warm,gcOnResolve,rtResolve -Xnocompressedrefs 

[2023-03-30T17:15:40.144Z] DLT 12:31:51.517 - Completed 100.0%. Number of tests started=10209 (+0)
[2023-03-30T17:15:40.144Z] STF 12:31:51.546 - **FAILED** Process DLT has hung
[2023-03-30T17:15:40.144Z] STF 12:31:51.546 - Collecting dumps for: DLT

[2023-03-30T17:15:40.145Z] DaaLoadTest_daa2_special_5m_7_FAILED

[Akira1Saitoh]

It seems that the process DLT is treated as timeout earlier than specified.
For example, in https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_special.system_aarch64_mac_Personal_testList_1/53/consoleText, timeout was specified as 1 hour and 5 minutes.

[2023-04-01T16:58:00.535Z] STF 02:58:00.290 - +------ Step 1 - Run daa load test
[2023-04-01T16:58:00.535Z] STF 02:58:00.290 - | Run foreground process
[2023-04-01T16:58:00.535Z] STF 02:58:00.290 - | Program: /Users/jenkins/workspace/Test_openjdk17_j9_special.system_aarch64_mac_Personal_testList_1/openjdkbinary/j2sdk-image/bin/java
[2023-04-01T16:58:00.535Z] STF 02:58:00.290 - | Mnemonic: DLT
[2023-04-01T16:58:00.535Z] STF 02:58:00.290 - | Echo: ECHO_ON
[2023-04-01T16:58:00.535Z] STF 02:58:00.290 - | Expectation: CLEAN_RUN within 1h5m

However, STF said that DLT timed out after only 5 minutes passed.

[2023-04-01T16:58:00.536Z] STF 02:58:00.294 - Monitoring processes: DLT
[2023-04-01T16:58:14.449Z] DLT 02:58:13.015 - Load test parameters
[2023-04-01T16:58:14.449Z] DLT 02:58:13.102 - Time limited = true
[2023-04-01T16:58:14.449Z] DLT 02:58:13.102 - Time limit = 5m
...
[2023-04-01T17:02:56.049Z] DLT 03:02:54.268 - Completed 93.7%. Number of tests started=18448 (+1325)
[2023-04-01T17:03:01.579Z] STF 03:03:00.355 - Heartbeat: Process DLT is still running
[2023-04-01T17:03:02.413Z] STF 03:03:01.367 - FAILED Process DLT has timed out

[Akira1Saitoh]

All timeouts except ones on internal Jenkins happened around 5 p.m on April 1st in UTC.
According to javacore, the timezone of the machine is AEST (Australian Eastern Time).

│1TITIMEZONE Timezone: UTC+10 (AEST)

It seems that Australia had a clock change at that time. I suspect that this was the reason of the timeout because STF uses localtime() to get the current time.
https://github.com/adoptium/STF/blob/89020fe1391aecaa6d2cf4f5402b60d200705c2a/stf.core/scripts/stf/ProcessMgmt/UNIX.pm#L66

https://www.timeanddate.com/time/change/australia/sydney

When local daylight time was about to reach
Sunday, 2 April 2023, 03:00:00 clocks were turned backward 1 hour to
Sunday, 2 April 2023, 02:00:00 local standard time instead.

[Akira1Saitoh]

The failures in #17090 (comment) are not related to Australian time change.

What is weird about them is that the STF process (a perl script) did not report heartbeat as expected. The interval of heartbeat is 5 minutes, but it did not report the second heartbeat event when running MathLoadTest_autosimd_special_5m_2.
https://github.com/adoptium/STF/blob/89020fe1391aecaa6d2cf4f5402b60d200705c2a/stf.core/scripts/stf/Commands.pm#L91-L94

[2023-03-30T15:07:30.409Z] MLT 11:07:29.072 - Completed 82.0%. Number of tests started=2495648 (+149802)
[2023-03-30T15:24:38.693Z] STF 11:07:43.217 - Heartbeat: Process MLT is still running
[2023-03-30T15:24:38.693Z] MLT 11:07:49.125 - Completed 88.9%. Number of tests started=2756482 (+260834)
[2023-03-30T15:24:38.693Z] STF 11:24:13.970 - FAILED Process MLT has timed out
[2023-03-30T15:24:38.693Z] STF 11:24:13.977 - Collecting dumps for: MLT

For DaaLoadTest_daa2_special_5m_7, the first heartbeat was reported after 15 minutes elapsed, which was also strange.

[2023-03-30T17:15:40.144Z] DLT 12:15:18.720 - Completed 66.8%. Number of tests started=10011 (+626)
[2023-03-30T17:15:40.144Z] STF 12:31:51.045 - Heartbeat: Process DLT is still running
[2023-03-30T17:15:40.144Z] DLT 12:31:51.076 - Completed 100.0%. Number of tests started=10209 (+198)
[2023-03-30T17:15:40.144Z] DLT 12:31:51.077 - POSSIBLE HANG DETECTED
[2023-03-30T17:15:40.144Z] DLT 12:31:51.188 - Completed 100.0%. Number of tests started=10209 (+0)
[2023-03-30T17:15:40.144Z] DLT 12:31:51.284 - Thread completed. Suite=0 thread=5

It looks as if the machine was in sleep mode for 16.5 minutes in the middle of those tests.

Launched /job/Grinder/32110/ to see if it is reproducible.

[Akira1Saitoh]

No failures in /job/Grinder/32110/ and /job/Grinder/32129/. Launched /job/Grinder/32150/.

[Akira1Saitoh]

No failures in /job/Grinder/32150/ and /job/Grinder/32164/.

[pshipton]

Internal build
MathLoadTest_autosimd_special_5m_2
-Xgcpolicy:optthruput -Xjit:count=0,optlevel=hot,gcOnResolve,rtResolve -Xnocompressedrefs

Artifacts

MLT 10:23:36.600 - Completed 6.8%. Number of tests started=354451
MLT 10:23:55.385 - Completed 13.7%. Number of tests started=852326 (+497875)
MLT 10:24:15.392 - Completed 20.5%. Number of tests started=1345458 (+493132)
MLT 10:24:39.440 - Completed 28.7%. Number of tests started=1713505 (+368047)
STF 10:40:11.761 - **FAILED** Process MLT has timed out