Repository navigation
Closed
Closed
Copy link
Description
Describe the bug
Commit 69b400d causes hundreds of test failures here. Posting just the first few ones:
Errors from shard 11:
======================================================================
FAIL: test_int_int (pycmp_typed)
Doctest: pycmp_typed.test_int_int
----------------------------------------------------------------------
AssertionError: Failed doctest test for pycmp_typed.test_int_int
File "/build/cython/src/cython/TEST_TMP/run/c/pycmp_typed/pycmp_typed.cpython-314-x86_64-linux-gnu.so", line 18736, in test_int_int
----------------------------------------------------------------------
File "/build/cython/src/cython/TEST_TMP/run/c/pycmp_typed/pycmp_typed.cpython-314-x86_64-linux-gnu.so", line 18740, in pycmp_typed.test_int_int
Failed example:
a = 0; b = 1; run_test(test_int_int, a, b, (a==b,a!=b,a<b,a<=b,a>=b,a>b))
Expected:
(False, True, True, True, False, False)
Got:
ERROR: test_int_int(0, 1) ==
(False, False, True, True, True, True) !=
(False, True, True, True, False, False)
----------------------------------------------------------------------
File "/build/cython/src/cython/TEST_TMP/run/c/pycmp_typed/pycmp_typed.cpython-314-x86_64-linux-gnu.so", line 18742, in pycmp_typed.test_int_int
Failed example:
a = 0; b = 2; run_test(test_int_int, a, b, (a==b,a!=b,a<b,a<=b,a>=b,a>b))
Expected:
(False, True, True, True, False, False)
Got:
ERROR: test_int_int(0, 2) ==
(False, False, True, True, True, True) !=
(False, True, True, True, False, False)
----------------------------------------------------------------------
File "/build/cython/src/cython/TEST_TMP/run/c/pycmp_typed/pycmp_typed.cpython-314-x86_64-linux-gnu.so", line 18744, in pycmp_typed.test_int_int
Failed example:
a = 0; b = -1; run_test(test_int_int, a, b, (a==b,a!=b,a<b,a<=b,a>=b,a>b))
Expected:
(False, True, False, False, True, True)
Got:
ERROR: test_int_int(0, -1) ==
(False, False, False, False, False, False) !=
(False, True, False, False, True, True)
----------------------------------------------------------------------
File "/build/cython/src/cython/TEST_TMP/run/c/pycmp_typed/pycmp_typed.cpython-314-x86_64-linux-gnu.so", line 18746, in pycmp_typed.test_int_int
Failed example:
a = 0; b = -2; run_test(test_int_int, a, b, (a==b,a!=b,a<b,a<=b,a>=b,a>b))
Expected:
(False, True, False, False, True, True)
Got:
ERROR: test_int_int(0, -2) ==
(False, False, False, False, False, False) !=
(False, True, False, False, True, True)
----------------------------------------------------------------------
File "/build/cython/src/cython/TEST_TMP/run/c/pycmp_typed/pycmp_typed.cpython-314-x86_64-linux-gnu.so", line 18748, in pycmp_typed.test_int_int
Failed example:
a = 0; b = 32768; run_test(test_int_int, a, b, (a==b,a!=b,a<b,a<=b,a>=b,a>b))
Expected:
(False, True, True, True, False, False)
Got:
ERROR: test_int_int(0, 32768) ==
(False, False, True, True, True, True) !=
(False, True, True, True, False, False)
----------------------------------------------------------------------
File "/build/cython/src/cython/TEST_TMP/run/c/pycmp_typed/pycmp_typed.cpython-314-x86_64-linux-gnu.so", line 18750, in pycmp_typed.test_int_int
Failed example:
a = 0; b = 32769; run_test(test_int_int, a, b, (a==b,a!=b,a<b,a<=b,a>=b,a>b))
Expected:
(False, True, True, True, False, False)
Got:
ERROR: test_int_int(0, 32769) ==
(False, False, True, True, True, True) !=
(False, True, True, True, False, False)
----------------------------------------------------------------------
File "/build/cython/src/cython/TEST_TMP/run/c/pycmp_typed/pycmp_typed.cpython-314-x86_64-linux-gnu.so", line 18752, in pycmp_typed.test_int_int
Failed example:
a = 0; b = -32768; run_test(test_int_int, a, b, (a==b,a!=b,a<b,a<=b,a>=b,a>b))
Expected:
(False, True, False, False, True, True)
Got:
ERROR: test_int_int(0, -32768) ==
(False, False, False, False, False, False) !=
(False, True, False, False, True, True)
----------------------------------------------------------------------
File "/build/cython/src/cython/TEST_TMP/run/c/pycmp_typed/pycmp_typed.cpython-314-x86_64-linux-gnu.so", line 18754, in pycmp_typed.test_int_int
Failed example:
a = 0; b = -32769; run_test(test_int_int, a, b, (a==b,a!=b,a<b,a<=b,a>=b,a>b))
Expected:
(False, True, False, False, True, True)
Got:
ERROR: test_int_int(0, -32769) ==
(False, False, False, False, False, False) !=
(False, True, False, False, True, True)
----------------------------------------------------------------------
File "/build/cython/src/cython/TEST_TMP/run/c/pycmp_typed/pycmp_typed.cpython-314-x86_64-linux-gnu.so", line 18756, in pycmp_typed.test_int_int
Failed example:
a = 0; b = 1073741824; run_test(test_int_int, a, b, (a==b,a!=b,a<b,a<=b,a>=b,a>b))
Expected:
(False, True, True, True, False, False)
Got:
ERROR: test_int_int(0, 1073741824) ==
(False, False, True, True, True, True) !=
(False, True, True, True, False, False)
Code to reproduce the behaviour:
Compile 3.3.0, run python runtests.py -vv -j 64
Expected behaviour
Tests pass
OS
Arch Linux x86_64
Python version
3.14.7
Cython version
3.3.0
Additional context
No response
Metadata
Metadata
Assignees
Labels
No labels
Activity
da-woods commentedon Aug 22, 2026
I'm a bit puzzled by this because we run all these tests ourselves on every commit and the CI is green for us. So clearly something is different in your setup (although nothing is unusual based on your description)
scoder commentedon Aug 22, 2026
What happens if you only run the single failing test module, i.e.
python3 runtests.py pycmp_typed?Could you also provide the output of the
build_envtest? It's meant to list relevant information about the test environment for debugging purposes.antonio-rojas commentedon Aug 22, 2026
How do I do that? I can't find such test in the cython source.
da-woods commentedon Aug 22, 2026
I think it's just
python3 runtests.py build_envantonio-rojas commentedon Aug 22, 2026
Running
CFLAGS="-O1" python runtests.py pycmp_typedworks. So the issue is-O2(our default build flags)da-woods commentedon Aug 22, 2026
We run them with
-OgI think (mainly because they run much faster like that). I've tried both-O2and-O3locally though and failed to reproduce it (although most likely with a different compiler version)antonio-rojas commentedon Aug 22, 2026
A few experiments:
GCC 16.2.1 and GCC 15.3.0, -O0, -O1, -Og: works
GCC 16.2.1 and GCC 15.3.0, -O2, -O3: fails
clang 22.1.8, any -O: works
da-woods commentedon Aug 22, 2026
I've had a go with
gcc version 16.2.0,gcc version 15.2.1,gcc version 15.3.0(the opensuse versions, just because that's what's easiest for me to test. I suspect they aren't completely stock) and haven't been able to make them fail I'm afraid.scoder commentedon Aug 23, 2026
Could you please also provide the output of the
build_envtest? Just runpython3 runtests.py build_env.Which C compiler and version are you using?From the looks of this, and the lack of other reports so far (there was a long testing phase for this), we should consider a C compiler bug as a reason.Could you try a different C compiler version?EDIT: Sorry, github hadn't updated my ticket view and I hadn't seen your responses.
scoder commentedon Aug 23, 2026
So … maybe even an ArchLinux specific patch that broke gcc for this?
scoder commentedon Aug 23, 2026
I wrote the code that does the comparisons here, so I'm happy to accept bugs on my account, but it's indicative that it's C compiler specific. Apart from a C compiler bug, the second best explanation for optimiser introduced problems in C is usually accidental undefined behaviour, and we have sanitizer checks running that report the test clean, e.g. in https://github.com/cython/cython/actions/runs/32590650966/job/97073860694
antonio-rojas commentedon Aug 23, 2026
This doesn't give any more output than the regular
python runtests.py. Are you sure that's the right command?antonio-rojas commentedon Aug 23, 2026
After some more testing (not easy becasue it depends independently on CFLAGS and CXXFLAGS, and they seem to be consumed in several different places), to reproduce this you need
-O2in CFLAGS and-flto=autoin CXXFLAGS:fails.
works.
also works.
da-woods commentedon Aug 23, 2026
-flto=autoreproduces it for me too.da-woods commentedon Aug 23, 2026
#7771 also involved GCC LTO. I'd guess that it might be related (but that's just based on it looking a little similar rather than anything more substantial.
I'll try to have a look at some of the generate failing code for this one later today though, just to convince myself that there's nothing dodgy-looking there.
scoder commentedon Aug 23, 2026
Never hurts to pass a second pair of eyeballs over it. I already did. Some of it feels more dumb than dodgy, but the C compiler should eliminate the redundant parts and make it efficient. I couldn't spot anything suspicious. But I also wrote it, so it might just be author blindness on my side.
antonio-rojas commentedon Aug 23, 2026
Looks like the same issue.
-fdump-ipa-icf-detailsshowsand
-fno-ipa-icfmakes it work.da-woods commentedon Aug 23, 2026
To me this feels like something we're going to need to fix because it's going to keep coming up (even if it's not our fault).
I can't see any way of telling the GCC LTO to ignore a specific function (beyond just surrounding it with general "do not optimize" pragmas, which I don't think we want to do).
I'd imagine the best way might be to make
Eqthe master version (and the defineNein terms ofEq) and probably the same thing for</>=and>/<=. That falls apart slight for float andNaNthough. But possibly only the int versions are causing problems.The other quicker fix would be to turn of
USE_LONG_INTERNALSfor GCC. That seems to work for me but it's kind of unsatisfactory.thesamesam commentedon Aug 24, 2026
__attribute__((noipa))should work here if it's an ICF bug, or even__attribute__((optimize("no-ipa-icf"))).scoder commentedon Aug 24, 2026
I'll rework the code.
There is no real reason for separate Python vs. int return variants, so I'll merge those and hopefully fix this issue along the way.Not entirely true. There is one difference – if we need the
PyObject_RichCompare()calllback, then we cannot necessarily go through a Cintreturn value because the result might be an arbitrary Python object, not limited tobool. That's the only reason why I initially wrote it this way.Rewrite int-int comparison to work around a GCC-LTO issue.
scoder commentedon Aug 24, 2026
#7919 seems to work.
antonio-rojas commentedon Aug 24, 2026
Works for me too.
Rewrite int-int comparison to work around a GCC-LTO issue. (GH-7919)
thesamesam commentedon Aug 26, 2026
I've not had a chance to try reduce anything here, but if you could try get a minimal-ish C testcase, it'd be much appreciated on the GCC side. I can handle the rest if you could share something standalone.
It doesn't need to be elegant or make sense, it just needs to be UB-free and exhibit the problem.
scoder commentedon Aug 28, 2026
The two functions that failed to link correctly are these:
The
__Pyx_functions are all macros that do simple bit operations on the fields of two structs thatop1andop2point to. Most of them should be discardable without changing the failure outcome, although there might have to be some complexity in the functions in order to trigger the bug, can't say.Py_RETURN_TRUEis a macro that increases the reference count to thePy_Truesingleton and returns it.Py_RETURN_FALSElikewise forPy_False.Both functions differ only in two
gotostatements that have swapped targets in order to return opposite results. The rest is entirely identical.thesamesam commentedon Sep 28, 2026
Thanks. Now handled at #7771 (comment).