Remove Dilithium

[xuganyu96]

This pull request removes Dilithium source code, tests, configurations, and documentation from the repository. This pull request closes #2156 . See reviewer's guide below for details:

• [YES] Does this PR change the input/output behaviour of a cryptographic algorithm (i.e., does it change known answer test values)? (If so, a version bump will be required from x.y.z to x.(y+1).0.)
• [YES] Does this PR change the list of algorithms available -- either adding, removing, or renaming? Does this PR otherwise change an API? (If so, PRs in fully supported downstream projects dependent on these, i.e., oqs-provider will also need to be ready for review and merge by the time this is merged.)

Reviewer's guide to "removing Dilithium"

The north star of this pull request is a grep command that recursively searched through the entire repository for the keyword "dilithium":

grep -ril \
    --exclude-dir .git \
    --exclude-dir "ml_kem" \
    --exclude-dir "ml_dsa" \
    --exclude-dir build \
    --exclude-dir .pytest_cache \
    --exclude-dir patches \
    --exclude ml_dsa.yml \
    --exclude ml_dsa.md \
    --exclude RELEASE.md \
    --exclude copy_from_upstream.yml \
    "dilithium" .

• -i makes the search case insensitive
• .git, .pytest_cache, build are excluded because they are build/VCS artifacts
• ml_kem is excluded because src/kem/ml_kem mentions Dilithium only as a academic reference/bibliography; similarly we don't care about mentions of Dilithium in ML-DSA code and/or documentation
• copy_from_upstream.yml is excluded because many ML-DSA integrations have patches that mention Dilithium. CI pipeline will confirm that this does not affect copy_from_upstream.py

With main at 1ae1b92302 piping the grep command into wc -l gives 194. With this pull request, the number of matches is 1.
Here is a summary of the changes (in this order) that got us to the state of this pull request:

1. 41b34ee deleted two sections from copy_from_upstream.yml (the upstream pqcrystals-dilithium and the signature scheme dilithium). It also deleted everything under src/sig/dilithium. Re-running copy_from_upstream.py then produced downstream changes in the files below. After this commit, grep produces 20 matches.
   • .CMake/alg_support.cmake
   • CMakeLists.txt
   • src/CMakeLists.txt
   • src/oqsconfig.h.cmake
   • tests/kat_sig.c
2. c2ac9ab deleted Dilithium from tests/KATs/sig/kats.json. grep 19 matches
3. a2c7f66 deleted Dilithium from tests/constant_time/sig. grep at 14
4. 329f601 deleted docs/algorithm/sig/dilithium.yml and docs/algorithm/sig/dilithium.md. Re-running copy_from_upstream.py then modified README.md, and cbom.json. grep produces 11 matches.
5. 2273d55 updated the documentation in FUZZING.md. 10 matches.
6. 9f214d9 removed Dilithium entries from the "licensing" section in README.md. Still at 10 matches because README.md contains one more mention of Dilithium saying "Dilithium-AES has been deprecated", which I chose not to remove.
7. 6d7175c updated the example in CONFIGURE.md to use ML-KEM-768 and ML-DSA-44 instead of Kyber-768 and Dilithium-2. 9 matches.
8. 232493c updated the C++ linking test to use ML-DSA-44 instead of Dilithium-2. This is done with a straightforward s/dilithium_2/ml_dsa_44/g search and replace.
9. db3b29e and 7d1fd49 replaced Dilithium with ML-DSA in CI pipeline and in Zephyr's examples, respectively.

[bhess]

Thanks @xuganyu96 for taking care of the Dilithium removal. I've added two comments inline.

[baentsch]

LGTM. Only thought (not meant to stop this PR): Worthwhile adding a HISTORY section somewhere stating the algorithms that OQS no longer supports (plus maybe a rationale for each alg (family) removal)?

[dstebila]

LGTM. Only thought (not meant to stop this PR): Worthwhile adding a HISTORY section somewhere stating the algorithms that OQS no longer supports (plus maybe a rationale for each alg (family) removal)?

This would be a long list and require a fair bit of digging back in Git history to properly reconstruct. An alternative would be to have a clearly stated policy somewhere on what our algorithm inclusion policy is. I wouldn't hold this PR for that.

[baentsch]

I wouldn't hold this PR for that.

Fine. It would have been good to hold the merge to follow through our stated questionnaire, though: The "Yes" above meant that a fitting downstream PR should (have) be(en) ready by the time of merge. This did not happen. In consequence, all downstream PRs and "main" CI fail now :-( Time to do away with the questionnaire and go back to dealing with problems only as they arise? That's not really efficient, though and impacts many people (at least active maintainers and PR contributors).

[dstebila]

I wouldn't hold this PR for that.

Fine. It would have been good to hold the merge to follow through our stated questionnaire, though: The "Yes" above meant that a fitting downstream PR should (have) be(en) ready by the time of merge. This did not happen. In consequence, all downstream PRs and "main" CI fail now :-( Time to do away with the questionnaire and go back to dealing with problems only as they arise? That's not really efficient, though and impacts many people (at least active maintainers and PR contributors).

I just didn't think to go back and check that before merging, my mistake; sorry. Maybe we can reformulate the text/questionnaire slightly -- Github's PR list will show how many checkboxes remain unchecked. So maybe it should be less of a questionnaire, and more of a "remove the lines that don't apply, and for the ones that do apply, leave the checkboxes unchecked until they are resolved"?