Skip to content

feat: update support for KCLE dataset v2 - #2307

Merged
mzr1996 merged 9 commits into
open-compass:mainfrom
xgao922:dev
Nov 20, 2025
Merged

mzr1996 merged 9 commits into
open-compass:mainfrom
xgao922:dev

Conversation

@xgao922

@xgao922 xgao922 commented Nov 14, 2025

Copy link
Copy Markdown
Contributor

Motivation

This PR adds evaluation support for the KCLE dataset v2. The goal is to enable benchmarking of LLMs on KCLE, which is designed for evaluating models on STEM-related multiple-choice questions.

You can run the following command to evaluate the KCLE dataset v2:

python run.py --models lmdeploy_qwen3_0_6b --datasets kcle_llm_judge_gen_60327a

Modification

Two new files are introduced to support the KCLE dataset v2:

  • opencompass/datasets/kcle.py: the dataset loading and preprocessing script.

  • opencompass/configs/datasets/kcle/kcle_llm_judge_gen_60327a.py: the evaluation configuration for LLM judgment.

Checklist

Before PR:

  • Pre-commit or other linting tools are used to fix the potential lint issues.
  • Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
  • The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  • The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

  • If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
  • CLA has been signed and all committers have signed the CLA in this PR.

@tonysy

tonysy commented Nov 18, 2025

Copy link
Copy Markdown
Collaborator

Please resolve the conflicts

@mzr1996
mzr1996 merged commit 5685728 into open-compass:main Nov 20, 2025
2 checks passed
Jensen246 pushed a commit to Jensen246/opencompass that referenced this pull request Nov 27, 2025
* feat(dataset): add support for kcle dataset

* fix: apply code style fixes from pre-commit

* Merge feature updates from dev into main

* Delete local model path

* feat: update support for KCLE dataset v2

* fix: update config file suffix

* fix

* fix

---------

Co-authored-by: root <root@gaoxuan.gaoxuan.ailab-hx.svc.pjlab.local>
Co-authored-by: Myhs_phz <demarcia2014@126.com>
iamkaia pushed a commit to iamkaia/opencompass that referenced this pull request Feb 4, 2026
* feat(dataset): add support for kcle dataset

* fix: apply code style fixes from pre-commit

* Merge feature updates from dev into main

* Delete local model path

* feat: update support for KCLE dataset v2

* fix: update config file suffix

* fix

* fix

---------

Co-authored-by: root <root@gaoxuan.gaoxuan.ailab-hx.svc.pjlab.local>
Co-authored-by: Myhs_phz <demarcia2014@126.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants