Skip to content

Releases

0.5.5

Choose a tag to compare

@Myhs-phz Myhs-phz released this 06 Nov 08:08
6288ea0

OpenCompass v0.5.5 Release Notes

🌟 Highlights

✨ Integrated More Benchmarks: Integrated new benchmarks spanning physics, reasoning, long-context, alignment, hallucination, maritime, biomedical, translation and multimodal evaluation, including CritPt, General365, AA-OmniScience v1.0 and v1.1, AA-LCR v1.1, HLE-Diamond, arena-hard-v2, ReasonZoo, LG-Bench, CHHallu-Src, GrandPhysics, PromptCBLUE Life Science, MaritimeBench, WMT19, LogiQA and ethics, plus ChartQAPro, MicroVQA, SFE and SciDocBench from VLMEvalKit (#2610, #2626, #2658, #2713, #2362, #2256, #2690, #2525, #2118, #2073, #2018, #1503, #516, #495, #2665).

✨ Documentation Refactor: Restructured the Chinese and English documentation into a new layout (get started, user guides, evaluation, execution, extension, tools and FAQ), rewrote the README, installation and quick start guides, and added configs for the latest models (#2666).

✨ Refactored CI: Refactored the mock API tests, fixed the PR stage CI, and extended the PR stage CI to cover Python 3.9 and 3.12 (#2698, #2670, #2715).

✨ Dropped Python 3.8 and Fixed Python 3.9 Compatibility: Removed Python 3.8 support and fixed the PEP 604 annotations that broke imports on Python 3.9 (#2715, #2674).


🚀 New Features

🔧 Added new benchmark configurations: CritPt, General365, AA-OmniScience, AA-LCR v1.1, HLE-Diamond, arena-hard-v2, ReasonZoo, LG-Bench, CHHallu-Src v1, GrandPhysics, PromptCBLUE Life Science, MaritimeBench, WMT19, LogiQA and ethics, with AA-LCR, AA-OmniScience (v1.1) and HLE-Diamond configs and the prompt template updated (#2610, #2626, #2658, #2713, #2362, #2256, #2690, #2525, #2118, #2073, #2018, #1503, #516, #495).

🔧 Added ChartQAPro, MicroVQA, SFE and SciDocBench datasets through VLMEvalKit (#2665).

🔧 Added retry and verbose options to the TurboMind API for get_ppl and get_ll (#2650, #2652).


🐛 Bug Fixes

🔧 Fixed reasoning parsing in the OpenAI, OpenAISDK and OpenAISDKStreaming models, and surfaced inference failures in OpenICLInferConcurrentTask (#2653, #2651).

🔧 Fixed dataset loading and downloads for AA-LCR, MBPP-CN, C-Eval (ModelScope and HF generation for causal LMs) and SciReasoner 1.5 (with a new llm_extract config), and fixed MATH JudgeLLM config imports (#2630, #2695, #2416, #2640, #2662).

🔧 Fixed VLMEvalKit empty predictions and JSONL resume with Unicode separators (#2691).


⚙ Enhancements and Refactors

⚙ Refactored the OpenCompass documentation (#2666).

⚙ Refactored the mock API tests and fixed the PR stage CI (#2698, #2670).

⚙ Dropped Python 3.8 support, added Python 3.9 and 3.12 to the PR stage CI, and fixed PEP 604 annotations that broke imports on Python 3.9 (#2715, #2674).

⚙ Fixed VLM installation dependencies and requirement declarations (#2679, #2676).


🎉 Welcome New Contributors

A warm welcome and special thanks to our newest contributors who made this release possible:


Full Changelog: 0.5.4...0.5.5

Thank you to all contributors who made OpenCompass v0.5.5 possible!

0.5.4

Choose a tag to compare

@Myhs-phz Myhs-phz released this 26 Aug 06:41
20586a2

OpenCompass v0.5.4 Release Notes

🌟 Highlights

✨ Multimodal Evaluation with VLMEvalKit: OpenCompass now integrates VLMEvalKit for native multimodal dataset loading, OpenAI-compatible inference, and official VLMEvalKit evaluation metrics (#2586).

✨ Multi-Round Evaluation: GenInferencer now supports multi-round inference, accompanied by the new Multi-IF benchmark for evaluating multi-turn instruction following (#2557).

✨ Expanded Model API Ecosystem: Added LiteLLM as an AI gateway backend and extended TurboMind API capabilities with PPL evaluation support (#2441, #2510).

✨ Broader Benchmark Coverage: Added new evaluations spanning finance, long-context understanding, alignment, information extraction, and code reasoning, including BuySideFinBench, ELBench, Inverse IFEval, AA-LCR, CRUXEval, and updated LongBench/MRCR V2 configurations (#2446, #2495, #2514, #2558).


🚀 New Features

🔧 Integrated VLMEvalKit datasets and evaluators into the OpenCompass workflow, with MMBench and MMMU-Pro evaluation examples (#2586).

🔧 Added multi-round inference to GenInferencer and introduced Multi-IF dataset support (#2557).

🔧 Added LiteLLM AI Gateway model support and the TurboMind API /get_ppl endpoint (#2441, #2510).

🔧 Added and expanded benchmark configurations for CRUXEval, LongBench v1, Qasper, MRCR v2, Fake-Alignment, and more (#2572, #2592, #2549, #2553).


🐛 Bug Fixes

🔧 Strengthened evaluation correctness by rejecting deceptive return values in code evaluators and preventing invalid SIQA generations from being scored as correct (#2565, #2450).

🔧 Fixed generation behavior across HuggingFace and vLLM, including max_out_len=None, default torch dtype loading, accelerator generation config conversion, and chat-template BOS handling (#2459, #1969, #2519, #2554).

🔧 Improved evaluator reliability for BBH free-form answers, LiveCodeBench memory limits, Wikitext PPL without references, and official Fake-Alignment prompts (#2548, #2538, #2532, #2567).

🔧 Fixed dataset loading, local path resolution, download mappings, and prompts across AGIEval, BoolQ, GSM8K, C-Eval, SVAMP, TACO, and other benchmark configurations (#2497, #2498, #2540, #2546).


⚙ Enhancements and Refactors

⚙ Improved multi-round observability by fixing per-turn statistics in dump-res-length and repeat detection for multi-round conversations (#2569).

⚙ Improved model API robustness for Qwen response extraction, DeepSeek retry timeouts, Claude custom headers, and TurboMind proxy compatibility (#2521, #2530, #2445, #2529).

⚙ Improved data and task handling with safer SizePartitioner cost calculation and corrected TopK retriever metadata collation (#2438, #2556).

⚙ Refactored CI workflows, pinned third-party CI actions, and removed misleading error logs from successful RJob submissions (#2570, #2533, #2588).


📦 Compatibility and Packaging

📦 Made OpenCC optional to restore Windows installation compatibility (#2512).

📦 Pinned the mmengine-lite runtime dependency for more reproducible installations (#2531).

📦 Resolved the Dingo/Pillow dependency conflict and fixed Dingo evaluation behavior (#2547, #2607).


🎉 Welcome New Contributors

A warm welcome and special thanks to our newest contributors who made this release possible:


Full Changelog: 0.5.3...0.5.4

Thank you to all contributors who made OpenCompass v0.5.4 possible!

0.5.3

Choose a tag to compare

@ssiq ssiq released this 29 Jun 03:17
a25fdd2

OpenCompass v0.5.3 Release Notes

🌟 Highlights

✨ 🧪 Extensive New Benchmarks and Dataset Support: Added support for a wide range of new benchmarks and datasets, including AIME2026, HMMT Feb 2026, SimpleQA-Verified, AdvancedIF, HLE-Verified, MRCR-V2, Molecular-IQ, SciReasoner 1.5, MP20, PerspectiveGap, ZebraLogic, ArxivRollBench, CL-Bench, and more.

✨ 🧩 RawPromptTemplate Support: Introduced RawPromptTemplate and expanded its support across dataset configs, OpenAISDKStreaming, ChatML datasets, retrievers, and multiple benchmark configurations.

✨ 🤖 New Model & API Support: Added support for the OpenAI Responses API, Gemini SDK API and Claude SDK thinking content.

✨ 🛠️ Infrastructure & Evaluation Enhancements: Improved concurrent inference, repeat analysis, summarizer configuration, judge post-processing, CI coverage, and evaluation robustness.


🚀 New Features

🔧 Added support for new datasets and benchmarks, including AIME2026 and HMMT Feb 2026 (#2404), Molecular-IQ (#2431), SimpleQA-Verified (#2436), AdvancedIF (#2461), HLE-Verified (#2454), MRCR-V2 (#2467), S2-ToM-G Bench (#2476), SciReasoner 1.5 (#2477, #2479), MP20 (#2482), PerspectiveGap (#2484), ZebraLogic (#2464), and ArxivRollBench (#2458).

🔧 Introduced RawPromptTemplate and new dataset configs (#2407).

🔧 Added RawPromptTemplate support for OpenAISDKStreaming and ChatMLDatasets (#2414).

🔧 Added SciReasoner config and support for retrievers using RawPromptTemplate (#2422).

🔧 Added repeat configs for HMMT2025 and UGD_hard (#2425).

🔧 Added new subjective dataset configs and fixed AdvancedIF related configs (#2466).

🔧 Added RawPromptTemplate configs for AlignBench v1.1 (#2472), RULER, NeedleBench, LongBench v2, and BABILong (#2474).

🔧 Added support for repeat analysis (#2455).

🔧 Added OpenAI Responses API model support (#2481).

🔧 Added Gemini SDK API model support (#2494).

🔧 Added support for Claude SDK thinking content (#2487).

🔧 Added CL Bench support (#2483).


🐛 Bug Fixes

🔧 Fixed IFEval path issue (#2406).

🔧 Fixed tag matching in generic_llmjudge_postprocess (#2417).

🔧 Fixed issues in generic.py (#2419).

🔧 Fixed ChatML and dataset utility issues (#2421).

🔧 Fixed OlympiadBench compatibility with RawPromptTemplate (#2423).

🔧 Fixed ellipsis extraction in bio_data (#2429).

🔧 Fixed Molecular-IQ PR test issues (#2432).

🔧 Added multi-round support from api_meta_template (#2435).

🔧 Fixed result metrics in SimpleQA (#2439).

🔧 Fixed no-prediction cases in SciReasoner biology instruction evaluation (#2456).

🔧 Fixed hub version issue (#2457).

🔧 Fixed SimpleQA prompt (#2463).

🔧 Added unsafe pattern handling for MathEvaluator (#2468).

🔧 Fixed MTBench101, WildBench, and AdvancedIF issues (#2470).

🔧 Added subprocess timeout to MATHVerifyEvaluator to prevent parse and verify hangs (#2478).

🔧 Resolved dependency issues for Python 3.12 (#2480).

🔧 Restored LongBench v2 answer prompt punctuation (#2452).

🔧 Fixed MP20 metrics (#2488).

🔧 Prioritized Responses reasoning content (#2490).


⚙ Enhancements and Refactors

⚙ Evaluation and Runtime Improvements:

  • Supported concurrent inference across tasks (#2403).
  • Reduced RJOB sleep time (#2411).
  • Added new template for CompassAcademic (#2447).
  • Expanded config options of the summarizer group (#2448).
  • Added --no-progress parameter for repeat analysis to suppress detailed logs (#2473).
  • Added prediction postprocessor support for judge models in GenericLLMEvaluator (#2491).

⚙ RawPromptTemplate Documentation and Compatibility:

  • Added guide for RawPromptTemplate (#2420).
  • Improved RawPromptTemplate coverage across benchmark and dataset configs (#2407, #2414, #2422, #2472, #2474).

⚙ Metadata and Release Updates:

  • Updated dataset index metadata (#2492).
  • Bumped version to 0.5.3 (#2493).

⚙ CI/CD Improvements:

  • Refactored ETE test cases and added more unit tests (#2408).
  • Added inference test cases using mock API (#2428).
  • Added API test cases into PR tests (#2460).
  • Removed host network from CI settings (#2462).

⚙ Documentation:

  • Fixed minor documentation bugs (#2469).

🎉 Welcome New Contributors

A warm welcome and special thanks to our newest contributors who made this release possible:


Full Changelog: 0.5.2...0.5.3

Thank you for using OpenCompass! These updates bring broader benchmark coverage, stronger API support, and more reliable evaluation workflows. Keep exploring, and stay tuned for future innovations! 🌟

0.5.2

Choose a tag to compare

@Myhs-phz Myhs-phz released this 14 Feb 03:46
9741792

OpenCompass v0.5.2 Release Notes

🌟 Highlights

✨ 🧪 Extensive New Benchmarks Support: We have introduced comprehensive support for Scientific and General Benchmarks, including SciReasoner, Biology Instructions, Mol Instructions, CMPhysBench, IFBench, LCB-pro, etc.
✨ 🤖 New Model & API Support: Added support for Intern-S1-Pro and TeleChat API evaluation examples.
✨ 🛠️ Infrastructure & Enhancements: Fixed bugs, improved evaluation pipelines and updated CI.


🚀 New Features

🔧 Introduced support for HMMT2025 (#2305), AMO-Bench (#2305), IMO-Bench (#2305), ATLAS (#2297), OpenSWI (#2312), CMPhysBench (#2313), Biology Instructions (#2314), Mol Instructions (#2326), ARC_AGI_2 (#2330), IFBench (#2354), SciReasoner (#2360), PI-LLM (#2283), ProcessBench (#2274), and LCB_pro (#2361).
🔧 Supported monitoring of multi-dimensional evaluation metrics, including output length, logprobs, and finish reasons (#2351).
🔧 Added support for Intern-S1-Pro evaluation examples (#2394).
🔧 Added support for TeleChat API inference (#2371).
🔧 Added LLM-judge-based config for C-Eval (#2398).


🐛 Bug Fixes

🔧 Fixed OpenAISDKStreaming regarding output completeness and related issues (#2367, #2389, #2399).
🔧 Removed Pyext in runtime requirement (#2306).
🔧 Fixed pattern match in Smolinstruct (#2384).
🔧 Fixed buffer-related error in the LiveCodeBench evaluation (#2393).


⚙ Enhancements and Refactors

⚙ Infrastructure Refactors:

  • Updated LCBench (#2166).
  • Added headers as input param in BigCodeBench (#2302).
  • Updated rjob with metadata name (#2316).
  • Parametrized timeout in OpenAISDK (#2352).
  • Added meta logger in OpenICLInferTask (#2383).

⚙ CI/CD Improvements:

  • Refactored dailytest (#2308).
  • Added CI for new datasets (#2358, #2369).
  • Changed github runner (#2373).
  • Added uni-test (#2390).

🎉 Welcome New Contributors

A warm welcome and special thanks to our newest contributors who made this release possible:


Full Changelog: 0.5.1.post1...0.5.2

Thank you for using OpenCompass! These updates empower deeper insights and more reliable evaluations. Keep exploring, and stay tuned for future innovations! 🌟

0.5.1.post1

Choose a tag to compare

@Myhs-phz Myhs-phz released this 17 Oct 10:17
ecc86a2

OpenCompass v0.5.1 Release Notes

🌟 Highlights

✨ A New Method to quickly Integrate and Evaluate Your Datasets: Added a fast dataset integration and evaluation method based on ChatML format, simplifying the previously complex dataset integration process.
✨ New Datasets: Integrated new benchmarks including SeedBench and BeyondAIME.
✨ Infrastructure & Enhancements: Fixed several bugs and updated CI.


🚀 New Features

🔧 Introduced a new approach for dataset integration and evaluation based on ChatML Template with evaluation examples (#2277).
🔧 Added SeedBench dataset (#2020).
🔧 Added BeyondAIME dataset (#2192).


🐛 Bug Fixes

🔧 Fixed Module Registers(#2262, #2266)
🔧 Fixed duplicate engine config update in TurboMindModelwithChatTemplate (#2276)
🔧 Fixed torchrun to avoid unexpected PATH environment (#2269)
🔧 Fixed the None return value case in cascade evaluator (#2211)


⚙ Enhancements and Refactors

⚙ Infrastructure Refactors:

  • Update rjob.py and subjective_eval.py (#2263)

⚙ CI/CD Improvements:

  • Updated testcase (#2257)
  • Updated pr_test (#2281)
  • Fixed pr_test installation (#2290)

🎉 Welcome New Contributors

A warm welcome and special thanks to our newest contributors who made this release possible:


Full Changelog: 0.5.0...0.5.1.post1

Thank you for using OpenCompass! These updates empower deeper insights and more reliable evaluations. Keep exploring, and stay tuned for future innovations! 🌟

0.5.0

Choose a tag to compare

@Myhs-phz Myhs-phz released this 01 Sep 06:20
a88f268

OpenCompass v0.5.0 Release Notes

🌟 Highlights

✨ ​Comprehensive Scientific Benchmarks: Integrated 10+ specialized datasets (MedXpertQA, ClimaQA, SmolInstruct, etc.), covering multiple scientific fields such as chemistry, physics, biology, and earth sciences
✨ Cascade Evaluator: Supported cascading eval methods from rules to LLM judgments.
✨ New Runner: Supported for Rjob Runner has now been completed.
✨ OpenAISDK Streaming: Provided a more stable OpenAI API method.
✨ New Evaluation Examples: Published the real-time evaluation config of CompassAcademic Leaderboard and the Intern-S1 related benchmark evaluation config.


🚀 New Features

🔧 Cascade Evaluator (#1992)
🔧 Rjob Runner (#2144)
🔧 OpenAISDK Streaming (#2208)
🔧 Evaluation Example for CompassAcademic Leaderboard. (#2202)
🔧 Evaluation Example for Intern-S1 and Scientific Benchmarks. (#2220)
🔧 So Many New Scientific Datasets!

  1. MedXpertQA for expert-level medical knowledge evaluation (#2002)
  2. ClimaQA for climate question evaluation (#2017)
  3. HealthBench for better measuring capabilities of AI systems for health (#2099)
  4. ProteinLMBench for protein related tasks (#2064)
    ...

📖 Documentation

📝 Fixed 404 links between Chinese/English docs (#2001)
📝 Added CompassAcademic Leaderboard task tutorial (#2202)
📝 Added Intern-S1 evaluation task tutorial (#2220)
📝 Fixed format problems of the dataset statistics page (#2170)
📝 Align NIAH CLI command guide to the actual CLI argument parser (#2194)
📝 Set correct paths for the examples (#2198)


🐛Bug Fixes

🔧 Fixed compare error base_evaluator (#2010)
🔧 Fixed OpenICL Math Evaluator Config (#2007)
🔧 Added Error Case for content filter (#2167)
🔧 Fix the OpenAI SDK to adapt to gpt-5 (#2236)
🔧 Fixed dataset repeat by concatenating (#2039)
🔧 Concat OpenaiSDK reasoning content (#2041)


⚙ Enhancements and Refactors

⚙ Infrastructure Refactors:

  • Set dump-eval-details as default behavior (#1999)
  • Refactorized openicl eval task (#1990)
  • Added openai_extra_kwargs for API customization (#2210)

⚙ CI/CD Improvements:

  • Fixed baseline score (#2000)
  • Updated baseline for kernal change of vllm and lmdeploy (#2011)
  • Updated baseline and fix lmdeploy version (#2098)
  • Added check rule (#2101)
  • Updated testcases' baseline (#2184)
    ...

🎉 Welcome New Contributors

A warm welcome to our newest contributors:


Full Changelog: 0.4.2...0.5.0

Thank you for using OpenCompass! These updates empower deeper insights and more reliable evaluations. Keep exploring, and stay tuned for future innovations! 🌟

0.4.2

Choose a tag to compare

@MaiziXiao MaiziXiao released this 02 Apr 10:06
dc8deb6

OpenCompass v0.4.2

The OpenCompass team is thrilled to announce the release of OpenCompass v0.4.2! This version introduces powerful new datasets, robust evaluation enhancements, and critical refinements to elevate your benchmarking workflows. Let’s explore the key updates!


🌟 Highlights

✨ SuperGPQA Benchmark: Added support for the SuperGPQA dataset, enabling advanced reasoning evaluation with refined subset metrics. (#1924, #1966)
✨ MultiPL-E & Code Evaluator: Introduced MultiPL-E dataset integration and a dedicated code evaluator for comprehensive programming task assessments. (#1963)
✨ OlympiadMath Dataset: Expanded mathematical reasoning benchmarks with the OlympiadMath dataset. (#1982)


🚀 New Features

🔧 MMLU Pro Evaluation: Implemented generic LLM evaluation for MMLU Pro, broadening assessment flexibility. (#1923)
🔧 LLM Judge Configuration: Enhanced LLM judge capabilities with updated dataset configurations. (#1940)
🔧 Model Configurations:

  1. Added Intervl-8B and Intervl-38B model configurations. (#1978)
  2. Introduced Qwen-72B model support. (#1959)

📖 Documentation

📝 Results Persistence Guide: Documented evaluation results persistence workflows for better reproducibility. (#1908)
📝 Dataset Statistics: Added TBD token clarifications in dataset statistics documentation. (#1986)
📝 Typo Fixes: Corrected typos in DeepSeed-R1 documentation. (#1916)


🐛 Bug Fixes

🔧 Math Verification: Fixed math-verify evaluator logic to ensure accurate mathematical reasoning checks. (#1917)
🔧 Summarizer Logic: Resolved summarizer inconsistencies for reliable result aggregation. (#1953)
🔧 Model Compatibility:

  1. Fixed model_kwargs handling for vLLM accelerator compatibility. (#1958)
  2. Patched AIME-2024 configuration errors. (#1974)
  3. Addressed OpenAI model tokenization constraints. (#1960)

⚙ Enhancements and Refactors

⚙ Dataset Optimization:

  1. Refined dataset configurations for KorBench, LiveMathBench, and others. (#1937, #1967)
  2. Added no max_out_len configurations for flexible evaluations. (#1968)

⚙ Infrastructure Upgrades:

  1. Increased memory allocation for VOLC Runner CPU jobs. (#1962)
  2. Updated OlympiadBench and LLM Judge pipelines. (#1954)

⚙ CI/CD Improvements:

  1. Fixed baseline scoring in daily tests for consistent benchmarking. (#1996)

🎉 Welcome New Contributors

🎊 A warm welcome to our newest contributors:

  1. @kangreen0210 for adding SuperGPQA support in #1924
  2. @Jiajun0425 for resolving vLLM accelerator issues in #1958

Full Changelog: 0.4.1...0.4.2

Thank you for using OpenCompass! These updates empower deeper insights and more reliable evaluations. Keep exploring, and stay tuned for future innovations! 🌟

0.4.1

Choose a tag to compare

@MaiziXiao MaiziXiao released this 04 Mar 10:35
5547fd1

OpenCompass v0.4.1 Release Notes

The OpenCompass team is thrilled to announce the release of OpenCompass v0.4.1! This version brings exciting new features, enhancements, and bug fixes to improve your experience. Let’s dive into the highlights and changes!


🌟 Highlights

✨ Omni-Math Support: OpenCompass now supports the Omni-Math dataset, expanding its capabilities in mathematical reasoning. (#1837)
✨ OlympiadBench Benchmark: Added support for the OlympiadBench benchmark, enabling evaluation on advanced problem-solving tasks. (#1841)
✨ HLE Dataset: Introduced the Humanity's Last Exam (HLE) dataset, providing a challenging new benchmark for AI models. (#1902)


🚀 New Features

🔧 Math Verification: Added model post-processor for math verification, enhancing accuracy in mathematical evaluations. (#1881)
🔧 Dataset Repeat & G-Pass Compute: Support for dataset repetition and G-Pass computation for each evaluator, improving flexibility in evaluations. (#1886)
🔧 General Math & LLM Judge Evaluator: Added a general math evaluator and LLM judge evaluator for broader evaluation scenarios. (#1892)
🔧 AIME-24 Evaluation: Supported AIME-24 evaluation for DeepSeek-R1 series, enhancing competition-level benchmarking. (#1888)


📖 Documentation

📝 Broken Links Fix: Fixed broken links in README.md to ensure a smoother onboarding experience. (#1852)
📝 Supported Datasets List: Added a list of supported datasets on the HTML page for better accessibility. (#1850)


🐛 Bug Fixes

🔧 Compatibility Issue: Fixed a compatibility issue to ensure smoother integration with existing workflows. (#1904)
🔧 Hard Configs Fix: Updated hard configs with General GPassK for improved evaluation consistency. (#1906)


⚙ Enhancements and Refactors

⚙ Daily Test Updates: Updated daily test scores and scheduler for better CI/CD pipeline reliability. (#1854, #1898)
⚙ Benchmark Updates: Updated BigCodeBench, LCBench, and LiveMathBench configurations for enhanced evaluation accuracy. (#1857, #1862, #1826)
⚙ OpenAI Model Update: Updated OpenAI models and BigCodeBench configurations for better performance. (#1879)


🎉 Welcome New Contributors

🎊 A warm welcome to our new contributors:

  1. @sudanl for their first contribution in #1841
  2. @Pablohn26 for their first contribution in #1852
  3. @Zhudongsheng75 for their first contribution in #1857

Full Changelog: 0.4.0...0.4.1

Thank you for using OpenCompass! We’re excited to see how you leverage these new features and improvements. Stay tuned for more updates! 🚀

0.4.0

Choose a tag to compare

@MaiziXiao MaiziXiao released this 22 Jan 06:41
35ec307

The OpenCompass team is thrilled to announce the release of OpenCompass v0.4.0!

🌟 Highlights
✨ This version introduces several new features and improvements that enhance the user experience and expand the capabilities of OpenCompass. Notable changes include support for Longbenchv2, compatibility with InternLM3 models and also refactorization of Openai model class. We are also consolidating all AMOTIC configuration files (previously located in ./configs/datasets, ./configs/models, and ./configs/summarizers) into the opencompass package.

🚀 New Features
-🆕 Support for Longbenchv2 has been added to provide more comprehensive model evaluation. (#1801)
-🆕 The Bradley-Terry Subjective Evaluation method has been extended to the Arena Hard dataset. (#1802)
-🆕 Predicted Win Probabilities have been added to CompassArenaBradleyTerrySummarizer for better insights into model performance. (#1815)
-🆕 Support for InternLM3 models is now available, expanding our model library. (#1829)
-🆕 MMLU-CF Benchmark support has been introduced to further enrich our suite of benchmarks. (#1775)

📖 Documentation
-📚 Documentation on adding new datasets (new_dataset.md) has been updated for clarity. (#1827)
-📚 The installation guide (installation.md) has been revised to reflect the latest setup process. (#1830)

🐛 Bug Fixes
-🔧 A path conflict within the CI pipeline has been resolved. (#1814)
-🔧 The logic for max_out_len in OpenAI models has been corrected. (#1839)

⚙ Enhancements and Refactors
-💪 Code refactoring has been performed to improve project structure and maintainability. (#1831)
-💪 The threshold for CI checks has been updated for more stringent quality control. (#1812)
-💪 LiveMathBench has been updated to ensure it remains a cutting-edge benchmarking tool. (#1809)

🎉 Welcome New Contributors
@fistyee contributed support for the MMLU-CF Benchmark. (#1775)
@Myhs-phz improved documentation on adding new datasets. (#1827)
@thejishnunair updated the installation guide. (#1830)

Thank you for being part of the OpenCompass community! Your support and contributions make each release possible. We look forward to your feedback on this release.

0.3.9

Choose a tag to compare

@MaiziXiao MaiziXiao released this 31 Dec 09:28
f322043

The OpenCompass team is thrilled to announce the release of OpenCompass v0.3.9!

🌟 Highlights
✨ This version introduces a number of new features and improvements that enhance the user experience and expand the capabilities of OpenCompass. Notable changes include support for G-Pass@k and LiveMathBench, as well as the introduction of the Bradley-Terry subjective evaluation method.

🚀 New Features
-🆕 Support for G-Pass@k and LiveMathBench metrics to better evaluate model performance. (#1772)
-🆕 Theorem QA 0shot CoT configuration has been added for more comprehensive evaluation scenarios. (#1783)
-🆕 A customizable tokenizer for RULER offers greater flexibility in processing inputs. (#1731)
-🆕 Added LiveStemBench Dataset to enrich our collection of datasets. (#1794)
-🆕 Integration of JudgeLLM into o1 evaluation for improved assessment accuracy. (#1795)
-🆕 Implementation of the Bradley-Terry subjective evaluation method on wildbench, alpacaeval, and compassarena datasets. (#1791)

📖 Documentation
-📚 Updated OC academic content to the most recent information as of December 2024. (#1771)

🐛 Bug Fixes
-🔧 Fixed Order error which was causing issues with sequence handling. (#1767)
-🔧 Resolved an issue where the lark report was returning None. (#1769)
-🔧 Corrected the path for saving Local Runner parameters. (#1768)
-🔧 Amended the summarizer abbreviation for models to ensure proper identification. (#1789)
-🔧 Fixed output_path errors to improve file handling reliability. (#1798)

⚙ Enhancements and Refactors
-💪 Fullbench testcase has been integrated into the CI pipeline. (#1766)
Volc status exception handling has been updated for more robust responses. (#1780)
-💪 Removed daily step retry mechanism and updated PR score calculation for efficiency. (#1782)
-💪Deploy Python version has been updated to the latest stable release. (#1784)
-💪Pypi deploy workflow has been refined for smoother deployments. (#1786)

Thank you for being part of the OpenCompass community! Your support and contributions make each release possible.