MERA Code

   

MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks.

🚀 About

MERA Code brings together a rich collection of code-focused evaluation tasks—both private and public—under one roof. Built on top of the Language Model Evaluation Harness (v0.4.9), it enables researchers and practitioners to:

• Compare models on identical tasks and metrics
• Reproduce results with fixed prompts and few-shot settings
• Submit standardized ZIP archives for leaderboard integration

🔍 Datasets Overview

🛠 Getting Started

First, you need to clone the MERA_CODE repository and load the submodule:

Now, you can choose one of two evaluation regimes, depending on whether you want to obtain the metrics for public tasks locally or intend to use our remote scoring via the website.

Remote Scoring

Remote Scoring (default): quick setup for cloud-based scoring — install only core dependencies, run the evaluation, and submit the resulting ZIP archive to our website to get the score.

You will not get the metrics even for public datasets (for each dataset, you will see a "bypass" placeholder instead of actual metrics) in the terminal.

You may also need additional libraries for model inference or evaluation. Use lm-eval compatible libraries and their versions:

Local Scoring

Local Scoring (optional): full setup for on-premise evaluation — install extra dependencies with metrics and run Docker containers. Available only for Public sets.

Ensure you have a stable internet connection, sufficient disk space, and adequate CPU resources.

Now, proceed to the evaluations, but with the flag

More details on

📁 Repository Structure

💪 How to Join the Leaderboard

Follow these steps to see your model on the Leaderboard:

1. Run Remote Scoring   Evaluate the benchmark in the Remote Scoring regime (see 🛠 Getting Started above). You may run Local Scoring, but you will have to wait twice for submission scoring.

You’ll end up with a logs folder and a ready-to-submit zip archive like

Qwen2.5-0.5B-Instruct_submission.zip

.

2. Submit on the website   Head over to Create Submission, upload the archive, and move on to the form.

3. Fill in Model Details   Provide accurate information about the model and evaluation. These details are crucial for reproducibility—if something is missing, administrators may ping you (or your Submission might be rejected).

4. Wait for Scoring ⏳   Scoring usually wraps up in ~2 hours. There is a progress bar to track the scoring process.

Keep in mind that if you submit more than one archive, they are scored sequentially, one after another (not in parallel).

5. Publish your result   Once scoring finishes, click "Submit for moderation". After approval, your model goes Public and appears on the Leaderboard.  

Good luck, and happy benchmarking! 🎉

🤝 Contributing

We are interested in improving the MERA Code and invite the community to contribute to the development of new, complex tasks and the project's codebase.

Steps to Add a New Task:  

0. Develop a dataset (on the contributor's side; see task requirements)  
1. Convert the dataset to MERA format (guide)  
2. Upload the dataset to 🤗HF Hub (guide)  
3. Submit the dataset for MERA organizer review (guide)  
4. Write evaluation code using lm-harness (guide)  
5. Benchmark state-of-the-art baseline models on the dataset
6. Final moderation, and your dataset is officially added!    

📝 License

Distributed under the MIT License. See LICENSE for details.

📑 Cite as

@misc{chervyakov2025meracodeunifiedframework,
      title={MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks}, 
      author={Artem Chervyakov and 
        Alexander Kharitonov and 
        Pavel Zadorozhny and 
        Adamenko Pavel and 
        Rodion Levichev and 
        Dmitrii Vorobev and 
        Dmitrii Salikhov and 
        Aidar Valeev and 
        Alena Pestova and 
        Maria Dziuba and 
        Ilseyar Alimova and 
        Artem Zavgorodnev and 
        Aleksandr Medvedev and 
        Stanislav Moiseev and 
        Elena Bruches and 
        Daniil Grebenkin and 
        Roman Derunets and 
        Vikulov Vladimir and 
        Anton Emelyanov and 
        Dmitrii Babaev and 
        Vladimir V. Ivanov and 
        Valentin Malykh and 
        Alena Fenogenova},
      year={2025},
      eprint={2507.12284},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2507.12284}, 
}

Read the paper on arXiv