Doubao Team Open Sources Multi-SWE-bench: A New Starting Point for Large Models’ ‘Automatic Bug Fixing’ Capabilities

Doubao Team’s Major Open Source Initiative Empowers Large Models’ ‘Bug Fixing’ Capabilities

In the rapid development of large model technology, the ability to fix code has become one of the key indicators of its performance. Recently, the Doubao team has stood out by open-sourcing the first multi-language code repair benchmark, Multi-SWE-bench. This initiative marks a new milestone in the field of large model technology, promising to significantly enhance the ‘automatic bug fixing’ capabilities of large models, and has sparked widespread attention and discussion within the technical community.

Multi-SWE-bench: A Solid Benchmark for Multi-Language Code Repair

1. Comprehensive Multi-Language Coverage: One of the significant advantages of Multi-SWE-bench is its comprehensive coverage of various programming languages. It includes mainstream programming languages such as Python, Java, and C++, as well as some niche languages that are widely used in specific fields. This multi-language support allows large models to be effectively evaluated and improved in their code repair capabilities across different language environments. For instance, whether it is a logical error in a Python script or a syntax issue in a Java enterprise application, large models trained and optimized based on Multi-SWE-bench can attempt to provide precise repair solutions, meeting the needs of different developer groups and application scenarios.
2. Real-World Scenario Simulation: This benchmark meticulously simulates real-world code error scenarios. It extracts common and representative error samples from a large number of real code repositories, including open-source projects and actual development cases, covering various types of errors such as syntax errors, semantic errors, logical errors, and runtime errors. These errors are not simply fabricated but represent issues that can realistically occur during actual development. By using Multi-SWE-bench, large models can train and test in an environment that closely resembles reality, thereby better learning how to handle complex and variable code errors, enhancing their practical application capabilities for ‘automatic bug fixing’.

Profound Impact on the Development of Large Models

1. Enhancing Code Repair Performance: Multi-SWE-bench provides large models with richer and more precise training and evaluation resources, significantly improving their code repair performance. In the past, due to the lack of comprehensive and realistic code repair benchmarks, large models often performed poorly when handling code errors, failing to accurately understand the nature of the errors and provide effective repair solutions. Now, with optimization based on Multi-SWE-bench, large models can learn more deeply about the syntax rules, semantic logic, and common error patterns of different languages, enabling them to quickly and accurately locate issues and provide reliable repair suggestions when faced with actual code errors, saving developers a significant amount of debugging time and effort.
2. Promoting Cross-Language Code Repair Research: The multi-language feature of Multi-SWE-bench strongly promotes research in cross-language code repair. Although different programming languages have varying syntax and characteristics, there are certain commonalities in code logic and error types. By training large models on Multi-SWE-bench, researchers can explore how to leverage these commonalities to develop more universal code repair models. This not only helps enhance the transfer learning capabilities of large models across various languages but may also provide new ideas and tools for cross-language programming development, fostering communication and collaboration among developers of different languages.

Significance in the Technical Community

1. Open Source Sharing Promotes Collaboration: The Doubao team’s open-sourcing of Multi-SWE-bench fully embodies the spirit of open source, creating a collaborative platform for researchers and developers worldwide. Everyone can share their research results, optimization algorithms, and improvement ideas based on this benchmark. This model of open-source sharing can gather global wisdom and accelerate the development of large model code repair technology. For example, research teams from different countries can use Multi-SWE-bench for experiments and exchange experiences in technical forums, collectively tackling the challenges faced by large models in code repair.
2. Standardizing Code Repair Evaluation Criteria: Before the emergence of Multi-SWE-bench, there was a lack of unified and authoritative standards for code repair evaluation, making it difficult to accurately compare and assess different research results. The open-sourcing of Multi-SWE-bench provides the entire technical community with a unified evaluation framework, making the assessment of large models’ code repair capabilities more scientific, objective, and comparable. This helps guide research directions and promotes the development of large model code repair technology towards a more standardized and regulated approach, enhancing the quality and level of research in the entire field.