Recent advances in coding agents have sparked excitement around AI-assisted modernization. But an important question remains:
Existing software engineering benchmarks have demonstrated impressive progress in bug fixing and code generation, but framework migration presents a fundamentally different challenge. Success requires not only translating code, but also preserving behavior, adapting build systems, and navigating runtime dependencies.
To address this gap, we introduce ScarfBench (Self-Contained Application Refactoring Benchmark), an open benchmark for evaluating AI agents on cross-framework migration tasks in Enterprise Java.
Unlike traditional benchmarks that compare generated code against reference implementations, ScarfBench evaluates whether migrated applications actually build, deploy, and preserve behavior.
A simple repository migration can require changes across dependency injection, persistence configuration, queries, and framework descriptors. Small mistakes in any of these pieces can prevent successful deployment.
ScarfBench provides a systematic way to evaluate AI agents on enterprise Java framework migration tasks.
Starting from a JSR-based enterprise Java taxonomy, expert migrations create verified implementations across Spring, Jakarta EE, and Quarkus.
Despite strong performance on traditional software engineering benchmarks, framework migration remains difficult. Success rates vary considerably across framework pairs and whole-application migrations remain particularly challenging.
Compile success consistently exceeds deploy success, which in turn exceeds behavioral success. Build success alone significantly overestimates migration quality.
Migration difficulty depends strongly on the target framework, with Jakarta EE proving particularly challenging.
Beyond measuring success rates, ScarfBench helps us understand how agents behave during modernization.
A migrated application is only useful if it actually builds and runs.
Claude Code reported successful builds for 29 out of 30 whole applications.
Source link
