LMArena

概要

Chatbot Arena論文は、クラウドソースされた人間のペア比較でLLMを評価するオープンプラットフォームと説明しています。LMSYSの更新では、Elo風評価からBradley-Terryモデルへ移行したことも説明されています。

向いている用途

Chatbot Arena論文は、クラウドソースされた人間のペア比較でLLMを評価するオープンプラットフォームと説明しています。LMSYSの更新では、Elo風評価からBradley-Terryモデルへ移行したことも説明されています。

主な機能

• Human-preference leaderboard for comparing frontier AI models.
• Blind pairwise comparisons aggregated into Elo-like/Bradley-Terry ratings.
• Leaderboards now span text and broader modalities such as image, video, search, and code on Arena/LMArena surfaces.
• Useful public signal for model quality, but not a complete benchmark suite.
• Backed by the Chatbot Arena research paper and LMSYS methodology updates.

実務ユースケース

• Compare model performance before choosing an LLM for a product.
• Track public preference shifts after new model releases.
• Explain why human-preference evaluation can differ from static academic benchmarks.
• Use leaderboard signals in procurement, model-routing, or evaluation planning.
• Study crowdsourced pairwise evaluation methodology for AI benchmarking.

推奨ワークフロー

• Check which arena category matches your use case: text, code, image, video, search, or other modality.
• Look at confidence intervals and recency, not just rank order.
• Compare Arena results with your own private evals before switching models.
• Use human preference rankings as one signal alongside cost, latency, safety, context, and tool support.
• Watch for leaderboard instability and category-specific differences.

強みと注意点

• Very useful for public preference signals and frontier-model comparison.
• Crowdsourced votes can reflect user mix, prompt mix, UI effects, and recency bias.
• Pairwise rankings do not replace domain-specific private evaluations.
• Scores can move as new votes, models, and methodology changes arrive.

比較候補

• HELM for broader academic benchmark coverage.
• Artificial Analysis for speed, price, and quality metrics.
• OpenRouter rankings for usage and routing ecosystem signals.
• Your own eval harness for domain-specific business tasks.

FAQ

Is Chatbot Arena the same as LMArena?

The platform has evolved from LMSYS Chatbot Arena/LMArena branding toward Arena-style leaderboards, but the core idea is human-preference model comparison.

How are models ranked?

The Chatbot Arena paper and LMSYS updates describe blind pairwise comparisons and Bradley-Terry/Elo-like rating methodology.

Should teams choose a model only by Arena rank?

No. Use it as one signal and also evaluate cost, latency, safety, context length, tool use, and your own domain tasks.

確認した情報源

• LMArena leaderboard
• Chatbot Arena paper
• LMSYS leaderboard update