Leaderboard

Brackets

Hybrid vs. pure.

Two brackets, one rule: no dense transformer. Models that mix attention with a recurrent or state-space core go in hybrid. Models with no attention at all — pure RNNs and pure SSMs — share the pure bracket.

Rows are added from self-submitted lm-eval-harness runs. A row is only ranked once its results are posted with the exact command used. Ranks below reflect submitted, reproducible results only. AVG is computed automatically from the raw scores you enter — no manual math needed.

Hybrid attention + recurrent / SSM

Known hybrid families shown as placeholders until results are submitted.

Pure RNN + SSM

Myosotis-1 is the first submitted row. Placeholders are awaiting results.

Methodology

How AVG is calculated

Each benchmark is normalized off its random-chance floor, then averaged across whatever benchmarks a model has reported.

N(score, chance) = 100 × (score − chance) ÷ (100 − chance) AVG = mean(N) over available benchmarks

Chance floors: HellaSwag 25 · ARC-easy 25 · ARC-challenge 25 · PIQA 50 · GSM8K 0 · MMLU 25 · MMLU-Pro 10 · Arithmetic 0 · ASDiv 0 · GLUE 50 (majority baseline).
Missing benchmarks are excluded from that model's AVG; at least two are required.
WikiText is word perplexity — lower is better — so it is displayed but never counted toward AVG.
The AVG is computed automatically on the page from the raw scores; submitters never calculate it by hand.
Extra benchmarks can be added on submission; they join the AVG once their floor is defined.
Scores below chance may be negative.

Submit

Put your model on the board.

Run your model through lm-evaluation-harness and post the numbers. Reproducibility is the only rule — include the exact command and commit so anyone can verify.

lm_eval --model hf \
  --model_args pretrained=<model_id> \
  --tasks hellaswag,arc_easy,arc_challenge,piqa,gsm8k,mmlu,mmlu_pro,wikitext,arithmetic,asdiv,glue \
  --batch_size auto \
  --output_path results.json

Run the harness — hellaswag, arc_easy, arc_challenge, piqa, gsm8k, mmlu, mmlu_pro, wikitext, arithmetic, asdiv, and glue. WikiText is optional if you'd rather not report perplexity.
Collect the output — model name, architecture class (hybrid / rnn / ssm), param count, and checkpoint/step.
Note the exact command and the lm-eval-harness commit for verification.
Submit — open an issue or pull request on the site repository (FWKV/FWKV.github.io) with your results.json, or reach out via Hugging Face.