Evaluation & Benchmarks

langfuse/langfuse

🪢 Open source agent evals & observability: Trace, evaluate, and improve LLM applications with one open platform.

confident-ai/deepeval

The LLM Evaluation Framework

raga-ai-hub/RagaAI-Catalyst

Python SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced anal

zilliztech/VectorDBBench

Benchmark for vector databases.

FareedKhan-dev/all-agentic-architectures

35 production-grade agentic AI architectures (Reflexion, LATS, GraphRAG, MemGPT, Voyager, BrowserAgent, ...) — a Python library and runnable textbook with multi-provider LLM support and a 17-task benc

CLUEbenchmark/CLUE

中文语言理解测评基准 Chinese Language Understanding Evaluation Benchmark: datasets, baselines, pre-trained models, corpus and leaderboard

chirpz-ai/pandaprobe

open source agent engineering platform: traces, evals, and metrics to debug and improve your AI agents. Integrates with LangGraph, CrewAI, Claude Agent SDK, and more.

google-research/android_world

AndroidWorld is an environment and benchmark for autonomous agents

open-mmlab/mmrazor

OpenMMLab Model Compression Toolbox and Benchmark.

SakanaAI/text-to-lora

Hypernetworks that adapt LLMs for specific benchmark tasks using only textual task description as the input

Saivineeth147/lora-speedrun

Speedrunning LoRA fine-tuning: frozen task, frozen hardware, public wall-clock leaderboard. modded-nanogpt for fine-tuning.

SouthpawIN/turbofit

Adaptive local inference for Hermes — TurboFit Check, evidence-built TurboFit List, repaired discovery/benchmark campaigns, Qwen 3.8 DFlash2, Bonsai DSpark, FreeToken, and Sirvir.

onejune2018/Awesome-LLM-Eval

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表，主要面向基础大模型评测，旨在探求生成式AI的技术边界.

AudarAI/Audar-ASR-V1

Arabic-first generative speech recognition — Audar-ASR-V1 (Flash + Turbo). #1 on the Open Universal Arabic ASR Leaderboard. Model cards, benchmarks & inference.

ChrisChen667788/Your-First-LLM-Studio

First LLM Studio: local-first LLM workbench for Apple Silicon + MLX. 本地优先 LLM 工作台，覆盖 Agent、对比、微调、评测与运行时遥测。

oneal2000/SR-Agents

SRA-Bench and SR-Agents: a benchmark and toolkit for skill-retrieval-augmented LLM agents.

hopit-ai/Moda

MODA: open fashion retrieval benchmark and models by Hopit AI. MODA (203M, open source), MODA Pro Lite (213M, open weights), MODA Pro (hosted). Full-corpus benchmarks vs FashionSigLIP, SigLIP-SO400M a

ELM-Research/ECG-Language-Models

A research-oriented training and evaluation framework for ECG-Language Models (ELMs)

david-g-3654/homebench

Benchmark your local LLMs: speed, memory, and quality, in one command. TUI leaderboard for Ollama, LM Studio, llama.cpp, and vLLM.

JingbiaoMei/ATM-Bench

ATM-Bench: A benchmark for long-term personalized memory QA spanning ~4 years of multimodal data (images, videos, emails). Features referential queries, evidence-grounded answering, and multi-source r

alesha-pro/tools

Tools, ComfyUI workflows and benchmark configs from a 4x RTX 3090 local-inference rig

AgentSafeLabs/safelabs-eval

Red-teaming and evaluation framework for AI agents, built around an OWASP-inspired ASI01–ASI10 taxonomy

cryptopoly/ChaosEngineAI

Local AI workstation — discover, run, chat, benchmark, and generate images from open-weight models. DFlash/DDTree speculative decoding, TurboQuant & TriAttention cache compression strategies, MLX + ll

famstack-dev/local-llm-bench

Local LLM benchmark tool for comparing engines (MLX vs llama.cpp), scenarios on Apple Silicon

kaivid-labs/evret

Evals framework for Information Retrieval Systems

ruc-datalab/CoDA-Bench

CoDA-Bench is a benchmark for code agents on data-intensive tasks. 🎈代码智能体能搞定数据密集型任务吗?

ikergarcia1996/MetaVec

A monolingual and cross-lingual meta-embedding generation and evaluation framework

xmpuspus/kb-arena

Benchmark 9 retrieval architectures (vector, contextual, QnA, knowledge graph, hybrid, RAPTOR, PageIndex, BM25, rerank) on your own docs. Automated hyperparameter search with bootstrap CIs and signifi

shilicon/LTX-LLM

Local LLM test benchmark for x86 hardware

gvkhosla/pi-tinker

Fine-tune open-source models with Tinker from inside Pi — managed improve loops, data prep, evals, smoke tests, deploy snippets, and checkpoint chat.

wontopos/WMB-100K

WMB-100K — The first 100,000-turn benchmark for AI memory systems

Shiyun-x/Benchmark-Agent

Benchmark Everything Everywhere All at Once, a fully autonomous agentic system for benchmark construction and customization.

OpenRaiser/OmniOpt

🌟 [Survey & Benchmark] A unified taxonomy, geometric framework, and cross-domain benchmark for modern optimizers

Achyuthan-S/moe-bench

Open benchmark for expert collapse and routing efficiency in sparse MoE LLMs

loganionian/0gmem

Long-term conversational memory system for AI agents — cell-based architecture with hybrid BM25 + semantic retrieval. 96% accuracy on LoCoMo benchmark.

OpenDCAI/DataFlex

[NeurIPS 2026] DataFlex: A Unified Benchmark and Evaluation Platform for Data-Centric Training of Large Language Models

harbor-framework/terminal-bench-1

A benchmark for LLMs on complicated tasks in the terminal

SemiAnalysisAI/InferenceX

Open Source AI Accelerator Research Platform Standard / 开源推理研究平台

usamahz/cpu-performance-engineering

A reading path for CPU performance engineering, from one instruction to production inference. Primary sources only, with a runnable benchmark for every section.

Tiger3807861189/DeepSeek-V4-J-Space-Capability-Realization-Report

DeepSeek V4 × J-Space capability realization report — benchmark evidence that J-Space reduces capability-realization loss on DeepSeek V4.

Tiger3807861189/GLM-5.3-Flash-J-Space-Capability-Realization-Report

GLM-5.3-Flash × J-Space capability realization — benchmark presentation of the J-Space Cognition Suite

TIGER-AI-Lab/ClawBench

Open-source benchmark for browser AI agents on daily tasks.

xinxuxin/keystone-bench

A clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.

ciprian-chelba/1-billion-word-language-modeling-benchmark

Formerly known as code.google.com/p/1-billion-word-language-modeling-benchmark

aidatatools/ollama-benchmark

LLM Benchmark for Throughput via Ollama (Local LLMs)

huggingface/optimum-benchmark

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.

Simreal-AI/Simreal-MLBench

[Public preview] Externally scored agentic ML research benchmark: 60 tasks, real competition ground truth. Open protocol, operated evaluation.

TristanBilot/mlx-benchmark

Benchmark of Apple MLX operations on all Apple Silicon chips (GPU, CPU) + MPS and CUDA.

wadeKeith/autoresearch-qwen

Autonomous Qwen3-VL training-code research on the official DocVQA benchmark. main: NVIDIA multi-GPU, mlx: Apple Silicon/MPS.

1173591564/Dynamics-memory

LLM agent 的有界自纠错长期记忆层：把连续、含矛盾的项目交互日志蒸馏成有界、可自更新的记忆库。Bounded, self-correcting long-term memory for LLM agents — candidate pool + value dynamics + tension adjudication + causal-replay evals.

cookiespiggy/agentic-rl

Agentic RL 中文零基础教程（33 章）：从概念到 GRPO 实战，含 TRL 最小可跑示例；26–33 章附一套可运行的三方判别模型实证工程（encoder vs LLM-LoRA vs 规则基线）。第 25 章讲清 Jev / TypeSafe System One 与 RL 的能力边界 | Chinese Agentic RL tutorial (33 chapters) + a r

ArtificialAnalysis/aa-agentperf-local

Benchmark local LLM serving by replaying real agent trajectories

PMZFX/intel-arc-pro-b70-benchmarks

Benchmark results and performance data for the Intel Arc Pro B70 GPU (Xe2/Battlemage) - LLM inference, video generation, dual-GPU scaling.

OpenDataBox/Workspace-Bench

Benchmark self-evolving Agent upon realistic large-scale file workspaces

DanTup/spark-evals

Some benchmark results of small models and quants that fit on DGX Spark

InternScience/ResearchHarness

A lightweight, general-purpose harness for tool-using LLM agents, fair benchmark evaluation, harness baselines, and personal assistant workflows.

xiaofengShi/MechVQA

ICML 2026 benchmark and MechVL models for multimodal understanding of mechanical engineering drawings.

thc1006/qwen3.6-speculative-decoding-rtx3090

llama.cpp speculative decoding measured on one RTX 3090, Qwen3.6-35B-A3B UD-Q4_K_XL, commit 3737e4137. Published figures are re-derived from the committed data by a checker that fails on drift, a cove

michailfragkiskos/EdgeMosaic

Rapid on-device AI evaluation lab: manifest validation, benchmark plans, latency/memory/thermal aggregation, JSON+Markdown reports.