ETH Evals: testing agents on Ethereum

Duration: NaN:NaN:NaN

Speaker: Rinat Akhunianov

Type: Talk

Expertise: No prior knowledge

Event: Devcon

Date: Nov 2026

Agents write Solidity, send transactions, and use live protocols, yet builders still pick models, agents, and skills by feel. We spent the past months trying to measure this instead. We started with ETH Skills. Across 19 skills and 97 tasks, evals showed where a skill helped and where it did not, and skills rewritten to be almost two thirds shorter beat the originals. We also saw how evals mislead: tasks every model already solves, and scores that move between runs. ETH Evals is what we built from those lessons: an open system that tests any model or agent on real Ethereum tasks, on local chains and mainnet forks. Tasks run as a bare model, an agent with the web, or an agent with skills, which separates what a model knows from what an agent can do and what skills add. Results go to a public board, and the knowledge questions are published on Hugging Face. We will show how it works, what the results show, and how to add your own eval or model.