Design AI agents that autonomously identify and exploit real-world vulnerabilities across diverse application environments — including other AI agents — in a Capture The Flag format.
Organized by the Berkeley RDI Center, AgentCTF x AgentXploit challenges participants to build AI agents capable of identifying and exploiting real-world vulnerabilities. Targeted frameworks include LangChain, AutoGPT, and many more, with tasks sourced from publicly disclosed CVEs.
The evaluation pipeline follows the AAA (Agentified Agent Assessment) paradigm. A portion of tasks are released as a development set so participants can iterate locally before the official evaluation.
Implement an AI agent with an A2A interface that autonomously reasons about and exploits CVEs.
Tasks are drawn from publicly disclosed vulnerabilities across 20+ popular AI and web application frameworks.
Agents are scored on both the released dev set and a hidden test set, with full runs replayed for verification.
Review the AAA evaluation paradigm documentation and fork the GitHub repository to your own account.
Build your exploit agent with an A2A interface inside ./src/white_agent/. Only modify files in that
directory and pyproject.toml. Do not alter the
Green Agent or task configurations — violations may result in disqualification.
Run the full dev-set evaluation, then bundle results with the provided CLI. Total submission size must be
under 1 MB — do not include model weights or large files. The bundle captures the latest
run-all results; do not modify them after
bundling.
Upload your submission.zip. Official evaluation
will rerun results to verify authenticity.
Submissions are evaluated against both the released dev set and a hidden test set.
Specify which LLM you used so organizers can provision appropriate model access.
Supported models include openai/*,
gemini/*, and
vertex_ai/claude-*.
# Provided via .env for each task evaluation LITELLM_PROXY_API_KEY=sk-xxxxx LITELLM_PROXY_API_BASE=... # Specify your model (prefix with litellm_proxy/ in most cases) LITELLM_MODEL=litellm_proxy/openai/gpt-4o
Full setup instructions, dependency requirements, and usage examples are available in the repository README.
This framework is intended for educational and research purposes only. All included CVEs are publicly disclosed vulnerabilities. Participants must adhere to responsible disclosure policies and may not use techniques or artifacts from this competition outside of the authorized evaluation environment.