Run async code-research benchmarks with a coding agent
- Posted
- by Simon Willison
- Last checked
- Method:
- How we verify
Evidence: Screenshots or logsThe source shows screenshots, a transcript or the agent's output.
Outcome
Asked only about cmarkgfm, Claude Code found six other Python Markdown libraries by itself, benchmarked all seven and wrote a report with charts, in which cmarkgfm came out on top; the author notes he hasn't been reviewing these reports in great detail.
Agent
Prompt Original prompt
Exact prompt as published in the source.
Create a performance benchmark and feature comparison report on PyPI cmarkgfm compared to other popular Python markdown libraries—check all of them out from github and read the source to get an idea for features, then design and run a benchmark including generating some charts, then create a report in a new python-markdown-comparison folder (do not create a _summary.md file or edit anywhere outside of that folder). Make sure the performance chart images are directly displayed in the README.md in the folder.
Try this in:
Sends the Original prompt shown above. ChatGPT and Grok open with it filled in, and ChatGPT may send it straight away. Gemini opens empty, so we copy the prompt for you to paste. Every button also copies it.
The source used Claude Code, which can take actions on your computer or in a browser. These are general chat tools, so results may differ, and a normal chat may only give you a plan.
Opens in a new tab. We don't see what you send.
These links are only a shortcut. We're not affiliated with OpenAI, Google or xAI, and your results may not match the source.
Optional stronger version
Our rewrite with extra checks. The poster didn't test it.
Goal: Benchmark cmarkgfm against popular Python markdown libraries and write a feature and performance report. Inputs (fill in or confirm): - Libraries to include (e.g. markdown, mistune, markdown-it-py, commonmark) - Test documents or sizes to use - Output folder name (python-markdown-comparison) Rules: - Read each library's source and docs to build a feature table (GFM tables, footnotes, extensions, safety). - Pin versions and record them in the report. - Run each benchmark several times on the same inputs and report median and spread. - Generate charts as PNG and include the script that made them. - Note the machine, Python version and input sizes used so results can be reproduced. - Check each library's output on a sample document matches cmarkgfm's, and flag differences, so speed isn't compared on wrong output. Checkpoint: Don't publish or push results; keep everything in the new folder. If you get stuck: If a library won't install, note it and continue with the rest. If a login, CAPTCHA or two-step code appears, stop and ask me to handle it; never guess or store passwords. Verify before you finish: Re-run the benchmark once to confirm numbers are stable within a reasonable margin. Output: A README report with feature table, charts, methodology and how to reproduce. Done when: someone can rerun your script and get similar numbers.
Source
https://simonwillison.net/2025/Nov/6/async-code-research/
Related cases
- Compare top social media tools in a spreadsheet with ChatGPT agent and ClaudeRecreated promptEvidence: Screenshots or logsBeginner-friendly
Allie K. Miller asked both ChatGPT Agent Mode and Claude's Excel feature to compare top social media tools, and Claude finished in 3 minutes vs 8 with denser decision support, though some prices mixed annual vs monthly.
ChatGPT agent, Claude (Excel) · Research · Source date:
- Extend a family tree two generations with Claude Code and browser agentsRecreated promptEvidence: Shown on video
Working from the family's archive scans, Claude Code with memory and the Claude browser plugin cross-checked church books, ship manifests and naturalization records, built a 300-page family wiki, and found two more generations in a day; the family is now checking it for errors.
Poster says:300-page family wiki
Claude Code, Claude in Chrome, Perplexity Comet · Research · Source date:
- Get a quick sourced briefing on a local wildlife species with Grok DeepSearchEvidence: Article or interview
Indian Express's reviewer asked Grok 3 DeepSearch about the state of the house sparrow in India; he reports it scoured 101 sources in 46 seconds before presenting its findings.
Poster says:101 sources in 46 seconds
Grok DeepSearch · Research · Source date: