Building Calm Toad, a Free Desktop SEO Crawler

An ex-coworker recently asked me if Screaming Frog was still the thing to use for technical SEO audits. I've been using it for years for technical SEO and content audits. I also didn't particularly want to buy another license.

That conversation helped kick off another attempt at something I've tried building many times: the crawler I actually want to use.

I've built plenty of crawlers over the years, including WCAG tools and Site Spell Checker. Most of my attempts at a newer SEO crawler were web based. And I kept running into the same problems. Crawling and scraping are hard enough without trying to convince a website that requests coming from a datacenter IP are perfectly fine.

So this time I built a desktop app. Requests come from your computer. Download it, enter a URL, crawl.

It's called Calm Toad. The opposite of Screaming Frog. That's the idea, anyway: fast and simple. I wanted less bloat and no artificial page limits.

An excuse to try Rust

I also wanted to play with Rust. It keeps coming up, I wanted to see the speed for myself, and Rune from Anomaly gave me a nice kickstart for building a native interface. Learn something, build something cross platform. Why not try again?

Calm Toad is built in Rust with Rune and Rune Kit. The interface is mostly tables, which is what I want when I'm auditing a website. Let me sort the URLs, filter down to the issues, and inspect a page without losing where I was. There are light and dark themes and a drawer for page details. Nothing particularly exotic. The crawl results are the point.

Underneath that interface are parallel requests, results that stream in as the crawl runs, cancellation, robots.txt support, and retry/backoff handling. You can set your own page and depth limits. There isn't a product-imposed page cap you have to pay to remove, although your computer and the website you're crawling still have their limits.

The speed did not disappoint. I recorded a side-by-side crawl with Screaming Frog, and Calm Toad was much faster in that run. You can watch the recording rather than take my word for it.

Getting the first 30 percent

I started with a basic prompt to get about 30 percent of the way there in one shot. Then it was continuous iteration.

I used a Codex orchestration agent to help design the system, with subagents and OpenCode agents doing implementation. OpenCode running DeepSeek let me do a lot of that work cheaply. I also had an OpenCode session running in a loop on performance: run the benchmarks, inspect the results, improve the code, repeat.

That loop had actual things to measure. The local benchmark harness exercises crawling, HTML parsing, auditing, and counting issues using synthetic pages. It gives changes something repeatable to run against without depending on how quickly a public website happens to respond.

Some of the improvements were straightforward once somebody went looking. Parsing went from eight passes through the HTML to one. The interface stopped rebuilding the entire audit every time it polled for results and started updating the audit as pages arrived. Those are the kinds of changes that matter when results keep coming in and the table needs to stay usable.

Rust was part of why I wanted to try this. There was still plenty of work to do in how the app used it.

The updater was the hard part

The most challenging part so far has been in-app updates. Mostly because I'd never built that before.

Fetching a new version sounds simple until you have to replace an installed application, restart it, and recover if something goes wrong. Across operating systems. With verified downloads.

This was where independent reviews from Codex and OpenCode earned their place. Reviews caught rollback and concurrency problems, including a failure path that could delete the backup before the new installation was confirmed. Another issue let the update transaction's lock go too early. These were details that needed fixing before trusting the updater with somebody's installation.

The updater now uses signed manifests and verifies the package size and SHA-256 hash. Installation happens after a user-confirmed restart, with backup and rollback handling.

Windows supplied another little lesson. Git's checkout converted signed test fixtures to CRLF line endings. The files looked like text, but their signatures depended on the exact bytes. The fix was to preserve those bytes with Git attributes. You don't fix a signature test by quietly changing what gets verified.

Getting agents to write the implementation was one part of this project. Reviewing the failure paths, testing them, and coordinating the releases was a substantial part too.

Already finding things on my own sites

I started using Calm Toad on my own sites, including PMPrompt because it's big. It surfaced indexing issues on a bunch of pages I hadn't realized were there. Already useful.

At v0.3.0 it checks titles, meta descriptions, H1s, response codes, redirects, canonicals, image alt text, and internal links. You can export the current tab to CSV with its filters and sort order preserved.

The latest release also discovers and audits sitemaps. It finds them through robots.txt or common paths, reads XML indexes and gzip files, and can discover URLs that aren't linked from other pages. It reports errors, redirects, indexing directives, and entries it hasn't checked.

There are downloads for Windows, Linux, Apple Silicon Mac, and Intel Mac. GitHub Release Notes and the website changelog come from the same release content, so there's one place to describe what changed.

The basic crawler is free. I don't think those basic features should be artificially limited. If people use it, I'll build cloud features: crawl history, comparisons, and possibly AI analysis. Those aren't shipped. JavaScript rendering and accessibility auditing are other things I'd like to explore.

For now, I finally have a desktop crawler of my own. Download Calm Toad or look through the public repository.