What is Colly?
Colly is an open-source Go library for building crawlers, scrapers, and spiders. It extracts HTTP-based content with goquery and CSS selectors rather than a headless browser. It handles cookies and sessions automatically and runs in sync, async, or parallel modes, with per-domain concurrency and delay controls. It also supports distributed scraping, built-in caching, and robots.txt handling. It is Apache-2.0 licensed, free for commercial use, and claims throughput of over 1,000 requests per second on a single core.
Our verdict
Colly fits Go teams who want a fast, code-first crawler they embed and control directly rather than an out-of-the-box service. It has no JavaScript rendering, so sites that depend on client-side rendering are out of scope. The project shows steady issue and PR activity, though how recently the underlying code itself was last committed could not be confirmed.
Categories:
Crawling frameworks give engineering teams full control over their pipeline: where it runs, how it scales, and what it collects. The trade is that anti-bot work and infrastructure become your problem rather than a vendor's, which is why they pair well with a proxy network.
