Open Source SEO Tools: Self-Hosted Crawlers, Rank Trackers and APIs

Screaming Frog SEO Spider is free for crawls up to 500 URLs. A paid license lets you crawl more than 500 URLs and unlocks its advanced features. Judge each open source crawler against that license, because the license is the cost you avoid.

LibreCrawl is a web-based crawler for SEO analysis and site audits, built on Python Flask. Its README says it will always be free and open source. It compares itself with a $279 per year Screaming Frog license.

It renders JavaScript sites built with React, Vue or Angular, and it exports results to CSV, XLSX, JSON or XML. Its PageSpeed Insights feature for Core Web Vitals calls Google's service, so that one feature has an outside dependency.

1. 1Install Docker, which the README recommends, or Python 3.8 or later.
2. 2For JavaScript rendering, install the Chromium browser that Playwright uses, with the command below.
3. 3Open the app at http://localhost:5000 in your browser in local mode.

playwright install chromium

SiteOne Crawler was rewritten in Rust. It ships as one native binary for Windows, macOS and Linux, with zero runtime dependencies, so a build server needs no language runtime installed.

Its exit code 10 on failure enables automated deployment blocking. Browser rendering and an AI assistant are optional features that you switch on when you need them.

SEOnaut is a web app built with Go and a MySQL database. It sorts issues into critical, high and low severity. The issue types include broken links, redirect problems, missing or duplicate meta tags and heading order.

You provide the database. The README compose file starts a MySQL container for you. If you do not want to run the database yourself, a hosted version is available at seonaut.org.

python-seo-analyzer shows a license that GitHub reports as Other, not a standard open source license. Read its LICENSE file before you call it open source.

Crawlee and Scrapy are libraries. They fetch and parse pages. You write the checks, the report and the schedule. Build with them when you need custom extraction or a crawl that a ready tool cannot run.

Crawlee is a web scraping and browser automation library for JavaScript and TypeScript. It needs Node.js 22.13 or higher. It does not bundle a browser library, so you install Playwright yourself for pages that need JavaScript.

• You write the crawl logic and the extraction logic.
• You write the audit checks, because Crawlee brings no issue list.
• A Python version of Crawlee lives in a separate repo.

advertools includes a customizable SEO crawler built with Scrapy. By default it extracts titles, header tags, body text, status codes and headers. It accepts CSS and XPath selectors, and it can stop and resume large crawls.

The same package turns log files, XML sitemaps and robots.txt files into DataFrames. Join those with crawl output in Python when one report needs several sources.

SerpBear tracks Google positions for unlimited domains and keywords. It sends email notifications, has a built-in API and connects to Search Console. It does not query Google directly. It uses third-party scrapers or your own proxy IPs to check whether your domain appears for each keyword.

Unless you run your own proxy IPs, a paid scraper is the running cost of this free tracker. The SerpBear README lists these scraper prices.

The same README table marks three services as no longer working: SpaceSerp, ScrapingAnt and ScrapingRobot. Scrapers change or block traffic, so a tracker built on one of them stops returning ranks.

The Getting Started steps still tell you to get a free key from ScrapingRobot. That step points you at a service the README marks as no longer working.

SerpBear shows keyword volume and new keyword ideas through a Google Ads test account. Actual visits per keyword need Search Console linked to the tracker.

None of the open source tools in this guide ships its own keyword volume. OpenSEO gets volume from DataForSEO, and OpenSERP returns ranks, not volume.

Google's Custom Search JSON API is closed to new customers. The service ends on January 1, 2027. Additional requests cost $5 per 1,000 queries.

A new project cannot rely on this API, so tools that wrap it, such as advertools serp_goog, do not work for new projects. Choose a self-hosted SERP source or a paid SERP API, and test it on your own keywords first.

OpenSERP is a self-hosted REST API and CLI. It loads results in a browser for Google, Bing, Yandex, Baidu, DuckDuckGo and Ecosia, and it returns ranks as JSON. It needs no API keys and has no per-search billing.

Bing localizes results by exit IP, so some regions need a proxy in the target country. Engines can still return CAPTCHAs, rate limits or unrelated results, even in browser mode.

docker run --rm -p 127.0.0.1:7000:7000 karust/openserp serve -a 0.0.0.0 -p 7000

serposcope, a self-hosted rank tracker, is archived on GitHub. Its last code push came on January 22, 2024, and its repo description reads LEGACY CODE. Do not start new work on it.

Lighthouse is Google's open source audit engine, licensed under Apache-2.0. Unlighthouse runs Lighthouse across a whole site, with a UI and smart sampling. Lighthouse CI runs, saves and asserts Lighthouse results, which stops regressions in accessibility, SEO and performance.

Unlighthouse runs with npx, and its README requires Node.js 22.18 or higher. It saves reports to a local .unlighthouse output folder, and the README recommends that you add that folder to .gitignore.

The Lighthouse CI README quick start is a GitHub Actions workflow. Its key step runs the command below on every commit.

lhci autorun

1. 1Set score budgets for performance, accessibility and SEO.
2. 2Run lhci autorun in your CI workflow on every commit.
3. 3Fail the build when a score drops below its budget.

The optional LHCI server stores history and shows a dashboard. It runs in any Node environment with persistent disk storage, or with network access to a Postgres or MySQL database. The newest GitHub release came out in June 2025, so pin a version.

GoAccess is an open source, real-time web log analyzer, licensed under MIT. It runs in a terminal on *nix systems, or in your browser as a live HTML dashboard.

You point it at your access log, pick the log format and let it parse. No data feed is involved, so the only cost is the server that runs it.

GoAccess puts AI crawlers and Fediverse traffic in their own categories, apart from traditional search engine crawlers. The README also shows a quick estimate of crawler hits: count the log lines that contain bot. Use that count for a rough trend only.

advertools converts log files into DataFrames. Use it when you want to join logs with a crawl, a sitemap or a URL list in Python.

OpenSEO is an MIT licensed open source alternative to Semrush and Ahrefs. It covers keyword research, rank tracking, competitor insights, backlinks, site audits and AI visibility.

It needs a DataForSEO API key to get SEO data. When you self-host, you pay DataForSEO directly. For the full setup and the costs from a five-site run, read OpenSEO on the DataForSEO API.

The 28% is how the hosted service earns its margin. Self-hosting skips that fee, so your DataForSEO bill becomes the only data cost.

Docker is the quickest way to try OpenSEO on your own machine. You need Docker Engine or Docker Desktop with Compose. A prebuilt image, ghcr.io/every-app/open-seo-standalone, also runs under Coolify, Portainer and Dokploy.

The Cloudflare path deploys with one command and works on Cloudflare's free plan. It needs Node.js 22.6 or newer, pnpm, and a Cloudflare account with R2 enabled. R2 requires a payment method on file, even within its free tier.

Four optional keys extend OpenSEO. OPENROUTER_API_KEY enables AI conversations, research and summaries. A Google OAuth app links Search Console and Analytics. CONTEXT_API_KEY turns on JavaScript rendering for site audits, and GOOGLE_MAPS_API_KEY powers the local SEO map.

OpenSEO exposes an MCP server, so AI agents such as Claude Code can use your SEO data directly.

Stars and releases as of October 31, 2026. Both change, so open the repo before you decide. Crawl, audit and log tools need no paid data. Rank, keyword and backlink tools do need one.

• Crawl a site you own: LibreCrawl or SiteOne Crawler. Neither needs a license, and both fetch your site directly.
• Gate a release on a crawl: SiteOne Crawler, which exits with code 10 on failure.
• Score pages in CI: Lighthouse CI for budgets, Unlighthouse for site-wide Lighthouse runs.
• Read server logs: GoAccess for a live view, advertools for joins with crawl data in Python.
• Track ranks: budget for a scraper, proxies or a SERP API, because the rank trackers here all depend on search results from one of them.
• Keyword volume and backlinks: OpenSEO on a DataForSEO key, the only all-in-one in this guide.

Crawl, audit and log jobs cost only hosting. Rank, keyword and backlink data come from a paid or fragile feed, so price that feed before you commit to a tool.