About the tool
WebCrawler Pro is a browser-based site auditor that crawls your website and scores every page across three dimensions: classic SEO (Google and Bing ranking signals), AEO (Answer Engine Optimization — readiness for AI assistants and featured snippets), and GEO (Generative Engine Optimization — signals that make LLMs like ChatGPT, Perplexity, and Claude cite your content).
It runs entirely in your browser — no desktop app, no installation. Paste a URL, hit Crawl, and get a full audit in under a minute.
No account is required to start. You get 500 proxy requests per day as a guest — enough to crawl most small to medium sites. Sign in with Google to unlock 2,000 requests per day and persistent scan history that follows you across devices.
Any publicly accessible website. In HTML mode, it fetches the raw server response — ideal for traditional CMS sites (WordPress, Drupal, etc.). In DOM Render mode, it executes JavaScript before extracting content — essential for React, Next.js, Vue, Angular, and other single-page applications where content is injected client-side.
Private pages behind login walls, pages blocked by robots.txt disallow rules, and sites using advanced bot detection may not be fully crawlable.
Yes. After a crawl, click 💾 Save Scan and you'll get a unique URL for that report. Share it with a client, teammate, or bookmark it for later. Anyone with the link can view the report — no login required to view.
If you're signed in, all your scans are stored privately in your account and listed in the History panel. Guest scans are stored anonymously — they remain accessible via their URL but are not listed anywhere.
Plans and request limits
Each page fetched through our server proxy counts as one request. The limit resets daily at midnight server time. If you crawl a 30-page site, that uses 30 requests. The usage bar in the top-right header shows your current consumption.
- Guest (anonymous): 500 requests/day per IP address
- Signed in with Google: 2,000 requests/day
Cached responses (pages you've crawled recently) do not count against your limit — the proxy serves them from cache.
Yes, completely free to use. There are no paid tiers, no credit card required, and no trial period. Signing in with Google is free and simply increases your daily limit from 500 to 2,000 requests and enables persistent history.
The crawler stops automatically and a modal explains that you've reached your daily limit, when it resets, and how signing in increases it. Any pages already crawled are fully preserved — you can still view and save that partial scan.
Technical details
Browsers block cross-origin requests — a webpage on our domain cannot directly fetch HTML from yoursite.com due to CORS (Cross-Origin Resource Sharing) restrictions. Our PHP proxy runs on the server, fetches the target URL server-side, and returns it to your browser. This bypasses CORS while adding caching, gzip decompression, and rate limiting.
If you're self-hosting, you can point the proxy selector to your own instance or use one of the public proxy options (allorigins.win, corsproxy.io) — though public proxies may be slower and have their own rate limits.
In DOM Render mode, fetched HTML is injected into a hidden <iframe> in your browser. The iframe executes all JavaScript on the page, then after 900ms (enough for most frameworks to render), sends the final rendered DOM back to the crawler via postMessage. Extraction then runs on the rendered output rather than the raw server HTML.
This is slower (roughly 1-2 seconds per page versus 200-400ms in HTML mode) but accurately reflects what search engines see when they render your JavaScript.
Yes. The project is a standard PHP + vanilla JavaScript stack. You need PHP 7.4+ with the pdo, pdo_sqlite, and curl extensions. Upload the files to any web host, configure your Google OAuth credentials in api/auth.php, make api/scans/ and api/data/ writable, and it's ready. See the README for full deployment instructions.
All scan data is stored on the server as JSON files — in scans/public/ for anonymous saves and scans/users/{id}/ for signed-in users. Only you can see your scans when signed in. Anonymous scans are accessible via their unique URL but are not listed anywhere publicly. User accounts and sessions are stored in a SQLite database.
We do not share, sell, or analyse your scan data. On a self-hosted install, all data stays entirely on your own server.
SEO basics
SEO is the practice of improving a website so that search engines like Google and Bing rank it higher in organic (non-paid) search results. It covers three broad areas: technical SEO (crawlability, page speed, structured data), on-page SEO (titles, headings, content quality), and off-page SEO (backlinks and authority — not something a crawler can directly measure).
WebCrawler Pro focuses on technical and on-page SEO signals — the things you can fix directly on your own site.
The title tag is the clickable blue link in search results. Google uses it as a primary signal for what a page is about. It should be 50–60 characters, unique to each page, and contain the target keyword near the front.
The meta description is the grey snippet below the title. It doesn't directly affect rankings but strongly influences click-through rate — a compelling description brings more visitors. Keep it 140–160 characters. Google sometimes rewrites it, but a well-written one is used more often.
A canonical tag (<link rel="canonical" href="...">) tells search engines which version of a page is the "official" one when the same content is accessible at multiple URLs — for example, with and without a trailing slash, or with and without query parameters. Without canonicals, search engines may split ranking signals across duplicate URLs, weakening all versions.
Every page should have a self-referencing canonical tag pointing to its own preferred URL.
An orphan page is one with no internal links pointing to it from the rest of your site. Search engine crawlers discover pages by following links — if no link points to a page, crawlers may never find it, and it will never rank. Fix orphan pages by adding at least one contextually relevant internal link from another page on your site.
What WebCrawler Pro checks
For each page, the crawler checks:
- Title tag presence, length (50–60 chars), and uniqueness
- Meta description presence, length (140–160 chars), and uniqueness
- H1 tag — presence, count (exactly one), and uniqueness
- Canonical tag presence
- Viewport meta tag (mobile-friendliness signal)
- Open Graph tags (og:title, og:image) for social sharing
- Images missing
altattributes - Robots meta tag (
noindexdetection) - Page response time and size
- Word count (thin content detection, under 300 words)
- Internal link count (orphan page detection)
Across the full crawl, it also identifies duplicate titles, duplicate descriptions, duplicate H1s, slow pages, and pages with no schema markup.
The Schema section in the Report tab shows every JSON-LD schema type found across your site — Article, Product, FAQPage, BreadcrumbList, etc. — along with how many pages use each type. It also flags pages with no schema at all.
You can assign a page type (Homepage, Article, Product, FAQ, etc.) to any page and the tool will show you the recommended schema markup and a ready-to-copy JSON-LD template for that type.
Select any page in the sidebar and click the Content tab to see: total word count, sentence count, average words per sentence, Flesch reading ease score (0 = very difficult, 100 = very easy), reading level label, internal link count, external link count, content length rating, the introduction paragraph, and the top 10 key phrases ranked by frequency with their density percentage.
Understanding SEO scores
Each page starts at 100. Each error (missing title, missing meta description, no H1) deducts 20 points. Each warning (title too long, no canonical, no viewport, no OG image, images missing alt text) deducts 5 points. The score floors at 0.
The site-wide average score shown in the Report tab is the mean across all successfully crawled pages.
Go to the Action Report tab (last tab). It lists every issue across SEO, AEO, and GEO, sorted by severity. Fix errors first (red — each one costs 20 points), then warnings. The report groups issues by page so you can work through one URL at a time without jumping around.
What is AEO?
Answer Engine Optimization is the practice of structuring your content so that AI-powered answer engines — Google's featured snippets and AI Overviews, Siri, Alexa, Bing Copilot, and other voice assistants — pull from your page when answering a user's question.
Unlike traditional SEO where you compete for a top-10 ranking, AEO targets the single answer read out loud or shown in the answer box — often called "position zero". The winner gets enormous visibility. The technique requires a different kind of content structure than standard SEO.
SEO optimises for ranking — getting your page into the top results list. AEO optimises for being quoted — getting your page selected as the direct answer. The two overlap but require different execution:
- SEO cares about keyword density and backlinks. AEO cares about question-framed headings and direct answer paragraphs.
- SEO uses broad, informational meta descriptions. AEO needs concise 40–60 word paragraphs that could stand alone as answers.
- SEO benefits from long-form content. AEO requires scannable, structured content with clear Q&A patterns.
Both matter — a page needs to rank before it can be chosen as an answer.
Yes. Google's AI Overviews (formerly Search Generative Experience) appear in regular text search results on desktop and mobile — not just voice. When Google shows an AI-generated answer at the top of results and cites a source, the cited page can receive a significant traffic spike. AEO-optimised pages are more likely to be cited as that source.
AEO signals we measure
Each page is scored across four equal-weight signals (25 points each):
- FAQPage schema — JSON-LD markup using FAQPage with Question and acceptedAnswer. This is the single most impactful AEO action — it explicitly tells Google which questions your page answers.
- Question-based headings — at least two H2 or H3 headings phrased as questions (starting with "What", "How", "Why", "When", etc., or ending with "?"). AI assistants map user questions to headings when selecting answers.
- Entity clarity in H1 — the H1 is 3–12 words long and specific. Vague or very long H1s make it harder for AI to classify the page's topic.
- Featured-snippet paragraphs — at least one paragraph with 40–60 words. This is the sweet spot for Google featured snippets — long enough to be informative, short enough to quote in full.
Google's featured snippet box holds roughly 45–50 words comfortably. Paragraphs in this range are short enough to reproduce in full (which is what Google does for "paragraph snippets") but long enough to actually answer a question. Paragraphs under 40 words are often incomplete. Paragraphs over 60 words get truncated — the trailing "..." reduces click appeal and often signals to Google that the answer isn't crisp enough.
Place these concise paragraphs immediately after a question-framed heading for maximum snippet eligibility.
Improving your AEO score
In order of impact:
- Add FAQPage schema to any page that already has a Q&A section. This is free points — the content exists, you just need to mark it up. Use the Schema section in the Report tab for a ready-to-copy JSON-LD template.
- Rephrase 2+ subheadings as questions. Convert "Our Services" to "What services do we offer?" — the content doesn't change, only the framing.
- Write a direct answer paragraph. After each question heading, write a 40–60 word paragraph that directly answers it. No preamble, no "great question" — just the answer.
Yes. Product pages should answer the questions buyers ask: "What does this product do?", "Who is it for?", "How much does it cost?", "What are the system requirements?". Framing your product page sections as questions and adding concise answer paragraphs — along with FAQPage or Product schema — can significantly improve AEO on commercial pages, not just informational ones.
What is GEO?
Generative Engine Optimization is the practice of making your content more likely to be cited, referenced, or summarised by large language models — ChatGPT, Perplexity, Gemini, Claude, and AI-powered search engines like Bing Copilot and Google AI Overviews.
These systems don't just rank pages — they synthesise information from multiple sources and cite the ones they trust most. GEO is about becoming one of those trusted sources. It requires a combination of authoritativeness signals, content structure, factual density, and freshness that LLMs use to assess credibility.
AEO is about getting your content quoted in a specific answer to a specific question — one page, one answer, one moment. It's reactive (triggered by a query) and structural (depends on content format).
GEO is about becoming the type of source that AI systems treat as credible and citable across many queries — even ones not directly about your topic. It's about building the signals that LLMs use to assess whether a source is worth referencing: attribution, dates, facts, authority, and originality.
A page can score well on AEO with good structure but poorly on GEO if it lacks an author, has no dates, and contains no original data.
If your potential customers or audience use AI tools to research — yes, regardless of industry. Research from BrightEdge and others shows that AI-influenced search now affects significant percentages of discovery journeys in B2B, healthcare, finance, legal, education, and consumer markets. If a buyer asks ChatGPT "what's the best accounting software for small businesses" and your product isn't in the answer, you've missed a lead that never saw your site.
GEO signals we measure
Six signals, weighted by impact:
- Author schema or byline (20 pts) — Person schema or an HTML author/byline element. LLMs weight attributed content significantly higher than anonymous content.
- Publication date in schema (20 pts) —
datePublishedordateModifiedin Article/WebPage schema. LLMs strongly prefer dateable, fresh content and often refuse to cite pages with no date. - Organization schema (15 pts) — Organization JSON-LD with name and URL. Establishes brand identity and publisher credibility.
- Factual density (15 pts) — at least 1.5 specific numbers or statistics per 100 words. LLMs prefer to cite content with concrete data over vague claims.
- Authority outlinks (15 pts) — at least one link to a
.gov,.edu, or established authority domain (WHO, PubMed, Reuters, BBC, etc.). Signals that your content is grounded in verifiable sources. - Original research signals (15 pts) — language indicating proprietary data, surveys, or original analysis. Unique data is the most citable kind — it can't be found anywhere else.
LLMs are trained on data up to a cutoff date, but retrieval-augmented systems (like Perplexity and Bing Copilot) fetch live content. These systems heavily filter for recency because outdated information can cause them to produce incorrect answers — something they are specifically designed to avoid.
A page with no datePublished in its schema is treated as potentially stale regardless of when it was actually written. A page with a recent dateModified signals that it's been reviewed and updated, which is a strong positive signal.
Improving your GEO score
In order of effort versus impact:
- Add datePublished to your schema. If you have Article or WebPage schema, adding
"datePublished": "YYYY-MM-DD"is a one-line change with outsized impact. Do it today. - Add author attribution. Even a simple byline HTML element or Person schema on blog posts dramatically improves GEO. People trust content that has an identifiable author.
- Add Organization schema to your homepage. One block of JSON-LD covering your brand name, URL, logo, and social profiles acts as a GEO signal for your entire domain.
- Add specific numbers. Replace "most users prefer our product" with "78% of users in our 2024 survey preferred our product". Concrete numbers make LLMs more confident in citing you.
- Link to one authoritative source per article. If you're writing about health, link to PubMed or a .gov source. If finance, link to official regulatory data. This contextualises your content within the authoritative ecosystem LLMs prefer.
Ask directly. Open ChatGPT, Perplexity, or Claude and ask a question that your content directly addresses. If your site appears in the cited sources, you're already being picked up. If not, note what sources are being cited and study their content structure — they're doing something your pages aren't yet.
Perplexity is particularly useful for this research because it shows explicit inline citations, making it easy to see exactly which pages it favours for which types of queries.
No — the two are almost entirely complementary. Author attribution, clear dates, factual density, authoritative outlinks, and Organisation schema are all positive SEO signals too. GEO improvements are an extension of E-E-A-T (Experience, Expertise, Authoritativeness, Trust) — Google's framework for assessing content quality, which is already central to modern SEO.