Documentation

Frequently asked questions

Everything you need to know about the tool, SEO, Answer Engine Optimization, and Generative Engine Optimization.

🛠 General

About the tool

What is WebCrawler Pro?

WebCrawler Pro is a browser-based site auditor that crawls your website and scores every page across three dimensions: classic SEO (Google and Bing ranking signals), AEO (Answer Engine Optimization — readiness for AI assistants and featured snippets), and GEO (Generative Engine Optimization — signals that make LLMs like ChatGPT, Perplexity, and Claude cite your content).

It runs entirely in your browser — no desktop app, no installation. Paste a URL, hit Crawl, and get a full audit in under a minute.

Do I need to create an account?

No account is required to start. You get 500 proxy requests per day as a guest — enough to crawl most small to medium sites. Sign in with Google to unlock 2,000 requests per day and persistent scan history that follows you across devices.

What types of sites can it crawl?

Any publicly accessible website. In HTML mode, it fetches the raw server response — ideal for traditional CMS sites (WordPress, Drupal, etc.). In DOM Render mode, it executes JavaScript before extracting content — essential for React, Next.js, Vue, Angular, and other single-page applications where content is injected client-side.

Private pages behind login walls, pages blocked by robots.txt disallow rules, and sites using advanced bot detection may not be fully crawlable.

Can I save and share my scans?

Yes. After a crawl, click 💾 Save Scan and you'll get a unique URL for that report. Share it with a client, teammate, or bookmark it for later. Anyone with the link can view the report — no login required to view.

If you're signed in, all your scans are stored privately in your account and listed in the History panel. Guest scans are stored anonymously — they remain accessible via their URL but are not listed anywhere.

💳 Pricing & Limits

Plans and request limits

How does the request limit work?

Each page fetched through our server proxy counts as one request. The limit resets daily at midnight server time. If you crawl a 30-page site, that uses 30 requests. The usage bar in the top-right header shows your current consumption.

  • Guest (anonymous): 500 requests/day per IP address
  • Signed in with Google: 2,000 requests/day

Cached responses (pages you've crawled recently) do not count against your limit — the proxy serves them from cache.

Is WebCrawler Pro free?

Yes, completely free to use. There are no paid tiers, no credit card required, and no trial period. Signing in with Google is free and simply increases your daily limit from 500 to 2,000 requests and enables persistent history.

What happens when I hit the limit?

The crawler stops automatically and a modal explains that you've reached your daily limit, when it resets, and how signing in increases it. Any pages already crawled are fully preserved — you can still view and save that partial scan.

⚙️ Technical

Technical details

What is the PHP proxy and why is it needed?

Browsers block cross-origin requests — a webpage on our domain cannot directly fetch HTML from yoursite.com due to CORS (Cross-Origin Resource Sharing) restrictions. Our PHP proxy runs on the server, fetches the target URL server-side, and returns it to your browser. This bypasses CORS while adding caching, gzip decompression, and rate limiting.

If you're self-hosting, you can point the proxy selector to your own instance or use one of the public proxy options (allorigins.win, corsproxy.io) — though public proxies may be slower and have their own rate limits.

How does DOM Render mode work?

In DOM Render mode, fetched HTML is injected into a hidden <iframe> in your browser. The iframe executes all JavaScript on the page, then after 900ms (enough for most frameworks to render), sends the final rendered DOM back to the crawler via postMessage. Extraction then runs on the rendered output rather than the raw server HTML.

This is slower (roughly 1-2 seconds per page versus 200-400ms in HTML mode) but accurately reflects what search engines see when they render your JavaScript.

Can I self-host WebCrawler Pro?

Yes. The project is a standard PHP + vanilla JavaScript stack. You need PHP 7.4+ with the pdo, pdo_sqlite, and curl extensions. Upload the files to any web host, configure your Google OAuth credentials in api/auth.php, make api/scans/ and api/data/ writable, and it's ready. See the README for full deployment instructions.

Where is data stored? Is it private?

All scan data is stored on the server as JSON files — in scans/public/ for anonymous saves and scans/users/{id}/ for signed-in users. Only you can see your scans when signed in. Anonymous scans are accessible via their unique URL but are not listed anywhere publicly. User accounts and sessions are stored in a SQLite database.

We do not share, sell, or analyse your scan data. On a self-hosted install, all data stays entirely on your own server.

🔍 SEO

SEO basics

What is Search Engine Optimization?

SEO is the practice of improving a website so that search engines like Google and Bing rank it higher in organic (non-paid) search results. It covers three broad areas: technical SEO (crawlability, page speed, structured data), on-page SEO (titles, headings, content quality), and off-page SEO (backlinks and authority — not something a crawler can directly measure).

WebCrawler Pro focuses on technical and on-page SEO signals — the things you can fix directly on your own site.

Why do title tags and meta descriptions matter?

The title tag is the clickable blue link in search results. Google uses it as a primary signal for what a page is about. It should be 50–60 characters, unique to each page, and contain the target keyword near the front.

The meta description is the grey snippet below the title. It doesn't directly affect rankings but strongly influences click-through rate — a compelling description brings more visitors. Keep it 140–160 characters. Google sometimes rewrites it, but a well-written one is used more often.

What is a canonical tag and why does it matter?

A canonical tag (<link rel="canonical" href="...">) tells search engines which version of a page is the "official" one when the same content is accessible at multiple URLs — for example, with and without a trailing slash, or with and without query parameters. Without canonicals, search engines may split ranking signals across duplicate URLs, weakening all versions.

Every page should have a self-referencing canonical tag pointing to its own preferred URL.

What are orphan pages?

An orphan page is one with no internal links pointing to it from the rest of your site. Search engine crawlers discover pages by following links — if no link points to a page, crawlers may never find it, and it will never rank. Fix orphan pages by adding at least one contextually relevant internal link from another page on your site.

🔍 SEO

What WebCrawler Pro checks

What SEO signals does the crawler check?

For each page, the crawler checks:

  • Title tag presence, length (50–60 chars), and uniqueness
  • Meta description presence, length (140–160 chars), and uniqueness
  • H1 tag — presence, count (exactly one), and uniqueness
  • Canonical tag presence
  • Viewport meta tag (mobile-friendliness signal)
  • Open Graph tags (og:title, og:image) for social sharing
  • Images missing alt attributes
  • Robots meta tag (noindex detection)
  • Page response time and size
  • Word count (thin content detection, under 300 words)
  • Internal link count (orphan page detection)

Across the full crawl, it also identifies duplicate titles, duplicate descriptions, duplicate H1s, slow pages, and pages with no schema markup.

What is the schema / structured data inventory?

The Schema section in the Report tab shows every JSON-LD schema type found across your site — Article, Product, FAQPage, BreadcrumbList, etc. — along with how many pages use each type. It also flags pages with no schema at all.

You can assign a page type (Homepage, Article, Product, FAQ, etc.) to any page and the tool will show you the recommended schema markup and a ready-to-copy JSON-LD template for that type.

What readability signals does the content panel show?

Select any page in the sidebar and click the Content tab to see: total word count, sentence count, average words per sentence, Flesch reading ease score (0 = very difficult, 100 = very easy), reading level label, internal link count, external link count, content length rating, the introduction paragraph, and the top 10 key phrases ranked by frequency with their density percentage.

🔍 SEO

Understanding SEO scores

How is the SEO score calculated?

Each page starts at 100. Each error (missing title, missing meta description, no H1) deducts 20 points. Each warning (title too long, no canonical, no viewport, no OG image, images missing alt text) deducts 5 points. The score floors at 0.

The site-wide average score shown in the Report tab is the mean across all successfully crawled pages.

My score is low — what should I fix first?

Go to the Action Report tab (last tab). It lists every issue across SEO, AEO, and GEO, sorted by severity. Fix errors first (red — each one costs 20 points), then warnings. The report groups issues by page so you can work through one URL at a time without jumping around.

🎙️ AEO

What is AEO?

What is Answer Engine Optimization?

Answer Engine Optimization is the practice of structuring your content so that AI-powered answer engines — Google's featured snippets and AI Overviews, Siri, Alexa, Bing Copilot, and other voice assistants — pull from your page when answering a user's question.

Unlike traditional SEO where you compete for a top-10 ranking, AEO targets the single answer read out loud or shown in the answer box — often called "position zero". The winner gets enormous visibility. The technique requires a different kind of content structure than standard SEO.

How is AEO different from SEO?

SEO optimises for ranking — getting your page into the top results list. AEO optimises for being quoted — getting your page selected as the direct answer. The two overlap but require different execution:

  • SEO cares about keyword density and backlinks. AEO cares about question-framed headings and direct answer paragraphs.
  • SEO uses broad, informational meta descriptions. AEO needs concise 40–60 word paragraphs that could stand alone as answers.
  • SEO benefits from long-form content. AEO requires scannable, structured content with clear Q&A patterns.

Both matter — a page needs to rank before it can be chosen as an answer.

Does AEO matter if I don't do voice search?

Yes. Google's AI Overviews (formerly Search Generative Experience) appear in regular text search results on desktop and mobile — not just voice. When Google shows an AI-generated answer at the top of results and cites a source, the cited page can receive a significant traffic spike. AEO-optimised pages are more likely to be cited as that source.

🎙️ AEO

AEO signals we measure

What AEO signals does WebCrawler Pro check?

Each page is scored across four equal-weight signals (25 points each):

  • FAQPage schema — JSON-LD markup using FAQPage with Question and acceptedAnswer. This is the single most impactful AEO action — it explicitly tells Google which questions your page answers.
  • Question-based headings — at least two H2 or H3 headings phrased as questions (starting with "What", "How", "Why", "When", etc., or ending with "?"). AI assistants map user questions to headings when selecting answers.
  • Entity clarity in H1 — the H1 is 3–12 words long and specific. Vague or very long H1s make it harder for AI to classify the page's topic.
  • Featured-snippet paragraphs — at least one paragraph with 40–60 words. This is the sweet spot for Google featured snippets — long enough to be informative, short enough to quote in full.
Why is 40–60 words the target for paragraphs?

Google's featured snippet box holds roughly 45–50 words comfortably. Paragraphs in this range are short enough to reproduce in full (which is what Google does for "paragraph snippets") but long enough to actually answer a question. Paragraphs under 40 words are often incomplete. Paragraphs over 60 words get truncated — the trailing "..." reduces click appeal and often signals to Google that the answer isn't crisp enough.

Place these concise paragraphs immediately after a question-framed heading for maximum snippet eligibility.

🎙️ AEO

Improving your AEO score

What's the fastest way to improve AEO?

In order of impact:

  • Add FAQPage schema to any page that already has a Q&A section. This is free points — the content exists, you just need to mark it up. Use the Schema section in the Report tab for a ready-to-copy JSON-LD template.
  • Rephrase 2+ subheadings as questions. Convert "Our Services" to "What services do we offer?" — the content doesn't change, only the framing.
  • Write a direct answer paragraph. After each question heading, write a 40–60 word paragraph that directly answers it. No preamble, no "great question" — just the answer.
Can I have a good AEO score on a product page?

Yes. Product pages should answer the questions buyers ask: "What does this product do?", "Who is it for?", "How much does it cost?", "What are the system requirements?". Framing your product page sections as questions and adding concise answer paragraphs — along with FAQPage or Product schema — can significantly improve AEO on commercial pages, not just informational ones.

🤖 GEO

What is GEO?

What is Generative Engine Optimization?

Generative Engine Optimization is the practice of making your content more likely to be cited, referenced, or summarised by large language models — ChatGPT, Perplexity, Gemini, Claude, and AI-powered search engines like Bing Copilot and Google AI Overviews.

These systems don't just rank pages — they synthesise information from multiple sources and cite the ones they trust most. GEO is about becoming one of those trusted sources. It requires a combination of authoritativeness signals, content structure, factual density, and freshness that LLMs use to assess credibility.

How is GEO different from AEO?

AEO is about getting your content quoted in a specific answer to a specific question — one page, one answer, one moment. It's reactive (triggered by a query) and structural (depends on content format).

GEO is about becoming the type of source that AI systems treat as credible and citable across many queries — even ones not directly about your topic. It's about building the signals that LLMs use to assess whether a source is worth referencing: attribution, dates, facts, authority, and originality.

A page can score well on AEO with good structure but poorly on GEO if it lacks an author, has no dates, and contains no original data.

Does GEO matter for my industry?

If your potential customers or audience use AI tools to research — yes, regardless of industry. Research from BrightEdge and others shows that AI-influenced search now affects significant percentages of discovery journeys in B2B, healthcare, finance, legal, education, and consumer markets. If a buyer asks ChatGPT "what's the best accounting software for small businesses" and your product isn't in the answer, you've missed a lead that never saw your site.

🤖 GEO

GEO signals we measure

What GEO signals does WebCrawler Pro check?

Six signals, weighted by impact:

  • Author schema or byline (20 pts) — Person schema or an HTML author/byline element. LLMs weight attributed content significantly higher than anonymous content.
  • Publication date in schema (20 pts)datePublished or dateModified in Article/WebPage schema. LLMs strongly prefer dateable, fresh content and often refuse to cite pages with no date.
  • Organization schema (15 pts) — Organization JSON-LD with name and URL. Establishes brand identity and publisher credibility.
  • Factual density (15 pts) — at least 1.5 specific numbers or statistics per 100 words. LLMs prefer to cite content with concrete data over vague claims.
  • Authority outlinks (15 pts) — at least one link to a .gov, .edu, or established authority domain (WHO, PubMed, Reuters, BBC, etc.). Signals that your content is grounded in verifiable sources.
  • Original research signals (15 pts) — language indicating proprietary data, surveys, or original analysis. Unique data is the most citable kind — it can't be found anywhere else.
Why do publication dates affect GEO?

LLMs are trained on data up to a cutoff date, but retrieval-augmented systems (like Perplexity and Bing Copilot) fetch live content. These systems heavily filter for recency because outdated information can cause them to produce incorrect answers — something they are specifically designed to avoid.

A page with no datePublished in its schema is treated as potentially stale regardless of when it was actually written. A page with a recent dateModified signals that it's been reviewed and updated, which is a strong positive signal.

🤖 GEO

Improving your GEO score

What are the highest-leverage GEO improvements?

In order of effort versus impact:

  • Add datePublished to your schema. If you have Article or WebPage schema, adding "datePublished": "YYYY-MM-DD" is a one-line change with outsized impact. Do it today.
  • Add author attribution. Even a simple byline HTML element or Person schema on blog posts dramatically improves GEO. People trust content that has an identifiable author.
  • Add Organization schema to your homepage. One block of JSON-LD covering your brand name, URL, logo, and social profiles acts as a GEO signal for your entire domain.
  • Add specific numbers. Replace "most users prefer our product" with "78% of users in our 2024 survey preferred our product". Concrete numbers make LLMs more confident in citing you.
  • Link to one authoritative source per article. If you're writing about health, link to PubMed or a .gov source. If finance, link to official regulatory data. This contextualises your content within the authoritative ecosystem LLMs prefer.
How do I know if LLMs are already citing my content?

Ask directly. Open ChatGPT, Perplexity, or Claude and ask a question that your content directly addresses. If your site appears in the cited sources, you're already being picked up. If not, note what sources are being cited and study their content structure — they're doing something your pages aren't yet.

Perplexity is particularly useful for this research because it shows explicit inline citations, making it easy to see exactly which pages it favours for which types of queries.

Does improving GEO hurt SEO?

No — the two are almost entirely complementary. Author attribution, clear dates, factual density, authoritative outlinks, and Organisation schema are all positive SEO signals too. GEO improvements are an extension of E-E-A-T (Experience, Expertise, Authoritativeness, Trust) — Google's framework for assessing content quality, which is already central to modern SEO.