Skip to content
Web reading settings

Web reading settings

Every page PULSE reads from the web (search results, lead and buyer pages, trending posts, competitor channels, the site audit, the new-audit preview, tender searches) goes through ONE reading profile. TIDE uses the same pattern. This page lists every setting and its value. The profile lives in grpc/internal/metasearch/profile.go and robots.go. The engine is the shared org-runink/crawler at the commit pinned in grpc/crawler.pin.

The shared profile

These values are the crawler’s own defaults at the pinned commit. TestTheProfileIsTheCrawlersDefaults fails if a crawler bump changes the User-Agent under PULSE.

SettingValueWhere
User-AgentMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36. It is the only one. TestEveryWebReadSendsTheOneUserAgent fails on any othermetasearch.UserAgent
Page read timeout (body included)20 smetasearch.PageTimeout, crawler static reader
Rendered page timeout30 s, then 2 s settle (search result pages: 6 s settle)crawler stealth browser
Probe timeout (robots.txt, llms.txt, sitemap.xml, the http→https check, a linked stylesheet or logo)10 smetasearch.ProbeTimeout
Body cap10 MiBmetasearch.MaxBodyBytes
Redirectsat most 10, each hop SSRF-guardedmetasearch.HTTPClient, crawler
Retries2 on 429/503 for pages, honouring Retry-After (backoff 1 s to 20 s). None for search result pages: a refusing engine is benched and the next one askedcrawler static reader
Pacing, page reads4 at a time, up to 400 ms random pause eachnewReader
Pacing, search result pages2 at a time, up to 300 ms random pause eachnewSources
Rendered pages at once2 (search result pages: 1)MaxRenders
Search result cache15 min, process-widecrawler DefaultCacheTTL
robots.txtobeyed (owner, 2026-11-07). One read per site per hour. A missing or unreadable robots.txt allows; an explicit Disallow skips the pagemetasearch.CheckRobots, crawler websource.RobotsChecker
Egress guardloopback, private, link-local and metadata addresses refused, on the URL and on every redirectcrawler GuardAgainstSSRF

When robots.txt disallows a page

The page is not requested. The caller gets the reason skipped: robots.txt disallows <host>. The skip is counted and shown where the results are shown. It is never an empty answer and never a failure that is retried.

  • Audit research stages (leads, trending posts, competitor channels, guests): the stage line says N page(s) skipped: robots.txt disallows <hosts> and counts robots_skipped.
  • Lead search run on its own: the progress log says the same line, and the run’s report carries the count and the hosts.
  • Site audit: a disallowed audited page carries the reason as its fetch error. It is not rendered or probed. A disallowed stylesheet or logo is skipped with the reason.
  • New-audit preview: the wizard shows the reason instead of the page.
  • Trending posts outside an audit: the skips are logged with their reason. Showing them on that page needs a new response field (pending).

The site audit still reads a site’s robots.txt, llms.txt and sitemap as audit signals. Reading robots.txt is how a disallow is learned, and it is not reading a page.

The search engines’ own result pages obey robots.txt too. An engine whose robots.txt disallows its result pages is never asked: when last read (2026-10-26) that was Bing, Brave Search, Mojeek and Google, so search runs mainly on DuckDuckGo and Startpage, and the AI-answer check reads DuckDuckGo only. Each search records the engines it skipped like any other skip (“skipped: robots.txt disallows www.bing.com”), so the counts and hosts above include them. Every result page an engine lists is checked before it is read.

PULSE’s own settings

SettingValueWhy
Least time between two searches, process-wide2.5 s (PULSE_SEARCH_MIN_GAP_MS)a burst of searches got PULSE refused by the result engines (measured 2026-10-24)
Lead and buyer search: per search, per page45 sa lead pass reads up to 60 company pages
Assistant research and grounding: per search, per page20 sanswers inside a chat turn
Public tender APIs (TED, OCDS, PNCP, CanadaBuys, SAM.gov, job boards)90 s client timeout, 48 MiB body cap, 0.5 requests/s per source, 6 h cachedocumented APIs, not pages: one User-Agent, no robots.txt (it governs crawling a site)

Signed-in vendor APIs (Google Ads, Google Analytics, HubSpot) identify themselves as Runink-PULSE/1.0. They read the customer’s own account with the customer’s grant, not the web.

Changing a setting

Change the profile in grpc/internal/metasearch/profile.go. To move the crawler, bump grpc/crawler.pin and the crawler require in grpc/go.mod together. TestTheCrawlerIsPinnedOnce keeps them, CI’s checkout and the image build on one commit.