Web reading settings
Every page PULSE reads from the web (search results, lead and buyer pages,
trending posts, competitor channels, the site audit, the new-audit preview,
tender searches) goes through ONE reading profile. TIDE uses the same
pattern. This page lists every setting and its value. The profile lives in
grpc/internal/metasearch/profile.go and robots.go. The engine is the
shared org-runink/crawler at the commit pinned in grpc/crawler.pin.
The shared profile
These values are the crawler’s own defaults at the pinned commit.
TestTheProfileIsTheCrawlersDefaults fails if a crawler bump changes the
User-Agent under PULSE.
| Setting | Value | Where |
|---|---|---|
| User-Agent | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36. It is the only one. TestEveryWebReadSendsTheOneUserAgent fails on any other | metasearch.UserAgent |
| Page read timeout (body included) | 20 s | metasearch.PageTimeout, crawler static reader |
| Rendered page timeout | 30 s, then 2 s settle (search result pages: 6 s settle) | crawler stealth browser |
| Probe timeout (robots.txt, llms.txt, sitemap.xml, the http→https check, a linked stylesheet or logo) | 10 s | metasearch.ProbeTimeout |
| Body cap | 10 MiB | metasearch.MaxBodyBytes |
| Redirects | at most 10, each hop SSRF-guarded | metasearch.HTTPClient, crawler |
| Retries | 2 on 429/503 for pages, honouring Retry-After (backoff 1 s to 20 s). None for search result pages: a refusing engine is benched and the next one asked | crawler static reader |
| Pacing, page reads | 4 at a time, up to 400 ms random pause each | newReader |
| Pacing, search result pages | 2 at a time, up to 300 ms random pause each | newSources |
| Rendered pages at once | 2 (search result pages: 1) | MaxRenders |
| Search result cache | 15 min, process-wide | crawler DefaultCacheTTL |
| robots.txt | obeyed (owner, 2026-11-07). One read per site per hour. A missing or unreadable robots.txt allows; an explicit Disallow skips the page | metasearch.CheckRobots, crawler websource.RobotsChecker |
| Egress guard | loopback, private, link-local and metadata addresses refused, on the URL and on every redirect | crawler GuardAgainstSSRF |
When robots.txt disallows a page
The page is not requested. The caller gets the reason
skipped: robots.txt disallows <host>. The skip is counted and shown where
the results are shown. It is never an empty answer and never a failure that
is retried.
- Audit research stages (leads, trending posts, competitor channels,
guests): the stage line says
N page(s) skipped: robots.txt disallows <hosts>and countsrobots_skipped. - Lead search run on its own: the progress log says the same line, and the run’s report carries the count and the hosts.
- Site audit: a disallowed audited page carries the reason as its fetch error. It is not rendered or probed. A disallowed stylesheet or logo is skipped with the reason.
- New-audit preview: the wizard shows the reason instead of the page.
- Trending posts outside an audit: the skips are logged with their reason. Showing them on that page needs a new response field (pending).
The site audit still reads a site’s robots.txt, llms.txt and sitemap as audit signals. Reading robots.txt is how a disallow is learned, and it is not reading a page.
The search engines’ own result pages obey robots.txt too. An engine whose robots.txt disallows its result pages is never asked: when last read (2026-10-26) that was Bing, Brave Search, Mojeek and Google, so search runs mainly on DuckDuckGo and Startpage, and the AI-answer check reads DuckDuckGo only. Each search records the engines it skipped like any other skip (“skipped: robots.txt disallows www.bing.com”), so the counts and hosts above include them. Every result page an engine lists is checked before it is read.
PULSE’s own settings
| Setting | Value | Why |
|---|---|---|
| Least time between two searches, process-wide | 2.5 s (PULSE_SEARCH_MIN_GAP_MS) | a burst of searches got PULSE refused by the result engines (measured 2026-10-24) |
| Lead and buyer search: per search, per page | 45 s | a lead pass reads up to 60 company pages |
| Assistant research and grounding: per search, per page | 20 s | answers inside a chat turn |
| Public tender APIs (TED, OCDS, PNCP, CanadaBuys, SAM.gov, job boards) | 90 s client timeout, 48 MiB body cap, 0.5 requests/s per source, 6 h cache | documented APIs, not pages: one User-Agent, no robots.txt (it governs crawling a site) |
Signed-in vendor APIs (Google Ads, Google Analytics, HubSpot) identify
themselves as Runink-PULSE/1.0. They read the customer’s own account with
the customer’s grant, not the web.
Changing a setting
Change the profile in grpc/internal/metasearch/profile.go. To move the
crawler, bump grpc/crawler.pin and the crawler require in grpc/go.mod
together. TestTheCrawlerIsPinnedOnce keeps them, CI’s checkout and the image
build on one commit.