Crawler, metasearch & CRM
Web collection and CRM access. CrawlerService scrapes and crawls public web pages.
MetasearchService runs the sovereign metasearch engine and carries the RPCs used to administer
the prompt guardrail’s rules. HubspotService reads deals and contacts from HubSpot. All three run inside
the FACE deployment: no third-party search or AI API is involved.
Summary
| Service | RPC | Kind | Purpose |
|---|---|---|---|
| CrawlerService | Scrape | Unary | Fetch and extract one page |
| CrawlerService | Crawl | Unary | Crawl from a seed URL and record the pages |
| CrawlerService | GetCrawlStatus | Unary | Read a finished crawl’s pages |
| MetasearchService | SearchTerms | Unary | Ranked, content-extracted web search |
| MetasearchService | EvaluateBias | Unary | Guardrail rule administration |
| MetasearchService | ListBiasRules | Unary | Guardrail rule administration |
| HubspotService | Connect | Unary | Verify a HubSpot key |
| HubspotService | GetDeals | Unary | Read deals from HubSpot |
| HubspotService | GetContacts | Unary | Read contacts from HubSpot |
CrawlerService
Full name semantics.v1.CrawlerService.
Scrape
rpc Scrape(ScrapeRequest) returns (ScrapeResponse);- Kind: Unary.
- Auth: Bearer session, writer, admin or service account.
- Errors:
INVALID_ARGUMENTifurlis empty. A page that cannot be fetched, or no free browser slot forrender_js, returnssuccess: falsewitherror_messageinstead of an error.
Fetches one page and extracts its title and main content. With render_js: true, the page is
rendered in the server’s headless browser first, which waits for a browser slot. Otherwise it is
fetched as plain HTML.
Request: ScrapeRequest
| Field | Type | Description |
|---|---|---|
url | string | Required. The page to fetch. |
render_js | bool | Render the page in a headless browser before extracting. The server implements it. |
Response: ScrapeResponse
| Field | Type | Description |
|---|---|---|
success | bool | Whether the page was fetched and extracted. |
url | string | The URL that was fetched. |
title | string | Page title. |
content | string | Main detected content, as Markdown or plain text. |
metadata | map<string, string> | Not populated by the server. |
error_message | string | Why the scrape failed, when success is false. |
Crawl
rpc Crawl(CrawlRequest) returns (CrawlResponse);- Kind: Unary. The call returns when the crawl has finished.
- Auth: Bearer session, writer, admin or service account.
- Errors:
INVALID_ARGUMENTifurlis empty or not an absolutehttp(s)URL with a host, or ifdepthorlimitis negative. If the call’s deadline passes or the call is cancelled mid-crawl, the matching status is returned (DEADLINE_EXCEEDEDorCANCELLED).
Crawls breadth-first from the seed. Only http and https links are followed. The crawl stops
when limit pages have been attempted. The result is kept under the returned crawl_id in the
serving process’s memory, for GetCrawlStatus.
depth counts link hops from the seed: the seed is at depth 0, and links are followed from any
page whose depth is below depth. For example, depth: 1 fetches the seed and the pages it
links to.
Request: CrawlRequest
| Field | Type | Description |
|---|---|---|
url | string | Required. Seed URL. It must be an absolute http(s) URL with a host. |
depth | int32 | Maximum link hops from the seed. 0 means the default, 2. Capped at 5. |
limit | int32 | Maximum pages to attempt. 0 means the default, 10. Capped at 500. |
allow_external | bool | Follow links to other hosts. When false (the default), the crawl stays on the seed’s host or its subdomains, on the same port. This is a host rule, not a registrable-domain rule: with seed www.example.com, a link to example.com counts as external. |
Response: CrawlResponse
| Field | Type | Description |
|---|---|---|
crawl_id | string | Id for GetCrawlStatus. |
status | string | Completed, or Failed when at least one page was attempted and none was fetched. The call is synchronous, so the server never returns Started. |
pages_visited | int32 | Pages fetched successfully. |
GetCrawlStatus
rpc GetCrawlStatus(GetCrawlStatusRequest) returns (GetCrawlStatusResponse);- Kind: Unary.
- Auth: Bearer session.
- Errors: None beyond authentication. An unknown
crawl_idreturnsstatus: "Unknown"and no pages.
Returns the pages recorded for a crawl. Crawl results are held in memory by the process that ran the crawl. They do not survive a restart and are not shared between replicas.
Request: GetCrawlStatusRequest
| Field | Type | Description |
|---|---|---|
crawl_id | string | The id returned by Crawl. |
Response: GetCrawlStatusResponse
| Field | Type | Description |
|---|---|---|
status | string | Completed, Failed or Unknown. |
pages | repeated CrawledPage | Every page attempted, in visit order. |
MetasearchService
Full name semantics.v1.MetasearchService.
This service returns only numbers that the search engine or the guardrail computed. It reports no invented score, ratio or percentage.
SearchTerms
rpc SearchTerms(SearchTermsRequest) returns (SearchTermsResponse);- Kind: Unary.
- Auth: Bearer session.
- Errors:
INVALID_ARGUMENTifqueryis empty or only whitespace.UNAVAILABLEif the search fails.
Runs a live search through the sovereign metasearch engine. The engine drives a headless browser, fetches and extracts candidate pages, then ranks them. This is an expensive call, not a typeahead: bind it to an explicit user action.
Request: SearchTermsRequest
| Field | Type | Description |
|---|---|---|
query | string | Required. The search query. Leading and trailing whitespace is removed. |
max_results | int32 | Ranked results to return. 0 or negative means 5. Capped at 20. |
Response: SearchTermsResponse
| Field | Type | Description |
|---|---|---|
query | string | The query as searched, echoed so responses can be matched to requests. |
results | repeated SearchTermResult | Ranked hits. There may be fewer than requested. |
requested_results | int32 | How many results the engine was asked for, after the default and cap were applied. |
EvaluateBias
rpc EvaluateBias(EvaluateBiasRequest) returns (EvaluateBiasResponse);- Kind: Unary.
- Auth: Bearer session, org admin only.
- Errors:
INVALID_ARGUMENTifagent_nameis not a known rule set (the message lists the known names), or iftextis empty or only whitespace.
An administrative RPC used when maintaining an agent’s guardrail rules.
Request: EvaluateBiasRequest
| Field | Type | Description |
|---|---|---|
text | string | Required. The text to evaluate. |
agent_name | string | The rule set to use. See Rule sets. An unknown name is refused. |
Response: EvaluateBiasResponse
| Field | Type | Description |
|---|---|---|
agent_name | string | The rule set used. |
rules_evaluated | int32 | Rule administration field. |
hits | repeated BiasRule | Rule administration field. |
blocked | bool | Rule administration field. |
ListBiasRules
rpc ListBiasRules(ListBiasRulesRequest) returns (ListBiasRulesResponse);- Kind: Unary.
- Auth: Bearer session, org admin only.
- Errors:
INVALID_ARGUMENTifagent_nameis not a known rule set.
An administrative RPC used when maintaining an agent’s guardrail rules.
Request: ListBiasRulesRequest
| Field | Type | Description |
|---|---|---|
agent_name | string | Same names and refusal as EvaluateBiasRequest.agent_name. |
Response: ListBiasRulesResponse
| Field | Type | Description |
|---|---|---|
agent_name | string | The rule set returned. |
rules | repeated BiasRule | The rule set. |
Rule sets
agent_name accepts exactly these values. They name agents’ rule sets, not gRPC services, so
most of them are not services in this reference. An empty agent_name means
HypothesisService.
AnalysisService, ClaimsLeakageService, ClaimsService, ComplianceService, CopService,
CounterpartyCheckService, CustomerAnalysisService, DocumentationService, FetchService,
FinanceService, FraudDetectionService, FreightMarketService, FulfilmentService,
HypothesisService, PostureService, PredictiveMaintenanceService,
ReverseLogisticsService, RulesService, SelfHealingService, SurveillanceService,
TelemetryService, TwinsService, UnderwritingService, VisionService, VoiceService.
HubspotService
Full name semantics.v1.HubspotService.
HubSpot access uses a key that the operator configures on the deployment. A key sent to
Connect is only verified: it is never stored or used for later calls.
Connect
rpc Connect(HubspotServiceConnectRequest) returns (HubspotServiceConnectResponse);- Kind: Unary.
- Auth: Bearer session, org admin only.
- Errors:
INTERNALif the verification request cannot be built. Every other outcome is reported insuccessandmessage.
Verifies api_key by making one read-only request to HubSpot. The key is not stored. For
GetDeals and GetContacts to work, the operator must configure a HubSpot key on the
deployment.
Request: HubspotServiceConnectRequest
| Field | Type | Description |
|---|---|---|
api_key | string | HubSpot API key or private-app access token. Write-only: never returned, logged or stored. |
Response: HubspotServiceConnectResponse
| Field | Type | Description |
|---|---|---|
success | bool | true if HubSpot accepted the key. |
message | string | Outcome, including HubSpot’s HTTP status on rejection. |
GetDeals
rpc GetDeals(GetDealsRequest) returns (GetDealsResponse);- Kind: Unary.
- Auth: Bearer session.
- Errors:
FAILED_PRECONDITIONif no HubSpot key is configured on the deployment. A HubSpot transport, HTTP or decoding failure is returned without a specific status code (clients seeUNKNOWN).
Returns one page of deals from HubSpot’s default page size, with the properties below.
Request: GetDealsRequest
| Field | Type | Description |
|---|---|---|
pipeline_id | string | Not applied by the server. Deals from all pipelines are returned. |
limit | int32 | Not applied by the server. |
Response: GetDealsResponse
| Field | Type | Description |
|---|---|---|
deals | repeated HubspotDeal | The deals. |
GetContacts
rpc GetContacts(GetContactsRequest) returns (GetContactsResponse);- Kind: Unary.
- Auth: Bearer session.
- Errors: Same as
GetDeals.
Returns one page of contacts from HubSpot’s default page size.
Request: GetContactsRequest
| Field | Type | Description |
|---|---|---|
limit | int32 | Not applied by the server. |
Response: GetContactsResponse
| Field | Type | Description |
|---|---|---|
contacts | repeated HubspotContact | The contacts. |
Messages
CrawledPage
| Field | Type | Description |
|---|---|---|
url | string | Page URL. |
title | string | Page title. Empty on error. |
content_snippet | string | The first 500 bytes of extracted content, followed by ... when truncated. |
status | string | Success or Error. |
SearchTermResult
| Field | Type | Description |
|---|---|---|
title | string | Page title. |
url | string | Page URL. |
content | string | Clean readable text extracted from the page, truncated by the engine. Not a summary: no model has processed it. |
score | double | The engine’s relevance score for this hit. Comparable within one response only. Do not display it as a confidence or a percentage. |
image_url | string | A representative image for the page, when the engine found one. |
BiasRule
A guardrail rule, as used in rule administration.
| Field | Type | Description |
|---|---|---|
rule_id | string | Rule id. |
description | string | The rule’s description. |
severity | string | The rule’s severity. |
category | string | The rule’s family. |
compliance | repeated string | Compliance standards the rule is cited under. |
is_domain_scope | bool | Rule administration field. |
HubspotDeal
| Field | Type | Description |
|---|---|---|
id | string | HubSpot deal id. |
dealname | string | Deal name. |
amount | string | Amount, as HubSpot returns it. |
pipeline | string | Pipeline id. |
dealstage | string | Deal stage id. |
closedate | string | Close date, as HubSpot returns it. |
HubspotContact
| Field | Type | Description |
|---|---|---|
id | string | HubSpot contact id. |
firstname | string | First name. |
lastname | string | Last name. |
email | string | Email address. |