Skip to content
Crawler, metasearch & CRM

Crawler, metasearch & CRM

Web collection and CRM access. CrawlerService scrapes and crawls public web pages. MetasearchService runs the sovereign metasearch engine and carries the RPCs used to administer the prompt guardrail’s rules. HubspotService reads deals and contacts from HubSpot. All three run inside the FACE deployment: no third-party search or AI API is involved.

Summary

ServiceRPCKindPurpose
CrawlerServiceScrapeUnaryFetch and extract one page
CrawlerServiceCrawlUnaryCrawl from a seed URL and record the pages
CrawlerServiceGetCrawlStatusUnaryRead a finished crawl’s pages
MetasearchServiceSearchTermsUnaryRanked, content-extracted web search
MetasearchServiceEvaluateBiasUnaryGuardrail rule administration
MetasearchServiceListBiasRulesUnaryGuardrail rule administration
HubspotServiceConnectUnaryVerify a HubSpot key
HubspotServiceGetDealsUnaryRead deals from HubSpot
HubspotServiceGetContactsUnaryRead contacts from HubSpot

CrawlerService

Full name semantics.v1.CrawlerService.

Scrape

rpc Scrape(ScrapeRequest) returns (ScrapeResponse);
  • Kind: Unary.
  • Auth: Bearer session, writer, admin or service account.
  • Errors: INVALID_ARGUMENT if url is empty. A page that cannot be fetched, or no free browser slot for render_js, returns success: false with error_message instead of an error.

Fetches one page and extracts its title and main content. With render_js: true, the page is rendered in the server’s headless browser first, which waits for a browser slot. Otherwise it is fetched as plain HTML.

Request: ScrapeRequest

FieldTypeDescription
urlstringRequired. The page to fetch.
render_jsboolRender the page in a headless browser before extracting. The server implements it.

Response: ScrapeResponse

FieldTypeDescription
successboolWhether the page was fetched and extracted.
urlstringThe URL that was fetched.
titlestringPage title.
contentstringMain detected content, as Markdown or plain text.
metadatamap<string, string>Not populated by the server.
error_messagestringWhy the scrape failed, when success is false.

Crawl

rpc Crawl(CrawlRequest) returns (CrawlResponse);
  • Kind: Unary. The call returns when the crawl has finished.
  • Auth: Bearer session, writer, admin or service account.
  • Errors: INVALID_ARGUMENT if url is empty or not an absolute http(s) URL with a host, or if depth or limit is negative. If the call’s deadline passes or the call is cancelled mid-crawl, the matching status is returned (DEADLINE_EXCEEDED or CANCELLED).

Crawls breadth-first from the seed. Only http and https links are followed. The crawl stops when limit pages have been attempted. The result is kept under the returned crawl_id in the serving process’s memory, for GetCrawlStatus.

depth counts link hops from the seed: the seed is at depth 0, and links are followed from any page whose depth is below depth. For example, depth: 1 fetches the seed and the pages it links to.

Request: CrawlRequest

FieldTypeDescription
urlstringRequired. Seed URL. It must be an absolute http(s) URL with a host.
depthint32Maximum link hops from the seed. 0 means the default, 2. Capped at 5.
limitint32Maximum pages to attempt. 0 means the default, 10. Capped at 500.
allow_externalboolFollow links to other hosts. When false (the default), the crawl stays on the seed’s host or its subdomains, on the same port. This is a host rule, not a registrable-domain rule: with seed www.example.com, a link to example.com counts as external.

Response: CrawlResponse

FieldTypeDescription
crawl_idstringId for GetCrawlStatus.
statusstringCompleted, or Failed when at least one page was attempted and none was fetched. The call is synchronous, so the server never returns Started.
pages_visitedint32Pages fetched successfully.

GetCrawlStatus

rpc GetCrawlStatus(GetCrawlStatusRequest) returns (GetCrawlStatusResponse);
  • Kind: Unary.
  • Auth: Bearer session.
  • Errors: None beyond authentication. An unknown crawl_id returns status: "Unknown" and no pages.

Returns the pages recorded for a crawl. Crawl results are held in memory by the process that ran the crawl. They do not survive a restart and are not shared between replicas.

Request: GetCrawlStatusRequest

FieldTypeDescription
crawl_idstringThe id returned by Crawl.

Response: GetCrawlStatusResponse

FieldTypeDescription
statusstringCompleted, Failed or Unknown.
pagesrepeated CrawledPageEvery page attempted, in visit order.

MetasearchService

Full name semantics.v1.MetasearchService.

This service returns only numbers that the search engine or the guardrail computed. It reports no invented score, ratio or percentage.

SearchTerms

rpc SearchTerms(SearchTermsRequest) returns (SearchTermsResponse);
  • Kind: Unary.
  • Auth: Bearer session.
  • Errors: INVALID_ARGUMENT if query is empty or only whitespace. UNAVAILABLE if the search fails.

Runs a live search through the sovereign metasearch engine. The engine drives a headless browser, fetches and extracts candidate pages, then ranks them. This is an expensive call, not a typeahead: bind it to an explicit user action.

Request: SearchTermsRequest

FieldTypeDescription
querystringRequired. The search query. Leading and trailing whitespace is removed.
max_resultsint32Ranked results to return. 0 or negative means 5. Capped at 20.

Response: SearchTermsResponse

FieldTypeDescription
querystringThe query as searched, echoed so responses can be matched to requests.
resultsrepeated SearchTermResultRanked hits. There may be fewer than requested.
requested_resultsint32How many results the engine was asked for, after the default and cap were applied.

EvaluateBias

rpc EvaluateBias(EvaluateBiasRequest) returns (EvaluateBiasResponse);
  • Kind: Unary.
  • Auth: Bearer session, org admin only.
  • Errors: INVALID_ARGUMENT if agent_name is not a known rule set (the message lists the known names), or if text is empty or only whitespace.

An administrative RPC used when maintaining an agent’s guardrail rules.

Request: EvaluateBiasRequest

FieldTypeDescription
textstringRequired. The text to evaluate.
agent_namestringThe rule set to use. See Rule sets. An unknown name is refused.

Response: EvaluateBiasResponse

FieldTypeDescription
agent_namestringThe rule set used.
rules_evaluatedint32Rule administration field.
hitsrepeated BiasRuleRule administration field.
blockedboolRule administration field.

ListBiasRules

rpc ListBiasRules(ListBiasRulesRequest) returns (ListBiasRulesResponse);
  • Kind: Unary.
  • Auth: Bearer session, org admin only.
  • Errors: INVALID_ARGUMENT if agent_name is not a known rule set.

An administrative RPC used when maintaining an agent’s guardrail rules.

Request: ListBiasRulesRequest

FieldTypeDescription
agent_namestringSame names and refusal as EvaluateBiasRequest.agent_name.

Response: ListBiasRulesResponse

FieldTypeDescription
agent_namestringThe rule set returned.
rulesrepeated BiasRuleThe rule set.

Rule sets

agent_name accepts exactly these values. They name agents’ rule sets, not gRPC services, so most of them are not services in this reference. An empty agent_name means HypothesisService.

AnalysisService, ClaimsLeakageService, ClaimsService, ComplianceService, CopService, CounterpartyCheckService, CustomerAnalysisService, DocumentationService, FetchService, FinanceService, FraudDetectionService, FreightMarketService, FulfilmentService, HypothesisService, PostureService, PredictiveMaintenanceService, ReverseLogisticsService, RulesService, SelfHealingService, SurveillanceService, TelemetryService, TwinsService, UnderwritingService, VisionService, VoiceService.

HubspotService

Full name semantics.v1.HubspotService.

HubSpot access uses a key that the operator configures on the deployment. A key sent to Connect is only verified: it is never stored or used for later calls.

Connect

rpc Connect(HubspotServiceConnectRequest) returns (HubspotServiceConnectResponse);
  • Kind: Unary.
  • Auth: Bearer session, org admin only.
  • Errors: INTERNAL if the verification request cannot be built. Every other outcome is reported in success and message.

Verifies api_key by making one read-only request to HubSpot. The key is not stored. For GetDeals and GetContacts to work, the operator must configure a HubSpot key on the deployment.

Request: HubspotServiceConnectRequest

FieldTypeDescription
api_keystringHubSpot API key or private-app access token. Write-only: never returned, logged or stored.

Response: HubspotServiceConnectResponse

FieldTypeDescription
successbooltrue if HubSpot accepted the key.
messagestringOutcome, including HubSpot’s HTTP status on rejection.

GetDeals

rpc GetDeals(GetDealsRequest) returns (GetDealsResponse);
  • Kind: Unary.
  • Auth: Bearer session.
  • Errors: FAILED_PRECONDITION if no HubSpot key is configured on the deployment. A HubSpot transport, HTTP or decoding failure is returned without a specific status code (clients see UNKNOWN).

Returns one page of deals from HubSpot’s default page size, with the properties below.

Request: GetDealsRequest

FieldTypeDescription
pipeline_idstringNot applied by the server. Deals from all pipelines are returned.
limitint32Not applied by the server.

Response: GetDealsResponse

FieldTypeDescription
dealsrepeated HubspotDealThe deals.

GetContacts

rpc GetContacts(GetContactsRequest) returns (GetContactsResponse);
  • Kind: Unary.
  • Auth: Bearer session.
  • Errors: Same as GetDeals.

Returns one page of contacts from HubSpot’s default page size.

Request: GetContactsRequest

FieldTypeDescription
limitint32Not applied by the server.

Response: GetContactsResponse

FieldTypeDescription
contactsrepeated HubspotContactThe contacts.

Messages

CrawledPage

FieldTypeDescription
urlstringPage URL.
titlestringPage title. Empty on error.
content_snippetstringThe first 500 bytes of extracted content, followed by ... when truncated.
statusstringSuccess or Error.

SearchTermResult

FieldTypeDescription
titlestringPage title.
urlstringPage URL.
contentstringClean readable text extracted from the page, truncated by the engine. Not a summary: no model has processed it.
scoredoubleThe engine’s relevance score for this hit. Comparable within one response only. Do not display it as a confidence or a percentage.
image_urlstringA representative image for the page, when the engine found one.

BiasRule

A guardrail rule, as used in rule administration.

FieldTypeDescription
rule_idstringRule id.
descriptionstringThe rule’s description.
severitystringThe rule’s severity.
categorystringThe rule’s family.
compliancerepeated stringCompliance standards the rule is cited under.
is_domain_scopeboolRule administration field.

HubspotDeal

FieldTypeDescription
idstringHubSpot deal id.
dealnamestringDeal name.
amountstringAmount, as HubSpot returns it.
pipelinestringPipeline id.
dealstagestringDeal stage id.
closedatestringClose date, as HubSpot returns it.

HubspotContact

FieldTypeDescription
idstringHubSpot contact id.
firstnamestringFirst name.
lastnamestringLast name.
emailstringEmail address.