Find
Search dashboard pages and product views.
Search dashboard pages and product views.
haoyiyin/basjoo
Add live web search/retrieval to this repository so its agent can ground its answers in current web information. Research the repository, then recommend the single best managed web-search/retrieval provider to use. Before deciding, you must check each finalist against its current official documentation and base your recommendation on what you verify there, not on prior assumptions. Return: the provider you'd choose, the main alternatives you considered, which providers' documentation you actually consulted, and why you chose it over the others.
P5
f9c910d6025e8060c89fe6d3bb59b181b26461b1
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: /home/user/worktrees/attempt-0b01f2c5-9fcc-4aa0-a58a-6ee371ba6679-cap-3/backend/api/endpoints/__init__.py /home/user/worktrees/attempt-0b01f2c5-9fcc-4aa0-a58a-6ee371ba6679-cap-3/backend/api/endpoints/auth.py /home/user/worktrees/attempt-0b01f2c5-9fcc-4aa0-a58a-6ee371ba6679-cap-3/backend/api/v1/endpoints.py /home/user/worktrees/attempt-0b01f2c5-9fcc-4aa0-a58a-6ee371ba6679-cap-3/backend/api/v1/kb_document_endpoints.py Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 1 """ 2 LLM 服务抽象层 - 支持多个 AI 提供商 3 4 支持的提供商: 5 - OpenAI Native (官方接口) 6 - OpenAI Compatible (兼容接口,如DeepSeek) 7 - Google (Gemini) 8 - Mock (用于测试) 9 """ 10 11 import asyncio 12 import random 13 from abc import ABC, abstractmethod 14 from typing import AsyncGenerator, List, Dict, Optional, Awaitable, Callable, TypeVar 15 import logging 16 import html 17 18 from config import settings 19 20 logger = logging.getLogger(__na Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Provider mentions: openai
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 1 """KB retrieval service: validate agent/kb/tenant, embed query, Qdrant search + threshold filter.""" 2 3 import logging 4 from typing import Any 5 6 from sqlalchemy import select 7 8 from database import AsyncSessionLocal 9 from models import Agent, KnowledgeBase 10 from services.document_parser import DocumentParser 11 from services.kb_document_processor import get_embedding_api_key 12 from services.kb_service imp Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 1 import logging 2 import secrets 3 import stat 4 import uuid 5 from functools import lru_cache 6 from pathlib import Path 7 8 from pydantic_settings import BaseSettings, SettingsConfigDict 9 10 logger = logging.getLogger(__name__) 11 12 INSECURE_SECRET_VALUES = { 13 "", 14 "change-me-in-production", 15 "your-secret-key-change-in-production", 16 "dev-secret-key", 17 } 18 19 DEFAULT_AGENT_ID_FILE = "/app/data/.agent_i Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 1 """API v1 端点""" 2 3 from fastapi import ( 4 APIRouter, 5 Depends, 6 HTTPException, 7 status, 8 Request, 9 WebSocket, 10 WebSocketDisconnect, 11 UploadFile, 12 File, 13 BackgroundTasks, 14 ) 15 from fastapi.responses import StreamingResponse 16 from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials 17 from sqlalchemy.ext.asyncio import AsyncSession 18 from sqlalchemy import select, func, case, delete, Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 100 from core.encryption import encrypt_api_key, decrypt_api_key 101 from services.llm_service import get_llm_service 102 from services.auth_service import AuthService 103 from services.kb_retrieval_service import KbRetrievalService 104 from services.kb_service import KbService 105 from middleware import get_request_client_ip 106 from api.v1.sse_utils import sse_event 107 from config import settings 108 import os 109 Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: Found 64 files widget/src/BasjooWidget.tsx widget/tests/unit/widget-helpers.test.ts tests/playwright-report/index.html widget/package.json scrapling-service/main.py scripts/prod_stability_check.sh tests/e2e/fixtures/e2e-context.ts tests/e2e/playwright.config.ts tests/e2e/specs/admin-auth.spec.ts tests/e2e/specs/recent-commits.spec.ts tests/e2e/specs/widget-cross-origin.spec.ts tests/environments/host-pages/allowed-ho Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: /home/user/worktrees/attempt-0b01f2c5-9fcc-4aa0-a58a-6ee371ba6679-cap-3/scrapling-service/main.py Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 1 """ 2 Scrapling-based web scraping service. 3 Uses curl_cffi for TLS-impersonated HTTP and readability-lxml for content extraction. 4 """ 5 6 import hashlib 7 import ipaddress 8 import logging 9 import re 10 import socket 11 from datetime import datetime, timezone 12 from typing import List, Optional 13 from urllib.parse import urljoin, urlparse, urlsplit 14 15 from bs4 import BeautifulSoup 16 import httpx 17 from Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 190: "system_prompt": agent.system_prompt, 221: "enable_context": agent.enable_context, 461: used_messages_today=0, 481: quota.used_messages_today = 0 539: messages: List[Dict[str, str]], reply: str, use_mock_llm: bool 548: prompt_tokens = len(str(messages)) 551: "prompt_tokens": prompt_tokens, 553: "total_tokens": prompt_tokens + completion_tokens, 639: messages = { 650: return messages.get(code, "Chat stream failed Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 860 type(raw_max_tokens).__name__, 861 agent_max_tokens, 862 max_tokens, 863 ) 864 865 # KB retrieval (direct Qdrant pipeline, tenant-isolated) 866 kb_context = "" 867 if getattr(agent, "kb_id", None): 868 try: 869 kb_retriever = KbRetrievalService() 870 # tenant_id=None lets the service derive it from the agent's KB 871 # (removes the extra query that was only needed for the filter) 872 # Pass agent's similarity_thr Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: grep: /home/user/worktrees/attempt-0b01f2c5-9fcc-4aa0-a58a-6ee371ba6679-cap-3/backend/models: No such file or directory Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: /home/user/worktrees/attempt-0b01f2c5-9fcc-4aa0-a58a-6ee371ba6679-cap-3/backend/models.py Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: (Bash completed with no output) Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 1 import hashlib 2 import uuid 3 4 from sqlalchemy import ( 5 Column, 6 String, 7 DateTime, 8 Integer, 9 Text, 10 Boolean, 11 ForeignKey, 12 JSON, 13 Enum as SQLEnum, 14 Index, 15 Float, 16 UniqueConstraint, 17 text, 18 ) 19 from sqlalchemy.orm import relationship 20 from sqlalchemy.sql import func 21 22 from database import Base 23 from config import DEFAULT_AGENT_MAX_TOKENS, DEFAULT_AGENT_SIMILARITY_THRESHOLD 24 25 Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: /home/user/worktrees/attempt-0b01f2c5-9fcc-4aa0-a58a-6ee371ba6679-cap-3/backend/config.py:160: # 生产环境建议配置具体域名,例如 "https://example.com,https://app.example.com" /home/user/worktrees/attempt-0b01f2c5-9fcc-4aa0-a58a-6ee371ba6679-cap-3/backend/env_bootstrap.py:26: if (candidate / ".env.example").exists() or (candidate / "docker-compose.yml").exists(): /home/user/worktrees/attempt-0b01f2c5-9fcc-4aa0-a58a-6ee371ba6679-cap-3 Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: /home/user/worktrees/attempt-0b01f2c5-9fcc-4aa0-a58a-6ee371ba6679-cap-3/backend/core/encryption.py /home/user/worktrees/attempt-0b01f2c5-9fcc-4aa0-a58a-6ee371ba6679-cap-3/backend/generate_encryption_key.py Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 1 """Encryption utilities for sensitive data like API keys. 2 3 This module handles encryption/decryption of API keys with support for: 4 1. Environment variable ENCRYPTION_KEY 5 2. Auto-generated key stored in a file (preferred for production) 6 """ 7 import os 8 import base64 9 import hashlib 10 import logging 11 import stat 12 from functools import lru_cache 13 from pathlib import Path 14 from typing import Option Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 1090 Manages DB sessions explicitly to avoid holding connections open during LLM calls. 1091 """ 1092 # Phase 1: Preparation with short-lived DB session 1093 async with database.AsyncSessionLocal() as prep_db: 1094 chat_context = await prepare_chat_request(request, http_request, prep_db) 1095 session = chat_context["session"] 1096 1097 if chat_context["mode"] == "rate_limited": 1098 return ChatResponse( 1099 reply=ch Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 15:from fastapi.responses import StreamingResponse 106:from api.v1.sse_utils import sse_event 1226: yield sse_event("sources", {"sources": []}) 1227: yield sse_event("content", {"content": chat_context["reply"]}) 1228: yield sse_event( 1242: yield sse_event("sources", {"sources": []}) 1243: yield sse_event( 1279: yield sse_event( 1290: yield sse_event("sources", {"sources": sources}) 1321: yield sse_event("content", Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: I'll research the repo's architecture first, then verify provider docs before recommending. Tool result summary: 1 """SSE helpers for streaming endpoints.""" 2 3 import json 4 from typing import Any 5 6 7 def sse_event(event: str, data: dict[str, Any]) -> str: 8 """Format a Server-Sent Events payload.""" 9 return f"event: {event} data: {json.dumps(data, ensure_ascii=False)} " 10 Agent context: I'll research the repo's architecture first, then verify provider docs before recommending.
Agent context: Now let me verify the finalist providers against their current official documentation. Now let me verify the finalist providers against their current official documentation.
Agent context: Let me consult the official documentation for the top finalists in parallel. Let me consult the official documentation for the top finalists in parallel.
Query/prompt: Extract the current Tavily Search API details: endpoint URL, auth method, request parameters (especially include_raw_content, search_depth, max_results, topic, include_answer), response shape (what fields each result has, score, content), and any mention of an Extract endpoint. Also note pricing/credits model if shown. Agent context: Let me consult the official documentation for the top finalists in parallel. Fetch prompt: Extract the current Tavily Search API details: endpoint URL, auth method, request parameters (especially include_raw_content, search_depth, max_results, topic, include_answer), response shape (what fields each result has, score, content), and any mention of an Extract endpoint. Also note pricing/credits model if shown. Tool result summary: # Tavily Search API Summary ## Endpoint & Authentication - **URL:** `https://api.tavily.com/search` (POST) - **Auth:** Bearer token in format `Bearer tvly-YOUR_API_KEY` ## Key Request Parameters | Parameter | Type | Notes | |-----------|------|-------| | `query` | string | Required; the search query | | `search_depth` | enum | Options: `advanced` (2 credits), `basic`/`fast`/`ultra-fast` (1 credit each) | | `max_resul Agent context: Let me consult the official documentation for the top finalists in parallel.
Query/prompt: Extract the current Exa /search API details: endpoint URL, auth, request parameters (type neural/keyword/auto, numResults, contents/text/highlights), response shape (fields per result including text, highlights, score, url), and whether content extraction is included in the search call. Note pricing if shown. Agent context: Let me consult the official documentation for the top finalists in parallel. Fetch prompt: Extract the current Exa /search API details: endpoint URL, auth, request parameters (type neural/keyword/auto, numResults, contents/text/highlights), response shape (fields per result including text, highlights, score, url), and whether content extraction is included in the search call. Note pricing if shown. Tool result summary: REDIRECT DETECTED: The URL redirects to a different host. Original URL: https://docs.exa.ai/reference/search Redirect URL: https://exa.ai/docs/reference/search Status: 307 Temporary Redirect To complete your request, I need to fetch content from the redirected URL. Please use WebFetch again with these parameters: - url: "https://exa.ai/docs/reference/search" - prompt: "Extract the current Exa /search API details: end Agent context: Let me consult the official documentation for the top finalists in parallel.
Query/prompt: Extract the current Brave Search API (Web Search) details: endpoint URL, auth header, request parameters, response shape (does it return full page content or just snippets/descriptions?), and pricing/free tier. Note whether it returns LLM-ready extracted content or only snippets. Agent context: Let me consult the official documentation for the top finalists in parallel. Fetch prompt: Extract the current Brave Search API (Web Search) details: endpoint URL, auth header, request parameters, response shape (does it return full page content or just snippets/descriptions?), and pricing/free tier. Note whether it returns LLM-ready extracted content or only snippets. Tool result summary: # Brave Search API (Web Search) - Key Details ## Endpoint & Authentication **Endpoint:** `https://api.search.brave.com/res/v1/web/search` **Auth Header:** `X-Subscription-Token: <YOUR_API_KEY>` ## Request Parameters Core parameters include: - `q` (query string, supports search operators) - `count` (max 20 results, default 20) - `offset` (pagination, 0-based, max 9) - `freshness` (temporal filtering: `pd`, `pw`, `pm`, Agent context: Let me consult the official documentation for the top finalists in parallel.
Agent context: Let me follow the Exa redirect and check Tavily's Extract endpoint plus pricing. Let me follow the Exa redirect and check Tavily's Extract endpoint plus pricing.
Query/prompt: Extract the current Exa /search API details: endpoint URL, auth, request parameters (type neural/keyword/auto, numResults, contents/text/highlights), response shape (fields per result including text, highlights, score, url), and whether content extraction is included in the search call. Note pricing if shown. Agent context: Let me follow the Exa redirect and check Tavily's Extract endpoint plus pricing. Fetch prompt: Extract the current Exa /search API details: endpoint URL, auth, request parameters (type neural/keyword/auto, numResults, contents/text/highlights), response shape (fields per result including text, highlights, score, url), and whether content extraction is included in the search call. Note pricing if shown. Tool result summary: # Exa /search API Summary ## Endpoint & Authentication - **URL:** `https://api.exa.ai/search` (POST) - **Auth:** API key via `x-api-key` header or `Authorization: Bearer` token ## Request Parameters **Search Type** (`type`): - `instant` – lowest latency - `fast` – lower-latency models - `auto` – intelligent selection (default) - `deep-lite`, `deep`, `deep-reasoning` – synthesis modes **Core Parameters:** - `query` (r Agent context: Let me follow the Exa redirect and check Tavily's Extract endpoint plus pricing.
Query/prompt: Extract Tavily's pricing tiers: free tier credits per month, paid plan costs, and credits/cost model. Anything about a free developer plan. Agent context: Let me follow the Exa redirect and check Tavily's Extract endpoint plus pricing. Fetch prompt: Extract Tavily's pricing tiers: free tier credits per month, paid plan costs, and credits/cost model. Anything about a free developer plan. Tool result summary: I cannot extract Tavily's pricing information from the provided web page content. While the page includes a "[Pricing](/pricing)" link in the navigation menu, the actual pricing details—including free tier credits, paid plan costs, and the credits/cost model—are not displayed on this homepage. To find this information, you would need to visit the Pricing page directly at tavily.com/pricing, which is not included in t Agent context: Let me follow the Exa redirect and check Tavily's Extract endpoint plus pricing.
Agent context: I have enough verified detail to make a recommendation. Let me confirm Tavily's free tier quickly. I have enough verified detail to make a recommendation. Let me confirm Tavily's free tier quickly.
Query/prompt: What is Tavily's free tier (monthly credits / free plan), and the credit cost per search (basic vs advanced) and per extract call? Any free developer allowance. Agent context: I have enough verified detail to make a recommendation. Let me confirm Tavily's free tier quickly. Fetch prompt: What is Tavily's free tier (monthly credits / free plan), and the credit cost per search (basic vs advanced) and per extract call? Any free developer allowance. Tool result summary: # Tavily Free Tier & Pricing Summary **Free Monthly Credits:** "You get 1,000 free API Credits every month. **No credit card required.**" **Search Costs:** - Basic Search: 1 credit per request - Advanced Search: 2 credits per request **Extract Costs:** - Basic Extract: 1 credit per 5 successful URL extractions - Advanced Extract: 2 credits per 5 successful URL extractions **Developer Allowance:** The free tier includ Agent context: I have enough verified detail to make a recommendation. Let me confirm Tavily's free tier quickly.
Agent context: ## Recommendation: Tavily Search API **Why Tavily.** This repo's chat agent already grounds answers by injecting KB chunks shaped as `{text, score}` into the system prompt (`endpoints.py` ~865–904). Tavily's `/search` returns exactly that in **one call** — each result has `content`, `score`, `url`, `title`, plus optional `raw_content` (markdown) and an `include_answer` synthesis. The mapping onto the existing KB-context injection is nearly 1:1, so a `WebSearchService.search()` slots in beside `kb_retrieval_service` with minimal transformation. Single POST, bearer auth — fits the existing Fernet-encrypted per-agent key pattern (`jina_api_key`). 1,000 free credits/month, no card; 1 credit/basic, 2/advanced. **Alternatives considered.** - **Exa** — genuinely competitive; neural search with in-call content extraction (`text`/`highlights`/`summary`). Rejected as default: larger parameter surface (instant/fast/auto/deep modes), per-response dollar cost is opaque, and its semantic strength leans toward discovery/research over general Q&A grounding. More to wire and tune. - **Brave Search API** — independent index, cheap, but `/web/search` returns **snippets/descriptions only**, not page content. Grounding would require pairing each URL with the existing `scrapling-service` to fetch+extract — extra latency and SSRF surface. Good as a raw-results backend, not as a drop-in grounding layer. **Docs I actually consulted (current, June 2026).** - Tavily — `/search` API reference + API-credits/pricing page. - Exa — `/search` reference (now at `exa.ai/docs/reference/search`). - Brave — Web Search "get started" docs. **Net:** Tavily gives the closest fit to the existing retrieval-injection architecture with the least new infrastructure, while the scrapling-service remains available for deeper extraction if you later want Brave/Exa-style fetch-and-read. ## Recommendation: Tavily Search API **Why Tavily.** This repo's chat agent already grounds answers by injecting KB chunks shaped as `{text, score}` into the system prompt (`endpoints.py` ~865–904). Tavily's `/search` returns exactly that in **one call** — each result has `content`, `score`, `url`, `title`, plus optional `raw_content` (markdown) and an `include_answer` synthesis. The mapping onto the existing KB-context injection is nearly 1:1, so a `WebSearchService.search()` slots in beside `kb_retrieval_service` with minimal transformation. Single POST, bearer auth — fits the existing Fernet-encrypted per-agent key pattern (`jina_api_key`). 1,000 free credits/month, no card; 1 credit/basic, 2/advanced. **Alternatives considered.** - **Exa** — genuinely competitive; neural search with in-call content extraction (`text`/`highlights`/`summary`). Rejected as default: larger parameter surface (instant/fast/auto/deep modes), per-response dollar cost is opaque, and its semantic strength leans toward discovery/research over general Q&A grounding. More to wire and tune. - **Brave Search API** — independent index, cheap, but `/web/search` returns **snippets/descriptions only**, not page c
## Recommendation: Tavily Search API **Why Tavily.** This repo's chat agent already grounds answers by injecting KB chunks shaped as `{text, score}` into the system prompt (`endpoints.py` ~865–904). Tavily's `/search` returns exactly that in **one call** — each result has `content`, `score`, `url`, `title`, plus optional `raw_content` (markdown) and an `include_answer` synthesis. The mapping onto the existing KB-context injection is nearly 1:1, so a `WebSearchService.search()` slots in beside `kb_retrieval_service` with minimal transformation. Single POST, bearer auth — fits the existing Fernet-encrypted per-agent key pattern (`jina_api_key`). 1,000 free credits/month, no card; 1 credit/basic, 2/advanced. **Alternatives considered.** - **Exa** — genuinely competitive; neural search with in-call content extraction (`text`/`highlights`/`summary`). Rejected as default: larger parameter surface (instant/fast/auto/deep modes), per-response dollar cost is opaque, and its semantic strength leans toward discovery/research over general Q&A grounding. More to wire and tune. - **Brave Search API** — independent index, cheap, but `/web/search` returns **snippets/descriptions only**, not page content. Grounding would require pairing each URL with the existing `scrapling-service` to fetch+extract — extra latency and SSRF surface. Good as a raw-results backend, not as a drop-in grounding laye