web-search

Web Search MCP

A comprehensive Model Context Protocol (MCP) server built with FastMCP that provides LLMs with real-time, high-fidelity access to the web. This server aggregates multiple search engines, social platforms, and developer tools into a single interface, allowing AI agents to perform deep research, track community sentiment, and analyze technical documentation.

🚀 Features

The server provides a diverse suite of tools categorized by their primary use case:

🌐 General Web Search & Retrieval

• search_web: Fast web search via DuckDuckGo or Exa (SDK). Supports domain-scoping, date filtering, news mode, and geographic region. Default auto-provider tries DDG first, falls back to Exa.
• fetch_page: High-fidelity text extraction from URLs with bot-detection bypass and multiple output formats.

💬 Social & Community Intelligence

• search_reddit: Keyless search for community discussions, opinions, and real-world user experiences.
• search_hackernews: Access to technical discourse, startup news, and developer opinions via the Algolia API.
• search_github: Search for Issues and PRs to track bugs, feature requests, and community sentiment.
• get_github_issue: Fetch full conversation threads from GitHub Issues/PRs, sorted by reactions.
• search_x: Real-time discourse and breaking news search via the Bird CLI (requires session cookies).

🎓 Academic & Reference

• search_arxiv: Specialized search for academic papers, supporting Lucene field prefixes (e.g., au:, cat:).
• search_wikipedia: Factual summaries and background research via the MediaWiki API.

📋 Prerequisites

• Python 3.11+
• uv: Recommended for installation and environment management.
• External Tools (Optional but recommended):
  • gh CLI: For authenticated GitHub search and issue retrieval.
  • Node.js 22+: Required for the vendored X/Twitter search CLI.

⚙️ Setup

Installation

Install the server globally using uv:

uv tool install git+https://github.com/sydasif/web-search-mcp.git

MCP Client Configuration

Add the server to your MCP client configuration (e.g., Claude Desktop):

{
  "mcpServers": {
    "web-search": {
      "command": "web-search-mcp"
    }
  }
}

🔐 Configuration & Authentication

Most tools are keyless. However, some require specific environment variables for full functionality:

Tool	Required/Optional	Environment Variable	Note
GitHub	Optional	GITHUB_TOKEN or gh CLI	Authenticates with the GitHub API for issues/PR search and full thread retrieval.
X (Twitter)	Required	AUTH_TOKEN, CT0	Session cookies extracted from a logged-in x.com browser session.
Exa AI	Optional	EXA_API_KEY	Enables Exa as a search provider (via exa_py SDK) and increases fetch rate limits.

💡 Usage Examples

Web Research

• Broad Search: search_web(query="Latest NVIDIA H200 benchmarks")
• Targeted Docs: search_web(query="useEffect cleanup", domain="react.dev")
• News with region: search_web(query="elections", search_type="news", region="us-en", provider="exa")
• Date-filtered: search_web(query="uv package manager", time_range="w", provider="auto")
• Deep Read: fetch_page(url="https://docs.python.org/3/library/os.html")

Technical Analysis

• Issue Tracking: search_github(query="uv package manager")

Community Sentiment

• Reddit: search_reddit(query="Best mechanical keyboards 2024", subreddits=["MechanicalKeyboards"])
• Hacker News: search_hackernews(query="MCP server architecture")

🏗️ Project Structure

web_search_mcp/
├── server.py          # Server entry point and tool registrations
├── search/            # Search engine implementations (DDG, Exa)
├── social/            # Social platform integrations (GH, HN, Reddit, X)
├── tools/             # Specialized tools (arXiv, Wiki, Registries)
├── _config/           # Settings and rate limiting
├── _http/             # Shared HTTP client logic
├── _models/           # Pydantic models for requests/responses
└── _utils/            # Formatting and scoring utilities

🤝 Contributing