Docs Fetch MCP Server

Docs Fetch MCP Server

By wolfyy970 GitHub

MCP server for fetching web page content with recursive exploration capability

docs-fetch-mcp web-content-fetcher
Overview

what is Docs Fetch MCP?

Docs Fetch MCP is a Model Context Protocol (MCP) server designed for fetching web page content with recursive exploration capabilities, allowing LLMs to autonomously explore web pages and documentation to learn about specific topics.

how to use Docs Fetch MCP?

To use Docs Fetch MCP, you need to set up the server and call the fetch_doc_content tool with the required parameters, including the URL of the web page you want to fetch and an optional depth for link exploration.

key features of Docs Fetch MCP?

  • Clean content extraction from web pages
  • Recursive exploration of linked pages
  • Smart filtering of navigation links
  • Parallel processing for efficient crawling
  • Robust error handling for network issues
  • Dual-strategy approach for fetching content
  • Global timeout handling for reliable operation
  • Returns partial results when some pages fail to load

use cases of Docs Fetch MCP?

  1. Enabling LLMs to learn about specific topics by exploring documentation.
  2. Fetching and analyzing content from multiple related web pages.
  3. Assisting in research by gathering comprehensive information from various sources.

FAQ from Docs Fetch MCP?

  • Can Docs Fetch MCP handle any web page?

Yes! It is designed to fetch content from any web page while filtering out irrelevant elements.

  • How deep can the exploration go?

The exploration depth can be set from 1 to 5, depending on your needs.

  • Is there a limit to the number of pages that can be explored?

The server can explore multiple pages, but the total number depends on the specified depth and the structure of the linked content.

Content

Docs Fetch MCP Server

A Model Context Protocol (MCP) server for fetching web content with recursive exploration capabilities. This server enables LLMs to autonomously explore web pages and documentation to learn about specific topics.

Overview

The Docs Fetch MCP Server provides a simple but powerful way for LLMs to retrieve and explore web content. It enables:

  • Fetching clean, readable content from any web page
  • Recursive exploration of linked pages up to a specified depth
  • Same-domain link traversal to gather comprehensive information
  • Smart filtering of navigation links to focus on content-rich pages

This tool is particularly useful when users want an LLM to learn about a specific topic by exploring documentation or web content.

Features

  • Content Extraction: Cleanly extracts the main content from web pages, removing distractions like navigation, ads, and irrelevant elements
  • Link Analysis: Identifies and extracts links from the page, assessing their relevance
  • Recursive Exploration: Follows links to related content within the same domain, up to a specified depth
  • Parallel Processing: Efficiently crawls content with concurrent requests and proper error handling
  • Robust Error Handling: Gracefully handles network issues, timeouts, and malformed pages
  • Dual-Strategy Approach: Uses fast axios requests first with puppeteer as a fallback for more complex pages
  • Timeout Prevention: Implements global timeout handling to ensure reliable operation within MCP time limits
  • Partial Results: Returns available content even when some pages fail to load completely

Usage

The server exposes a single MCP tool:

fetch_doc_content

Fetches web page content with the ability to explore linked pages up to a specified depth.

Parameters:

  • url (string, required): URL of the web page to fetch
  • depth (number, optional, default: 1): Maximum depth of directory/link exploration (1-5)

Returns:

{
  "rootUrl": "https://example.com/docs",
  "explorationDepth": 2,
  "pagesExplored": 5,
  "content": [
    {
      "url": "https://example.com/docs",
      "title": "Documentation",
      "content": "Main page content...",
      "links": [
        {
          "url": "https://example.com/docs/topic1",
          "text": "Topic 1"
        },
        ...
      ]
    },
    ...
  ]
}

Installation

  1. Clone this repository:
git clone https://github.com/wolfyy970/docs-fetch-mcp.git
cd docs-fetch-mcp
  1. Install dependencies:
npm install
  1. Build the project:
npm run build
  1. Configure your MCP settings in your Claude Client:
{
  "mcpServers": {
    "docs-fetch": {
      "command": "node",
      "args": [
        "/path/to/docs-fetch-mcp/build/index.js"
      ],
      "env": {
        "MCP_TRANSPORT": "pipe"
      }
    }
  }
}

Dependencies

  • @modelcontextprotocol/sdk: MCP server SDK
  • puppeteer: Headless browser for web page interaction
  • axios: HTTP client for making requests

Development

To run the server in development mode:

npm run dev

License

MIT

No tools information available.
No content found.