WhatsUpDoc (downmarked)
Scrape any developer documentation and save it locally as a markdown file using anthropic's MCP to standardize communication between the cli and the documentation server
what is WhatsUpDoc?
WhatsUpDoc is a command-line tool designed to scrape developer documentation from any website and save it locally as a markdown file using the Model Context Protocol (MCP) for standardized communication.
how to use WhatsUpDoc?
To use WhatsUpDoc, install it globally using npm, and then run the command downmarked fetch <url> to fetch and save documentation as a markdown file.
key features of WhatsUpDoc?
- Fetch documentation from any website and convert it to Markdown
- Save documentation to any location on your system
- Target specific content using CSS selectors
- Recursively fetch linked documentation pages
- Split documentation by headers into separate files
- Uses the Model Context Protocol (MCP) for standardized communication
use cases of WhatsUpDoc?
- Fetching and saving React documentation for offline access.
- Scraping Python documentation for personal reference.
- Collecting and organizing documentation from various sources for development purposes.
FAQ from WhatsUpDoc?
- Is WhatsUpDoc free to use?
Yes! WhatsUpDoc is free to use and open-source.
- What programming languages does WhatsUpDoc support?
WhatsUpDoc can scrape documentation from any website, regardless of the programming language.
- How does WhatsUpDoc handle different documentation structures?
It uses CSS selectors to target specific content, allowing for flexibility in handling various documentation layouts.
WhatsUpDoc (downmarked)
A command-line tool for fetching and storing developer documentation locally using the Model Context Protocol (MCP).
Features
- Fetch documentation from any website and convert it to Markdown
- Save documentation to any location on your system
- Target specific content using CSS selectors
- Recursively fetch linked documentation pages
- Split documentation by headers into separate files
- Uses the Model Context Protocol (MCP) for standardized communication
Installation
# Install globally
npm install -g downmarked
# Or use with npx
npx downmarked fetch https://reactjs.org/docs/getting-started.html
Usage
Basic Usage
downmarked fetch <url>
This will prompt you for an output location and save the documentation as Markdown.
Options
# Fetch documentation with specific options
downmarked fetch https://reactjs.org/docs/getting-started.html \
-o ~/Documents/react-docs.md \
-s "main" \
-r \
-d 2 \
--split
Available Options
| Option | Description |
|---|---|
-o, --output <path> | Output path (absolute or relative) |
-s, --selector <selector> | CSS selector to target specific content |
-r, --recursive | Recursively fetch linked documentation pages |
-d, --max-depth <number> | Maximum depth for recursive fetching (default: 3) |
--split | Split documentation by headers into separate files |
Examples
Fetch React Documentation
# Save React documentation to a specific location
downmarked fetch https://reactjs.org/docs/getting-started.html -o ~/Documents/react-docs.md
# Target only the main content area
downmarked fetch https://reactjs.org/docs/getting-started.html -s "main"
# Recursively fetch linked pages up to 2 levels deep
downmarked fetch https://reactjs.org/docs/getting-started.html -r -d 2
Fetch Python Documentation
# Save Python documentation
downmarked fetch https://docs.python.org/3/tutorial/index.html -o python-tutorial.md
How It Works
WhatsUpDoc (downmarked) uses the Model Context Protocol (MCP) to standardize communication between the CLI and the documentation server. The tool:
- Fetches HTML content from the specified URL
- Parses the HTML using Cheerio
- Converts the HTML to Markdown using Turndown
- Saves the Markdown to the specified location
Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
- Fork the repository
- Create your feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add some amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
License
This project is licensed under the MIT License - see the LICENSE file for details.
Acknowledgments
- Model Context Protocol (MCP) for providing the communication framework
- Turndown for HTML to Markdown conversion
- Cheerio for HTML parsing