Web site scraper for LLMs with Airtop
Recursive Web Scraping Use Case Automating web scraping with recursive depth is ideal for collecting content across multiple linked pages—perfect for content...
🔌 Nodes & Integrations Used (5)
📖 Implementation & Setup Guide
Recursive Web Scraping
Use Case Automating web scraping with recursive depth is ideal for collecting content across multiple linked pages—perfect for content aggregation, lead generation, or research projects.
What This Automation Does This automation reads a list of URLs from a Google Sheet, scrapes each page, stores the content in a document, and adds newly discovered links back to the sheet. It continues this process for a specified number of iterations based on the defined scraping depth.
Input Parameters: - `Seed URL`: The starting URL to begin the scraping process. Example: `https://example.com/` - `Links must contain`: Restricts the links to those that contain this specified string. Example: `https://example.com/` - `Depth`: The number of iterations (layers of links) to scrape beyond the initial set. Example: `3`
How It Works 1. Starts by reading the `Seed URL` from the Google Sheet. 2. Scrapes each page and saves its content to the specified document. 3. Extracts new links from each page that match the `Links must contain` string, appends them to the Google Sheet. 4. Repeats steps 2–3 for the number of times specified by `Depth - 1`.
Setup Requirements 1. [Airtop API Key](https://portal.airtop.ai/api-keys) — free to generate. 2. Credentials set up for Google Docs (requires creating a project on Google Console). Read [how to](https://docs.n8n.io/integrations/builtin/credentials/google/). 3. Credentials set up for Google Spreadsheet.
Next Steps - **Add Filtering Rules**: Filter which links to follow based on domain, path, or content type. - **Combine with Scheduler**: Run this automation on a schedule to continuously explore newly discovered pages. - **Export Structured Data**: Extend the process to store extracted data in a CSV or database for analysis.
Read more about [website scraping for LLMS](https://www.airtop.ai/automations/scrape-website-data-n8n)
⚡ How to import this workflow into n8n:
- Click the Direct Download JSON button above to save
Web site scraper for LLMs with Airtop.json. - Log in to your n8n Cloud or self-hosted n8n instance.
- In your workflow canvas, click the menu in top-right and select Import from File (or paste the JSON directly).
- Configure your credentials (e.g. API keys, database URLs) in the corresponding node settings.
- Click Execute Workflow to test and switch to Active!
Related Web Scraping & Data Workflows
View All in Web Scraping & Data →Amazon Data Scraper with Bright Data API
Production-ready n8n automation template for Amazon Data Scraper with Bright Data API built by AhiXLight. Seamlessly connects HTTP Request, Read Write File with 16 configured nodes.
Amazon product search scraper with BrightData GPT 4 and Google
This workflow automates web scraping of Amazon search result pages by retrieving raw HTML, cleaning it to retain only the relevant product elements, and then...
Apify Scrape - Transcribe - Add to Sheet
Production-ready n8n automation template for Apify Scrape - Transcribe - Add to Sheet built by AhiXLight. Seamlessly connects HTTP Request, Limit, OpenAI with 12 configured nodes.
Apollo Scraping
Production-ready n8n automation template for Apollo Scraping built by AhiXLight. Seamlessly connects Execute Workflow Trigger, HTTP Request, Google Sheets with 6 configured nodes.