Generate AI ready llms txt files from Screaming Frog website cra
This workflow helps you generate an llms.txt file (if you're unfamiliar with it, check out this article) using a Screaming Frog export. Screaming Frog is a w...
🔌 Nodes & Integrations Used (6)
📖 Implementation & Setup Guide
This workflow helps you generate an **llms.txt** file (if you're unfamiliar with it, check out [this article](https://towardsdatascience.com/llms-txt-414d5121bcb3/)) using a **Screaming Frog export**.
[Screaming Frog](https://www.screamingfrog.co.uk/seo-spider/) is a well-known website crawler. You can easily crawl a website. Then, export the **"internal_html"** section in CSV format.
How It Works:
A **form** allows you to enter: - The **name of the website** - A **short description** - The **internal_html.csv** file from your Screaming Frog export
Once the form is submitted, the **workflow is triggered automatically**, and you can **download the llms.txt file directly from n8n**.
Downloading the File Since the last node in this workflow is **"Convert to File"**, you will need to **download the file directly from the n8n UI**. However, you can easily **add a node** (e.g., Google Drive, OneDrive) to automatically upload the file **wherever you want**.
AI-Powered Filtering (Optional): This workflow includes a **text classifier node**, which is **deactivated by default**. - You can **activate it** to apply a more **intelligent filter** to select URLs for the `llms.txt` file. - Consider modifying the **description** in the classifier node to specify the type of URLs you want to include.
How to Use This Workflow
- **Crawl the website** you want to generate an `llms.txt` file for using **Screaming Frog**.
- **Export the "internal_html"** section in CSV format.
- 
- In **n8n**, click **"Test Workflow"**, fill in the form, and **upload** the `internal_html.csv` file.
- Once the workflow is complete, go to the **"Export to File"** node and **download the output**.
**That's it! You now have your llms.txt file!**
**Recommended Usage:** Use this workflow **directly in the n8n UI by clicking** 'Test Workflow' and uploading the file in the form.
⚡ How to import this workflow into n8n:
- Click the Direct Download JSON button above to save
Generate_AI_ready_llms_txt_files_from_Screaming_Frog_website_cra.json. - Log in to your n8n Cloud or self-hosted n8n instance.
- In your workflow canvas, click the menu in top-right and select Import from File (or paste the JSON directly).
- Configure your credentials (e.g. API keys, database URLs) in the corresponding node settings.
- Click Execute Workflow to test and switch to Active!
Related Docs, PDF & OCR Workflows
View All in Docs, PDF & OCR →Add articles to a Notion list by accessing a Discord slash comma
This workflow allows you to add articles to a Notion reading list by accessing a Discord slash command. Prerequisites - A Notion account and [credentials](),...
Adobe PDF Services API Automation
Production-ready n8n automation template for Adobe PDF Services API Automation built by AhiXLight. Seamlessly connects HTTP Request, Execute Workflow Trigger, Dropbox with 20 configured nodes.
Adobe PDF Services API Workflow
Production-ready n8n automation template for Adobe PDF Services API Workflow built by AhiXLight. Seamlessly connects HTTP Request, Execute Workflow Trigger, Dropbox with 20 configured nodes.
AI Brand Voice Content Generator
Production-ready n8n automation template for AI Brand Voice Content Generator built by AhiXLight. Seamlessly connects Lm Chat Open Ai, Information Extractor, HTTP Request with 27 configured nodes.