GPT-Crawler
Open-source crawler that scrapes a website and outputs structured knowledge files you can use to build a custom GPT or AI assistant.
About
GPT-Crawler is a free, open-source tool built by Builder.io that automates the process of turning any website into training data for a custom AI assistant. You point it at one or more URLs, configure crawl patterns and CSS selectors, and it outputs a structured knowledge file ready to upload to OpenAI's custom GPT builder or use with the Assistants API.
It supports sitemaps, token limits, Docker deployments, and an optional API server mode. The tool itself is completely free under an ISC license, though accessing custom GPT features on OpenAI's platform requires a paid ChatGPT plan.
It supports sitemaps, token limits, Docker deployments, and an optional API server mode. The tool itself is completely free under an ISC license, though accessing custom GPT features on OpenAI's platform requires a paid ChatGPT plan.