Europe 2026
From MCP to Scale: Pipelines That Build Themselves — Rafael Levi, Bright Data
Overview
This talk explores building self-maintaining data collection pipelines using Large Language Models (LLMs) and specialized tools. It addresses the challenge of collecting large-scale data from websites, particularly those with anti-bot measures, by shifting from direct LLM parsing of every page to an agent-driven approach that builds and maintains scrapers. The core idea is to empower LLMs with the ability to create and manage the tools needed for data extraction, thereby saving significant computational resources and development time.
Who should watch
- AI Engineers looking to build scalable data collection systems.
- Product Managers and Builders who need to integrate web data into their applications.
- Developers struggling with website blocking and CAPTCHA systems during data scraping.
- Anyone interested in automating complex web interactions beyond simple data retrieval.
- Individuals seeking to reduce token consumption and costs associated with LLM web access.
Key takeaways
- LLM-powered agents can automate the creation and maintenance of web scrapers, overcoming the limitations of direct LLM parsing for large datasets.
- Tools like Bright Data's MCP (Multi-Cloud Platform) provide agents with capabilities to extract HTML, identify selectors, and build scrapers, even from heavily protected websites.
- This agentic approach significantly reduces token usage compared to having an LLM parse raw HTML for every data point, leading to cost savings.
- The system can handle dynamic websites and changing selectors, with agents automatically fixing broken scrapers, creating self-healing pipelines.
- Beyond scraping, agents can perform actions on websites, such as filling forms or simulating human-like browser interactions, though logging in to private data is not supported.
- The technology is applicable for both large-scale enterprise data collection and personal use cases, like finding specific items or monitoring for changes.
- Bright Data's infrastructure offers a vast IP network and remote browser capabilities to mimic real human behavior, bypassing bot detection systems.
- The platform emphasizes the use of public data, with legal precedents supporting the collection of publicly accessible information regardless of the method.
Notable quotes
*Instead of telling, Hey, LLM, can you go and parse this for me? build a scraper that's going to parse it for me.*
*The whole session is how do you build pipelines, right? With LLM.*
*Public data is public data. It doesn't matter how you collect it.*
Unofficial community note. Prefer the recording for nuance.