Question regarding Web Scraping

General
,
1

Hello,

In the toolkit section, if we check the provided implementation for any web scraping tools, the command sent is to summarize some URL. In the case where the content of the URL are more, then does the agent automatically handles chunking of data or how does the agent creates a summary in that case?

Thank you.

2

Hey @Tpardhi, thanks for reaching out and supporting Agno. I’ve shared this with the team, we’re working through all requests one by one and will get back to you soon.If it’s urgent, please let us know. We appreciate your patience!

3

Yes it would be great if you can answer the query fast as this will help me in deciding the approach for my project.

4

Hi @Tpardhi,

Agno handles large web content two ways:

Direct web scraping tools (Newspaper4k, Firecrawl) - No automatic chunking, relies on model’s context window.

Knowledge system (UrlKnowledge) - Yes, automatic chunking enabled by default. Splits content into manageable chunks (5000 chars).

from agno.knowledge.url import UrlKnowledge
from agno.document.chunking.document import DocumentChunking

agent = Agent(
    knowledge=UrlKnowledge(
        urls=["https://example.com/long-article"],
        reader=URLReader(chunking_strategy=DocumentChunking())
    )
)

Examples: agno/cookbook/teams/team_with_knowledge.py at main · agno-agi/agno · GitHub