Skip_Existing flag not working in load function

General
1

Hi,

I have been populating some knowledge into my pgvector using UrlKnowledge() and running knowledge.load(skip_existing=True).

However after running this i’m getting database conflict errors on primary keys overlapping which suggests that skip_existing is not working as intended.

For now I am using knowledge.load(upsert=True) but this is essentially writing rows that already exist which is a waste of resouces.

I am on agno v1.8.1 for context.

Its worth noting that I tried the asynchronous version using aload() instead but I got the same error.

If anyone has any suggestions that would be great.

2

Hey @Jester, thanks for reaching out and supporting Agno. I’ve shared this with the team, we’re working through all requests one by one and will get back to you soon.If it’s urgent, please let us know. We appreciate your patience!

3

Hi @Jester i tried with the following example- agno/cookbook/agent_concepts/knowledge/url_kb.py at v1 · agno-agi/agno · GitHub

I’m on 1.8.2 and it works, pls let me know if i’m missing anything?

image
image1315×822 110 KB

image
image1259×996 130 KB

4

Hi @kausmos, I have just upgraded to v1.8.2 which is the version you used, but I still get the same error.

I think the only considerable difference is that I use an sqlalchemy database engine in pgvector() rather than a db_url.

The reason I did this was because I was uploading large volumes of knowledge and I hit timeout thresholds.

Also i’m using a postgresql database.

Here’s a snippet of my code.

image
image558×456 15.7 KB

5

Hey @Jester

We recently released a new version of Agno, with a large refactor of the Knowledge class.
Included in this was the logic for skipping existing vectors.
Let me know if you would like to try out the new version, we would love some feedback on it.
Especially if you are testing with large datasets, it would be a good test case for us.

In the meantime, I am looking into this issue on Agno 1.0 to see if I can spot anything that might be causing this.

6

Hi @WillemdeJongh1,

Yeah I saw v2 has just been released, I believe its going to require me to refactor some of my code should I transition over.

Also thanks for looking into this issue in v1