Agent is not able to search knowledge base for the first time

I have a FastAPI app with two routes:

  1. Receive a URL and analyze the image.
  2. Chat using that image data.
    My approach starts with a blank text file as a knowledge base in MongoDB, which I update with responses from the 1st api response. I’ve confirmed that the knowledge base updates correctly in MongoDB. However, when I first attempt to chat, it doesn’t search the knowledge base. It only begins searching after the second message, continuing to do so for subsequent messages.
app = FastAPI()

db_url = os.getenv("MONGODB_URI")
knowledge_base = TextKnowledgeBase(
    path="knowledge.txt",
    vector_db=MongoDBVector(
        collection_name="knowledge",
        db_url=db_url,
        embedder=OpenAIEmbedder(model="text-embedding-3-small"),
    ),
)

origins = [
    "http://localhost:5173",
]

app.add_middleware(
    CORSMiddleware,
    allow_origins=origins,
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)


class AnalyzeURLRequest(BaseModel):
    url: str
    request_id: str


class AnalyzeResponse(BaseModel):
    Ingredients: list[str]
    HealthImplications: list[str]
    Considerations: list[str]
    NutritionInformation: list[str]
    Conclusion: str
    ProductName: str


class ChatRequest(BaseModel):
    message: str
    request_id: str


class ChatResponse(BaseModel):
    message: str


def get_agent():
    agent = Agent(
        model=OpenAIChat(id="gpt-4o"),
        system_prompt=SYSTEM_PROMPT,
        instructions=INSTRUCTIONS,
        tools=[DuckDuckGo()],
        structured_outputs=True,
        response_model=AnalyzeResponse,
    )
    return agent


def get_chat_agent(request_id: str):
    return Agent(
        model=OpenAIChat(id="gpt-4o"),
        session_id=request_id,
        show_tool_calls=True,
        search_knowledge=True,
        update_knowledge=True,
        knowledge_base=knowledge_base,
        add_chat_history_to_messages=True,
        read_chat_history=True,
        storage=MongoAgentStorage(
            collection_name="chat_history",
            db_url=db_url,
            db_name="product_analysis",
        ),
        instructions=[
            "Always search the knowledge base for the answer before using the tools.",
            "If you don't get the answer from the knowledge base, you can use the tools to search for the answer.",
        ],
        structured_outputs=True,
        response_model=ChatResponse,
    )


@app.post("/analyze-url")
async def analyze_url(request: AnalyzeURLRequest):
    response = analyze_image(request.url, request.request_id)
    return response


@app.post("/chat")
async def chat(request: ChatRequest):
    print("chat request_id", request.request_id)
    chat_agent = get_chat_agent(request.request_id)
    response = chat_agent.run(request.message)
    return response.content


def analyze_image(image_path, request_id):
    print("analyze_image request_id", request_id)
    agent = get_agent()
    response = agent.run(
        "Analyze the product image",
        images=[image_path],
    )

    text = prepare_knowledge_base(response)
    knowledge_base.load_text(text, upsert=True)

    return response.content


def prepare_knowledge_base(response: AnalyzeResponse):
    text = f"**Metadata**: \n\n"
    text += f"Product Name: {response.content.ProductName}\n"
    text += f"Ingredients: {', '.join(response.content.Ingredients)}\n"
    return text

Hi @Vishal
Thank you for reaching out and using Phidata! I’ve tagged the relevant engineers to assist you with your query. We aim to respond within 24 hours.
If this is urgent, please feel free to let us know, and we’ll do our best to prioritize it.
Thanks for your patience!

Hi @Vishal
To clarify, do you see any information in your DB after the analyze_image message, before you start to chat?

1 Like

Hey Dirk,
Yes I see it’s stored in my DB. But there’s nothing in the Knowledge base.

So you see no tool calls on your first message?

Hey @Vishal! Can you please confirm if you can see any tool calls in your terminal? I can see you have used show_tool_calls=True to make that happen.

Hey @Dirk @manthanguptaa , It’s searching knowledge second time only.

@Vishal according to your explanation of what analyze_image does, it would only use your normal agent without any knowledge base attached

def get_agent():
    agent = Agent(
        model=OpenAIChat(id="gpt-4o"),
        system_prompt=SYSTEM_PROMPT,
        instructions=INSTRUCTIONS,
        tools=[DuckDuckGo()],
        structured_outputs=True,
        response_model=AnalyzeResponse,
    )
    return agent

It does however load the response in the knowledge base.

Then your first “chat” request, which uses the chat agent with knowledge_base, then correctly uses the knowledge base.

Am I right? This looks like it behaves correctly?

@Dirk The analyze_image response is loaded in the knowledge_base, but it isn’t utilized in the first chat request.

It is difficult for me to troubleshoot this… have you progressed further? Perhaps don’t re-instantialize the agent every time, but change get_chat_agent to create the agent if it doesn’t exist, otherwise return the agent. To ensure you have the same agent every time.

@Dirk I got your point, but I need to create an agent every time, as I will have different sessions for each image uploaded, and they will have a unique knowledge base as well. That’s why I’m recreating the agent again.

Hi @Vishal have you made any progress? I am unable to recreate your issue.