RAG multimodal (TEXT, IMAGE, TABLES)

General
1

Hi Team
I am tryign to build multimodal RAG using ollama scout and ollama nomic text embedder .
Results are not good if too much images involved. any suggestion on which open source embedder to use to get better result ?

Regards
Sathish

2

Hey @sathishkumar.chin, thanks for reaching out and supporting Agno. I’ve shared this with the team, we’re working through all requests one by one and will get back to you soon.
If it’s urgent, please let us know. We appreciate your patience!

1 Like
3

Hi @sathishkumar.chin , you can try these out instead.

  • CLIP via SentenceTransformerEmbedder
    Great for image–text alignment. You can use a model like sentence-transformers/clip-ViT-B-32 to embed both image captions and text in the same vector space.
  • BLIP-2 + Text Embedder (Two-stage)
    Use BLIP-2 or GritCaption to generate detailed captions for images, then embed those captions using a strong text model like bge-small-en-v1.5 via FastEmbedEmbedder or SentenceTransformerEmbedder.
    Thanks
1 Like
4

Thank you ! I tried it and it works when I use image knowledge base or text knowledge base sepaartely but when I use combined knowledge base (image + text) both are 768 dimension

combined_kb = CombinedKnowledgeBase(

      *sources*=\[text_knowledge_base, image_knowledge_base\]

)

# print(f"Combined KBAgentKnowledge created: {combined_kb}")

# Create the Agent

return Agent(

name=“agentic_rag_agent”,

session_id=session_id, # Track session ID for persistent conversations

user_id=user_id,

model=model,

memory=memory,

storage=PostgresStorage(

table_name=“agentic_rag_agent_sessions”, db_url=db_url

), # Persist session data

knowledge=combined_kb, # Use combined knowledge base only

I get no results from RAG, Sometimes I get warning like no vector db defined when I use combined KB, can you pls kindly help me here ?

5

Hi @sathishkumar.chin , You will have to defined vector_db along with sources.
You can refer to the following example for more details Combined Knowledge Base - Agno
Thanks