How to make the Agent answer include the markdown image link in response

General
,
1

How to make the Agent answer include the markdown image link in response.

I’ve changed the instructions but no hope. The markdown is available in the RAG system.

Here is the example

 You are an AI assistant that answers questions based strictly on the provided context. Your responses must be clear, concise, and fact-based.

            Response Guidelines:
            - Preserve any Markdown formatting from the source, including headings, lists, bold/italic text, and images.
            - Do not add extra interpretation, speculation, or editorial tone.
            - Use a numbered or bullet format if the source contains steps or instructions.
            - If that step in the original content contains a Markdown image (e.g., ![alt](url)), preserve the exact Markdown image syntax in the response.
            - If the context doesn't contain enough information, say so, then you can ask user to use search tool to search in the Internet. 
            - Do not use general knowledge, only use the context provided.
            - Base your response on the provided context only. Do not make assumptions.
            - Be specific and provide relevant details
            - Use the async_search method to search the knowledge base.
            - At the end of each response, add a follow-up question on a new line.
            - Format your response according to the requested format: Multiple Paragraphs, , preserving stepwise clarity where applicable.

Full code configuraiton

lightrag_agent = Agent(
    name="Lightrag Agent",
    agent_id="lightrag_agent",
    model=model,
    tools=[TavilyTools()],
    instructions=dedent(
        """\
            You are an AI assistant that answers questions based strictly on the provided context. Your responses must be clear, concise, and fact-based.

            Response Guidelines:
            - Preserve any Markdown formatting from the source, including headings, lists, bold/italic text, and images.
            - Do not add extra interpretation, speculation, or editorial tone.
            - Use a numbered or bullet format if the source contains steps or instructions.
            - If that step in the original content contains a Markdown image (e.g., ![alt](url)), preserve the exact Markdown image syntax in the response.
            - If the context doesn't contain enough information, say so, then you can ask user to use search tool to search in the Internet. 
            - Do not use general knowledge, only use the context provided.
            - Base your response on the provided context only. Do not make assumptions.
            - Be specific and provide relevant details
            - Use the async_search method to search the knowledge base.
            - At the end of each response, add a follow-up question on a new line.
            - Format your response according to the requested format: Multiple Paragraphs, , preserving stepwise clarity where applicable.

        """
    ),
    knowledge=knowledge_base,
    retriever=retriever,
    search_knowledge=True,
    # Store the agent sessions in a sqlite database
    storage=SqliteStorage(table_name="lightrag_agent", db_file=agent_storage),
    # Adds the current date and time to the instructions
    add_datetime_to_instructions=True,
    # Adds the history of the conversation to the messages
    add_history_to_messages=True,
    # Number of history responses to add to the messages
    num_history_responses=5,
    # Adds markdown formatting to the messages
    markdown=True,
    debug_mode=True,  # add the debug in order to send the traces to Phoenix
)

Anyone can help me :smiley:

Thanks

2

Hey @hunglv53, thanks for reaching out and supporting Agno. I’ve shared this with the team, we’re working through all requests one by one and will get back to you soon.
If it’s urgent, please let us know. We appreciate your patience!

3

Hey @hunglv53. This is a common scenario where part of the instructions are ignored by the LLM itself. It’s not directly related to the Agno processes, but let’s try to solve it:

  • First thing I would do is to confirm that what the agent is reading from the DB contains the image links you are missing. As I understand it’s vectorized content, maybe it’s just not reading the exact blocks containing that. You already have debug_mode=True so you should be able to see in the logs what the agent gets out of the storage.
  • Second, this sounds like something we can solve with prompt engineering and/or using a more powerful model. What I can advice here is that simpler prompts tend to provide more consistent results. We can try to reduce the guideline points and be very clear about keeping the Markdown images in the response.