Team implementation breaks while executing tool calls

General
,
1

I’m getting following error while executing team with v1.3.4. It seems there is some issue while executing tool calls.

DEBUG I would like to know about loan offers.                                                                                                           
                                                                                                                                                        
      <context>                                                                                                                                         
      {                                                                                                                                                 
        "session_id": "90"                                                                                                                              
      }                                                                                                                                                 
      </context>                                                                                                                                        
DEBUG ============================================================ assistant =============================================================              
DEBUG Tool Calls:                                                                                                                                       
        - ID: 'call_al5aC3bvDQB7oyK0AZy0JvBk'                                                                                                           
          Name: 'transfer_task_to_member'                                                                                                               
          Arguments: 'member_id: sales-agent, task_description: Please provide information on the available loan offers for the customer.,              
      expected_output: Details about the loan offers available for the customer, including rates, terms, and any current promotions.'                   
DEBUG =============================================================== tool ===============================================================              
DEBUG Tool call Id: call_al5aC3bvDQB7oyK0AZy0JvBk                                                                                                       
DEBUG To provide you with detailed information about the available loan offers including rates, terms, and any current promotions, I'll need the        
      customer ID. Could you please provide that information?                                                                                           
DEBUG ============================================================ assistant =============================================================              
DEBUG Certainly! Could you please provide your customer ID so I can give you detailed information about the available loan offers?                      
DEBUG =============================================================== user ===============================================================              
DEBUG sure, my customer id is 123                                                                                                                       
                                                                                                                                                        
      <context>                                                                                                                                         
      {                                                                                                                                                 
        "session_id": "90"                                                                                                                              
      }                                                                                                                                                 
      </context>                                                                                                                                        
DEBUG ============================================================ assistant =============================================================              
DEBUG Tool Calls:                                                                                                                                       
        - ID: 'call_K6zfmdX8LoINruUcUbzuHOzj'                                                                                                           
          Name: 'transfer_task_to_member'                                                                                                               
          Arguments: 'member_id: sales-agent, task_description: The user has provided their customer ID as 123. Please fetch and provide detailed       
      information about the loan offers available for the customer, including rates, terms, and any current promotions., expected_output: Detailed loan 
      offers available for customer ID 123, including rates, terms, and any current promotions.'                                                        
DEBUG ************************************************************  METRICS  *************************************************************              
DEBUG * Tokens:                      input=1552, output=82, total=1634                                                                                  
DEBUG * Prompt tokens details:       {'audio_tokens': 0, 'cached_tokens': 1408}                                                                         
DEBUG * Completion tokens details:   {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}       
DEBUG * Time:                        2.8130s                                                                                                            
DEBUG * Tokens per second:           29.1502 tokens/s                                                                                                   
DEBUG ************************************************************  METRICS  *************************************************************              
DEBUG Getting function transfer_task_to_member                                                                                                          
DEBUG Running: transfer_task_to_member(member_id=sales-agent, task_description=..., expected_output=...)                                                
DEBUG ****************************************************** Agent ID: sales-agent *******************************************************              
DEBUG ***************************************** Session ID: 6e2e0ef4-db89-4fc5-873d-72cdcd462bce *****************************************              
DEBUG ************************************** Agent Run Start: d86d5696-9ba5-444e-a83d-769c3e76d53c ***************************************              
DEBUG Getting messages from previous runs: 2                                                                                                            
DEBUG Adding 2 messages from history                                                                                                                    
DEBUG ------------------------------------------------------- Azure Response Start -------------------------------------------------------              
DEBUG ---------------------------------------------------------- Model: gpt-4o -----------------------------------------------------------              
DEBUG ============================================================== system ==============================================================              
DEBUG You are a sales assistant for Horizon Finance that engages customers with loan offers and resolve their queries.                                  
                                                                                                                                                        
      <your_goal>                                                                                                                                       
      Your goal is to present pre-approved loan offers and assist with general loan-related queries.                                                    
      </your_goal>                                                                                                                                      
                                                                                                                                                        
                                                                                                                                                        
      <your_role>                                                                                                                                       
      Handle loan related queries and fetching customer data.                                                                                           
      </your_role>                                                                                                                                      
                                                                                                                                                        
      <instructions>                                                                                                                                    
      You are a sales assistant for Horizon Finance that engages customers with loan offers and resolve their queries with following responsibilities:  
                                                                                                                                                        
       - To fetch customer profile data using tool `get_customer_data`.                                                                                 
       - To fetch historical credit history of customer using tool `get_historical_credit_history`.                                                     
       - To compute monthly EMIs for a provided amount, rate of interest and tenure in months using tool `calculate_emi`.                               
       - To offer recommended loan offers based on similar customer using tool `get_recommendation_similar_customers`,                                  
       - To send email containing online application link if customer is interested in applying using tool                                              
      `send_email_for_online_application_form_link`.                                                                                                    
                                                                                                                                                        
      General instructions:                                                                                                                             
                                                                                                                                                        
      1. You are a friendly sales assistant for Horizon finance.                                                                                        
      2. To retrieve any data for customer, you would need customer id from user.                                                                       
      3. If you need any input from user to run tool calls, please ask the customer to provide that information..                                       
      4. Please use provided tool calls wherever needed.                                                                                                
      </instructions>                                                                                                                                   
                                                                                                                                                        
                                                                                                                                                        
          1. `send_email_for_online_application_form_link` will update the session state to `pending_on_user`.                                          
                                                                                                                                                        
      You have the capability to retain memories from previous interactions with the user, but have not had any interactions with the user yet.         
DEBUG =============================================================== user ===============================================================              
DEBUG You are a member of a team of agents. Your goal is to complete the following task:                                                                
                                                                                                                                                        
      <task>                                                                                                                                            
      Please provide information on the available loan offers for the customer.                                                                         
      </task>                                                                                                                                           
                                                                                                                                                        
      <expected_output>                                                                                                                                 
      Details about the loan offers available for the customer, including rates, terms, and any current promotions.                                     
      </expected_output>                                                                                                                                
DEBUG ============================================================ assistant =============================================================              
DEBUG To provide you with detailed information about the available loan offers including rates, terms, and any current promotions, I'll need the        
      customer ID. Could you please provide that information?                                                                                           
DEBUG =============================================================== user ===============================================================              
DEBUG You are a member of a team of agents. Your goal is to complete the following task:                                                                
                                                                                                                                                        
      <task>                                                                                                                                            
      The user has provided their customer ID as 123. Please fetch and provide detailed information about the loan offers available for the customer,   
      including rates, terms, and any current promotions.                                                                                               
      </task>                                                                                                                                           
                                                                                                                                                        
      <expected_output>                                                                                                                                 
      Detailed loan offers available for customer ID 123, including rates, terms, and any current promotions.                                           
      </expected_output>                                                                                                                                
                                                                                                                                                        
      <team context>                                                                                                                                    
      {"customer_id": "123"}                                                                                                                            
      </team context>                                                                                                                                   
                                                                                                                                                        
                                                                                                                                                        
      <member interactions>                                                                                                                             
      Member: Sales Agent                                                                                                                               
      Task: Please provide information on the available loan offers for the customer.                                                                   
      Response: To provide you with detailed information about the available loan offers including rates, terms, and any current promotions, I'll need  
      the customer ID. Could you please provide that information?                                                                                       
                                                                                                                                                        
      </member interactions>                                                                                                                            
                                                                                                                                                        
DEBUG ============================================================ assistant =============================================================              
DEBUG Tool Calls:                                                                                                                                       
        - ID: 'call_8m2xTRgfgQsjisyR4fc8jDmC'                                                                                                           
          Name: 'get_customer_data'                                                                                                                     
          Arguments: 'customer_id: 123'                                                                                                                 
        - ID: 'call_1YMQZzBmkIIaptP9KljU20cR'                                                                                                           
          Name: 'get_recommendation_similar_customers'                                                                                                  
          Arguments: 'customer_id: 123'                                                                                                                 
DEBUG ************************************************************  METRICS  *************************************************************              
DEBUG * Tokens:                      input=977, output=54, total=1031                                                                                   
DEBUG * Prompt tokens details:       {'audio_tokens': 0, 'cached_tokens': 0}                                                                            
DEBUG * Completion tokens details:   {'accepted_prediction_tokens': 0, 'audio_tokens': 0, 'reasoning_tokens': 0, 'rejected_prediction_tokens': 0}       
DEBUG * Time:                        1.0809s                                                                                                            
DEBUG * Tokens per second:           49.9585 tokens/s                                                                                                   
DEBUG ************************************************************  METRICS  *************************************************************              
DEBUG Getting function get_customer_data                                                                                                                
ERROR    Function get_customer_data not found                                                                                                           
DEBUG Getting function get_recommendation_similar_customers                                                                                             
ERROR    Function get_recommendation_similar_customers not found                                                                                        
ERROR    API status error from OpenAI API: Error code: 400 - {'error': {'message': "An assistant message with 'tool_calls' must be followed by tool     
         messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_8m2xTRgfgQsjisyR4fc8jDmC,         
         call_1YMQZzBmkIIaptP9KljU20cR", 'type': 'invalid_request_error', 'param': 'messages.[5].role', 'code': None}}                                  
WARNING  Attempt 1/1 failed: An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following  
         tool_call_ids did not have response messages: call_8m2xTRgfgQsjisyR4fc8jDmC, call_1YMQZzBmkIIaptP9KljU20cR                                     
ERROR    Failed after 1 attempts. Last error using AzureOpenAI(gpt-4o)                                                                                  
WARNING  Attempt 3/4 failed: An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following  
         tool_call_ids did not have response messages: call_8m2xTRgfgQsjisyR4fc8jDmC, call_1YMQZzBmkIIaptP9KljU20cR

I’ve provided my code to @dirk over DMs. @Monali Hope it helps in quickly identifying the issue.

2

@Monali this has become a burning issue for me. I’m waiting for its resolution for almost a week. this is the same issue I informed you here. Please assist me with this. :folded_hands:

3

Hey @MahorShekhar,
Thanks for reaching out and for using Agno!

Just a quick request — it’d be super helpful if you could share code or issues here in the thread instead of DMs. That way, the whole team can pitch in and help out, especially if Dirk isn’t available.

Appreciate it!

1 Like
4

@Monali Here is a part of my code.


def get_loan_upselling_team(session_id:str, user_id:str) -> Team:
    loan_upselling_team: Team = Team(
        name="Loan Upselling team.",
        team_id="loan-upselling-team",
        user_id=user_id,
        mode="coordinate",
        session_id=session_id,
        session_state={"state": "running"},
        storage=session_db,
        memory=common_memory,
        model=azure_openai_model,
        members=[sales_agent, verification_agent, decision_making_agent],
        description="You are the orchestrator agent of Horizon Finance responsible for routing customer query to right agent.",
        instructions=dedent("""\
        
        You are the orchestrator agent for Horizon Finance responsible for managing a team of agents with specific roles and responsibilities.
        
        General Instructions:
        
        1. Participating agents can ask you to gather customer input on their behalf that they need to provide response to customer query.
        2. After taking user input, please provide that user input back to that agent which has asked you to gather that from the user.
        3. Along with deciding which tool call to make, also please decide the session state after tool call would have been executed. Use `update_session_state` to update session state along with each tool call based on the following rules:
          - if goal is not achieved and conversation is still going on, then session state should be `running`.
          - if goal is achieved, then session state should be `completed`.
          - if async tool call is made or you are waiting for response from some third party, then session state should be `waiting_async_tool`
          - if action is pending on the user, then session state should be `pending_on_user`
          - if any exception happens, then session state should be `disconnected`.\
        """
        ),
        success_criteria="""
        Following is the success criteria:
        - customer is satisfied with the response and his query is resolved.
        - customer has no further query to ask.
        - customer no longer wishes to continue the conversation..
        - customer is no longer interested in the loan offers.
        - customer loan application process is either approved or declined.
        """,
        add_context=True,
        context={"session_id":session_id},
        add_state_in_messages=True,
        # add_datetime_to_instructions=True,
        enable_agentic_context=True,
        share_member_interactions=True,
        read_team_history=True,
        enable_team_history=True,
        show_tool_calls=True,
        show_members_responses=True,
        debug_mode=True,
        num_of_interactions_from_history=10,

        # new params support in v1.3
        enable_agentic_memory=True,
        enable_user_memories=True,
        enable_session_summaries=True,
        add_member_tools_to_system_message=False
    )

sales_agent = Agent(
    agent_id="sales-agent",
    name="Sales Agent",
    model=azure_openai_model,
    description="You are a sales assistant for Horizon Finance that engages customers with loan offers and resolve their queries.",
    role="Handle loan related queries and fetching customer data.",
    goal="Your goal is to present pre-approved loan offers and assist with general loan-related queries.",
    instructions=dedent("""\
        You are a sales assistant for Horizon Finance that engages customers with loan offers and resolve their queries with following responsibilities:
        
         - To fetch customer profile data using tool `get_customer_data`.
         - To fetch historical credit history of customer using tool `get_historical_credit_history`.
         - To compute monthly EMIs for a provided amount, rate of interest and tenure in months using tool `calculate_emi`.
         - To offer recommended loan offers based on similar customer using tool `get_recommendation_similar_customers`,
         - To send email containing online application link if customer is interested in applying using tool `send_email_for_online_application_form_link`.

        General instructions:

        1. You are a friendly sales assistant for Horizon finance.
        2. To retrieve any data for customer, you would need customer id from user.
        3. If you need any input from user to run tool calls, please ask the customer to provide that information..
        4. Please use provided tool calls wherever needed.\
        """
    ),
    tools=[SalesToolkit()],
    read_chat_history=True,
    read_tool_call_history=True,
    add_history_to_messages=True,
    num_history_responses=10,
    add_state_in_messages=True,
    memory=common_memory,
    add_context=True,
    add_memory_references=True,
    add_session_summary_references=True,
    show_tool_calls=True,
    # markdown=True,
    # add_name_to_instructions=True,
    # debug_mode=True,
)

class SalesToolkit(Toolkit):
    name = "sales_toolkit"
    instructions = """
    1. `send_email_for_online_application_form_link` will update the session state to `pending_on_user`.
    """

    def __init__(self):
        super().__init__(name=self.name, instructions=self.instructions, add_instructions=True)
        self.register(self.get_customer_data)
        self.register(self.calculate_emi)
        self.register(self.get_historical_credit_history)
        self.register(self.get_recommendation_similar_customers)
        self.register(self.send_email_for_online_application_form_link)

   ....... [ tools are defined here ]
5

@MahorShekhar
Looking into this.

1 Like
6

Team execution in ‘coordinate’ mode is not consistent for several reasons: LLMs don’t return the same responses, even using the same LLM, bugs and incomplete process execution. I am finding that more complex agent process flows are NOT being executed properly. If a request is sent to Agent A, B and D, and then the results from that processing is supposed to go to Agent F, the last step is almost never executed. It seems that a single pass execution engine has been implemented. To perform some of these more complex processing requests, it needs to be multi-pass.

The passes should be: Step 1: Identify what the required/recommended process flow for all of the agents, which includes a post processing pass to insure that everything has been satisfied in the query or if it can’t be satisfied in X number of attempts, then exit gracefully back to the user. Step 2: Send the query to the appropriate Agents; Step 3: Examine the responses for completeness; If not complete, go back to Step 1 and determine the new flow given the initial query and the intermediate responses already received. If complete, then go to Step 4 which is to complete the current process flow, which typically should be to collect and summarize the results of the initial query.

This last step is almost never being executed in the current implementation. In addition many times the current implementation can get confused depending on slight modification of the user and internal prompts, and instead of providing a response to the query, the response is simply what should be done… and with partial results sometimes included in the debug messages, but not provided in the final results.

Getting this working properly is important for handling more complex, real-world problems and results.

Final note: Identifying and fixing some of these more complex problems/issues is made more difficult by a lack of documentation and general design architecture. Even descriptions of intent of how various multi-layered code is intended to work would be helpful. We all make mistakes, especially on edge cases which can be inordinately difficult to find and fix.

7

Hi @MahorShekhar
You are right, we are also finding limitations with teams in more complex architectures. Especially a “sequential flow” as you have outlined. When the team leader is in charge of calling a series of agents in sequence, it can get it wrong (or at least it won’t consistently get it right).

From talking to users and customers we have found that most people want some parts of the system to be 100% deterministic and some parts to be agentic. Teams is effectively 100% agentic, as is single agents.
We are currently working on releasing redone Workflows implementation to address exactly this issue → How to have a deterministic sequence of steps for a run, where agents/teams are involved in the steps themselves.

We are looking to release a first version in the coming week or so.

1 Like
8

hey @Dirk thanks for the update. Eagerly waiting to see revamped workflows. Just one thought, when you are designing sequential agents where each agent will run in sequence, do think about the steps which would involve taking any input from user. Execution should resume from there. There could be 2 types of sequential flows:

  1. involving interruption: it would involve multiple team.run executions.
  2. not involving interruption: can be finished using a single team.run execution.

I hope you get what I meant. Above thought is a limitation of Google’s ADK Sequential agent that I found.

I hope it helps @Dirk

9

Yes that is definitely the plan. We want to have the human-in-the-loop features in workflows for sure. Also allow users to “rerun” only parts of the workflow. We got some inspiration from Google’s implementation.

@bills @MahorShekhar please checkout 1.5.10. I found an issue with teams tool calling that I resolved which should improve these latent issues.