Handoff Orchestration sample - doesn't work with OpenAI?

[brainmurphy]

I've tried to run Step04_Handoff against OpenAI with gpt-4o-2024-08-06. I'm getting this back:

An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_Yd1hIw4CVtwFdpzuxgPrZAIu))))))

All I've done is git clone, set the secrets (OpenAI:ChatModelId and OpenAI:ApiKey), then dotnet test --filter Step04_Handoff.

Should this just work?

The conversation log:

# INPUT:
I am a customer that needs help with my orders

REGISTER HandoffOrchestration Start: HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68/default
REGISTER ACTOR HandoffOrchestration MEMBER #4: HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_4
REGISTER ACTOR HandoffOrchestration MEMBER #3: HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_3
REGISTER ACTOR HandoffOrchestration MEMBER #2: HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_2
REGISTER ACTOR HandoffOrchestration MEMBER #1: HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_1
REGISTER HandoffOrchestration Complete: HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68/default
INVOKE HandoffOrchestration: HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68/default
YIELD HandoffOrchestration: HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68/default
AWAIT HandoffOrchestration: HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68/default
INIT HandoffOrchestration: HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Boot/default
START HandoffOrchestration: HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Boot/default
REQUEST Handoff agent [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_1/default]

# RESPONSE assistant - TriageAgent: Could you please provide more details about the specific issue you're facing with your order? This will help me direct you to the appropriate support agent. Are you looking to check the status of an order, process a return, or request a refund?

# STREAMED Assistant - TriageAgent: ("Could")(" you")(" please")(" provide")(" more")(" details")(" about")(" the")(" specific")(" issue")(" you\u0027re")(" facing")(" with")(" your")(" order")("?")(" This")(" will")(" help")(" me")(" direct")(" you")(" to")(" the")(" appropriate")(" support")(" agent")(".")(" Are")(" you")(" looking")(" to")(" check")(" the")(" status")(" of")(" an")(" order")(",")(" process")(" a")(" return")(",")(" or")(" request")(" a")(" refund")("?")

RESULT Handoff agent [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_1/default]: Could you please provide more details about the specific issue you're facing with your order? This will help me direct you to the appropriate support agent. Are you looking to check the status of an order, process a return, or request a refund?

# INPUT: I'd like to track the status of my order

TOOL Handoff [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_1/default]: OrderStatusAgent
RESULT Handoff agent [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_1/default]: 
REQUEST Handoff agent [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_2/default]

# RESPONSE assistant - OrderStatusAgent: Please provide the order ID for the order you would like to track, and I will check the status for you.

# STREAMED Assistant - OrderStatusAgent: ("Please")(" provide")(" the")(" order")(" ID")(" for")(" the")(" order")(" you")(" would")(" like")(" to")(" track")(",")(" and")(" I")(" will")(" check")(" the")(" status")(" for")(" you")(".")

RESULT Handoff agent [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_2/default]: Please provide the order ID for the order you would like to track, and I will check the status for you.

# INPUT: My order ID is 123


# RESPONSE assistant - OrderStatusAgent: Your order (ID: 123) has been shipped and is expected to arrive in 2-3 days. If you have any more questions or need further assistance, feel free to ask!

# STREAMED Assistant - OrderStatusAgent: ("Your")(" order")(" (")("ID")(":")(" ")("123")(")")(" has")(" been")(" shipped")(" and")(" is")(" expected")(" to")(" arrive")(" in")(" ")("2")("-")("3")(" days")(".")(" If")(" you")(" have")(" any")(" more")(" questions")(" or")(" need")(" further")(" assistance")(",")(" feel")(" free")(" to")(" ask")("!")

RESULT Handoff agent [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_2/default]: 

# INPUT: I want to return another order of mine

TOOL Handoff [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_2/default]: TriageAgent
RESULT Handoff agent [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_2/default]: 
REQUEST Handoff agent [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_1/default]
TOOL Handoff [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_1/default]: OrderReturnAgent
RESULT Handoff agent [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_1/default]: 
REQUEST Handoff agent [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_3/default]
TOOL Handoff [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_3/default]: TriageAgent
RESULT Handoff agent [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_3/default]: 
REQUEST Handoff agent [HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Agent_1/default]
FAILURE HandoffOrchestration: HandoffOrchestration_500b0d48b5bd40569fa7a1a0a4f2de68_Boot/default

[brainmurphy]

This might be #12663 🤔

[brainmurphy]

Running the same sample with Bedrock/Claude, I get:

One or more errors occurred. (One or more exceptions occurred while processing the message. (One or more exceptions occurred while processing the message. (The text field in the ContentBlock object at messages.3.content.0 is blank. Add text to the text field, and try again.)))

I modified BaseTest.AddChatClientToKernel to look like this:

    protected IChatClient AddChatClientToKernel(IKernelBuilder builder)
    {
#pragma warning disable CA2000 // Dispose objects before losing scope
        IChatClient chatClient;

        var bedrock = new AmazonBedrockRuntimeClient();
        chatClient = bedrock.AsIChatClient("eu.anthropic.claude-3-7-sonnet-20250219-v1:0");

        IChatClient functionCallingChatClient = chatClient.AsBuilder().UseKernelFunctionInvocation().Build();
        builder.Services.AddSingleton(functionCallingChatClient);
        return functionCallingChatClient;
#pragma warning restore CA2000 // Dispose objects before losing scope
    }

It looks like the tool back-and-forth results in the last FunctionResultContent becoming transformed into an empty string which gets passed as part of the conversation? Not sure I follow the logic in ChatClientChatCompletionService.GetChatMessageContentsAsync

[moonbox3]

@TaoChenOSU any thoughts here?