Question about: Streaming response text with the 8.0.1 streaming feature for Assistants

[rkuiper1]

Hi,

I've been trying to get the highly anticipated streaming feature for the assistant thread to work, but for some reason I can't get it to actually stream the response text.
The Create thread and run streaming sample code works and the events are received, but it seems the streamEvent (as MessageResponse) never contains Content or a Delta until the MessageResponse.Status == "Completed". That last event will contain the entire response (when i retrieve it with the PrintContent method).

The 'Create thread and run streaming' sample ends with this code:

var messages = await thread.ListMessagesAsync();
foreach (var response in messages.Items.Reverse())
{
    Console.WriteLine($"{response.Role}: {response.PrintContent()}");
}

which hints that the response is not actually streamed but needs to be retrieved with a separate request. (which doesn't seem to be necessary when that last MessageResponse was captured)

The OpenAI doc 'context with streaming', especially 'Step 4' seem to hint that 'delta.value' is available for the on_text_delta in the AssistantEventHandler, which makes me think that response streaming is supported.

Are my expectations way off? Or am I misunderstanding the streaming feature, or am I doing something wrong?

Thanks,
Richard

[rkuiper1]

Hi @StephenHodgson ,
I basically copied the example from the documentation:

            var i = 0;
            using var api = new OpenAIClient(new OpenAIAuthentication(OpenAI_key, OpenAI_org));
            var assistantRequest = new CreateAssistantRequest(instructions: "You are a helpful assistant.");
            var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(assistantRequest);
            ThreadResponse thread = null;
            async void StreamEventHandler(IServerSentEvent streamEvent)
            {
                switch (streamEvent)
                {
                    case ThreadResponse threadResponse:
                        thread = threadResponse;
                        break;
                    case RunResponse runResponse:
                        break;
                    case MessageResponse messageResponse:
                        Console.WriteLine($"{i++} {messageResponse.Status}: {messageResponse.PrintContent()}");
                        break;
                    default:
                        Console.WriteLine(streamEvent.ToJsonString());
                        break;
                }
            }

            var run = await assistant.CreateThreadAndRunAsync("Who was Mozart?", StreamEventHandler);
            run = await run.WaitForStatusChangeAsync();

When I run this code I get this output:

{"id":"step_E6EZvvZLoSB","object":"thread.run.step","created_at":1718292737,"assistant_id":"asst_PimbMdHzENgoXAD","thread_id":"thread_DfIhiyygVf7nC5wG7NUaQrDH","run_id":"run_0B5uoe","type":"message_creation","status":"in_progress","step_details":{"message_creation":{"message_id":"msg_4w8HUPJBOsK"}}}
{"id":"step_E6EZvvZLoSB","object":"thread.run.step","created_at":1718292737,"assistant_id":"asst_PimbXHzENgoXAD","thread_id":"thread_DfIhiyygVf7nC5wG7NUaQrDH","run_id":"run_0B5uokZaZvve","type":"message_creation","status":"in_progress","step_details":{"message_creation":{"message_id":"msg_4w8HULFOsK"}}}
0 InProgress:
1 InProgress:
2 InProgress:
3 InProgress:
...
269 InProgress:
270 InProgress:
271 InProgress:
272 Completed: Wolfgang Amadeus Mozart was an influential and prolific composer of the Classical period. 
...
His influence on Western music cannot be overstated, and he is often cited as one of the greatest composers in the history of Western music.
{"id":"step_E6EZvkj8oLoSB","object":"thread.run.step","created_at":1718292737,"assistant_id":"asst_PimbX2HzEgoXAD","thread_id":"thread_DfIhiyygVUaQrDH","run_id":"run_0B5uokZvcZvve","type":"message_creation","status":"completed","step_details":{"message_creation":{"message_id":"msg_4w8HULFABpOsK"}},"completed_at":1718292742,"usage":{"prompt_tokens":34,"completion_tokens":272,"total_tokens":306}}

MessageResponse 272 Shows the Completed status and contains the full text response.
I would expect MessageResponse 1 to 272 to contain the delta's (which would be in line with "completion_tokens":272 in the final event), but as mentioned before; my expectations might be off.

[StephenHodgson]

Strange, in your debug I don't see it requesting the stream.

[StephenHodgson]

It's missing the create thread debug output

[StephenHodgson]

I see the first one to create the asst but I don't see the call to create and run the thread. Just the callback debug output from serialized event.

[rkuiper1]

Your remarks make me think that the code should indeed 'stream' the response text through the MessageResponse events, but for some reason it doesn't.
I'm still using .NET6 for my solution, but I find it hard to imagine that would play a role in this.
I copied and pasted the entire project (except for the credentials) so your might try to reproduce things (if you haven't already).
If there's anything I should try, please let me know.

[StephenHodgson]

should look something like this:

OpenAI-DotNet/OpenAI-DotNet-Tests/TestFixture_03_Threads.cs

Lines 218 to 255 in 41eac5c

	public async Task Test_06_01_02_CreateStreamingRun()
	{
	Assert.NotNull(OpenAIClient.ThreadsEndpoint);
	var assistant = await OpenAIClient.AssistantsEndpoint.CreateAssistantAsync(
	new CreateAssistantRequest(
	name: "Math Tutor",
	instructions: "You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.",
	model: Model.GPT4o,
	responseFormat: ChatResponseFormat.Json));
	Assert.NotNull(assistant);
	var thread = await OpenAIClient.ThreadsEndpoint.CreateThreadAsync();
	try
	{
	Assert.NotNull(thread);
	var message = await thread.CreateMessageAsync("I need to solve the equation `3x + 11 = 14`. Can you help me?");
	Assert.NotNull(message);
	var run = await thread.CreateRunAsync(assistant, streamEvent =>
	{
	Console.WriteLine(streamEvent.ToJsonString());
	});
	Assert.IsNotNull(run);
	Assert.IsTrue(run.Status == RunStatus.Completed);
	var messages = await thread.ListMessagesAsync();
	foreach (var response in messages.Items.Reverse())
	{
	Console.WriteLine($"{response.Role}: {response.PrintContent()}");
	}
	}
	finally
	{
	await assistant.DeleteAsync();
	await thread.DeleteAsync(deleteToolResources: true);
	}
	}

Test_06_01_02_CreateStreamingRun

 Test_06_01_02_CreateStreamingRun
   Source: TestFixture_03_Threads.cs line 218
   Duration: 2.3 sec

  Standard Output: 
CreateAssistantAsync -> [POST:200] https://localhost:80/v1/assistants
{
  "Request": {
    "Headers": {
      "User-Agent": [
        "OpenAI-DotNet"
      ],
      "OpenAI-Beta": [
        "assistants=v2"
      ],
      "Authorization": [
        "Bearer sess-aAbBcCdDeE123456789"
      ]
    },
    "Body-Headers": {
      "Content-Type": [
        "application/json; charset=utf-8"
      ]
    },
    "Body": {
      "model": "gpt-4o",
      "name": "Math Tutor",
      "instructions": "You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.",
      "response_format": {
        "type": "json_object"
      }
    }
  },
  "Response": {
    "Headers": {
      "Date": [
        "Mon, 10 Jun 2024 06:49:54 GMT"
      ],
      "openai-version": [
        "2020-10-01"
      ],
      "openai-organization": [
        "pillow-9"
      ],
      "X-Request-ID": [
        "req_ca3e516ce1c0fe6ca6b8ea96641d0f3f"
      ],
      "openai-processing-ms": [
        "66"
      ],
      "Strict-Transport-Security": [
        "max-age=15724800; includeSubDomains"
      ],
      "CF-Cache-Status": [
        "DYNAMIC"
      ],
      "Server": [
        "cloudflare"
      ],
      "CF-RAY": [
        "89176c909ad43b08-IAD"
      ]
    },
    "Body": {
      "Content": {
        "id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
        "object": "assistant",
        "created_at": 1718002194,
        "name": "Math Tutor",
        "description": null,
        "model": "gpt-4o",
        "instructions": "You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.",
        "tools": [],
        "top_p": 1.0,
        "temperature": 1.0,
        "tool_resources": {},
        "metadata": {},
        "response_format": {
          "type": "json_object"
        }
      }
    }
  }
}
CreateThreadAsync -> [POST:200] https://localhost:80/v1/threads
{
  "Request": {
    "Headers": {
      "User-Agent": [
        "OpenAI-DotNet"
      ],
      "OpenAI-Beta": [
        "assistants=v2"
      ],
      "Authorization": [
        "Bearer sess-aAbBcCdDeE123456789"
      ]
    }
  },
  "Response": {
    "Headers": {
      "Date": [
        "Mon, 10 Jun 2024 06:49:54 GMT"
      ],
      "openai-version": [
        "2020-10-01"
      ],
      "openai-organization": [
        "pillow-9"
      ],
      "X-Request-ID": [
        "req_5eed0c0ce99861e01da483e626ebbadd"
      ],
      "openai-processing-ms": [
        "34"
      ],
      "Strict-Transport-Security": [
        "max-age=15724800; includeSubDomains"
      ],
      "CF-Cache-Status": [
        "DYNAMIC"
      ],
      "Server": [
        "cloudflare"
      ],
      "CF-RAY": [
        "89176c917ba03b08-IAD"
      ]
    },
    "Body": {
      "Content": {
        "id": "thread_F42J8yHpyK548ZnU2LLda5EU",
        "object": "thread",
        "created_at": 1718002194,
        "metadata": {},
        "tool_resources": {}
      }
    }
  }
}
CreateMessageAsync -> [POST:200] https://localhost:80/v1/threads/thread_F42J8yHpyK548ZnU2LLda5EU/messages
{
  "Request": {
    "Headers": {
      "User-Agent": [
        "OpenAI-DotNet"
      ],
      "OpenAI-Beta": [
        "assistants=v2"
      ],
      "Authorization": [
        "Bearer sess-aAbBcCdDeE123456789"
      ]
    },
    "Body-Headers": {
      "Content-Type": [
        "application/json; charset=utf-8"
      ]
    },
    "Body": {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "I need to solve the equation `3x + 11 = 14`. Can you help me?"
        }
      ]
    }
  },
  "Response": {
    "Headers": {
      "Date": [
        "Mon, 10 Jun 2024 06:49:54 GMT"
      ],
      "openai-version": [
        "2020-10-01"
      ],
      "openai-organization": [
        "pillow-9"
      ],
      "X-Request-ID": [
        "req_a05a00b91e21af77a8d0afd5e9c7baba"
      ],
      "openai-processing-ms": [
        "79"
      ],
      "Strict-Transport-Security": [
        "max-age=15724800; includeSubDomains"
      ],
      "CF-Cache-Status": [
        "DYNAMIC"
      ],
      "Server": [
        "cloudflare"
      ],
      "CF-RAY": [
        "89176c923c353b08-IAD"
      ]
    },
    "Body": {
      "Content": {
        "id": "msg_koo7m9nqB9eIL3DFT3bpW4ke",
        "object": "thread.message",
        "created_at": 1718002194,
        "assistant_id": null,
        "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
        "run_id": null,
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": {
              "value": "I need to solve the equation `3x + 11 = 14`. Can you help me?",
              "annotations": []
            }
          }
        ],
        "attachments": [],
        "metadata": {}
      }
    }
  }
}
{"id":"run_mM8WNPj28oO60IEx8r6VZeOG","object":"thread.run","created_at":1718002194,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","status":"queued","expires_at":1718002794,"model":"gpt-4o","instructions":"You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.","tools":[],"metadata":{},"temperature":1,"top_p":1,"truncation_strategy":{"type":"auto"},"tool_choice":"auto","parallel_tool_calls":true,"response_format":{"type":"json_object"}}
{"id":"run_mM8WNPj28oO60IEx8r6VZeOG","object":"thread.run","created_at":1718002194,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","status":"queued","expires_at":1718002794,"model":"gpt-4o","instructions":"You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.","tools":[],"metadata":{},"temperature":1,"top_p":1,"truncation_strategy":{"type":"auto"},"tool_choice":"auto","parallel_tool_calls":true,"response_format":{"type":"json_object"}}
{"id":"run_mM8WNPj28oO60IEx8r6VZeOG","object":"thread.run","created_at":1718002194,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","status":"in_progress","expires_at":1718002794,"started_at":1718002195,"model":"gpt-4o","instructions":"You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.","tools":[],"metadata":{},"temperature":1,"top_p":1,"truncation_strategy":{"type":"auto"},"tool_choice":"auto","parallel_tool_calls":true,"response_format":{"type":"json_object"}}
{"id":"step_10HZORF6qyVVTiXmRtSLbWlo","object":"thread.run.step","created_at":1718002195,"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","type":"message_creation","status":"in_progress","step_details":{"message_creation":{"message_id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe"}}}
{"id":"step_10HZORF6qyVVTiXmRtSLbWlo","object":"thread.run.step","created_at":1718002195,"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","type":"message_creation","status":"in_progress","step_details":{"message_creation":{"message_id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe"}}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"in_progress","role":2,"content":[],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe","object":"thread.message","created_at":1718002195,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","status":"completed","completed_at":1718002195,"role":2,"content":[{"type":"text","text":{"value":"{\n  \"solution\": \"x = 1\"\n}","annotations":[]}}],"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","metadata":{}}
{"id":"step_10HZORF6qyVVTiXmRtSLbWlo","object":"thread.run.step","created_at":1718002195,"assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","run_id":"run_mM8WNPj28oO60IEx8r6VZeOG","type":"message_creation","status":"completed","step_details":{"message_creation":{"message_id":"msg_WzAKpu8qrN0JTPjCs9FtaxSe"}},"completed_at":1718002195,"usage":{"prompt_tokens":74,"completion_tokens":13,"total_tokens":87}}
{"id":"run_mM8WNPj28oO60IEx8r6VZeOG","object":"thread.run","created_at":1718002194,"thread_id":"thread_F42J8yHpyK548ZnU2LLda5EU","assistant_id":"asst_Gj4efmelAHprrUs8k4QvFuQS","status":"completed","expires_at":1718002794,"started_at":1718002195,"completed_at":1718002195,"model":"gpt-4o","instructions":"You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.","tools":[],"metadata":{},"usage":{"prompt_tokens":74,"completion_tokens":13,"total_tokens":87},"temperature":1,"top_p":1,"truncation_strategy":{"type":"auto"},"tool_choice":"auto","parallel_tool_calls":true,"response_format":{"type":"json_object"}}
StreamEventsAsync -> [POST:200] https://localhost:80/v1/threads/thread_F42J8yHpyK548ZnU2LLda5EU/runs
{
  "Request": {
    "Headers": {
      "User-Agent": [
        "OpenAI-DotNet"
      ],
      "OpenAI-Beta": [
        "assistants=v2"
      ],
      "Authorization": [
        "Bearer sess-aAbBcCdDeE123456789"
      ]
    },
    "Body-Headers": {
      "Content-Type": [
        "application/json; charset=utf-8"
      ]
    },
    "Body": {
      "assistant_id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
      "model": "gpt-4o",
      "instructions": "You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.",
      "tools": [],
      "temperature": 1,
      "top_p": 1,
      "stream": true,
      "response_format": {
        "type": "json_object"
      }
    }
  },
  "Response": {
    "Headers": {
      "Date": [
        "Mon, 10 Jun 2024 06:49:55 GMT"
      ],
      "openai-version": [
        "2020-10-01"
      ],
      "openai-organization": [
        "pillow-9"
      ],
      "X-Request-ID": [
        "req_97e850e704f6053d82be857fc2aefcf5"
      ],
      "openai-processing-ms": [
        "589"
      ],
      "Strict-Transport-Security": [
        "max-age=15724800; includeSubDomains"
      ],
      "CF-Cache-Status": [
        "DYNAMIC"
      ],
      "Server": [
        "cloudflare"
      ],
      "CF-RAY": [
        "89176c933cfe3b08-IAD"
      ]
    },
    "Body": {
      "Events": [
        {
          "event": "thread.run.created",
          "data": {
            "id": "run_mM8WNPj28oO60IEx8r6VZeOG",
            "object": "thread.run",
            "created_at": 1718002194,
            "assistant_id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
            "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
            "status": "queued",
            "started_at": null,
            "expires_at": 1718002794,
            "cancelled_at": null,
            "failed_at": null,
            "completed_at": null,
            "required_action": null,
            "last_error": null,
            "model": "gpt-4o",
            "instructions": "You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.",
            "tools": [],
            "tool_resources": {
              "code_interpreter": {
                "file_ids": []
              }
            },
            "metadata": {},
            "temperature": 1.0,
            "top_p": 1.0,
            "max_completion_tokens": null,
            "max_prompt_tokens": null,
            "truncation_strategy": {
              "type": "auto",
              "last_messages": null
            },
            "incomplete_details": null,
            "usage": null,
            "response_format": {
              "type": "json_object"
            },
            "tool_choice": "auto",
            "parallel_tool_calls": true
          }
        },
        {
          "event": "thread.run.queued",
          "data": {
            "id": "run_mM8WNPj28oO60IEx8r6VZeOG",
            "object": "thread.run",
            "created_at": 1718002194,
            "assistant_id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
            "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
            "status": "queued",
            "started_at": null,
            "expires_at": 1718002794,
            "cancelled_at": null,
            "failed_at": null,
            "completed_at": null,
            "required_action": null,
            "last_error": null,
            "model": "gpt-4o",
            "instructions": "You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.",
            "tools": [],
            "tool_resources": {
              "code_interpreter": {
                "file_ids": []
              }
            },
            "metadata": {},
            "temperature": 1.0,
            "top_p": 1.0,
            "max_completion_tokens": null,
            "max_prompt_tokens": null,
            "truncation_strategy": {
              "type": "auto",
              "last_messages": null
            },
            "incomplete_details": null,
            "usage": null,
            "response_format": {
              "type": "json_object"
            },
            "tool_choice": "auto",
            "parallel_tool_calls": true
          }
        },
        {
          "event": "thread.run.in_progress",
          "data": {
            "id": "run_mM8WNPj28oO60IEx8r6VZeOG",
            "object": "thread.run",
            "created_at": 1718002194,
            "assistant_id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
            "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
            "status": "in_progress",
            "started_at": 1718002195,
            "expires_at": 1718002794,
            "cancelled_at": null,
            "failed_at": null,
            "completed_at": null,
            "required_action": null,
            "last_error": null,
            "model": "gpt-4o",
            "instructions": "You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.",
            "tools": [],
            "tool_resources": {
              "code_interpreter": {
                "file_ids": []
              }
            },
            "metadata": {},
            "temperature": 1.0,
            "top_p": 1.0,
            "max_completion_tokens": null,
            "max_prompt_tokens": null,
            "truncation_strategy": {
              "type": "auto",
              "last_messages": null
            },
            "incomplete_details": null,
            "usage": null,
            "response_format": {
              "type": "json_object"
            },
            "tool_choice": "auto",
            "parallel_tool_calls": true
          }
        },
        {
          "event": "thread.run.step.created",
          "data": {
            "id": "step_10HZORF6qyVVTiXmRtSLbWlo",
            "object": "thread.run.step",
            "created_at": 1718002195,
            "run_id": "run_mM8WNPj28oO60IEx8r6VZeOG",
            "assistant_id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
            "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
            "type": "message_creation",
            "status": "in_progress",
            "cancelled_at": null,
            "completed_at": null,
            "expires_at": 1718002794,
            "failed_at": null,
            "last_error": null,
            "step_details": {
              "type": "message_creation",
              "message_creation": {
                "message_id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe"
              }
            },
            "usage": null
          }
        },
        {
          "event": "thread.run.step.in_progress",
          "data": {
            "id": "step_10HZORF6qyVVTiXmRtSLbWlo",
            "object": "thread.run.step",
            "created_at": 1718002195,
            "run_id": "run_mM8WNPj28oO60IEx8r6VZeOG",
            "assistant_id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
            "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
            "type": "message_creation",
            "status": "in_progress",
            "cancelled_at": null,
            "completed_at": null,
            "expires_at": 1718002794,
            "failed_at": null,
            "last_error": null,
            "step_details": {
              "type": "message_creation",
              "message_creation": {
                "message_id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe"
              }
            },
            "usage": null
          }
        },
        {
          "event": "thread.message.created",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message",
            "created_at": 1718002195,
            "assistant_id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
            "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
            "run_id": "run_mM8WNPj28oO60IEx8r6VZeOG",
            "status": "in_progress",
            "incomplete_details": null,
            "incomplete_at": null,
            "completed_at": null,
            "role": "assistant",
            "content": [],
            "attachments": [],
            "metadata": {}
          }
        },
        {
          "event": "thread.message.in_progress",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message",
            "created_at": 1718002195,
            "assistant_id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
            "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
            "run_id": "run_mM8WNPj28oO60IEx8r6VZeOG",
            "status": "in_progress",
            "incomplete_details": null,
            "incomplete_at": null,
            "completed_at": null,
            "role": "assistant",
            "content": [],
            "attachments": [],
            "metadata": {}
          }
        },
        {
          "event": "thread.message.delta",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message.delta",
            "delta": {
              "content": [
                {
                  "index": 0,
                  "type": "text",
                  "text": {
                    "value": "{\n",
                    "annotations": []
                  }
                }
              ]
            }
          }
        },
        {
          "event": "thread.message.delta",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message.delta",
            "delta": {
              "content": [
                {
                  "index": 0,
                  "type": "text",
                  "text": {
                    "value": " "
                  }
                }
              ]
            }
          }
        },
        {
          "event": "thread.message.delta",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message.delta",
            "delta": {
              "content": [
                {
                  "index": 0,
                  "type": "text",
                  "text": {
                    "value": " \""
                  }
                }
              ]
            }
          }
        },
        {
          "event": "thread.message.delta",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message.delta",
            "delta": {
              "content": [
                {
                  "index": 0,
                  "type": "text",
                  "text": {
                    "value": "solution"
                  }
                }
              ]
            }
          }
        },
        {
          "event": "thread.message.delta",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message.delta",
            "delta": {
              "content": [
                {
                  "index": 0,
                  "type": "text",
                  "text": {
                    "value": "\":"
                  }
                }
              ]
            }
          }
        },
        {
          "event": "thread.message.delta",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message.delta",
            "delta": {
              "content": [
                {
                  "index": 0,
                  "type": "text",
                  "text": {
                    "value": " \""
                  }
                }
              ]
            }
          }
        },
        {
          "event": "thread.message.delta",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message.delta",
            "delta": {
              "content": [
                {
                  "index": 0,
                  "type": "text",
                  "text": {
                    "value": "x"
                  }
                }
              ]
            }
          }
        },
        {
          "event": "thread.message.delta",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message.delta",
            "delta": {
              "content": [
                {
                  "index": 0,
                  "type": "text",
                  "text": {
                    "value": " ="
                  }
                }
              ]
            }
          }
        },
        {
          "event": "thread.message.delta",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message.delta",
            "delta": {
              "content": [
                {
                  "index": 0,
                  "type": "text",
                  "text": {
                    "value": " "
                  }
                }
              ]
            }
          }
        },
        {
          "event": "thread.message.delta",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message.delta",
            "delta": {
              "content": [
                {
                  "index": 0,
                  "type": "text",
                  "text": {
                    "value": "1"
                  }
                }
              ]
            }
          }
        },
        {
          "event": "thread.message.delta",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message.delta",
            "delta": {
              "content": [
                {
                  "index": 0,
                  "type": "text",
                  "text": {
                    "value": "\"\n"
                  }
                }
              ]
            }
          }
        },
        {
          "event": "thread.message.delta",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message.delta",
            "delta": {
              "content": [
                {
                  "index": 0,
                  "type": "text",
                  "text": {
                    "value": "}"
                  }
                }
              ]
            }
          }
        },
        {
          "event": "thread.message.completed",
          "data": {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message",
            "created_at": 1718002195,
            "assistant_id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
            "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
            "run_id": "run_mM8WNPj28oO60IEx8r6VZeOG",
            "status": "completed",
            "incomplete_details": null,
            "incomplete_at": null,
            "completed_at": 1718002195,
            "role": "assistant",
            "content": [
              {
                "type": "text",
                "text": {
                  "value": "{\n  \"solution\": \"x = 1\"\n}",
                  "annotations": []
                }
              }
            ],
            "attachments": [],
            "metadata": {}
          }
        },
        {
          "event": "thread.run.step.completed",
          "data": {
            "id": "step_10HZORF6qyVVTiXmRtSLbWlo",
            "object": "thread.run.step",
            "created_at": 1718002195,
            "run_id": "run_mM8WNPj28oO60IEx8r6VZeOG",
            "assistant_id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
            "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
            "type": "message_creation",
            "status": "completed",
            "cancelled_at": null,
            "completed_at": 1718002195,
            "expires_at": 1718002794,
            "failed_at": null,
            "last_error": null,
            "step_details": {
              "type": "message_creation",
              "message_creation": {
                "message_id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe"
              }
            },
            "usage": {
              "prompt_tokens": 74,
              "completion_tokens": 13,
              "total_tokens": 87
            }
          }
        },
        {
          "event": "thread.run.completed",
          "data": {
            "id": "run_mM8WNPj28oO60IEx8r6VZeOG",
            "object": "thread.run",
            "created_at": 1718002194,
            "assistant_id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
            "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
            "status": "completed",
            "started_at": 1718002195,
            "expires_at": null,
            "cancelled_at": null,
            "failed_at": null,
            "completed_at": 1718002195,
            "required_action": null,
            "last_error": null,
            "model": "gpt-4o",
            "instructions": "You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.",
            "tools": [],
            "tool_resources": {
              "code_interpreter": {
                "file_ids": []
              }
            },
            "metadata": {},
            "temperature": 1.0,
            "top_p": 1.0,
            "max_completion_tokens": null,
            "max_prompt_tokens": null,
            "truncation_strategy": {
              "type": "auto",
              "last_messages": null
            },
            "incomplete_details": null,
            "usage": {
              "prompt_tokens": 74,
              "completion_tokens": 13,
              "total_tokens": 87
            },
            "response_format": {
              "type": "json_object"
            },
            "tool_choice": "auto",
            "parallel_tool_calls": true
          }
        }
      ]
    }
  }
}
ListMessagesAsync -> [GET:200] https://localhost:80/v1/threads/thread_F42J8yHpyK548ZnU2LLda5EU/messages
{
  "Request": {
    "Headers": {
      "User-Agent": [
        "OpenAI-DotNet"
      ],
      "OpenAI-Beta": [
        "assistants=v2"
      ],
      "Authorization": [
        "Bearer sess-aAbBcCdDeE123456789"
      ]
    }
  },
  "Response": {
    "Headers": {
      "Date": [
        "Mon, 10 Jun 2024 06:49:56 GMT"
      ],
      "openai-version": [
        "2020-10-01"
      ],
      "openai-organization": [
        "pillow-9"
      ],
      "X-Request-ID": [
        "req_634361dac8b4d56d0bc2bdef4bdeecfa"
      ],
      "openai-processing-ms": [
        "58"
      ],
      "Strict-Transport-Security": [
        "max-age=15724800; includeSubDomains"
      ],
      "CF-Cache-Status": [
        "DYNAMIC"
      ],
      "Server": [
        "cloudflare"
      ],
      "CF-RAY": [
        "89176c9cbc743b08-IAD"
      ]
    },
    "Body": {
      "Content": {
        "object": "list",
        "data": [
          {
            "id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
            "object": "thread.message",
            "created_at": 1718002195,
            "assistant_id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
            "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
            "run_id": "run_mM8WNPj28oO60IEx8r6VZeOG",
            "role": "assistant",
            "content": [
              {
                "type": "text",
                "text": {
                  "value": "{\n  \"solution\": \"x = 1\"\n}",
                  "annotations": []
                }
              }
            ],
            "attachments": [],
            "metadata": {}
          },
          {
            "id": "msg_koo7m9nqB9eIL3DFT3bpW4ke",
            "object": "thread.message",
            "created_at": 1718002194,
            "assistant_id": null,
            "thread_id": "thread_F42J8yHpyK548ZnU2LLda5EU",
            "run_id": null,
            "role": "user",
            "content": [
              {
                "type": "text",
                "text": {
                  "value": "I need to solve the equation `3x + 11 = 14`. Can you help me?",
                  "annotations": []
                }
              }
            ],
            "attachments": [],
            "metadata": {}
          }
        ],
        "first_id": "msg_WzAKpu8qrN0JTPjCs9FtaxSe",
        "last_id": "msg_koo7m9nqB9eIL3DFT3bpW4ke",
        "has_more": false
      }
    }
  }
}
User: I need to solve the equation `3x + 11 = 14`. Can you help me?
Assistant: {
  "solution": "x = 1"
}
DeleteAssistantAsync -> [DELETE:200] https://localhost:80/v1/assistants/asst_Gj4efmelAHprrUs8k4QvFuQS
{
  "Request": {
    "Headers": {
      "User-Agent": [
        "OpenAI-DotNet"
      ],
      "OpenAI-Beta": [
        "assistants=v2"
      ],
      "Authorization": [
        "Bearer sess-aAbBcCdDeE123456789"
      ]
    }
  },
  "Response": {
    "Headers": {
      "Date": [
        "Mon, 10 Jun 2024 06:49:56 GMT"
      ],
      "openai-version": [
        "2020-10-01"
      ],
      "openai-organization": [
        "pillow-9"
      ],
      "X-Request-ID": [
        "req_20fd8e700fd2d497fbf0a29711425a1e"
      ],
      "openai-processing-ms": [
        "32"
      ],
      "Strict-Transport-Security": [
        "max-age=15724800; includeSubDomains"
      ],
      "CF-Cache-Status": [
        "DYNAMIC"
      ],
      "Server": [
        "cloudflare"
      ],
      "CF-RAY": [
        "89176c9dad3d3b08-IAD"
      ]
    },
    "Body": {
      "Content": {
        "id": "asst_Gj4efmelAHprrUs8k4QvFuQS",
        "object": "assistant.deleted",
        "deleted": true
      }
    }
  }
}
DeleteThreadAsync -> [DELETE:200] https://localhost:80/v1/threads/thread_F42J8yHpyK548ZnU2LLda5EU
{
  "Request": {
    "Headers": {
      "User-Agent": [
        "OpenAI-DotNet"
      ],
      "OpenAI-Beta": [
        "assistants=v2"
      ],
      "Authorization": [
        "Bearer sess-aAbBcCdDeE123456789"
      ]
    }
  },
  "Response": {
    "Headers": {
      "Date": [
        "Mon, 10 Jun 2024 06:49:56 GMT"
      ],
      "openai-version": [
        "2020-10-01"
      ],
      "openai-organization": [
        "pillow-9"
      ],
      "X-Request-ID": [
        "req_382d5baa1aa4e02e50cdcf10dd1c9129"
      ],
      "openai-processing-ms": [
        "38"
      ],
      "Strict-Transport-Security": [
        "max-age=15724800; includeSubDomains"
      ],
      "CF-Cache-Status": [
        "DYNAMIC"
      ],
      "Server": [
        "cloudflare"
      ],
      "CF-RAY": [
        "89176c9e5df63b08-IAD"
      ]
    },
    "Body": {
      "Content": {
        "id": "thread_F42J8yHpyK548ZnU2LLda5EU",
        "object": "thread.deleted",
        "deleted": true
      }
    }
  }
}

[rkuiper1]

Firstly I'd like to thank you for putting so much effort in answering my question and I'm a big fan of OpenAI-DotNet and the API it provides.

What I see in your log file is exactly the same as in mine: the MessageResponse events (that should probably contain the delta's) are empty until the last one (Status: Completed) which contains the entire response.
The rest of the code to show the response by getting the it with an extra request (thread.ListMessagesAsync()) and then reading messages.Items.Reverse() seems unnecessary because the last MessageReponse event contains the entire response text. That's why I commented that part out in my test project. At that point the 'message_creation' thread step has completed, as you can see in your logfile.
Fetching the the response with thread.ListMessagesAsync() after event streaming has completed seems to defeat the purpose of 'streaming'.
I've been using the streaming functionality with your Chat endpoint and then the delta's were received as they were generated by the model without the need for a fetch afterward. I simply assumed the assistant streaming would work the same.
In my current 'production' project I use your version 7 with the Assistant interfaces and it seems that the your proposed streaming implementation has no real advantages over the v7 implementation when the response delta's are not actually streamed as they are generated by the model.
Unless, of course, I'm misunderstanding things

[StephenHodgson]

response delta's are not actually streamed as they are generated by the model.

I believe they are streamed. I am able to get the message deltas and print those out.

The rest of the code to show the response by getting the it with an extra request (thread.ListMessagesAsync()) and then reading messages.Items.Reverse()

Yeah I was doing that just to verify that those messages were the same as the delta messages.

I'm in the process of refactoring the unit tests. I'll be sure to verify that the message delta events are being raised correctly and compare those to the final list message output results.

[StephenHodgson]

@rkuiper1 if you wanna swing by my discord server today, maybe we work on it some. I'll try to live stream for a bit and you can ask me questions in realtime.

[rkuiper1]

I appreciate the invite, but I'm on vacation and at this time I have very limited online access. I can do some occasional emailing, but I’m afraid anything else is out of my league right now.

[rkuiper1]

Hi @StephenHodgson ,
Today I was lucky enough to have a decent internet connection and I took a look at your streaming implementation.
I focused on ThreadsEndpoint.cs and in particular the StreamRunAsync method.
There I found that all the delta's are neatly received and processed, but not passed to the streamEventHandler, which explains the empty content we see in our logfiles.
This tiny tweak made the difference:

                        case "thread.message.incomplete":
                            var partialMessage = sseResponse.Deserialize<MessageResponse>(ssEvent, client);
                            if (message == null)
                            {
                                message = partialMessage;
                            }
                            else
                            {
                                message.AppendFrom(partialMessage);
                            }

481                         streamEventHandler?.Invoke(partialMessage);
                            return;

Line 481 originally contains: streamEventHandler?.Invoke(message); where message does not contain the actual delta.
Changing it to streamEventHandler?.Invoke(partialMessage); will pass the message event that does contain the delta.

Next, I can print the delta's in my test project like this:

                    case MessageResponse messageResponse:
                        if (messageResponse.Delta != null)
                        {
                            Console.Write($"{messageResponse.Delta.Content.FirstOrDefault()?.Text}");
                        }
                        break;

Running the test project will show the familiar chat-gpt delta streaming I was trying to accomplish.
I know this is a quick hack and I can't oversee all consequences of this change, but the way it works would be good enough for me.

[StephenHodgson]

I see, it is slightly different from my ChatEndpoint streaming behaviour.

OpenAI-DotNet/OpenAI-DotNet/Chat/ChatEndpoint.cs

Lines 62 to 77 in 8294a63

	using var response = await this.StreamEventsAsync(GetUrl("/completions"), payload, (sseResponse, ssEvent) =>
	{
	var partialResponse = sseResponse.Deserialize<ChatResponse>(ssEvent, client);
	if (chatResponse == null)
	{
	chatResponse = new ChatResponse(partialResponse);
	}
	else
	{
	chatResponse.AppendFrom(partialResponse);
	}
	resultHandler?.Invoke(partialResponse);
	}, cancellationToken);

Regardless, I was copying over all of the Delta data in the AppendFrom method, and attaching it to the response we have so far.

OpenAI-DotNet/OpenAI-DotNet/Threads/MessageResponse.cs

Lines 184 to 200 in 8294a63

	if (other.Delta != null)
	{
	if (Role == 0 &&
	other.Delta.Role > 0)
	{
	Role = other.Delta.Role;
	}
	if (other.Delta.Content != null)
	{
	content ??= new List<Content>();
	content.AppendFrom(other.Delta.Content);
	}
	// bail early since we only care about the delta content
	return;
	}

I intentionally wanted to pass back the full message and not just the partial one. Maybe something inside the AppendFrom is broken. I'll take a look.

[StephenHodgson]

Btw you can also deserialize any/all of the events yourself if needed:

                var run = await thread.CreateRunAsync(assistant, streamEvent =>
                {
                    Console.WriteLine(streamEvent.ToJsonString());
                    var partialMessage = JsonSerializer.Deserialize<MessageResponse>(streamEvent.ToJsonString());
                });

I left this just in case they add new events not already handled by the library, but should be a good workaround in the meantime.

[StephenHodgson]

new bug ticket for tracking

• Message and StepRun Deltas missing from eventMessage callback object #331

[StephenHodgson]

@rkuiper1 I think I fixed it in

• OpenAI-DotNet 8.0.3 #332

Try it out and let me know. The Delta object should show be properly populated.

[rkuiper1]

@StephenHodgson
It works wonderfully. the Delta.PrintContent method is very helpful.
You might want to take a look at this exception which is what the Delta.PrintContent returns for the last MessageResponse event in many cases:

System.InvalidOperationException: Attempting to append a different object than the original!
   at OpenAI.Threads.RunStepResponse.AppendFrom(RunStepResponse other)
   at OpenAI.Threads.ThreadsEndpoint.<>c__DisplayClass23_0.<StreamRunAsync>b__0(HttpResponseMessage sseResponse, ServerSentEvent ssEvent)

It could be that I should test something before calling the PrintContent method to avoid this exception, but I'm not sure what.

[StephenHodgson]

Most likely you should only print if Delta != null

[StephenHodgson]

But I suspect that a new run object has been created and we need to make a new one.

Yes still a few edge cases to fix.

[rkuiper1]

I do the Delta != null test, just in case, but nevertheless...
I also observed that the behavior occurs without using the PrintContent method:

                await Console.Out.WriteAsync($"{message.Delta.PrintContent()}");
                await Console.Out.WriteAsync($"{message.Delta.Content.FirstOrDefault()?.Text}");

The two lines above show the same output...