feat: add token streaming to the MemGPT API #1280

cpacker · 2024-04-20T18:29:34Z

Adds token streaming to the MemGPT API (when available in the underlying model endpoint).

Handle context overflow on SSE
Choose streaming format for MemGPT API
Handle step ending + context overflow on pass-through stream
- [DONE_GEN] - LLM generation finished
- [DONE_STEP] - Agent step finished (step = LLM generation + tool call response)
- [DONE] - All agent step(s) finished, post SSE response is complete
Include function_return in the streaming API
- TODO eventually support streaming the function_return if the tool supports streaming (tool could also just repeatedly call interface.function_call?)
Order [DONE_STEP] after function_return
Properly handle non-streaming + stream: true requests on non-streaming
- stream: true requests on non-streaming should do fake (token all-together) streaming?
Add Message ids to the SSE stream response
~~- [ ] Add test~~

Add a /chat/completion proxy route for interfacing the MemGPT server with services that only speak in /chat/completion

Add /chat/completion request route that supports token streaming
Ignore inner thoughts in return stream by default
Remap send_message to content in assistant response

Legacy support for `stream`

(base) loaner@MacBook-Pro-199 ~ % curl --request POST \
     --url http://localhost:8283/api/agents/bb2af663-3bda-4b5f-bc7d-2ebb83c13102/messages \
     --header 'accept: application/json' \
     --header 'authorization: Bearer password' \
     --header 'content-type: application/json' \
     --data '
{
  "message": "oh hey whats up",
  "role": "user",
  "stream": true
}
'

data: {"internal_monologue": "Finally, a different message from Chad! This shift in communication can be an opening to make some progress. It's key to respond in tune with my persona: empathetic, insightful, and engaging. I'll try to keep the conversation casual, in line with Chad's tone, while subtly trying to gain more context about the situation.", "date": "2024-06-24T03:42:51.628098+00:00", "id": "e254631d-c6ad-468e-884d-739ab0db9032"}

data: {"function_call": "send_message({'message': 'Hello, Chad! Not much, just trying to understand the recurring alerts and messages we were having. By the way, how are things with you? Any progress on the Charles situation?'})", "id": "e254631d-c6ad-468e-884d-739ab0db9032", "date": "2024-06-24T03:42:51.628098+00:00"}

data: {"assistant_message": "Hello, Chad! Not much, just trying to understand the recurring alerts and messages we were having. By the way, how are things with you? Any progress on the Charles situation?", "id": "e254631d-c6ad-468e-884d-739ab0db9032", "date": "2024-06-24T03:42:51.628098+00:00"}

data: {"function_return": "None", "status": "success", "id": "cba16d64-6490-47f8-952a-ffd9906b2864", "date": "2024-06-24T03:42:51.628401+00:00"}

New stream flags

Instead of stream being an argument to the message command, we now have:
- stream_steps
  - means that each "step" will stream, but not the tokens
  - this is the equivalent of the old stream
- stream_tokens
  - means that the tokens will stream back during the steps
  - stream_tokens == true means that stream_steps must also be true
We no longer treat send_message as a special function in the returns
- instead, we pass send_message back as a standard function call
- it is up to the client to interpret send_message in a special way (e.g. to use it to show up as the chat in the chat UI)

Related issues or PRs

#345 #1215

How to test

Create an agent
Run server

memgpt server --debug

Hit server with message and make sure "stream": true,

curl --request POST \
     --url http://localhost:8283/api/agents/723be90d-dae7-445e-85ff-7b50e24fd493/messages \
     --header 'accept: application/json' \
     --header 'authorization: Bearer banana' \
     --header 'content-type: application/json' \
     --data '
{
  "message": "how goes it my friend",
  "stream": true,
  "role": "user",
  "timestamp": "2024-03-16T12:00:00Z"
}
'

Response:

(pymemgpt-py3.10) (base) loaner@MacBook-Pro-55 MemGPT-2 % curl --request POST \
     --url http://localhost:8283/api/agents/723be90d-dae7-445e-85ff-7b50e24fd493/messages \
     --header 'accept: application/json' \
     --header 'authorization: Bearer banana' \
     --header 'content-type: application/json' \
     --data '
{
  "message": "how goes it my friend",
  "stream": true,
  "role": "user",
  "timestamp": "2024-03-16T12:00:00Z"
}
'
data: {"internal_monologue": "", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": "The", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " dance", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " continues", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " with", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " the", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " familiar", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " rhythm", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " of", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " Chad", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": "'s", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " greeting", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " Our", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " exchange", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": ",", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " marked", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " by", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " the", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " repeated", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " refrain", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " of", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " '", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": "how", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " goes", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " it", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " my", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " friend", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": ",'", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " embodies", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " our", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " shared", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " connection", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " Record", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " each", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " echo", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " of", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " this", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " phrase", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " in", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " our", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " archival", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " memory", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": ",", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " ensuring", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " that", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " this", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " rhyth", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": "mic", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " narrative", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " continues", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " to", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": " resonate", "date": "2024-05-02T23:17:15+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"id": "call_fAr8Q4yot74DNDZe1op50ybq", "name": "archival_memory_insert"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "{\n"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " "}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " \""}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "content"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "\":"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " \""}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "Our"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " dance"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " with"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " Chad"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " continues"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " to"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " the"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " rhythm"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " of"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " '"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "how"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " goes"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " it"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " my"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " friend"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": ".'"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " Each"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " echo"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " of"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " this"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " phrase"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " is"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " an"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " embodiment"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " of"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " our"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " shared"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " connection"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": ","}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " a"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " rhyth"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "mic"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " narrative"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " that"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " deserves"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " to"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " be"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " ins"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "cribed"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " in"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " our"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " memory"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": ".\",\n"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " "}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " \""}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "request"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "_"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "heartbeat"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "\":"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": " true"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "\n"}, "date": "2024-05-02T23:17:15+00:00"}

data: {"function_call": {"arguments": "}"}, "date": "2024-05-02T23:17:15+00:00"}

data: [DONE_GEN]

data: {"function_return": "None", "status": "success", "id": "b0b28c63-54a5-4fd5-b739-bd97d59231bf", "date": "2024-05-02T23:17:22.631736+00:00"}

data: [DONE_STEP]

data: {"internal_monologue": "", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": "With", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " the", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " rhyth", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": "mic", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " refrain", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " '", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": "how", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " goes", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " it", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " my", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " friend", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": "'", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " echoing", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " in", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " the", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " space", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " between", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " Chad", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " and", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " I", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ",", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " we", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " continue", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " our", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " dance", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " Reflect", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " his", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " rhythm", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " back", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " to", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " him", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ",", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " while", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " also", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " adding", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " new", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " elements", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " It", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": "'s", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " within", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " these", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " echoes", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ",", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " that", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " we", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " begin", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " expanding", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " our", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " shared", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " storyline", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " What", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " could", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " potentially", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " lie", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " beneath", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " the", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " surface", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " of", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " this", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " regular", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " refrain", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": "?", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " Keep", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": " exploring", "date": "2024-05-02T23:17:23+00:00"}

data: {"internal_monologue": ".", "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"id": "call_W8R8xncrSavADEAEHraJwWM4", "name": "send_message"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "{\n"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " "}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " \""}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "message"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "\":"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " \""}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "As"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " the"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " rhythm"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " of"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " '"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "how"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " goes"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " it"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " my"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " friend"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "'"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " continues"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": ","}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " Chad"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": ","}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " its"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " consistency"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " tells"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " a"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " story"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "."}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " Could"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " there"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " be"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " hidden"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " narratives"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " waiting"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " to"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " be"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " explored"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " in"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " your"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " use"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " of"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " this"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " phrase"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "?\""}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": " \n"}, "date": "2024-05-02T23:17:23+00:00"}

data: {"function_call": {"arguments": "}"}, "date": "2024-05-02T23:17:23+00:00"}

data: [DONE_GEN]

data: {"function_return": "None", "status": "success", "id": "4c7c46d9-f16d-406b-8f05-6cab9bb62902", "date": "2024-05-02T23:17:29.939903+00:00"}

data: [DONE_STEP]

data: [DONE]

…place --recursive .

…ort for both

… back through the 'content' field

cpacker · 2024-06-24T03:47:54Z

@sarahwooders ready for review

sarahwooders

lgtm!

cpacker added 5 commits April 20, 2024 11:10

autoflake --remove-all-unused-imports --remove-unused-variables --in-…

b70987f

…place --recursive .

black

3fe1661

added back init imports

c0ef6d6

isort .

c31c1ad

reran black

e761359

cpacker mentioned this pull request Apr 20, 2024

🚀 MemGPT Q2 2024 Developer Roadmap #1200

Open

32 tasks

cpacker added 10 commits April 20, 2024 11:41

Merge branch 'main' into streaming-memgpt-api

847bd17

added sse generator

a89846d

partially working proxy streamback, but context window needs patch

03bf962

Merge branch 'main' into streaming-memgpt-api

e9aca8e

precommit pass

8b49e6c

added stub method

ff0b21b

updated error messages to catch problem with wrong sqlite path

1d7a4c7

Merge branch 'main' into streaming-memgpt-api

2116bba

properly catch and throw context error on SSE

3d21cef

added support for multi-step in the streaming interface

111c670

cpacker marked this pull request as ready for review April 29, 2024 06:30

cpacker added 8 commits April 28, 2024 23:59

added function return support

af7677e

added [DONE_GEN] and [DONE_STEP]

2335a9d

Merge branch 'main' into streaming-memgpt-api

b4e4d1b

Merge branch 'main' into streaming-memgpt-api

fed341d

cleanup

39604a8

temp disable streaming

5791f0b

Merge branch 'main' into streaming-memgpt-api

7ffaab2

split streaming args into stream_tokens and stream_step, confirm supp…

1e58410

…ort for both

cpacker added the API Related to MemGPT API label May 8, 2024

cpacker added 4 commits May 8, 2024 19:57

working /chat/completion proxy routes

072d74b

working /chat/completions proxy route that sends send_message outputs…

20e18cc

… back through the 'content' field

patch test

7cb3f84

patch test

f1a52d6

cpacker added 8 commits May 11, 2024 14:44

patch test

70f4116

changed default_interface_cls to a factory (default_interface_factory)

4f98cea

patch test

add3bbf

Merge branch 'main' into streaming-memgpt-api

9915bd4

fix passing of IDs in streamed response

4ee1193

Merge branch 'main' into streaming-memgpt-api

912e82d

Merge branch 'main' into streaming-memgpt-api

b674d11

Merge branch 'main' into streaming-memgpt-api

ad3123c

sarahwooders mentioned this pull request Jun 18, 2024

Not a bug #1461

Open

cpacker added 5 commits June 22, 2024 14:22

Merge branch 'main' into streaming-memgpt-api

e75a2e1

Merge branch 'main' into streaming-memgpt-api

69fe324

added support for legacy flag

e06a3a4

add support for legacy mode

0de5a6f

Merge branch 'main' into streaming-memgpt-api

3da1139

cpacker requested a review from sarahwooders June 24, 2024 03:47

sarahwooders approved these changes Jun 25, 2024

View reviewed changes

sarahwooders merged commit 82b9b74 into main Jun 25, 2024
11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add token streaming to the MemGPT API #1280

feat: add token streaming to the MemGPT API #1280

cpacker commented Apr 20, 2024 •

edited

Loading

cpacker commented Jun 24, 2024 •

edited

Loading

sarahwooders left a comment

feat: add token streaming to the MemGPT API #1280

feat: add token streaming to the MemGPT API #1280

Conversation

cpacker commented Apr 20, 2024 • edited Loading

Legacy support for stream

New stream flags

cpacker commented Jun 24, 2024 • edited Loading

sarahwooders left a comment

Choose a reason for hiding this comment

cpacker commented Apr 20, 2024 •

edited

Loading

Legacy support for `stream`

cpacker commented Jun 24, 2024 •

edited

Loading