Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Storybot interaction issues #216

Open
rwmjhb opened this issue Jun 6, 2024 · 3 comments
Open

Storybot interaction issues #216

rwmjhb opened this issue Jun 6, 2024 · 3 comments

Comments

@rwmjhb
Copy link

rwmjhb commented Jun 6, 2024 •
edited
Loading

I modified the llm service code to local ollama. After the program started, everything was fine at first. The robot asked me what kind of story I wanted to hear. I answered it, and then it told me the first paragraph. When it finished telling this When I was asked about my next step needs, the microphone on the page turned gray, and there was no error message in the log. The page was obviously waiting for my response, but because the microphone turned gray, I couldn't answer it. I observed that the Chrome tab shows that the microphone is running normally, I posted the log below, hoping someone can help me.

  • (venv) (base) pope@Hengs-MBP storytelling-chatbot % python src/server.py
  • INFO: Started server process [7067]
  • INFO: Waiting for application startup.
  • INFO: Application startup complete.
  • INFO: Uvicorn running on http://0.0.0.0:7860 (Press CTRL+C to quit)
  • INFO: 127.0.0.1:50611 - "GET / HTTP/1.1" 200 OK
  • INFO: 127.0.0.1:50619 - "POST /create HTTP/1.1" 200 OK
  • INFO: 127.0.0.1:50619 - "POST /start HTTP/1.1" 200 OK
  • 2024-06-06 11:02:09.055 | DEBUG | main:main:55 - Transport created for room:https://popeking.daily.co/S9xQJMm9alpJQ5nWPcSM
  • 2024-06-06 11:02:09.055 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking PipelineSource#0 -> OllamaLLMService#0
  • 2024-06-06 11:02:09.056 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking OllamaLLMService#0 -> ElevenLabsTTSService#0
  • 2024-06-06 11:02:09.056 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking ElevenLabsTTSService#0 -> DailyOutputTransport#0
  • 2024-06-06 11:02:09.056 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking DailyOutputTransport#0 -> PipelineSink#0
  • 2024-06-06 11:02:09.056 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking Source#0 -> Pipeline#0
  • 2024-06-06 11:02:09.056 | DEBUG | main:main:109 - Waiting for participant...
  • 2024-06-06 11:02:09.056 | DEBUG | pipecat.pipeline.runner:run:28 - Runner PipelineRunner#0 started running PipelineTask#0
  • 2024-06-06 11:02:09.056 | INFO | pipecat.transports.services.daily:_join:224 - Joining https://popeking.daily.co/S9xQJMm9alpJQ5nWPcSM
  • 2024-06-06 11:02:10.374 | INFO | pipecat.transports.services.daily:on_participant_joined:423 - Participant joined 89f7ab49-1166-4d7c-b044-318db56cf91e
  • 2024-06-06 11:02:10.375 | DEBUG | main:on_first_participant_joined:113 - Participant joined, storytime commence!
  • 2024-06-06 11:02:11.894 | INFO | pipecat.transports.services.daily:_handle_join_response:285 - Joined https://popeking.daily.co/S9xQJMm9alpJQ5nWPcSM
  • 2024-06-06 11:02:11.895 | INFO | pipecat.transports.services.daily:_handle_join_response:288 - Enabling transcription with settings language='en' tier='nova' model='2-conversationalai' profanity_filter=True redact=False endpointing=True punctuate=True includeRawResponse=True extra={'interim_results': True}
  • 2024-06-06 11:02:12.534 | DEBUG | pipecat.transports.services.daily:on_transcription_started:450 - Transcription started: {'language': 'en', 'startedBy': '817bd6c7-896f-4686-a824-217fb6129d26', 'tier': 'nova', 'transcriptId': '6056d8f3-d20c-478c-a906-b45d15642987', 'model': '2-conversationalai'}
  • 2024-06-06 11:02:23.432 | DEBUG | pipecat.services.elevenlabs:run_tts:35 - Generating TTS: [Welcome to my whimsy workshop! I'm thrilled to spin a tale just for you. What kind of story would you love to embark on? Would you like it to be adventurous, mysterious, romantic, or perhaps something entirely unexpected? Let me know, and we'll conjure up a fantastical journey together!]
  • 2024-06-06 11:02:25.426 | DEBUG | pipecat.pipeline.runner:run:32 - Runner PipelineRunner#0 finished running PipelineTask#0
  • 2024-06-06 11:02:25.427 | DEBUG | pipecat.transports.services.daily:init:70 - Loaded native WebRTC VAD
  • 2024-06-06 11:02:25.427 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking PipelineSource#1 -> DailyInputTransport#0
  • 2024-06-06 11:02:25.427 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking DailyInputTransport#0 -> LLMUserResponseAggregator#0
  • 2024-06-06 11:02:25.427 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking LLMUserResponseAggregator#0 -> OllamaLLMService#0
  • 2024-06-06 11:02:25.428 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking OllamaLLMService#0 -> StoryProcessor#0
  • 2024-06-06 11:02:25.428 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking StoryProcessor#0 -> StoryImageProcessor#0
  • 2024-06-06 11:02:25.428 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking StoryImageProcessor#0 -> ElevenLabsTTSService#0
  • 2024-06-06 11:02:25.428 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking ElevenLabsTTSService#0 -> DailyOutputTransport#0
  • 2024-06-06 11:02:25.428 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking DailyOutputTransport#0 -> LLMAssistantResponseAggregator#0
  • 2024-06-06 11:02:25.428 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking LLMAssistantResponseAggregator#0 -> PipelineSink#1
  • 2024-06-06 11:02:25.428 | DEBUG | pipecat.processors.frame_processor:link:37 - Linking Source#1 -> Pipeline#1
  • 2024-06-06 11:02:25.428 | DEBUG | pipecat.pipeline.runner:run:28 - Runner PipelineRunner#0 started running PipelineTask#1
  • 2024-06-06 11:02:43.758 | DEBUG | pipecat.transports.services.daily:_on_transcription_message:840 - Transcription (from: 89f7ab49-1166-4d7c-b044-318db56cf91e): [Perhaps.]
  • 2024-06-06 11:02:45.488 | DEBUG | pipecat.transports.services.daily:_on_transcription_message:840 - Transcription (from: 89f7ab49-1166-4d7c-b044-318db56cf91e): [Something like]
  • 2024-06-06 11:02:47.290 | DEBUG | pipecat.transports.services.daily:_on_transcription_message:840 - Transcription (from: 89f7ab49-1166-4d7c-b044-318db56cf91e): [mysteries.]
  • 2024-06-06 11:03:13.508 | DEBUG | pipecat.services.fal:run_image_gen:55 - Generating image from prompt: illustrative art of a tiny, curious fairy with iridescent wings, a brown fluffy dog, and a tiny red cat in a whimsical forest with sparkling fireflies. In the style of Studio Ghibli. colorful, whimsical, painterly, concept art.
  • 2024-06-06 11:03:15.136 | DEBUG | pipecat.services.fal:run_image_gen:69 - Image generated at: https://fal.media/files/koala/ghhFxFx4ZIzGwDQEgUA-r.png
  • 2024-06-06 11:03:15.136 | DEBUG | pipecat.services.fal:run_image_gen:72 - Downloading image https://fal.media/files/koala/ghhFxFx4ZIzGwDQEgUA-r.png ...
  • 2024-06-06 11:03:16.505 | DEBUG | pipecat.services.fal:run_image_gen:74 - Downloaded image https://fal.media/files/koala/ghhFxFx4ZIzGwDQEgUA-r.png
  • 2024-06-06 11:03:16.973 | DEBUG | pipecat.services.elevenlabs:run_tts:35 - Generating TTS: [In the heart of the enchanted woods, a tiny fairy named Luna befriended a playful pup and a mischievous kitty who stumbled upon a hidden glade . <a delicate, shimmering portal surrounded by vines and twinkling flowers> As they explored the clearing, they discovered a glowing portal that seemed to pulse with an otherworldly energy, beckoning them to venture forth . <a soft, ethereal mist swirling around the trio as they step through the portal> With a collective sense of wonder, Luna, the dog, and the cat stepped through the shimmering gateway, and into a realm beyond their wildest dreams . How would you like the story to continue?]
  • 2024-06-06 11:03:50.483 | DEBUG | pipecat.services.fal:run_image_gen:55 - Generating image from prompt: illustrative art of = N <= B`.<a whimsical forest with towering trees, twinkling fireflies, and a crescent moon. In the style of Studio Ghibli. colorful, whimsical, painterly, concept art.
  • 2024-06-06 11:03:51.997 | DEBUG | pipecat.services.fal:run_image_gen:69 - Image generated at: https://fal.media/files/lion/ImN8FiXw8jRjKVPrQHks0.png
  • 2024-06-06 11:03:51.997 | DEBUG | pipecat.services.fal:run_image_gen:72 - Downloading image https://fal.media/files/lion/ImN8FiXw8jRjKVPrQHks0.png ...
  • 2024-06-06 11:03:53.843 | DEBUG | pipecat.services.fal:run_image_gen:74 - Downloaded image https://fal.media/files/lion/ImN8FiXw8jRjKVPrQHks0.png
  • 2024-06-06 11:03:54.481 | DEBUG | pipecat.services.elevenlabs:run_tts:35 - Generating TTS: [You're thinking of a way to generate random numbers within a specific range. Here's how you can do it in Python: import random # Generate a random number between A and B (inclusive) A = 1 B = 100 random_number = random.randint(A, B) print(random_number) In this code, randint(A, B) generates a random integer N such that `A In the heart of this enchanted forest, a curious adventurer stumbled upon a hidden path <the adventurer: a young girl with wild curly hair, wearing a flowing white dress, holding a lantern> She had been searching for the legendary Moonflower, said to bloom only once a year under the gentle light of the crescent moon How would you like the story to continue?]
@jptaylor
Copy link
Contributor

jptaylor commented Jun 6, 2024

Did you remove the DailyTransportMessageFrame(CUE_USER_TURN) frame from the pipeline? Or does it get shown in the logs? This is an app message that gets sent to the UI in order to trigger the user's turn.

I will give this a try with a local LLM and see if I can replicate.

@rwmjhb
Copy link
Author

rwmjhb commented Jun 7, 2024

I just modified the code about the llm service part to ollama, and the other parts were completely untouched.

@rwmjhb
Copy link
Author

rwmjhb commented Jun 11, 2024

Did you remove the DailyTransportMessageFrame(CUE_USER_TURN) frame from the pipeline? Or does it get shown in the logs? This is an app message that gets sent to the UI in order to trigger the user's turn.

I will give this a try with a local LLM and see if I can replicate.

Hi,Is there any progress?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jptaylor @rwmjhb