Session brief
Storyteller: Building Multi-modal Apps with TS & ModelFusion - Lars Grammel, PhD
Overview
This talk details the creation of Storyteller, an application designed to generate short audio stories for preschool children. The system utilizes TypeScript and the ModelFusion AI orchestration library, accepting a voice input to produce approximately two-minute audio narratives. Key technical challenges addressed include achieving responsiveness, ensuring quality, and maintaining consistency in the generated content.
Who should watch
- AI engineers and builders interested in multi-modal application development.
- Developers working with AI orchestration libraries and frameworks.
- Product managers and engineers focused on shipping AI-powered creative tools.
- Those facing challenges with AI model response times and output consistency.
Key takeaways
- Storyteller is a client-server application with a React client and a Fastify server, communicating via server-sent events.
- OpenAI Whisper is used for quick audio transcription, and GPT-3 Turbo Instruct is employed for rapid story outline and title generation.
- Image generation involves using GPT-4 to extract keywords from the story for a prompt fed into Stability AI Stable Diffusion XL.
- The most time-consuming part, full audio story generation, is addressed by streaming the story structure and synthesizing speech for completed parts.
- ModelFusion facilitates partial parsing of streamed results, allowing the application to determine when a story segment is ready for narration.
- Voice generation for new speakers involves creating a voice description using GPT-3.5 Turbo, embedding voices, and retrieving suitable options based on gender and uniqueness.
- Speech synthesis is supported by providers like ElevenLabs, with generated audio files stored virtually and accessed by the client.
- Responsiveness is enhanced through parallel processing, streaming, and a loading state that updates as more content becomes available, allowing playback to begin while generation continues.
Notable quotes
*The main challenges were responsiveness meaning getting results to the user as quickly as possible uh quality and consistency.*
*The key idea is streaming the structure that's a little bit more difficult than just streaming characters token by token.*
Unofficial community note. Prefer the recording for nuance.