Session brief
Lessons from building GenAI based applications — Juan Peredo
Overview
This talk shares practical lessons learned from building Generative AI applications over the past year and a half. It emphasizes that while AI can accelerate development, integrating it introduces significant new complexities. Key challenges include model selection, preventing hallucinations, managing infrastructure costs (especially GPUs), ensuring correct outputs, and understanding the operational overhead of agentic systems. The speaker advocates for careful evaluation, externalizing prompts, and robust observability to navigate these complexities effectively.
Who should watch
- AI Engineers
- Product Managers
- Developers building AI-powered applications
- Anyone interested in the practical challenges and solutions for deploying GenAI
Key takeaways
- AI coding tools can speed up development, but they don't build entire applications autonomously.
- Integrating AI into applications requires addressing new concerns like model choice, prompt engineering, hallucination prevention, and fine-tuning.
- Hosting AI models, especially those requiring GPUs, introduces significant infrastructure and cost considerations beyond traditional application hosting.
- Ensuring AI-generated responses are accurate is challenging, requiring techniques like prompt engineering, guardrails, Retrieval Augmented Generation (RAG), or fine-tuning, none of which guarantee perfect results.
- Continuous evaluation of models and applications throughout the development lifecycle is crucial, using tools like Ama, Hugging Face Spaces, OpenRouter, and Lsmith.
- Externalizing prompts from code allows for easier collaboration with domain experts and future-proofing applications against model updates.
- Agents, which allow LLMs to interact with the real world via tools, represent a powerful future direction but introduce latency and significant cost concerns.
- Careful model selection based on cost and performance is vital, as cheaper, smaller models can often suffice and drastically reduce operational expenses.
Notable quotes
*Building a chat bot is very simple; making sure that chatbot gives you the correct answers is really hard.*
*Prompts should never be encoded in your code.*
*Small is beautiful and better for the environment and better for your wallet.*
Unofficial community note. Prefer the recording for nuance.