Session brief
AI Engineering 201: The Rest of the Owl
Overview
This talk explores the engineering challenges and emerging patterns in building AI systems beyond just the inference engine. It posits that the current wave of AI development is focused on creating language user interfaces, analogous to the shift from text-based terminals to graphical user interfaces. The presentation delves into various architectural patterns, including retrieval-augmented generation (RAG) and structured outputs, and discusses the critical need for robust monitoring, observability, and evaluation in the AI engineering lifecycle.
Who should watch
- AI Engineers
- Product Managers
- Builders working with language models
- Those interested in the practical application and engineering of AI systems
- Individuals facing challenges in deploying and improving AI products
Key takeaways
- The primary focus is shifting from inference engineering to building comprehensive AI products, with language user interfaces (LUIs) being a key development, similar to the impact of GUIs on computing.
- Retrieval-augmented generation (RAG) is a stable pattern for providing language models with necessary context, often leveraging vector databases or hybrid search approaches.
- Structured outputs, derived from tool use, are crucial for improving model robustness and enabling AI systems to interact with other systems, with techniques like prompt engineering, fine-tuning, and grammar-based sampling being employed.
- Agents represent a more advanced form of AI, capable of memory, skill accumulation, and tool use, with ongoing research exploring cognitive architectures to enhance their capabilities.
- Effective AI engineering requires a strong emphasis on monitoring, observability, and evaluation from the outset, adopting a "ship to learn" philosophy rather than "learn to ship."
- Challenges in AI evaluation, particularly for open-ended generations, necessitate strategies like iterated decomposition, reference matching, and leveraging user feedback or AI annotators.
- The development of AI systems is an iterative process, with continuous learning from production data and user behavior being essential for improvement.
Notable quotes
*The thing that we're building right now are language user interfaces.*
*The dominant ideology right now is that you should ship to learn rather than learning to ship.*
*The problem with fine-tuning and an annoying thing about prompting is that if there is not a kind of shared...*
Unofficial community note. Prefer the recording for nuance.