Session brief
Optimizing LLMs in Insurance with DSPy: Jeronim Morina
Overview
This talk emphasizes a return to first principles for AI engineers, moving beyond superficial prompt engineering and tool tinkering. It argues that AI models are most effective when integrated into larger systems designed to solve real-world problems, particularly in complex domains like insurance. The presentation advocates for a structured approach to development, including clear problem definition, robust evaluation, and modular system design, with DSPy presented as a tool to aid this process.
Who should watch
- AI Engineers struggling with prompt engineering and tool integration.
- Product Managers and builders seeking to develop robust AI solutions.
- Developers looking to move beyond toy problems and tackle complex, real-world challenges.
- Those interested in applying AI within the insurance industry or similar data-intensive fields.
Key takeaways
- Effective AI development requires a focus on solving real-world problems rather than just experimenting with tools and datasets.
- AI models are components within larger systems; their utility depends on the system's design and problem definition.
- Thorough problem definition, often involving domain experts and crafting specific examples, is crucial before implementing AI solutions.
- Manual prompt writing and inspection of tool behavior are essential for understanding and debugging AI systems.
- Robust evaluation is critical; avoid vague metrics like "looks good to me" and instead develop annotated datasets and clear objective metrics.
- DSPy can help modularize complex AI programs, but it has a steep learning curve and requires a well-defined problem structure.
- Custom metrics and evaluators may be necessary, especially for non-English languages, when using tools like DSPy.
- Iterative, small adjustments and a focus on measurement are key to improving AI system performance.
Notable quotes
*We are Engineers you know we have to reconsider the thing we are doing these these days which is like prompt engineering and start programming is systems again.*
*Please don't fall into these tools will solve the problem for me trap so get your hands dirty inspect what these tools are really doing for you.*
*You cannot improve anything which you don't measure.*
Unofficial community note. Prefer the recording for nuance.