World's Fair 2024
Understanding AI Stakes to Break Production Code: Philip Rathle
Overview
This talk explores the relationship between the stakes of an AI application and the obstacles encountered during production. Higher stakes, defined by potential impacts on finances, reputation, health, or safety, correlate with increased challenges. The discussion contrasts vector-based retrieval-augmented generation (RAG) with knowledge graphs, suggesting that structured data and reasoning from knowledge graphs can enhance reliability for high-stakes use cases.
Who should watch
- AI engineers and builders facing production challenges.
- Product Managers and engineers focused on shipping AI products.
- Those working on applications where accuracy and reliability are critical.
- Individuals exploring advanced data representation beyond simple vectors.
Key takeaways
- The complexity and obstacles in AI production increase with the stakes of the application.
- While vector-based RAG is effective for many use cases, knowledge graphs offer a more structured approach for higher-stakes scenarios.
- LLMs can be used to generate code for specific tasks, effectively removing the LLM from the execution loop for deterministic problems.
- User adoption of new query methods, like semantic search, requires explicit guidance and education, as users may default to familiar keyword searches.
- Focusing on a few projects with clear production potential is more effective than broad experimentation, as many experimental projects may not meet user accuracy expectations.
- Starting with larger, more capable models (like GPT-4 or Claude 3.5) to validate product needs is often more successful than premature optimization with smaller models.
- Building an evaluation framework and ensuring the availability of necessary data should begin at the start of an AI project, not as an afterthought.
- Transparency with end-users about AI-generated content, through slow generation, UI cues, and validation tools, can manage expectations and encourage vigilance.
Notable quotes
*It's so easy to build an amazing prototype but then getting to production is harder.*
*The lesson is kill more projects than experimentation.*
*Start with GPT-40 or I guess the now now maybe Claude 35 and if that can't do it then the smaller models probably won't.*
Unofficial community note. Prefer the recording for nuance.