Session brief
Trust, but Verify: Shreya Rajpal
Overview
This talk introduces a new programming paradigm for generative AI applications called Trust, but Verify. It addresses the inherent non-deterministic nature of machine learning models, which leads to issues like hallucinations and unreliability, especially when moving from prototyping to production. The proposed solution involves implementing a verification suite around LLM outputs to ensure correctness and build trust in AI applications.
Who should watch
- AI Engineers
- Product Managers
- Builders of generative AI applications
- Those experiencing unreliability in AI applications after prototyping
- Developers seeking to add correctness guarantees to LLM outputs
Key takeaways
- Generative AI applications often suffer from lower retention rates compared to traditional software due to inherent non-determinism and issues like hallucinations.
- Unlike traditional software systems that provide deterministic outputs, LLM APIs can produce different responses even for the same input, leading to compounding errors.
- The proposed "Trust, but Verify" paradigm involves a verification suite that checks LLM outputs for functional inconsistencies and risks before they are used.
- Guardrails AI is an open-source framework that acts as a safety firewall, orchestrating the validation and verification process for LLM outputs.
- This framework allows for custom validators, a catalog of common guardrails, and automatic prompt compilation to communicate requirements to the LLM.
- Instead of relying solely on prompt engineering or fine-tuning, which offer no guarantees, this approach ensures outputs are valid and can trigger self-correction mechanisms in LLMs.
- Guardrails can be implemented using various methods, including grounding in external systems, rule-based heuristics, traditional ML classifiers, and LLM self-reflection.
- The framework supports policies like re-prompting, filtering, fixing, or falling back on other systems when verification fails, ensuring only trusted outputs are returned.
Notable quotes
*Add a constraint checker to check for valid generation. On violation inject what was generated and the rule violation and regenerate.*
*Guardrails acts as a safety firewall around your LLMs and this kind of fundamentally introduces a novel paradigm.*
*The paradigm that we propose is that only use large language model outputs if your verification suite passes.*
Unofficial community note. Prefer the recording for nuance.