Session brief
Your LLM Ran Out of Knowledge — Now What?
Overview
This talk addresses the challenge of applying large language models (LLMs) to domains where structured training data is scarce. The core thesis is that by providing LLMs with explicit rules, heuristics, and guidelines, similar to how junior professionals are mentored, their powerful reasoning capabilities can be effectively leveraged even in low-knowledge areas. This approach aims to bridge the gap between domains with abundant data and those lacking it, enabling LLMs to assist in complex problem-solving across a wider range of professions.
Who should watch
- AI engineers and builders working with LLMs in specialized or data-scarce domains.
- Product Managers seeking to expand LLM applications beyond well-established areas.
- Professionals in fields like corporate negotiations or risk management who lack structured data for AI training.
- Anyone interested in practical methods for enhancing LLM performance and domain expertise.
Key takeaways
- LLMs excel in domains with ample structured data (e.g., coding, math) but struggle in areas with implicit or undocumented knowledge.
- A technique is proposed to equip LLMs with domain-specific rules and heuristics, akin to providing guidelines to a junior employee.
- This method involves a parsing engine that identifies the problem type, reformats the input, and applies a curated set of rules.
- The approach uses a comprehensive prompt system that integrates domain-specific rules with general LLM reasoning capabilities.
- This technique can be applied to various domains, including corporate negotiations and geopolitics, by defining clear, provable rules.
- The system allows for flexibility in rule sets and can incorporate external simulators (like world simulations) for added realism and constraints.
- While human oversight is still recommended, this method enables LLMs to generate and evaluate numerous scenarios much faster than human experts.
- The approach offers a way to overcome knowledge gaps and apply LLM intelligence to underexplored areas.
Notable quotes
*The problem that we have is that we also have other areas where we don't have well structured information.*
*We're going to give them a set of rules to follow for specific domains and ask them to apply those on top of the very powerful reasoning capabilities that they now have.*
*The point is we can see now that it is not only using its reasoning ability the intelligence that we know the model has but it's also overcoming perhaps domain expertise or a lack of insight.*
Unofficial community note. Prefer the recording for nuance.