Session brief
AI Engineering at Jane Street - John Crepezzi
Overview
This talk details Jane Street's approach to integrating large language models (LLMs) into their developer workflow, focusing on overcoming challenges posed by their unique technology stack, particularly their extensive use of OCaml. The core thesis is that building custom LLM solutions, including fine-tuned models and tailored editor integrations, is necessary to maximize value when off-the-shelf tools are insufficient due to specialized languages and internal development environments.
Who should watch
- AI Engineers
- Product Managers
- Developers working with niche or internal programming languages
- Teams looking to build custom AI-powered developer tools
- Those interested in LLM fine-tuning and evaluation strategies
Key takeaways
- Jane Street uses OCaml as its primary development language, necessitating custom tooling as standard LLMs struggle with this obscure language.
- The company has built its own build systems, code review tools (Iron), and even uses Mercurial for its monorepo, further complicating off-the-shelf integration.
- To effectively fine-tune models for OCaml, Jane Street created a data generation pipeline using workspace snapshotting to capture context, build errors, and successful fixes.
- A Code Evaluation Service (CES) was developed to accelerate the reinforcement learning phase, allowing for rapid testing of model-generated code against build and test statuses.
- Editor integrations for Neovim, VS Code, and Emacs were unified under a single service called Aid, which acts as a sidecar application for flexibility and easier updates.
- Aid allows for A/B testing of different models and prompting strategies, enabling data-driven decisions on model performance and developer acceptance.
- The team prioritizes a pluggable architecture for Aid, allowing for easy swapping of models, context-building strategies, and the addition of domain-specific tools across all supported editors.
- Meaningful evaluation is crucial to prevent models from producing nonsensical outputs, as demonstrated by a code review model that responded with "I'll do it tomorrow."
Notable quotes
*The models themselves are just not very good at OCaml.*
*We wanted the ability to kind of take LLMs and apply them to different parts of our development flow and light up different parts.*
*I'll do it tomorrow.*
Unofficial community note. Prefer the recording for nuance.