Steering the Shoggoth: Taming LLMs with Sequential Monte Carlo

In this blog post, we present our findings from an exciting direction in controlling text generation with large language models. Using sequential Monte Carlo (SMC) steering, we can programmatically define constraints on the output of a model, ensuring it adheres to specific formats or styles. SMC offers optimized search for completions and supports both hard and soft constraints. The article introduces SMC as an alternative to beam search and token masking, explains the Feynman–Kac formulation, and discusses a new lampl library for implementing SMC steering. We demonstrate steering away from undesirable tokens, controlling length, and triggering generation based on entropy or control vectors.