Passing the lexical information in and requiring s-expressions be returned is a good idea. Put a cache on it to remember what the llm came up with last time and you have a legitimate, if somewhat weird, language implementation.