LLM evaluation pipeline development

While LLMs are claimed to be useful for an ever growing range of tasks, systematic evaluation is required to produce evidence of their actual performance. A core component of an automated LLM evaluation is the data used to prompt the model, i.e. the user’s “side” of the transcript. These inputs can be designed in a wide variety of ways – static or dynamic, single or multi turn, manually curated, scraped, or generated, and so on. In this project, you compare three prompting design choices: fixed multi-turn scripts, a bank of fixed replies selected dynamically by an LLM, and fully dynamic scripts generated by LLMs themselves. To do so, you will build an LLM evaluation pipeline which can be used with any of these approaches. You will then use the pipeline to generate transcripts on the same topic using each approach, allowing you to compare how different prompting approaches affect the resulting output.