The widespread adoption of commercial chatbots calls for reliable tools for systematically evaluating their behaviour. While most work uses APIs to conduct these evaluations, recent work [1,2] has shown that some chatbots’ performance differs when prompted through the API versus through the web browser. In this project, you will build and open source a browser automation tool that enables AI auditors and researchers to automatically prompt these commercial chatbots through their web interface. The tool should support automated prompting, chat-log management, and importing of resulting transcripts, ideally across different commercial chatbot platforms. This would be similar to re-implementing https://behaviors.transluce.org/mental-health?detail=page:app_C_subject_models#browser-automation. You will then demonstrate the tool by using it to run structured evaluations of a selected chatbot behaviour. If time permits, you will conclude by open sourcing this tool.
School of Computer Science and Statistics