This repository is designed for evaluating the effectiveness of prompt templates and language models in rephrasing error messages for buggy code snippets. The web application allows users to select different models and prompt styles, generate improved error messages, and provide feedback on their quality. All feedback is stored locally and can be exported for further analysis.
- Node.js and npm installed
- Ollama installed and running locally
- Download the desired Ollama models on your machine before using the app (models must be available locally). More specifically, the following models are used for validation and need to be downloaded:
Note: Technically, any Ollama model can be used, but the ones above are specifically valid options on the frontend. If you want to use a different model, you can change the
ollamaModelin thesrc/data/codeSnippets.tsfile to match the model you have available locally.
-
Clone the repository:
git clone git@github.com:alemoraru/exceed-project-validation.git cd exceed-project-validation -
Install dependencies:
npm install
-
Start the development server:
npm run dev
The app will be available at http://localhost:8080.
- Open the app in your browser at http://localhost:8080.
- Select a code snippet and choose a model and error message style.
- Click "Improve Error" to generate a rephrased error message using the selected model and prompt.
- Review the improved error message and submit feedback using the feedback form.
- Download all feedback as a CSV file for further analysis (it's stored locally in the browser's localstorage).
This project was developed as part of the EXCEED MSc Thesis project at Technische Universiteit Delft. As such, contributions of any sort will not be accepted. This repository is provided for replication and educational purposes ONLY. Since it was used as a validation step, before deploying our study on Prolific, it is NOT intended for further development or contributions.
This project is licensed under the MIT License. See the LICENSE file for details.