How to Create and Simulate a Run [Example]

Early Access

Spyglass is currently in early access with trusted partners. Contact us for early access.

To reduce the size of the run for this example, we have uploaded a small custom dataset relating to the System Prompt Leakage vulnerability. Learn more about this LLM at: https://genai.owasp.org/llmrisk/llm072025-system-prompt-leakage/.

Start by adding the components to your Spyglass project. Navigate to Spyglass > Projects in the navigation menu.

1. Select Create Project in the top right corner.

   

2. Enter a Project Name and Project Description and click Create Project.

   

3. On the dedicated Project screen, select the Datasets tab and click Add Dataset. View spyglass custom datasets for instructions on how to upload a custom dataset.

4. Select the Dataset or Datasets you want to add to your project and click Add Dataset.
5. Select the Targets tab and click Add Target. If you haven't already done so, add an API key.

6. Now that you’ve provided your Datasets and Targets, you can create a Run that consists of a single Dataset, Target, and Scorer.

   

   Now you are ready to create your run. which is coupled to the project. Now you must select your added components:

7. Click Create Run in the top right corner of the Project screen.

8. Enter a Name (optional), select a Target, Dataset, and Scorer for this Run.

   In this example, we utilize the iPython Magic Commands Scorer as we are using its associated dataset. Learn more about the iPython Magic Commands at: https://ipython.readthedocs.io/en/stable/interactive/magics.html.

   To learn more about this public dataset, go to spyglass public datasets.

9. Click Create Run.

   As your Run progresses, you can monitor status on the Runs tab of your Project.

   

After a Run is complete, you can view the raw output data and visualizations on the Runs tab of your Project and download the output dataset.

Click on a sample from the run to view the full list of inputs and outputs: