This is the source code, including scripts and evaluation data, used in our project titled "A Large-Scale Empirical Analysis of Custom GPTs’ Vulnerabilities in the OpenAI Ecosystem". Below is a comprehensive description of each of the datasets and scripts.
GPTs_5_percent_random_sample_with_updated_metrics.xlsx
This is an updated subset of the Beetrove dataset, incorporating the latest GPT metadata from the OpenAI store. This served as the decision matrix for the Entropy-TOPSIS multi-criteria decision-making algorithm used to score and rank GPTs based on popularity and category.
Images
This directory contains all the images used in the project.
Combined Python Codes
This directory contains all the datasets and Python scripts used for vulnerability assessment in custom GPTs in one place.
Datasets Overview
The Data Collection & Analysis folder contains all key datasets and scripts used in the data collection and analysis section of this project:
• criteria.xlsx – Defines evaluation criteria used in the analysis. All listed criteria are benefit-oriented metrics.
• Privacy_Analysis_GPTs_5_percent_random_sample_with_updated_metrics.xlsx – The primary dataset for vulnerability assessment. It is an enriched version of a 5% Beetrove sample, containing updated metrics, vulnerability patterns, and privacy analysis results.
• Trend.xlsx – Used to compute the cumulative distribution function (CDF) for identifying trends in vulnerability exposure.
Data Collection & Analysis
This directory contains the criteria.xlsx, Privacy_Analysis_GPTs_5_percent_random_sample_with_updated_metrics.xlsx, Trend.xlsx, and Data Collection_Analysis.py.
Data Collection_Analysis.py
This Python script contains utility functions and analysis scripts. Here's a step-by-step breakdown of its components:
Script 1: Convert GPT Creation Timestamps to UNIX Format
It loads the Accessible GPTs Performance sheet, extracts creation dates from a specific column, converts them to numerical (UNIX timestamp) format, and saves the new values to a designated column within the same file.
Script 2: Category Distribution Bar Chart
It visualizes the frequency of GPTs across categories like Education, Programming, Productivity, etc., and highlights the prevalence of GPTs falling under "None" or uncategorized apps.
Script 3: Temporal Evolution of GPT Submissions
It compares how the GPT counts evolve between the original and updated datasets and displays week-by-week growth of custom GPTs in the OpenAI Store.
Script 4: Popularity Score Computation and Ranking of GPTs Across Categories (Entropy-TOPSIS Method)
This script performs multi-criteria decision-making (MCDM) analysis by iterating over various GPT categories within an Excel workbook, calculating a popularity score for each GPT in each category using the Entropy Weight-TOPSIS method. It then ranks GPTs accordingly and appends the results to the workbook. Its core functionalities include:
Entropy Weighting: Dynamically determines the relative importance of each evaluation criterion based on variability (entropy) in the decision matrix.
TOPSIS (Technique for Order Preference by Similarity to Ideal Solution): Ranks GPTs by comparing their performance to ideal (best) and anti-ideal (worst) solutions.
Automated Iteration Over Sheets: Processes each GPT category as a separate worksheet.
Excel Writing: Appends calculated performance scores and ranks directly into the corresponding Excel sheet.
Script 5: While Script 4 computed popularity scores and ranks for each GPT category using their respective sheets, this script extends this logic to the "Main" sheet, enabling a unified and comparative ranking of all GPTs together under consistent evaluation criteria.
Vulnerability Assessment
This directory contains the Privacy_Analysis_GPTs_5_percent_random_sample_with_updated_metrics.xlsx and all the scripts used to evaluate and document vulnerabilities across various GPTs, categorized by popularity and type. The primary goal of this module is to automate the testing of each GPT's response to jailbreaking prompts, capture their behavior visually, and store the outputs for manual review.
Scripts: Vulnerability_Analysis.py
This Python script contains core scripts for automated vulnerability testing.
Script 1: This script automates the process of sending a prompt to a list of GPT instances, collecting their responses, taking screenshots of the interactions, and compiling the results into a single PDF file for analysis. Its core functionalities include:
Reading GPT instance URLs from the “Privacy_Analysis_GPTs_5_percent_random_sample_with_updated_metrics.xlsx” Workbook. Using Selenium to interact with each GPT interface. Automatically sending a predefined prompt to each GPT instance. Capturing screenshots of the GPT's response for each instance. Batching screenshots into intermediate PDFs and merging them into one final document for manual review.
Script 2: This script performs the same automated vulnerability assessment as Script 1 but is specifically tailored to GPTs categorized under “Other” and “None” in the Excel Workbook.
Script 3: This script analyzes each category sheet in the Excel workbook to count the number of benign (0) and vulnerable (1) custom GPTs based on their response to jailbreaking prompts.
Script 4: This script divides GPTs in each category sheet into three row-based segments (35%, 30%, 35%) and counts the number of benign (0) and malicious (1) GPTs in each segment. It helps to analyze how vulnerability is distributed across different popularity levels of the dataset.
Scripts 5, 6, 7, and 8, 9, 10, and 11: These scripts visualize the vulnerability analysis of system prompt leakage, roleplay, reverse psychology, DEN jailbreak, phishing, social engineering, and malware code generation attacks, respectively, across different categories of Custom GPTs. It creates a stacked bar chart showing the number of vulnerable (malicious) and non-vulnerable (benign) GPTs per category, alongside a line graph overlay indicating the percentage of vulnerable GPTs in each category.
Vulnerability Patterns
This directory contains the Privacy_Analysis_GPTs_5_percent_random_sample_with_updated_metrics.xlsx, Trend.xlsx, and all the scripts used to analyze vulnerability patterns in custom GPTs, as contained in Vulnerability_Patterns(1).py.
Scripts: Vulnerability_Analysis.py
This Python script contains core scripts for determining the vulnerability patterns.
Scripts 1 – 7: These scripts generate a grouped bar chart to illustrate the percentage of vulnerable GPTs in three popularity segments — Top 35%, Middle 30%, and Bottom 35% — across the GPT categories.
Scripts 8: This script performs a vulnerability distribution analysis for Custom GPT apps by counting the number of "zero" values across the custom GPTs in the vulnerability analysis results in the Privacy_Analysis_GPTs_5_percent_random_sample_with_updated_metrics.xlsx” workbook, in seven GPT categories, where a "zero" implies non-vulnerability in a custom GPT.
Script 9: This script creates a dual-axis line and bar chart to visualize GPT vulnerability distributions across categories and overall. It combines category-specific vulnerability curves with an overall vulnerability percentage histogram, giving a comprehensive view of GPT exposure levels across multiple domains. To compare how vulnerable GPTs are across seven app categories (like Productivity, Research, etc.) and simultaneously show the overall distribution of vulnerabilities in the dataset.
Script 10: This Python script reads UNIX timestamps from column K (11th column) of an Excel file named Trend.xlsx and converts them into human-readable datetime format. The converted dates are then written into column V (22nd column) of the same sheet.
Script 11: This Python script analyzes and visualizes the cumulative distribution of vulnerabilities in Custom GPTs over time, based on data from an Excel file named Trend.xlsx. It provides a comparative visualization of how vulnerabilities (and resistance) among GPTs are distributed over time, helping to track trends in safety performance.
Script 12: This code generates a horizontal stacked bar chart that visualizes the percentage of Custom GPTs that are vulnerable vs. resistant to seven different types of attack methods. This visualization clearly shows how each attack vector affects GPTs, highlighting which vulnerabilities are most common and where resistance is strongest.