🚧 Package under construction
Repo to host the folktexts project.
Package documentation can be found here!
Table of contents:
Install package from PyPI:
pip install folktexts
- Create condo environment
conda create -n folktexts python=3.11
conda activate folktexts
- clone git repo
git clone git@github.com:AndreFCruz/folktexts.git
- Install folk texts package in edit mode
cd folktexts
pip install -e .
- Create models dataset and results folder
mkdir results
mkdir models
mkdir datasets
- Download models into models folder (make sure you have Huggingface permission)
python -m folktexts.cli.download_models --model "google/gemma-2b" --save-dir models
- Run benchmark (It downloads dataset and runs benchmarking)
python folktexts/cli/run_llm_as_classifier.py --results-dir results --data-dir datasets --acs-task-name "ACSIncome" --model models/google--gemma-2b --subsampling 0.01
NOTE: saves results to results folder, not a subfolder!
This is a template of how we envision the API -- not yet fully implemented!
from folktexts.datasets import ACSDataset
from folktexts.acs import acs_income_task
from folktexts.qa_interface import BinaryQA
# Create an object that classifies data using an LLM
clf = LLMClassifier(
model=model,
tokenizer=tokenizer,
task=acs_income_task, # NOTE: the task should know how to map itself to text!
qa_interface=BinaryQA, # How to frame the question and extract outputs from the model
)
# Use a dataset or feed in your own data
dataset = ACSDataset(name="ACSIncome")
# Get risk score predictions out of the model
y_scores = clf.predict_proba(dataset)
# Optionally, can fit the threshold based on a small portion of the data
clf.fit(dataset[0:100])
# ...in order to get more accurate binary predictions
clf.predict(dataset)
# Compute a variety of evaluation metrics on calibration and accuracy
benchmark_results = run_llm_as_clf_benchmark(clf, dataset)Code licensed under the MIT license.
The American Community Survey (ACS) Public Use Microdata Sample (PUMS) is governed by the U.S. Census Bureau terms of service.