Quantifying Training Membership Information in the Hyperspherical Embedding Geometry of Face Recognition Models
Trains face-recognition models on WebFace4M with angular-margin losses, evaluates them on standard verification benchmarks, and runs the membership-inference (MIA) analysis: per-identity embedding-geometry statistics computed for members (training identities) and non-members, and their separability.
This README is a run guide -- requirements, install, path configuration, the exact data formats expected, and every command with what it reads and writes.
Contents: 1 Requirements | 2 Install | 3 Paths | 4 Data | 5 Quick check | 6 Pipeline | 7 Config reference | 8 Outputs | 9 Third-party models | 10 Reproducing the paper | 11 Command reference | 12 Layout | Citation
- Linux x86-64. The environment is locked for
linux-64only. - Python 3.10 or 3.11 (pinned in
pixi.toml). - NVIDIA GPU with CUDA 12.1 for training. The pinned build is
pytorch-cuda=12.1. Evaluation runs on CPU too (--device cpu/device: cpu), just slowly. - Disk for the WebFace4M RecordIO plus checkpoints: one IResNet-50
final.ptis ~175 MB, and each run also keeps 4 pinned epochs andkeep_last_nrolling checkpoints.
Training the full paper grid is a large job (9 backbone x loss combinations x 5 training-set sizes, 40 epochs each). Every evaluation and analysis step below is cheap by comparison and can be run against checkpoints you already have.
Pixi builds the environment from the committed pixi.toml + pixi.lock:
pixi install # create the environment from the lock file
pixi shell # activate itAll commands below assume the environment is active and that you are in the repo root
(some example paths are repo-relative). Without pixi shell, prefix commands with
pixi run, e.g. pixi run python -m face_mia.train ....
Nothing in the repo hard-codes an absolute path. Three roots control where data is read and results are written. All are environment variables; unset, they default under the repo root, so a fresh clone runs in place.
| variable | holds | default |
|---|---|---|
FACE_DATA |
benchmarks (filelists + images) and training data | ./data |
FACE_MIA_CHECKPOINTS |
trained model weights | ./checkpoints |
FACE_MIA_RESULTS |
embeddings, stats, tables, figures | ./results |
export FACE_DATA=/path/to/data
export FACE_MIA_CHECKPOINTS=/path/to/checkpoints
export FACE_MIA_RESULTS=/path/to/resultsFour specific inputs sit at a generic $FACE_DATA/<name> default and can each be
relocated independently, without touching $FACE_DATA:
| variable | default | holds (expected format) |
|---|---|---|
FACE_WEBFACE4M |
$FACE_DATA/WebFace4M-preprocessed |
WebFace4M RecordIO: train.rec + train.idx + train.lst (4.2) |
FACE_FILELISTS |
$FACE_DATA/filelist |
per-benchmark protocol filelists, one <ds>/ sub-directory each (4.1) |
FACE_IJBC_META |
$FACE_DATA/ijbc/meta |
IJB-C metadata: ijbc_face_tid_mid.txt + ijbc_name_5pts_score.txt (4.3) |
FACE_XQLFW_SCORES |
$FACE_DATA/xqlfw/xqlfw_scores.txt |
XQLFW per-image quality scores (4.3) |
All seven variables are defined in one place, face_mia/paths.py; configs refer to them as
${FACE_DATA} / ${FACE_WEBFACE4M} / ${FACE_MIA_CHECKPOINTS} placeholders, expanded at
load time. Per-benchmark image directories are deliberately not fixed in code: each is
an image_root: key in the eval configs, defaulting to $FACE_DATA/<ds>/images.
$FACE_DATA/
WebFace4M-preprocessed/ train.rec train.idx train.lst # training data (RecordIO)
filelist/<ds>/ <ds>.csv enroll.csv verif.csv ... # per-benchmark protocol
<ds>/images/ raw benchmark images # image_root: in configs/eval
ijbc/meta/ ijbc_face_tid_mid.txt ijbc_name_5pts_score.txt
xqlfw/xqlfw_scores.txt # XQLFW quality scores
<ds> is a benchmark name: lfw, xqlfw, cfp_ff, cfp_fp, cplfw, agedb, rfw,
ijbc. Any location above can be moved via its env var, or (for images) its config key.
None of the datasets can be redistributed here; obtain each from its original source. The code reads benchmarks through filelists plus a directory of raw images.
A benchmark <ds> is one directory $FACE_FILELISTS/<ds>/ containing:
| file | required columns | purpose |
|---|---|---|
<ds>.csv |
crop_id, subject_id, image_path, left_eye_x/y, right_eye_x/y, nose_x/y, left_mouth_x/y, right_mouth_x/y |
one row per face crop |
enroll.csv |
template_id, subject_id, crop_id |
verification enrolment templates |
verif.csv |
template_id, subject_id, crop_id |
verification probe templates |
comparisons.csv |
enroll_template_id, verif_template_id, is_genuine |
verification pairs |
mia_template.csv (optional) |
crop_id |
single-condition crop subset for MIA |
Notes, all load-bearing:
crop_idmust be an integer and unique within<ds>.csv; the other files reference it.image_pathis resolved relative toimage_rootfrom the config (image_root / image_path), so it may contain sub-directories.- The ten landmark columns are the 5 points (eyes, nose, mouth corners) in raw-image pixel coordinates. Alignment to 112x112 is done from these, not by a runtime detector, so results do not depend on a detector version.
is_genuineis 0/1 (the stringstrue/falseare also accepted).mia_template.csvrestricts the MIA statistic to one within-identity condition. Mixing conditions (frontal + profile, high + low quality) corrupts the compactness signal, so the confounded benchmarks (cfp_fp,cfp_ff,cplfw,xqlfw) use it. Verification is unaffected -- it always uses the full enrol + probe protocol. Generate these withpython -m face_mia.run.build_mia_templates(needsFACE_XQLFW_SCORESforxqlfw).- Extra columns (
split,source,template_id,subject_name, face boxes,pose,age,ethnicity, ...) are ignored by the loader. Some are used byanalysis.factor_attribution:pose(CFP),age(AgeDB),ethnicity(RFW).
filelist/example/ in this repo is a filled-in example of all five files; see
section 5.
MXNet RecordIO holding pre-aligned 112x112 crops, at FACE_WEBFACE4M, as three files:
| file | format |
|---|---|
train.rec |
the records: [4-byte magic][4-byte length][IRHeader][JPEG bytes], padded to 4 bytes. IRHeader is struct format IfQQ = (flag, label, id, id2), 24 bytes; length counts header + image |
train.idx |
one line per record, record_key<TAB>byte_offset, tab-separated |
train.lst |
one line per record, record_key<TAB>label, tab-separated (further columns ignored) |
train.lst provides the record-key -> identity-label map. It is required for the member /
non-member split (dump_webface4m reads it); the training dataset falls back to the label
in each record header if it is absent, with a warning. Read by face_mia/data/recordio.py.
Only needed for those two benchmarks:
FACE_IJBC_META--ijbc_face_tid_mid.txtandijbc_name_5pts_score.txt, in the standard InsightFace IJB-C layout. Used for media-aware template pooling and faceness weighting (face_mia.eval.ijbc --insightface,face_mia.run.eval_benchmarks).FACE_XQLFW_SCORES-- the officialxqlfw_scores.txt: tab-separated with columnsID,Num,Score. Used only bybuild_mia_templatesto pick the low-quality half.
filelist/ ships a runnable placeholder: 24 Gaussian-noise 112x112 images (6 identities
x 4 crops) plus the five protocol CSVs in the real schema. configs/eval/example.yaml
points at it. Use it to verify an install end-to-end without any real benchmark.
It needs one checkpoint. Put any encoder matching the config's arch: (iresnet50 by
default) where the config expects it, then run the three eval stages:
mkdir -p "$FACE_MIA_CHECKPOINTS/example"
cp /path/to/your/final.pt "$FACE_MIA_CHECKPOINTS/example/final.pt"
python -m face_mia.run.dump_embeddings configs/eval/example.yaml
python -m face_mia.run.build_verification configs/eval/example.yaml
python -m face_mia.run.build_identity_mia configs/eval/example.yamlThis writes $FACE_MIA_RESULTS/{embeddings,verification,identity_mia}/example/....
The images are noise, so the numbers are meaningless (EER ~ 0.5) -- only the fact that
each stage runs and writes its outputs is. The example doubles as a template: copy it,
swap in a real filelist_dir / image_root, and you have a working benchmark config.
Throughout: <combo> is <backbone>_<head> (e.g. iresnet50_arcface), <model> is a
name: inside a config, <ds> is a benchmark. Steps (c)-(g) need only checkpoints.
Not a separate command -- it is a training argument. The member set of size N is
chosen deterministically from num_identities=N + data_seed, fixed at training time, and
saved as identity_indices.npy beside the checkpoint (the original WebFace4M identity
labels). Every later step reads that file, so the split is reused verbatim and never
recomputed. Sizes in the paper: 1K / 10K / 50K / 100K / all.
python -m face_mia.train --config configs/train/arcface_iresnet50.yaml \
--num-identities 10000 # any config key can be overridden
# multi-GPU (DistributedDataParallel):
torchrun --nproc_per_node=4 -m face_mia.train --config configs/train/arcface_iresnet50.yaml- Configs (11):
configs/train/{arcface,cosface,magface}_{iresnet34,iresnet50,iresnet100}.yamlplusarcface_vit_t.yamlandcosface_vit_t.yaml. - Reads
data_root:(default${FACE_WEBFACE4M}). - Writes
$FACE_MIA_CHECKPOINTS/<backbone>_<head>_n<N>_s<seed>/, where<N>is the identity count orall(e.g.iresnet50_arcface_n10000_s42,iresnet50_arcface_nall_s42). The name is deterministic -- no timestamp -- so eval configs can name it in advance. Contents:final.pt, pinnedepoch_XXX.pt(frompin_epochs, default 5/10/20/40), rollingcheckpoint_epochXXX_stepXXXXXXX.pt(lastkeep_last_n),config.yaml,identity_indices.npy,train_log.jsonl. - CLI flags override config keys; run
--helpfor the full list. Useful ones:--epochs,--batch-size,--lr,--num-workers,--no-fp16,--data-seed,--output-dir,--resume.
python -m face_mia.run.dump_embeddings configs/eval/grid_<combo>.yamlEmbeds every needed crop once (image + horizontal mirror, raw/unnormalised) and caches it,
so verification and MIA never re-run the model. Writes
$FACE_MIA_RESULTS/embeddings/<combo>/<model>/<ds>.npz + _meta.json. Run this first:
steps (d) and (e) read the cache.
python -m face_mia.run.build_verification configs/eval/grid_<combo>.yamlPools embeddings into templates, scores comparisons.csv, computes EER / AUC / TMR@FMR.
Writes $FACE_MIA_RESULTS/verification/<combo>/<ds>/<model>/{summary.json,scores.npz}.
Non-member side (benchmarks) and member side (WebFace4M):
# non-members: per-identity statistics on each benchmark (reads the (c) cache)
python -m face_mia.run.build_identity_mia configs/eval/grid_<combo>.yaml
# members + non-member reference, from WebFace4M itself
python -m face_mia.run.dump_webface4m configs/eval/member_<combo>.yaml
python -m face_mia.run.build_member_stats configs/eval/member_<combo>.yamlWrites $FACE_MIA_RESULTS/identity_mia/<combo>/<ds>/<model>/identity_mia_stats.csv
and $FACE_MIA_RESULTS/webface4m_mia/<combo>/<model>/identity_mia_stats.csv
(dump_webface4m also caches embeddings/<combo>/<model>/webface4m.npz).
Columns are described in section 8. Membership AUC is member-vs-non-member
per statistic, computed by the analysis scripts below.
python -m analysis.generate_analysis # tab-eer.tex, tab-anova.tex, fig-grid-*.pdf
python -m analysis.generate_epoch_tables # tab-eer-ep{5,10,20,40}.texgenerate_analysis builds the combined EER + MIA-AUC landscape table, the model-level and
score-level ANOVA over backbone / loss / n_ids / epoch, and the score-distribution figure
grid. generate_epoch_tables is the same landscape table split per epoch.
python -m analysis.factor_attribution --all # pose / quality / age / ethnicity
python -m analysis.mlp_fusion --all # fuse the four statistics with an MLPfactor_attribution runs the natural experiments that separate membership signal from
input-distribution shift; it reads the benchmark filelists (landmarks, pose, age,
ethnicity columns), not just the results tree. mlp_fusion cross-validates an MLP over
the four statistics.
The scripts in
analysis/are hard-wired to the paper's grid -- backbonesiresnet34/50/100, headsarcface/cosface/magface, sizes 1K/10K/50K/100K/all, epochs 5/10/20/40 -- and glob the results tree for it. They are not parameterised by a config; missing cells render as--rather than failing. Adjust the constants at the top of each script to analyse a different grid.
output_dir: results/identity_mia/iresnet50_arcface # ONLY the last component is read
datasets:
- name: lfw # must match <name>.csv in filelist_dir
filelist_dir: ${FACE_DATA}/filelist/lfw
image_root: ${FACE_DATA}/lfw/images
protocol_filter: true # keep only enrol+probe crops
mia_template_csv: ${FACE_DATA}/filelist/cfp_fp/mia_template.csv # optional
models:
- name: iresnet50-arcface-1K-ep5 # names the output directory
arch: iresnet50 # must match the weights
weights: ${FACE_MIA_CHECKPOINTS}/iresnet50_arcface_n1000_s42/epoch_005.pt
batch_size: 256
device: cuda
fmr_targets: [1.0e-3, 1.0e-4, 1.0e-5, 1.0e-6] # optional, for TMR@FMR
output_diris not an output path. Only its last component is used, as the<combo>directory name under$FACE_MIA_RESULTS. The results root isFACE_MIA_RESULTS, and the sub-tree (embeddings/,verification/, ...) is fixed by the step you run. The leadingresults/identity_mia/is inert.
output_dir: results/webface4m_mia/iresnet50_arcface # again: last component = <combo>
data_root: ${FACE_WEBFACE4M}
n_member: 2500 # member identities to sample
n_nonmember: 2500 # non-member identities to sample (0 available for nall models)
sample_seed: 123 # sampling is deterministic given this seed
models: [ ... ] # as above; needs identity_indices.npy beside the weights
batch_size: 256
device: cudaKeys: backbone, head, embedding_size, dropout, scale, margin, head_kwargs,
num_identities, data_seed, data_root, skip_preprocess, num_workers, epochs,
batch_size, lr, weight_decay, momentum, optimizer, warmup_epochs,
lr_schedule, lr_milestones, lr_gamma, fp16, sample_rate, flip_prob,
low_res_prob, save_every_epochs, save_every_steps, keep_last_n, pin_epochs,
resume, log_every, tensorboard, output_dir. Every one has a CLI override.
Everything lands under $FACE_MIA_RESULTS:
| path | written by | contents |
|---|---|---|
embeddings/<combo>/<model>/<ds>.npz |
(c) | crop_ids (int64), emb_raw (N,D), emb_flip_raw (N,D) -- raw, unnormalised |
embeddings/<combo>/<model>/webface4m.npz |
(e) | subject_ids, is_member (bool), emb_raw, emb_flip_raw |
embeddings/<combo>/<model>/_meta.json |
(c) | arch, weights, head, scale, margin, head_kwargs |
verification/<combo>/<ds>/<model>/summary.json |
(d) | eer, auc, tmr_at_fmr (per FMR target), n_comparisons, n_valid, n_missing, n_genuine, n_impostor |
verification/<combo>/<ds>/<model>/scores.npz |
(d) | one score per comparisons.csv row + labels |
identity_mia/<combo>/<ds>/<model>/identity_mia_stats.csv |
(e) | non-member statistics |
webface4m_mia/<combo>/<model>/identity_mia_stats.csv |
(e) | member + non-member statistics, with is_member |
tables/*.tex, figures/*.pdf |
(f) | LaTeX tables, figures |
factor_attribution/, mlp_membership/mlp_summary.csv |
(g) | confound analyses |
identity_mia_stats.csv has one row per identity:
subject_id, n_crops, pairwise_cos, inertia, vmf_kappa, penalized_logit, proto_softmax_loss
plus is_member on the WebFace4M side. The four membership statistics are pairwise_cos
(mean intra-identity pairwise cosine), vmf_kappa (vMF concentration), penalized_logit
(mean leave-one-out penalised logit, head-specific) and proto_softmax_loss
(reconstructed-head cross-entropy); inertia (mean cosine to the identity centroid) is an
additional compactness measure. All need >= 2 crops per identity; identities with fewer get
NaN. Embedding caches are deletable intermediates -- delete embeddings/ to reclaim space.
Evaluation reads only embeddings, so any face encoder works. Point a config at it:
models:
- name: my-model
arch: iresnet100 # backbone the weights match
weights: ${FACE_MIA_CHECKPOINTS}/my_model/final.pt- Supported
archvalues:iresnet18,iresnet34,iresnet50,iresnet100,iresnet200,vit_t,vit_s,vit_b. - Accepted checkpoint layouts: a training checkpoint written here (a dict with a
backbonekey, plusconfigandepoch), or a barestate_dict. Amodule.prefix is stripped; the load is strict, soarchmust match the weights exactly. - Two of the four statistics need the training head.
penalized_logitandproto_softmax_lossare reconstructed from the head type +margin(+scale), read from the checkpoint'sconfig. A barestate_dictcarries no config, so both come outNaN;pairwise_cos,inertiaandvmf_kappaneed embeddings alone and are always computed. To recover them for a foreign checkpoint, edithead/margin/scalein the_meta.jsonthat step (c) writes next to the embeddings, then re-run step (e). - The member side (
dump_webface4m) additionally needsidentity_indices.npybeside the weights, listing the WebFace4M identity labels the model was trained on. Without it every identity is treated as a member, which yields a member-only reference and no membership AUC.
- Train 45 runs: 9 combos (
{iresnet34,iresnet50,iresnet100}x{arcface,cosface,magface}) x 5 sizes (--num-identities 1000/10000/50000/100000/-1,-1= all), 40 epochs,data_seed 42. This produces the checkpoint names the shipped eval configs already reference. - For each combo, run (c), (d) and (e) with
configs/eval/grid_<combo>.yamlandconfigs/eval/member_<combo>.yaml. Each config lists all 20 models (5 sizes x 4 pinned epochs), so one invocation covers the combo. - Build the MIA template lists once:
python -m face_mia.run.build_mia_templates. - Generate tables and figures:
analysis.generate_analysis,analysis.generate_epoch_tables,analysis.factor_attribution --all,analysis.mlp_fusion --all.
The vit_t configs (grid_vit_t_cosface.yaml, member_vit_t_cosface.yaml) cover the
generalisation check outside the main grid; the analysis scripts' default grid excludes them.
Every entry point. Config arguments are positional; --device overrides the config.
| command | argument | purpose |
|---|---|---|
face_mia.train |
--config <cfg> + overrides |
train (b) |
face_mia.run.dump_embeddings |
<eval cfg> |
embed + cache crops (c) |
face_mia.run.build_verification |
<eval cfg> [--device] |
EER / AUC / TMR@FMR (d) |
face_mia.run.build_identity_mia |
<eval cfg> [--device] |
non-member statistics (e) |
face_mia.run.dump_webface4m |
<member cfg> |
embed WebFace4M members + non-members (e) |
face_mia.run.build_member_stats |
<member cfg> [--device] |
member statistics (e) |
face_mia.run.build_mia_templates |
(none) | write mia_template.csv for the confounded benchmarks |
face_mia.run.eval_benchmarks |
<eval cfg> [--device] [--model-idx N] |
verification with flip-TTA + IJB-C pooling |
face_mia.eval.ijbc |
--weights --arch --image-root [--filelist-dir --meta-dir --insightface --no-flip --output] |
IJB-C 1:1 with InsightFace-style pooling |
analysis.generate_analysis |
(none) | tab-eer.tex, tab-anova.tex, fig-grid-*.pdf |
analysis.generate_epoch_tables |
[--epoch N ...] [--dry-run] |
tab-eer-ep{5,10,20,40}.tex |
analysis.factor_attribution |
`[--all] [--arch A --loss L] [--metric M | --all-metrics]` |
analysis.mlp_fusion |
[--all] [--combo C] [--nids ...] [--n-folds K] |
MLP fusion of the four statistics |
analysis.generate_roc_plots |
(none) | verification ROC / DET figures |
analysis.generate_mia_roc_plots |
(none) | MIA ROC / DET figures |
analysis.generate_all_grid_plots |
(none) | score-distribution grid, all combos |
analysis.plot_score_distributions |
`[--arch --loss --nids --epoch --metric | --all-metrics --outdir]` |
face_mia.eval.{verification,identity_mia,webface4m} are the library modules behind the
run/ entry points and can also be called directly; prefer run/, which shares the
embedding cache.
configs/train/ 11 training configs
configs/eval/ grid_* (verification/MIA), member_* (WebFace4M), example.yaml
face_mia/
paths.py all data / checkpoint / result locations
train.py trainer (python -m face_mia.train)
backbones/ heads/ IResNet + ViT; ArcFace/CosFace/MagFace/AdaFace/Softmax + PartialFC
data/ recordio.py (RecordIO reader) + dataset.py (training Dataset)
eval/ identity_mia, verification, webface4m, ijbc
run/ dump_* / build_* command-line entry points
analysis/ factor_attribution, mlp_fusion, generate_* / plot_* (tables, figures)
filelist/ runnable placeholder benchmark (see section 5)
tests/ end-to-end smoke test (pytest tests/, or python tests/test_smoke.py)
results/ generated output (gitignored)
@article{ozturk2026quantifying,
title = {Quantifying Training Membership Information in the Hyperspherical
Embedding Geometry of Face Recognition Models},
author = {{\"O}zt{\"u}rk, {\"U}nsal and Marcel, S{\'e}bastien},
year = {2026},
eprint = {2607.15084},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2607.15084}
}MIT -- see LICENSE.
Some files derive from InsightFace, which is likewise MIT; see THIRD-PARTY.md for what is derived and from where. No third-party datasets or pre-trained models are distributed here.