Skip to content

Repository files navigation

Quantifying Training Membership Information in the Hyperspherical Embedding Geometry of Face Recognition Models

Trains face-recognition models on WebFace4M with angular-margin losses, evaluates them on standard verification benchmarks, and runs the membership-inference (MIA) analysis: per-identity embedding-geometry statistics computed for members (training identities) and non-members, and their separability.

This README is a run guide -- requirements, install, path configuration, the exact data formats expected, and every command with what it reads and writes.

Contents: 1 Requirements | 2 Install | 3 Paths | 4 Data | 5 Quick check | 6 Pipeline | 7 Config reference | 8 Outputs | 9 Third-party models | 10 Reproducing the paper | 11 Command reference | 12 Layout | Citation


1. Requirements

  • Linux x86-64. The environment is locked for linux-64 only.
  • Python 3.10 or 3.11 (pinned in pixi.toml).
  • NVIDIA GPU with CUDA 12.1 for training. The pinned build is pytorch-cuda=12.1. Evaluation runs on CPU too (--device cpu / device: cpu), just slowly.
  • Disk for the WebFace4M RecordIO plus checkpoints: one IResNet-50 final.pt is ~175 MB, and each run also keeps 4 pinned epochs and keep_last_n rolling checkpoints.

Training the full paper grid is a large job (9 backbone x loss combinations x 5 training-set sizes, 40 epochs each). Every evaluation and analysis step below is cheap by comparison and can be run against checkpoints you already have.


2. Install

Pixi builds the environment from the committed pixi.toml + pixi.lock:

pixi install          # create the environment from the lock file
pixi shell            # activate it

All commands below assume the environment is active and that you are in the repo root (some example paths are repo-relative). Without pixi shell, prefix commands with pixi run, e.g. pixi run python -m face_mia.train ....


3. Paths

Nothing in the repo hard-codes an absolute path. Three roots control where data is read and results are written. All are environment variables; unset, they default under the repo root, so a fresh clone runs in place.

variable holds default
FACE_DATA benchmarks (filelists + images) and training data ./data
FACE_MIA_CHECKPOINTS trained model weights ./checkpoints
FACE_MIA_RESULTS embeddings, stats, tables, figures ./results
export FACE_DATA=/path/to/data
export FACE_MIA_CHECKPOINTS=/path/to/checkpoints
export FACE_MIA_RESULTS=/path/to/results

Named data locations

Four specific inputs sit at a generic $FACE_DATA/<name> default and can each be relocated independently, without touching $FACE_DATA:

variable default holds (expected format)
FACE_WEBFACE4M $FACE_DATA/WebFace4M-preprocessed WebFace4M RecordIO: train.rec + train.idx + train.lst (4.2)
FACE_FILELISTS $FACE_DATA/filelist per-benchmark protocol filelists, one <ds>/ sub-directory each (4.1)
FACE_IJBC_META $FACE_DATA/ijbc/meta IJB-C metadata: ijbc_face_tid_mid.txt + ijbc_name_5pts_score.txt (4.3)
FACE_XQLFW_SCORES $FACE_DATA/xqlfw/xqlfw_scores.txt XQLFW per-image quality scores (4.3)

All seven variables are defined in one place, face_mia/paths.py; configs refer to them as ${FACE_DATA} / ${FACE_WEBFACE4M} / ${FACE_MIA_CHECKPOINTS} placeholders, expanded at load time. Per-benchmark image directories are deliberately not fixed in code: each is an image_root: key in the eval configs, defaulting to $FACE_DATA/<ds>/images.

Default layout under $FACE_DATA

$FACE_DATA/
  WebFace4M-preprocessed/  train.rec  train.idx  train.lst    # training data (RecordIO)
  filelist/<ds>/           <ds>.csv enroll.csv verif.csv ...  # per-benchmark protocol
  <ds>/images/             raw benchmark images                # image_root: in configs/eval
  ijbc/meta/               ijbc_face_tid_mid.txt  ijbc_name_5pts_score.txt
  xqlfw/xqlfw_scores.txt                                       # XQLFW quality scores

<ds> is a benchmark name: lfw, xqlfw, cfp_ff, cfp_fp, cplfw, agedb, rfw, ijbc. Any location above can be moved via its env var, or (for images) its config key.


4. Data

None of the datasets can be redistributed here; obtain each from its original source. The code reads benchmarks through filelists plus a directory of raw images.

4.1 Benchmarks: filelists + images

A benchmark <ds> is one directory $FACE_FILELISTS/<ds>/ containing:

file required columns purpose
<ds>.csv crop_id, subject_id, image_path, left_eye_x/y, right_eye_x/y, nose_x/y, left_mouth_x/y, right_mouth_x/y one row per face crop
enroll.csv template_id, subject_id, crop_id verification enrolment templates
verif.csv template_id, subject_id, crop_id verification probe templates
comparisons.csv enroll_template_id, verif_template_id, is_genuine verification pairs
mia_template.csv (optional) crop_id single-condition crop subset for MIA

Notes, all load-bearing:

  • crop_id must be an integer and unique within <ds>.csv; the other files reference it.
  • image_path is resolved relative to image_root from the config (image_root / image_path), so it may contain sub-directories.
  • The ten landmark columns are the 5 points (eyes, nose, mouth corners) in raw-image pixel coordinates. Alignment to 112x112 is done from these, not by a runtime detector, so results do not depend on a detector version.
  • is_genuine is 0/1 (the strings true/false are also accepted).
  • mia_template.csv restricts the MIA statistic to one within-identity condition. Mixing conditions (frontal + profile, high + low quality) corrupts the compactness signal, so the confounded benchmarks (cfp_fp, cfp_ff, cplfw, xqlfw) use it. Verification is unaffected -- it always uses the full enrol + probe protocol. Generate these with python -m face_mia.run.build_mia_templates (needs FACE_XQLFW_SCORES for xqlfw).
  • Extra columns (split, source, template_id, subject_name, face boxes, pose, age, ethnicity, ...) are ignored by the loader. Some are used by analysis.factor_attribution: pose (CFP), age (AgeDB), ethnicity (RFW).

filelist/example/ in this repo is a filled-in example of all five files; see section 5.

4.2 Training data (WebFace4M)

MXNet RecordIO holding pre-aligned 112x112 crops, at FACE_WEBFACE4M, as three files:

file format
train.rec the records: [4-byte magic][4-byte length][IRHeader][JPEG bytes], padded to 4 bytes. IRHeader is struct format IfQQ = (flag, label, id, id2), 24 bytes; length counts header + image
train.idx one line per record, record_key<TAB>byte_offset, tab-separated
train.lst one line per record, record_key<TAB>label, tab-separated (further columns ignored)

train.lst provides the record-key -> identity-label map. It is required for the member / non-member split (dump_webface4m reads it); the training dataset falls back to the label in each record header if it is absent, with a warning. Read by face_mia/data/recordio.py.

4.3 IJB-C metadata and XQLFW scores

Only needed for those two benchmarks:

  • FACE_IJBC_META -- ijbc_face_tid_mid.txt and ijbc_name_5pts_score.txt, in the standard InsightFace IJB-C layout. Used for media-aware template pooling and faceness weighting (face_mia.eval.ijbc --insightface, face_mia.run.eval_benchmarks).
  • FACE_XQLFW_SCORES -- the official xqlfw_scores.txt: tab-separated with columns ID, Num, Score. Used only by build_mia_templates to pick the low-quality half.

5. Quick check (no data needed)

filelist/ ships a runnable placeholder: 24 Gaussian-noise 112x112 images (6 identities x 4 crops) plus the five protocol CSVs in the real schema. configs/eval/example.yaml points at it. Use it to verify an install end-to-end without any real benchmark.

It needs one checkpoint. Put any encoder matching the config's arch: (iresnet50 by default) where the config expects it, then run the three eval stages:

mkdir -p "$FACE_MIA_CHECKPOINTS/example"
cp /path/to/your/final.pt "$FACE_MIA_CHECKPOINTS/example/final.pt"

python -m face_mia.run.dump_embeddings    configs/eval/example.yaml
python -m face_mia.run.build_verification configs/eval/example.yaml
python -m face_mia.run.build_identity_mia configs/eval/example.yaml

This writes $FACE_MIA_RESULTS/{embeddings,verification,identity_mia}/example/.... The images are noise, so the numbers are meaningless (EER ~ 0.5) -- only the fact that each stage runs and writes its outputs is. The example doubles as a template: copy it, swap in a real filelist_dir / image_root, and you have a working benchmark config.


6. Pipeline

Throughout: <combo> is <backbone>_<head> (e.g. iresnet50_arcface), <model> is a name: inside a config, <ds> is a benchmark. Steps (c)-(g) need only checkpoints.

(a) Partition WebFace4M into members / non-members

Not a separate command -- it is a training argument. The member set of size N is chosen deterministically from num_identities=N + data_seed, fixed at training time, and saved as identity_indices.npy beside the checkpoint (the original WebFace4M identity labels). Every later step reads that file, so the split is reused verbatim and never recomputed. Sizes in the paper: 1K / 10K / 50K / 100K / all.

(b) Train

python -m face_mia.train --config configs/train/arcface_iresnet50.yaml \
       --num-identities 10000                    # any config key can be overridden

# multi-GPU (DistributedDataParallel):
torchrun --nproc_per_node=4 -m face_mia.train --config configs/train/arcface_iresnet50.yaml
  • Configs (11): configs/train/{arcface,cosface,magface}_{iresnet34,iresnet50,iresnet100}.yaml plus arcface_vit_t.yaml and cosface_vit_t.yaml.
  • Reads data_root: (default ${FACE_WEBFACE4M}).
  • Writes $FACE_MIA_CHECKPOINTS/<backbone>_<head>_n<N>_s<seed>/, where <N> is the identity count or all (e.g. iresnet50_arcface_n10000_s42, iresnet50_arcface_nall_s42). The name is deterministic -- no timestamp -- so eval configs can name it in advance. Contents: final.pt, pinned epoch_XXX.pt (from pin_epochs, default 5/10/20/40), rolling checkpoint_epochXXX_stepXXXXXXX.pt (last keep_last_n), config.yaml, identity_indices.npy, train_log.jsonl.
  • CLI flags override config keys; run --help for the full list. Useful ones: --epochs, --batch-size, --lr, --num-workers, --no-fp16, --data-seed, --output-dir, --resume.

(c) Embed crops

python -m face_mia.run.dump_embeddings configs/eval/grid_<combo>.yaml

Embeds every needed crop once (image + horizontal mirror, raw/unnormalised) and caches it, so verification and MIA never re-run the model. Writes $FACE_MIA_RESULTS/embeddings/<combo>/<model>/<ds>.npz + _meta.json. Run this first: steps (d) and (e) read the cache.

(d) Verification (EER)

python -m face_mia.run.build_verification configs/eval/grid_<combo>.yaml

Pools embeddings into templates, scores comparisons.csv, computes EER / AUC / TMR@FMR. Writes $FACE_MIA_RESULTS/verification/<combo>/<ds>/<model>/{summary.json,scores.npz}.

(e) MIA statistics

Non-member side (benchmarks) and member side (WebFace4M):

# non-members: per-identity statistics on each benchmark (reads the (c) cache)
python -m face_mia.run.build_identity_mia configs/eval/grid_<combo>.yaml

# members + non-member reference, from WebFace4M itself
python -m face_mia.run.dump_webface4m     configs/eval/member_<combo>.yaml
python -m face_mia.run.build_member_stats configs/eval/member_<combo>.yaml

Writes $FACE_MIA_RESULTS/identity_mia/<combo>/<ds>/<model>/identity_mia_stats.csv and $FACE_MIA_RESULTS/webface4m_mia/<combo>/<model>/identity_mia_stats.csv (dump_webface4m also caches embeddings/<combo>/<model>/webface4m.npz). Columns are described in section 8. Membership AUC is member-vs-non-member per statistic, computed by the analysis scripts below.

(f) Tables

python -m analysis.generate_analysis          # tab-eer.tex, tab-anova.tex, fig-grid-*.pdf
python -m analysis.generate_epoch_tables      # tab-eer-ep{5,10,20,40}.tex

generate_analysis builds the combined EER + MIA-AUC landscape table, the model-level and score-level ANOVA over backbone / loss / n_ids / epoch, and the score-distribution figure grid. generate_epoch_tables is the same landscape table split per epoch.

(g) Confound analyses

python -m analysis.factor_attribution --all   # pose / quality / age / ethnicity
python -m analysis.mlp_fusion --all           # fuse the four statistics with an MLP

factor_attribution runs the natural experiments that separate membership signal from input-distribution shift; it reads the benchmark filelists (landmarks, pose, age, ethnicity columns), not just the results tree. mlp_fusion cross-validates an MLP over the four statistics.

The scripts in analysis/ are hard-wired to the paper's grid -- backbones iresnet34/50/100, heads arcface/cosface/magface, sizes 1K/10K/50K/100K/all, epochs 5/10/20/40 -- and glob the results tree for it. They are not parameterised by a config; missing cells render as -- rather than failing. Adjust the constants at the top of each script to analyse a different grid.


7. Config reference

Eval config (configs/eval/grid_*.yaml, example.yaml)

output_dir: results/identity_mia/iresnet50_arcface   # ONLY the last component is read
datasets:
  - name: lfw                                   # must match <name>.csv in filelist_dir
    filelist_dir: ${FACE_DATA}/filelist/lfw
    image_root: ${FACE_DATA}/lfw/images
    protocol_filter: true                       # keep only enrol+probe crops
    mia_template_csv: ${FACE_DATA}/filelist/cfp_fp/mia_template.csv   # optional
models:
  - name: iresnet50-arcface-1K-ep5              # names the output directory
    arch: iresnet50                             # must match the weights
    weights: ${FACE_MIA_CHECKPOINTS}/iresnet50_arcface_n1000_s42/epoch_005.pt
batch_size: 256
device: cuda
fmr_targets: [1.0e-3, 1.0e-4, 1.0e-5, 1.0e-6]   # optional, for TMR@FMR

output_dir is not an output path. Only its last component is used, as the <combo> directory name under $FACE_MIA_RESULTS. The results root is FACE_MIA_RESULTS, and the sub-tree (embeddings/, verification/, ...) is fixed by the step you run. The leading results/identity_mia/ is inert.

Member config (configs/eval/member_*.yaml)

output_dir: results/webface4m_mia/iresnet50_arcface   # again: last component = <combo>
data_root: ${FACE_WEBFACE4M}
n_member: 2500          # member identities to sample
n_nonmember: 2500       # non-member identities to sample (0 available for nall models)
sample_seed: 123        # sampling is deterministic given this seed
models: [ ... ]         # as above; needs identity_indices.npy beside the weights
batch_size: 256
device: cuda

Train config (configs/train/*.yaml)

Keys: backbone, head, embedding_size, dropout, scale, margin, head_kwargs, num_identities, data_seed, data_root, skip_preprocess, num_workers, epochs, batch_size, lr, weight_decay, momentum, optimizer, warmup_epochs, lr_schedule, lr_milestones, lr_gamma, fp16, sample_rate, flip_prob, low_res_prob, save_every_epochs, save_every_steps, keep_last_n, pin_epochs, resume, log_every, tensorboard, output_dir. Every one has a CLI override.


8. Outputs

Everything lands under $FACE_MIA_RESULTS:

path written by contents
embeddings/<combo>/<model>/<ds>.npz (c) crop_ids (int64), emb_raw (N,D), emb_flip_raw (N,D) -- raw, unnormalised
embeddings/<combo>/<model>/webface4m.npz (e) subject_ids, is_member (bool), emb_raw, emb_flip_raw
embeddings/<combo>/<model>/_meta.json (c) arch, weights, head, scale, margin, head_kwargs
verification/<combo>/<ds>/<model>/summary.json (d) eer, auc, tmr_at_fmr (per FMR target), n_comparisons, n_valid, n_missing, n_genuine, n_impostor
verification/<combo>/<ds>/<model>/scores.npz (d) one score per comparisons.csv row + labels
identity_mia/<combo>/<ds>/<model>/identity_mia_stats.csv (e) non-member statistics
webface4m_mia/<combo>/<model>/identity_mia_stats.csv (e) member + non-member statistics, with is_member
tables/*.tex, figures/*.pdf (f) LaTeX tables, figures
factor_attribution/, mlp_membership/mlp_summary.csv (g) confound analyses

identity_mia_stats.csv has one row per identity:

subject_id, n_crops, pairwise_cos, inertia, vmf_kappa, penalized_logit, proto_softmax_loss

plus is_member on the WebFace4M side. The four membership statistics are pairwise_cos (mean intra-identity pairwise cosine), vmf_kappa (vMF concentration), penalized_logit (mean leave-one-out penalised logit, head-specific) and proto_softmax_loss (reconstructed-head cross-entropy); inertia (mean cosine to the identity centroid) is an additional compactness measure. All need >= 2 crops per identity; identities with fewer get NaN. Embedding caches are deletable intermediates -- delete embeddings/ to reclaim space.


9. Using already-trained (or third-party) models

Evaluation reads only embeddings, so any face encoder works. Point a config at it:

models:
  - name: my-model
    arch: iresnet100                                  # backbone the weights match
    weights: ${FACE_MIA_CHECKPOINTS}/my_model/final.pt
  • Supported arch values: iresnet18, iresnet34, iresnet50, iresnet100, iresnet200, vit_t, vit_s, vit_b.
  • Accepted checkpoint layouts: a training checkpoint written here (a dict with a backbone key, plus config and epoch), or a bare state_dict. A module. prefix is stripped; the load is strict, so arch must match the weights exactly.
  • Two of the four statistics need the training head. penalized_logit and proto_softmax_loss are reconstructed from the head type + margin (+ scale), read from the checkpoint's config. A bare state_dict carries no config, so both come out NaN; pairwise_cos, inertia and vmf_kappa need embeddings alone and are always computed. To recover them for a foreign checkpoint, edit head / margin / scale in the _meta.json that step (c) writes next to the embeddings, then re-run step (e).
  • The member side (dump_webface4m) additionally needs identity_indices.npy beside the weights, listing the WebFace4M identity labels the model was trained on. Without it every identity is treated as a member, which yields a member-only reference and no membership AUC.

10. Reproducing the paper grid

  1. Train 45 runs: 9 combos ({iresnet34,iresnet50,iresnet100} x {arcface,cosface,magface}) x 5 sizes (--num-identities 1000/10000/50000/100000/-1, -1 = all), 40 epochs, data_seed 42. This produces the checkpoint names the shipped eval configs already reference.
  2. For each combo, run (c), (d) and (e) with configs/eval/grid_<combo>.yaml and configs/eval/member_<combo>.yaml. Each config lists all 20 models (5 sizes x 4 pinned epochs), so one invocation covers the combo.
  3. Build the MIA template lists once: python -m face_mia.run.build_mia_templates.
  4. Generate tables and figures: analysis.generate_analysis, analysis.generate_epoch_tables, analysis.factor_attribution --all, analysis.mlp_fusion --all.

The vit_t configs (grid_vit_t_cosface.yaml, member_vit_t_cosface.yaml) cover the generalisation check outside the main grid; the analysis scripts' default grid excludes them.


11. Command reference

Every entry point. Config arguments are positional; --device overrides the config.

command argument purpose
face_mia.train --config <cfg> + overrides train (b)
face_mia.run.dump_embeddings <eval cfg> embed + cache crops (c)
face_mia.run.build_verification <eval cfg> [--device] EER / AUC / TMR@FMR (d)
face_mia.run.build_identity_mia <eval cfg> [--device] non-member statistics (e)
face_mia.run.dump_webface4m <member cfg> embed WebFace4M members + non-members (e)
face_mia.run.build_member_stats <member cfg> [--device] member statistics (e)
face_mia.run.build_mia_templates (none) write mia_template.csv for the confounded benchmarks
face_mia.run.eval_benchmarks <eval cfg> [--device] [--model-idx N] verification with flip-TTA + IJB-C pooling
face_mia.eval.ijbc --weights --arch --image-root [--filelist-dir --meta-dir --insightface --no-flip --output] IJB-C 1:1 with InsightFace-style pooling
analysis.generate_analysis (none) tab-eer.tex, tab-anova.tex, fig-grid-*.pdf
analysis.generate_epoch_tables [--epoch N ...] [--dry-run] tab-eer-ep{5,10,20,40}.tex
analysis.factor_attribution `[--all] [--arch A --loss L] [--metric M --all-metrics]`
analysis.mlp_fusion [--all] [--combo C] [--nids ...] [--n-folds K] MLP fusion of the four statistics
analysis.generate_roc_plots (none) verification ROC / DET figures
analysis.generate_mia_roc_plots (none) MIA ROC / DET figures
analysis.generate_all_grid_plots (none) score-distribution grid, all combos
analysis.plot_score_distributions `[--arch --loss --nids --epoch --metric --all-metrics --outdir]`

face_mia.eval.{verification,identity_mia,webface4m} are the library modules behind the run/ entry points and can also be called directly; prefer run/, which shares the embedding cache.


12. Layout

configs/train/           11 training configs
configs/eval/            grid_* (verification/MIA), member_* (WebFace4M), example.yaml
face_mia/
  paths.py               all data / checkpoint / result locations
  train.py               trainer                     (python -m face_mia.train)
  backbones/ heads/      IResNet + ViT; ArcFace/CosFace/MagFace/AdaFace/Softmax + PartialFC
  data/                  recordio.py (RecordIO reader) + dataset.py (training Dataset)
  eval/                  identity_mia, verification, webface4m, ijbc
  run/                   dump_* / build_* command-line entry points
analysis/                factor_attribution, mlp_fusion, generate_* / plot_* (tables, figures)
filelist/                runnable placeholder benchmark (see section 5)
tests/                   end-to-end smoke test (pytest tests/, or python tests/test_smoke.py)
results/                 generated output (gitignored)

Citation

@article{ozturk2026quantifying,
  title         = {Quantifying Training Membership Information in the Hyperspherical
                   Embedding Geometry of Face Recognition Models},
  author        = {{\"O}zt{\"u}rk, {\"U}nsal and Marcel, S{\'e}bastien},
  year          = {2026},
  eprint        = {2607.15084},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2607.15084}
}

License

MIT -- see LICENSE.

Some files derive from InsightFace, which is likewise MIT; see THIRD-PARTY.md for what is derived and from where. No third-party datasets or pre-trained models are distributed here.

About

Computes hyperspherical distributions of face embeddings to compare differences between training dataset member and non-member identities.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages