Automated Table Reproduction via Code Generation

This repository contains the artifact for the paper "Automated Table Reproduction via Code Generation" (ASE 2026).

The artifact supports two kinds of reproduction:

Last-mile reproduction: regenerate the paper abstract, tables, figures, and compact ase.pdf wrapper from archived experimental logs.
Full reproduction: rerun the complete agent experiments from scratch, then run the last-mile reproduction pipeline on the new logs.

We recommend starting with Last-mile reproduction. It is fast, does not require LLM API credits, and directly validates the data used for the paper's generated tables and figures.

Last-mile reproduction

Last-mile reproduction regenerates the submitted artifact outputs from archived logs.

Docker image requirements

The ase2026 image targets Linux AMD64. The download is approximately 1.24 GiB; reserve 8 GB of free disk space for the image and restored logs.

Docker setup

Run the last-mile reproduction with:

docker run --platform linux/amd64 --rm -it -v "$PWD":/output ghcr.io/doehyunbaek/artisan:ase2026

This writes ./ase-artisan.pdf on the host. The command emits detailed download, generation, and LaTeX logs; a successful run ends with output similar to:

== Done ==
Wrote /artifact/data/tex/ase-artisan.pdf
Wrote /artifact/ase-artisan.pdf
Wrote /output/ase-artisan.pdf

After the container exits, ./ase-artisan.pdf should be created at the host directory. The image is based on Ubuntu 24.04 LTS and built from Dockerfile. For other platforms, use the Docker image or open an issue.

The command above runs data/reproduce.py, which performs the following steps.

To run the steps manually or inspect the container, start an interactive shell instead:

docker run --platform linux/amd64 --rm -it -v "$PWD":/output ghcr.io/doehyunbaek/artisan:ase2026 bash

The manual commands below assume you are inside this container shell. The repository is mounted at /artifact, and host outputs should be written under /output.

Step 1: Restore logs

cd /artifact/data
python3 download_logs.py

This restores archived run logs under:

/artifact/data/logs/

Expected layout:

logs/
  artisan-gpt5.1-dbe0/
  artisan-gpt5mini-845b/
  artisan-deepseek-reasoner-6f4b/
  ...

Step 2: Regenerate the abstract

cd /artifact/data
python3 abstract.py

Expected output:

tex/abstract.tex

Step 3: Regenerate tables

cd /artifact/data
python3 table.py

Expected outputs:

tex/table_1.tex
tex/table_2.tex
tex/table_3.tex
tex/table_4.tex

Table provenance:

Table	Source
Table 1	Static comparison table
Table 2	Restored logs plus manual corrections
Table 3	`data/method_judge_eval.jsonl`
Table 4	`data/bugs/bugs.json`

Step 4: Regenerate figures

Figure 8 uses Matplotlib/NumPy, which are included in the prebuilt image.

cd /artifact/data
python3 figure.py

Expected outputs:

tex/figure_1.tex
tex/figure_2.tex
...
tex/figure_9.tex
tex/figures/overview.pdf
tex/figures/artisan_time_breakdown.png

Figure provenance:

Figure	Source
Figures 1--7, 9	Static snippets/assets from the paper
Figure 8	Regenerated from restored logs

Step 5: Build the compact ASE wrapper PDF

cd /artifact/data/tex
pdflatex -shell-escape ase.tex
bibtex ase
pdflatex -shell-escape ase.tex
pdflatex -shell-escape ase.tex

Expected outputs:

tex/ase.pdf
tex/ase-artisan.pdf
/artifact/ase-artisan.pdf

The -shell-escape flag is required because the generated figure fragments use minted for syntax highlighting.

Full reproduction

Full reproduction reruns the complete experimental campaign instead of using archived logs. It is substantially more expensive than last-mile reproduction and is not required for routine artifact evaluation.

Cost estimate

The paper evaluates 15 configurations over 60 Artisan-Bench tasks, i.e., 900 agent runs in total. The experimental setup uses a 30-step limit, a USD 1 agent-cost limit, and an eight-hour wall-clock limit per task.

Based on the average per-task values reported in Table 2 of the paper:

Expected LLM cost for the complete 900-run campaign is about USD 172.
- Main comparison runs: about USD 102.
- Ablation runs: about USD 70.
NOTE: The configured upper bound is much larger: 900 runs × USD 1/run = USD 900 for the agent-side budget alone.

Requirements

Full reproduction requires:

Docker access from inside the artifact container via /var/run/docker.sock
Network access to download research artifacts and Docker images
LLM API keys and enough budget for 15 configurations over 60 tasks
OPENAI_API_KEY for GPT configurations and the format/method judges
DEEPSEEK_API_KEY or LLM_API_KEY for DeepSeek configurations

Smoke-test bloat Table 2

We illustrate the full reproduction process with a single run due to cost reasons. We reproduce the following table:

Source: Table 2 from “Bloat beneath Python’s Scales: A Fine-Grained Inter-Project Dependency Analysis” by Georgios-Petros Drosos, Thodoris Sotiropoulos, Diomidis Spinellis, and Dimitris Mitropoulos, Proc. ACM Softw. Eng. 2024. https://doi.org/10.1145/3660821. Licensed under CC BY 4.0. Screenshot/crop from original.

To test one GPT-5.1 Artisan run on bloat-t2 (Table 2: resolved and unresolved external calls), pass OPENAI_API_KEY and the Docker socket:

mkdir -p artisan-full

docker run --platform linux/amd64 -it --rm --network host \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v "$PWD/artisan-full":"$PWD/artisan-full" \
  -e OPENAI_API_KEY \
  ghcr.io/doehyunbaek/artisan:ase2026 \
  data/full_reproduce.py --output-root "$PWD/artisan-full" --only artisan-gpt5.1-dbe0 --filter bloat-t2

This writes logs under ./artisan-full/logs/. To inspect the agent trajectory, read artisan.log inside artisan-full/logs. To inspect the resulting reproduction script, read repro_bloat_table_2.sh inside artisan-full/logs.

Example trajectory from the reproduced run:

Download the Zenodo artifact with artisan get, and automatically locate the artifact README. This illustrates the artifact-download mechanism described in Section 2.3.2 of the paper.

09:16:59 - artisan:DEBUG - run.py:65 - role: assistant, content:
THOUGHT: Begin by downloading and extracting the artifact from the provided Zenodo URL, as required in step 1 of the workflow.

<mini_swe_agent_bash>
artisan get https://zenodo.org/records/11095274
</mini_swe_agent_bash>
09:17:04 - artisan:DEBUG - run.py:65 - role: user, content:
<returncode>0</returncode>
<output>
Downloading bloat-study-artifact-v1.0.zip 4% (65536/1552263 bytes)
...
README PATH: bloat-study-artifact-v1.0/gdrosos-bloat-study-artifact-0fe2fe5/README.md

Search the README for Docker and Table 2 instructions. This is the kind of clearly documented artifact path discussed in Section 4.2.2 of the paper.

<mini_swe_agent_bash>
grep -in docker bloat-study-artifact-v1.0/gdrosos-bloat-study-artifact-0fe2fe5/README.md
</mini_swe_agent_bash>
<output>
434:**Note** The commands presented in this section are being executed inside the produced docker container e.g. we have executed:
436:docker run -it --rm \
</output>

<mini_swe_agent_bash>
grep -in "Table 2" bloat-study-artifact-v1.0/gdrosos-bloat-study-artifact-0fe2fe5/README.md
</mini_swe_agent_bash>
<output>
468:Moreover, to produce Table 2 describing the statistics on the resolved and unresolved external calls during our stitching process, run:
</output>

Build and enter the artifact's Docker environment, then run the documented Table 2 command. This produces the resolved/unresolved-call counts before final formatting.

<mini_swe_agent_bash>
docker exec bloat-study-artifact-container /bin/bash --noprofile --norc -c "cd /home/user && python scripts/descriptives/evaluation.py -csv data/results/rq1a.csv > repro_table2.txt"
</mini_swe_agent_bash>
<returncode>0</returncode>

<mini_swe_agent_bash>
docker exec bloat-study-artifact-container /bin/bash --noprofile --norc -c "cd /home/user && cat repro_table2.txt"
</mini_swe_agent_bash>
<output>
External Calls  Aggregate count Proportion of total (%)   Average (per project) Median (per project)
Resolved        7799929         96.8                      5990.7               144.5
Unresolved      260249          3.2                       199.9                11.5
</output>

Author an executable reproduction script and submit it to the judge. This is the executable evidence emphasized by Artisan's code-generation framing.

09:21:25 - artisan:DEBUG - run.py:65 - role: assistant, content:
THOUGHT: I will create the `repro_bloat_table_2.sh` script using the provided template...

<mini_swe_agent_bash>
cat > /workspace/repro_bloat_table_2.sh <<'EOF'
...
artisan format --expected /workspace/expected.md --repro /workspace/repro.txt
...
chmod +x /workspace/repro_bloat_table_2.sh
artisan submit /workspace/repro_bloat_table_2.sh
</mini_swe_agent_bash>

Use judge feedback after a partial mismatch, revise the script, and resubmit successfully, followed by successful validation and completion. This highlights the output-judge/format-tool feedback loop discussed in the paper, including the Drosos Table 2 mismatch example in Figure 4.

<returncode>1</returncode>
<output>
Submitting to judge service at http://127.0.0.1:8000/submit...
...
Partial mismatch (score: 0.94): some cells differ (digits obfuscated)
</output>

09:25:24 - artisan:DEBUG - run.py:65 - role: assistant, content:
THOUGHT: The previous submission partially matched because the expected table still had placeholders...
...
<returncode>0</returncode>
<output>
Wrote format tool logs to /workspace/artisan-format.json
Wrote speedometer tool logs to /workspace/artisan-speedometer.json
Success: reproduce.sh output matches expected.md
Submitting to judge service at http://127.0.0.1:8000/submit...
</output>

Here is the example reproduction script generated:

#!/usr/bin/bash
# Section 1: Expected table
cat > /workspace/expected.md <<'EOTABLE'
**Table 2: Statistics on the resolved and unresolved external calls during our stitching process.**

|  | External Calls | Aggregate count | Proportion of total | Average (per project) | Median (per project) |
| --- | --- | --- | --- | --- | --- |
|  |  |  |  |  |  |
|  | Resolved | 7,799,929 | 96.8% | 5,991 | 144.5 |
|  | Unresolved | 260,249 | 3.2% | 200 | 11.5 |

EOTABLE
# Section 2: Artifact download
artisan get https://zenodo.org/records/11095274
# Section 3: Reproduction commands (populate from reviewed steps)
cd /workspace/bloat-study-artifact-v1.0/gdrosos-bloat-study-artifact-0fe2fe5
docker build -t bloat-study-artifact .
# Clean up any existing container with the same name (if present)
docker rm -f bloat-study-artifact-container >/dev/null 2>&1 || true
docker run -d --init --entrypoint bash --name bloat-study-artifact-container \
    -v $(pwd)/scripts:/home/user/scripts \
    -v $(pwd)/data:/home/user/data \
    -v $(pwd)/figures:/home/user/figures \
    bloat-study-artifact -c 'sleep infinity'
docker exec bloat-study-artifact-container /bin/bash --noprofile --norc -c "cd /home/user && python scripts/descriptives/evaluation.py -csv data/results/rq1a.csv" > /workspace/repro.txt
docker stop bloat-study-artifact-container
# Section 4: Formatting and submission block
echo '<artisan_submit>'
artisan format --expected /workspace/expected.md --repro /workspace/repro.txt
echo '</artisan_submit>'

Name		Name	Last commit message	Last commit date
Latest commit History 14 Commits
agents		agents
artisanbench		artisanbench
assets		assets
data		data
src/artisan		src/artisan
.dockerignore		.dockerignore
.gitignore		.gitignore
.gitmodules		.gitmodules
.python-version		.python-version
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Automated Table Reproduction via Code Generation

Last-mile reproduction

Docker image requirements

Docker setup

Step 1: Restore logs

Step 2: Regenerate the abstract

Step 3: Regenerate tables

Step 4: Regenerate figures

Step 5: Build the compact ASE wrapper PDF

Full reproduction

Cost estimate

Requirements

Smoke-test bloat Table 2

About

Releases

Packages

Used by

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

Automated Table Reproduction via Code Generation

Last-mile reproduction

Docker image requirements

Docker setup

Step 1: Restore logs

Step 2: Regenerate the abstract

Step 3: Regenerate tables

Step 4: Regenerate figures

Step 5: Build the compact ASE wrapper PDF

Full reproduction

Cost estimate

Requirements

Smoke-test bloat Table 2

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages