WinSyn: Synthetic Procedural Model

WinSyn is a research project to provide a matched pair of synthetic and real images for machine learning tasks such as segmentation. You can learn more on the project page, reading the paper or watching a video showing how to run the procedural model.

This repository contains code for running the synthetic procedural model. We also took photos of 75k real windows.

setting up

We use Blender 3.3. Below we introduct options to run from a gui (a blender add-on; gui.py ), python script withing blender (go.py ), headless, & on a cluster.

run from Blender add-on

• Open Blender 3.3.
• add gui.py as an add-on:
  • Edit > Preferences > Add-ons > install... then select gui.py.
  • Enable the WinSyn addon: ensure "community" is highlighted (blue) and search for "WinSyn". Click the checkbox next to "Research: WinSyn". Close the preferences window.
  • The WinSyn tab should now be visible at the top-right of the 3D area. You may have to press the n key with your mouse over the 3D view to show the options (below the item and tool tabs).
  • You should now see the interface as above.
• You have the options to
  • Generate from a seed. Each different seed creates a different window.
  • Generate from a file. The parameters from the file will be read to generate a building.
  • Specify a parameter file.
  • Edit the parameter file - edit and save (alt-s) the parameters in json format. Requires a text-editor pane be open (shift-f11).
  • Set the resource path. By default this is the stripped down resource set in this repository, see below for instructions on how to download the full resource set.
  • Set the styles/variation. These are used in the paper to explore different graphics techniques.
  • Run the slow physics simulator or not
  • The "generate" button will create your scene & write out the parameters to the given location

running code from within Blender

• Open the winsyn.blend file in Blender 3.3.
• Open a text pane in blender with the go.py script
• Optional: Set the resource_path in config.py to where you downloaded the resource files (see below)
• Run the script! Generation time varies from 20 seconds to a few minutes. Blender hangs during generation. Some generation may take a very long time depending on the parameters selected.
• Debugging requires a more challenging setup, I use Pycharm with something like this combined with the commented out pydevd_pycharm.settrace lines in go.py. The workflow goes something like - edit code in pycharm, switch to blender to run, switch back to pycharm to set breakpoints/inspect elements.

running headless

• as well as resource_path as above...
• set the render_path in config.py to the location where renders should be written
• set the number of renders you want in render_number.
• set interactive to False in config.py.
• optional: set the style (variations) in config.py
• run with something like (the CUDA bit says to use an Nvidia GPU to accelerate rendering):

blender -b /path/to/winsyn/winsyn.blend --python /path/to/winsyn/src/go.py -- --cycles-device CUDA

running on a cluster

I deploy on our ibex/slurm cluster to render large datasets. I use the nytimes Blender docker image built as singularity container (singularity definition file) and a job script similar to the below. On ibex I rendered on the p100 and v100 nodes, and run about 10 machines to render 2k images overnight.

with a run.sh script:

echo "Welcome to Windows"
outdir="/ibex/wherever/you/want/output_dataset"
mkdir -p $outdir
style="rgb;labels"

while : # "robustness"
do
  SINGULARITYENV_WINDOWZ_STYLE="$1" singularity exec --nv --bind $outdir:/container/winsyn/output --bind /ibex/winsyn/resources:/container/winsyn/resources --bind /ibex/winsyn:/container/winsyn /ibex/wherever/you/put//blender_3_3.sif blender -b /container/winsyn/winsyn.blend --python /container/winsyn/src/go.py -- --cycles-device OPTIX
  echo "blender crashed. let's try that again..."
done

with config.py lines render_path=/container/winsyn/output and resource_path=/container/winsyn/resources.

resource files

The model requires a resources files with various textures and meshes from different sources. We include a single example resource of each type - these are enough to run the code, but do not have much diversity. Running the model with only these resources will not match our results... The config.py file defines resource_path which should be the location of the resources folder.

• The 3D clutter scenes can be downloaded from the KAUST datastore. They should be added added to the exterior_clutter folder.

• The script import_google_panos.py can be used to download the panoramas used for the published dataset. It takes a single argument: your resource folder, and downloads images here in the subfolder outside.

  • Alternately, download a different set of panoramas from google directly.

• Signs can be downloaded from the Kaust datastore. They should be in the signs folder of your resource folder. The downloaded and unzipped files can be split into folders large, medium, and small using the script split_signs.py. It takes two arguments - the root folder of the unzipped signs dataset and the resource folder.

• The interior textures are from matterport.

  • You can download them by following the instructions (involving sending them a form and getting a download script)
  • Run the script thusly to download all the interior panoramas:

  download_mp.py /a/location --type matterport_skybox_images 
  

  • extract and convert the downloaded skyboxes into the panoramic image format using the script import_matterport.py. It takes two arguments: the root of the downloaded panoramas and your resource folder.

• If you wish to generate the variant with many textures (texture_rot), download and unzip the dtd dataset into the dtd folder inside your resource folder.

variations

These are known as 'styles' in the code and change the behavior of the model (e.g., all-grey walls, or all-nighttime lighting - see the end of the paper's appendix for examples). They are defined in the config.py file or using the WINDOWZ_STYLE env variable. The sequences below render the variations for various sequences of paramters and create the labels where required.

• rgb;labels the default baseline model (and also render the labels).
• rgb;128nwall;64nwall;32nwall;16nwall;8nwall;4nwall;2nwall;1nwall;labels changes the number of wall materials
• 0monomat;0.33monomat;0.66monomat;1monomat;2monomat;4monomat;0multimat;0.33multimat;0.66multimat;1multimat;2multimat;4multimat;labels;all_brick changes the parameterization of the procedural materials. monomat is a single proc material for each object class. multi-mat is the baseline number of materials. The numbers are multipliers on the deviations for parameter generation.
• nosplitz;nosplitz_labels;mono_profile;mono_profile_labels;only_rectangles;only_rectangles_labels;no_rectangles;no_rectangles_labels;only_squares;only_squares_labels;single_window;single_windows_labels;wide_windows;wide_windows_labels the window-shape parameterization variation.
• lvl9;lvl8;lvl7;lvl6;lvl5;lvl4;lvl3;lvl2;lvl1;lvl9_labels;lvl8_labels;lvl7_labels;lvl6_labels;lvl5_labels;lvl4_labels;lvl3_labels;lvl2_labels;lvl1_labels these are the number of modeled labels (i.e., just starting will the wall label with lvl1.
• 0cen;3cen;6cen;12cen;24cen;48cen;96cen;labels;0cenlab;3cenlab;6cenlab;12cenlab;24cenlab;48cenlab;96cenlab these are the camera positions (over a circle).
• 1spp;2spp;4spp;8spp;16spp;32spp;64spp;128spp;256spp;512spp;nightonly;dayonly;notransmission;0cen;3cen;6cen;12cen;24cen;48cen;nosun;nobounce;fixedsun;monomat;labels;0cenlab;3cenlab;6cenlab;12cenlab;24cenlab;48cenlab these are the rendering samples per pixel.
• canonical;64ms;128ms;256ms;512ms;1024ms;2048ms;labels;edges;diffuse;normals;col_per_obj;texture_rot;voronoi_chaos,phong_diffuse these are the many varied materials experiments.

parameters

There are a variable number of parameters (sometimes thousands) depending on the code-path, and not all have human-friendly names. The parameters are described in the code samples which samples them from the RantomCache class in rantom.py:

r2.uniform(0.1, 0.22, "stucco_crack_size", "Size of stucco cracks")

The model writes out an attribute file to the attribs directory containing all the parameters used to generate a given scene. The file also contains assorted metadata including the random seed and render times. You can vary the model's output by changing the parameters. By default a random seed is created and used to generate the remainder of the parameters.

After a parameter name has been assigned ("stucco_crack_size"), asking for it again in the code will return the same value (even if it lies outside of the given distribution - invalid ranges ).

If you generate using the same random seed, it should always generate the same scene. However, changes in the code will change this, so consider creating a parameter list as below.

todo lists and parameter lists

There is a mechanism to render a dataset using fixed seeds/parameters in headless/non-interactive mode. If there is a todo.txt file in the config.render_path, the system will try to render for each random seed (e.g., a number) in the file. One seed (a number) per line. There is a robustness mechanism for multi-node rendering, but I have observed occasional failures and had to re-run some seeds manually.

In addition, if there is an existing parameter (attribs) file for that seed (i.e., the file render_path/attribs/random_seed.txt exists), those parameters will override the random values that would otherwise be used. Attributes that are required but not specified in this file are sampled as usual. This can be used to manually set the parameters for a derivation.

code outline

• go.py start reading here - the main loop (for step in range (config.render_number):) runs until all renders have completed.
• config.py contains the per-setup (for me this is laptop/desktop/cluster) configuration
• rantom.py contains the paramter-sampling and storage code
• cgb.py my CGAShape implementation. Very different extensions and limitations to other implementations :/
• cgb_building.py the main entry point for actual geometry generation. Uses CGA to create a basic building
• cgb_*.py the other major components which use CGA-list constructions
• materials.py responsible for adding materials to the scene's geometry, as well as all variations. pre_render and post_render are the most interesting, with go as the entrypoint for the main texturing routine.
• shape.py and subframe.py create bezier shaped windows and then add geometry to them

acknowledgements

We thank the blender procedural artists Gabriel de Laubier for the UCP Wood material and Simon Thommes for the fantastic Br'cks material. Both were modified and used in our procedural model.

cite

@inproceedings{winsyn,
  title={WinSyn: A High Resolution Testbed for Synthetic Data},
  author={Tom Kelly and John Femiani and Peter Wonka},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  month = {June},
  year={2024}
}