Skip to content

Commit b83eceb

Browse files
lappalainenjlappalainenj
committed
Update docs and package metadata
- Add Hydra configuration documentation to mkdocs - Update pyproject.toml to include YAML config files - Update changelog for release
1 parent f31da71 commit b83eceb

File tree

5 files changed

+313
-8
lines changed

5 files changed

+313
-8
lines changed

‎CHANGELOG.md

+18-8
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,26 @@
11
# Changelog
22

3-
## [v1.1.3] - 2024-11-20
3+
## [v1.1.1] - 2024-11-21
44

55
### Distribution
6-
- Update package-data handling
6+
- Moved configuration files into package structure
7+
- Switched to `importlib.resources` for resource management
8+
- Removed MANIFEST.in in favor of pyproject.toml configuration
9+
- Enhanced root directory resolution logic
710

8-
## [v1.1.2] - 2024-11-20
9-
10-
### Distribution
11-
- Update project metadata for pypi
11+
### Infrastructure
12+
- Improved cluster management with better dry-run and slurm support
1213

13-
## [v1.1.1] - 2024-11-20
14+
### CLI
15+
- Enhanced command-line interface error handling
16+
- Improved argument parsing for multiple commands
17+
- Added `init_config` for config-based customization of network and training
1418

1519
### Documentation
16-
- Remove broken badges from README for now
20+
- Updated package metadata for PyPI
21+
- Added project URLs to package configuration
22+
- Removed broken badges from README
23+
- Added explanations for hydra-config-based customization of network and training in `CLI Reference`
1724

1825
## [v1.1.0] - 2024-11-20
1926

@@ -27,10 +34,13 @@
2734

2835
### Documentation
2936
- Improved docs
37+
- Updated project metadata for PyPI
38+
- Removed broken badges from README
3039

3140
### Infrastructure
3241
- Removed strict version pins for better compatibility (particularly removed UMAP constraints)
3342
- Updated GitHub workflows
3443
- Updated README badges
44+
- Updated package-data handling
3545

3646
[v1.1.0]: https://github.com/TuragaLab/flyvis/releases/tag/v1.1.0

‎docs/docs/reference/hydra_default_config.md

+121
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
# Configuration System
2+
3+
`flyvis` uses [Hydra](https://hydra.cc/) for configuration management of network and training. The configuration system is organized hierarchically, allowing for modular configuration of different components.
4+
5+
## Configuration Components
6+
7+
### Configuration Components
8+
9+
#### Network Configuration
10+
11+
* `network/`: Core network architecture and parameters
12+
* `connectome/`: Neural connectivity specification
13+
* `dynamics/`: Neural dynamics parameters
14+
* `edge_config/`: Synapse-related configurations (sign, strength, count)
15+
* `node_config/`: Neuron-related configurations (bias, time constants)
16+
* `stimulus_config/`: Input stimulus parameters
17+
18+
#### Training Configuration
19+
20+
* `optim/`: Optimization parameters
21+
* `penalizer/`: Loss function penalties
22+
* `scheduler/`: Learning rate scheduling
23+
* `task/`: Task-specific settings and parameters
24+
25+
#### Top-level Configuration
26+
* `solver.yaml`: Training loop parameters
27+
28+
29+
## Default Configuration Structure
30+
31+
The default configuration is structured as follows:
32+
33+
```bash
34+
config
35+
├── network # Network configuration
36+
│   ├── connectome # Connectome configurations
37+
│   │   └── connectome.yaml # Default connectome configuration
38+
│   ├── dynamics # Dynamics configurations
39+
│   │   └── dynamics.yaml # Default dynamics configuration
40+
│   ├── edge_config # Edge configuration
41+
│   │   ├── edge_config.yaml # Default edge configuration
42+
│   │   ├── sign # Signaling configurations
43+
│   │   │   └── sign.yaml # Default signaling configuration
44+
│   │   ├── syn_count # Synaptic count configurations
45+
│   │   │   └── syn_count.yaml # Default synaptic count configuration
46+
│   │   └── syn_strength # Synaptic strength configurations
47+
│   │   └── syn_strength.yaml # Default synaptic strength configuration
48+
│   ├── network.yaml # Default network configuration
49+
│   ├── node_config # Node configuration
50+
│   │   ├── bias # Bias configurations
51+
│   │   │   └── bias.yaml # Default bias configuration
52+
│   │   ├── node_config.yaml # Default node configuration
53+
│   │   └── time_const # Time constant configurations
54+
│   │   └── time_const.yaml # Default time constant configuration
55+
│   └── stimulus_config # Stimulus configuration
56+
│   └── stimulus_config.yaml # Default stimulus configuration
57+
├── optim # Optimizer configuration
58+
│   └── optim.yaml # Default optimizer configuration
59+
├── penalizer # Penalizer configuration
60+
│   └── penalizer.yaml # Default penalizer configuration
61+
├── scheduler # Scheduler configuration
62+
│   └── scheduler.yaml # Default scheduler configuration
63+
├── solver.yaml # Default solver configuration
64+
└── task # Task configuration
65+
└── task.yaml # Default task configuration
66+
67+
15 directories, 16 files
68+
```
69+
70+
## Customizing Configurations
71+
72+
### Overriding Default Parameters
73+
74+
For quick parameter changes you can use hydra command-line overrides.
75+
76+
```bash
77+
# Example: Change the scale of the synaptic strength to 0.5
78+
flyvis train-single network.edge_config.syn_strength.scale=0.5 ensemble_and_network_id=0042/007 task_name=flow
79+
80+
# Example: Add a custom parameter to an existing config
81+
flyvis train-single +network.edge_config.syn_strength.custom_param=100 ensemble_and_network_id=0042/007 task_name=flow
82+
```
83+
84+
More information on hydra command-line overrides can be found [here](https://hydra.cc/docs/advanced/override_grammar/basic/).
85+
86+
### Using Custom Configurations
87+
88+
In many situations you will want to create custom configuration files, to either break
89+
the existing config structure for new code or to keep track of different sets of parameters.
90+
91+
This particularly applies when installing `flyvis` as a package instead of using the
92+
developer mode, where one can directly modify the source code configuration.
93+
94+
To create and maintain your own configuration files proceed as follows:
95+
96+
1. Initialize a local config directory:
97+
98+
```bash
99+
# Copy the default config structure into flyvis_config
100+
flyvis init-config
101+
```
102+
103+
2. Create your custom configurations in the generated `flyvis_config` directory.
104+
105+
3. Use your custom configs:
106+
107+
```bash
108+
# Using the full custom config directory
109+
flyvis train-single --config-path $(pwd)/flyvis_config [other options]
110+
111+
# Using a specific custom config
112+
flyvis train-single network/edge_config/syn_strength=custom_syn_strength --config-dir $(pwd)/flyvis_config
113+
```
114+
115+
Notice that **`config-path`** overrides the entire default config directory, while **`config-dir`** only adds another search path for resolving configs.
116+
117+
**Important:** It is recommended to use different file names than the defaults.
118+
This is because for partial usage with `config-dir` hydra resolves the config names in the default config directory first. I.e., your changes reflected in files of the same name in the custom config directory might not have the intended effect. E.g., name your config file `custom_syn_strength.yaml` instead of `syn_strength.yaml`.
119+
120+
121+
More information on hydra configuration can be found [here](https://hydra.cc/docs/advanced/search_path/).

‎docs/docs/release.md

+171
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# Release Process
2+
3+
## Prerequisites
4+
5+
1. Ensure all tests pass:
6+
```bash
7+
pytest
8+
```
9+
10+
2. Update and verify the documentation:
11+
Follow the instructions in the [readme](README.md) to update and verify the documentation.
12+
The deployment to github can be done last or via workflow.
13+
14+
3. Install required tools:
15+
```bash
16+
python -m pip install build twine
17+
```
18+
19+
## Release Steps
20+
21+
0. **Test PyPi before committing (optional)**
22+
23+
One can create the wheel and upload it to Test PyPi before committing to develop the package.
24+
This can be useful to test if the installation will work before commiting to a version number and
25+
push to the remote repository. However, the wheels that are uploaded to Test PyPi cannot
26+
be deleted so one should probably do this sparingly and not use version numbers one wants to reserve for
27+
the actual release.
28+
29+
### Upload to Test PyPi
30+
```bash
31+
# Clean previous builds
32+
rm -rf dist/
33+
34+
# Set version temporarily for this session manually (change version number)
35+
export SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0.dev4
36+
37+
# Now build and test
38+
python -m build
39+
40+
# Upload to Test PyPI first
41+
python -m twine upload --repository testpypi dist/*
42+
```
43+
44+
### Test Installation
45+
46+
In a clean environment, run these sporadic tests to verify the installation:
47+
```bash
48+
# Install in clean environment to test (change version number)
49+
python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ flyvis==0.0.0.dev4
50+
51+
# Test installation
52+
flyvis download-pretrained
53+
flyvis train --help
54+
flyvis train-single --help
55+
python -c "from flyvis import Network; n = Network()"
56+
python -c "from flyvis import EnsembleView; e = EnsembleView('flow/0000')"
57+
58+
59+
# Test custom configuration
60+
flyvis init-config
61+
# Add custom_syn_strength.yaml to flyvis_config/network/edge_config/syn_strength/
62+
cat > flyvis_config/network/edge_config/syn_strength/custom_syn_strength.yaml << 'EOL'
63+
defaults:
64+
- /network/edge_config/syn_strength/syn_strength@_here_
65+
- _self_
66+
67+
scale: 1.0
68+
EOL
69+
# Run training and check if custom config is loaded correctly
70+
flyvis train-single --config-dir $(pwd)/flyvis_config network/edge_config/syn_strength=custom_syn_strength ensemble_and_network_id=0 task_name=flow delete_if_exists=true 2>&1 | while read line; do
71+
echo "$line"
72+
if [[ "$line" == *"'syn_strength': {'type': 'SynapseCountScaling'"* && "$line" == *"'scale': 1.0"* ]]; then
73+
echo "Found custom scale parameter in config!"
74+
pkill -P $$
75+
break
76+
fi
77+
done
78+
# Delete custom config
79+
rm -rf flyvis_config
80+
81+
# Delete installation and downloaded models
82+
pip uninstall flyvis -y
83+
84+
# When done testing, unset it
85+
unset SETUPTOOLS_SCM_PRETEND_VERSION
86+
```
87+
88+
### Commit Changes
89+
90+
Commit all open changes to the repository.
91+
92+
### Update Changelog
93+
94+
- Append entry in `CHANGELOG.md` with new version number
95+
- Include all notable changes under appropriate sections, e.g.,
96+
- Breaking
97+
- Features
98+
- Documentation
99+
- Infrastructure
100+
- Distribution
101+
- Bug Fixes
102+
103+
```bash
104+
git add CHANGELOG.md
105+
git commit -m "docs: add changelog for vX.Y.Z"
106+
```
107+
108+
### Create and Push Tag
109+
110+
```bash
111+
# Create annotated tag using changelog
112+
git tag -a vX.Y.Z -F CHANGELOG.md
113+
114+
# Push to both remotes
115+
git push origin main
116+
git push origin vX.Y.Z
117+
git push public_repo main
118+
git push public_repo vX.Y.Z
119+
```
120+
121+
### Build and Upload to PyPI
122+
```bash
123+
# Clean previous builds
124+
rm -rf dist/
125+
126+
# Build package
127+
python -m build
128+
129+
# Set version temporarily for this session manually
130+
export SETUPTOOLS_SCM_PRETEND_VERSION=X.Y.Z
131+
132+
# Now build and test
133+
python -m build
134+
python -m twine upload --repository testpypi dist/*
135+
136+
# When done testing, unset it
137+
unset SETUPTOOLS_SCM_PRETEND_VERSION
138+
139+
# Upload to Test PyPI first (recommended)
140+
python -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ flyvis==X.Y.Z
141+
142+
# Upload to PyPI
143+
python -m twine upload dist/*
144+
```
145+
146+
### Create GitHub Release
147+
- Go to GitHub releases page
148+
- Create new release using the tag
149+
- Copy changelog entry into release description
150+
151+
## Post-release
152+
153+
1. Verify package can be installed from PyPI:
154+
```bash
155+
python -m pip install flyvis
156+
```
157+
158+
## Check documentation is updated on the documentation website
159+
160+
## Version Numbering
161+
162+
We follow semantic versioning (MAJOR.MINOR.PATCH):
163+
- MAJOR: Breaking changes
164+
- MINOR: New features (backwards compatible)
165+
- PATCH: Bug fixes
166+
167+
## Notes
168+
169+
- Always test on Test PyPI before releasing to PyPI
170+
- Ideally CI checks pass before releasing
171+
- Keep both origin and public_repo remotes in sync

‎docs/mkdocs.yml

+1
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ nav:
4242
- CLI Reference:
4343
- Command Line Interface: reference/cli.md
4444
- CLI Entry Point: reference/flyvis_cli/flyvis_cli.md
45+
- Hydra Configuration: reference/hydra_default_config.md
4546
- Training:
4647
- Run Training for Single Model: reference/flyvis_cli/training/train_single.md
4748
- Launch Ensemble Training on Compute Cloud: reference/flyvis_cli/training/train.md

‎pyproject.toml

+2
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ flyvis = [
1515
"analysis/visualization/matplotlibrc",
1616
"analysis/__main__.ipynb",
1717
"analysis/__main_per_model__.ipynb",
18+
"config/**/*.yaml",
19+
"config/**/*.yml",
1820
]
1921

2022
[project]

0 commit comments

Comments
 (0)