Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
8de8cb1
Bump tensorflow from 0.10.0rc0 to 1.15.2
dependabot[bot] Jan 28, 2020
c086e1f
deleted files and stored to other places
Sep 17, 2020
5420505
Merge remote-tracking branch 'origin/dependabot/pip/tensorflow-1.15.2'
Sep 17, 2020
b0ecd58
start torch
afeldman Feb 2, 2021
ac10254
lightning
afeldman Feb 2, 2021
103bdb6
start with dev
May 19, 2021
b509e9d
am git ignore
May 19, 2021
c2e4951
requirement file
May 20, 2021
4f2e2ba
opencv in requirement file
May 20, 2021
6f21cde
modify checkpoitns
May 27, 2021
ab65aed
modify checkpoitns
May 27, 2021
8754b88
modify checkpoitns
May 27, 2021
7495761
modify checkpoitns
May 27, 2021
97fd93f
modify checkpoitns
May 27, 2021
6226943
change setting
May 27, 2021
25da96f
change setting
May 27, 2021
b761d39
eval
Jun 7, 2021
9aec12c
start with DeepMoon Net implementation
Jun 8, 2021
0934cf1
build v shape
Jun 8, 2021
4cf899c
create model folder
Aug 4, 2021
d2ab9bb
tidy
Aug 5, 2021
dbcaf93
deepmoon v0.1
Aug 18, 2021
acd11a5
checkpoints with model name
Sep 7, 2021
cb5414d
change master
Feb 10, 2022
d018e1d
rename
Feb 10, 2022
616e09e
only learning not data processing
Feb 22, 2022
2342e36
renew the project
Feb 22, 2022
1fecb5f
add config
Feb 22, 2022
b86835e
start new training pipeline
Feb 22, 2022
4f0d20f
rename yml to yaml
Feb 22, 2022
66456ad
new requirements
Feb 22, 2022
e4f561d
change Exception to ValueError
Feb 22, 2022
0afaade
lighning module
Feb 22, 2022
2cb754c
not happy with accuracy
Feb 22, 2022
fc85b2a
still not happy with acc calculation
Feb 22, 2022
24efa57
accel loss on cpu
Feb 23, 2022
080f4d4
setup
Feb 23, 2022
acf99eb
rename project to torchmoon
Feb 23, 2022
505fa1d
rename
Feb 23, 2022
2814bdf
new version
Feb 23, 2022
8bb9ac0
change to ssim
Feb 23, 2022
bde5711
in all modes ssim measurement
Feb 23, 2022
2db04ff
versioning
Feb 23, 2022
aea14b6
error in setup.py
Feb 23, 2022
04c6ad7
versioning
Feb 23, 2022
4dee765
add torch to apu
Feb 28, 2022
8b87092
add apu to code
Feb 28, 2022
f080b3f
error with datasize
afeldman Sep 14, 2022
34ff43d
poetry
afeldman Aug 12, 2024
8aa07df
torchdata lightning >=2; poetry
afeldman Aug 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*pyd binary
31 changes: 31 additions & 0 deletions .github/workflows/publish-to-pypi.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
name: Publish Python 🐍 distributions 📦 to PyPI

on:
- push

jobs:
build-n-publish:
name: Build and publish Python 🐍 distributions 📦 to PyPI and TestPyPI
runs-on: ubuntu-18.04

strategy:
matrix:
python-version: [3.8]

steps:
- uses: actions/checkout@master
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip setuptools wheel twine apu
python -m pip install -r requirements.txt
- name: Publish distribution 📦 to PyPI
env:
TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
run: |
python setup.py sdist bdist_wheel
twine upload dist/*
14 changes: 8 additions & 6 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Compiled files
*.py[cod]
*.a
*.o
*.so
__pycache__
*None
*__py**__
[Bb]uild
*.json

.vscode*

# ipynb stuff
.ipynb_checkpoints/
Expand All @@ -16,3 +16,5 @@ docs/.ipynb_checkpoints/

# Mac OSX
.DS_Store
[Dd]ist/
*egg*
7 changes: 0 additions & 7 deletions LICENSE.md

This file was deleted.

20 changes: 20 additions & 0 deletions License
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
Copyright (c) 2021 Anton Feldmann

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
131 changes: 1 addition & 130 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,130 +1 @@
# DeepMoon - Lunar Crater Counting Through Deep Learning
Center for Planetary Sciences / Department of Astronomy & Astrophysics / Canadian Institute for Theoretical Astrophysics
University of Toronto

DeepMoon is a TensorFlow-based pipeline for training a convolutional neural
network (CNN) to recognize craters on the Moon, and determine their positions and
radii. It is the companion repo to the paper
[Lunar Crater Identification via Deep Learning](https://arxiv.org/abs/1803.02192), which
describes the motivation and development of the code, as well as results.

## Getting Started

### Overview

The DeepMoon pipeline trains a neural net using data derived from a global
digital elevation map (DEM) and catalogue of craters. The code is divided into
three parts. The first generates a set images of the Moon randomly cropped
from the DEM, with corresponding crater positions and radii. The second
trains a convnet using this data. The third validates the convnet's
predictions.

To first order, our CNN activates regions with high negative gradients, i.e.
pixels that decrease in value as you move across the image. Below illustrates
two examples of this, the first is a genuine DEM Lunar image from our dataset,
the second is a sample image taken from the web.
![CNN1](docs/CNN1.png?raw=true)
![CNN2](docs/CNN2.png?raw=true)

### Dependences

DeepMoon requires the following packages to function:

- [Python](https://www.python.org/) version 2.7 or 3.5+
- [Cartopy](http://scitools.org.uk/cartopy/) >= 0.14.2. Cartopy itself has a
number of [dependencies](http://scitools.org.uk/cartopy/docs/latest/installing.html#installing),
including the GEOS and Proj.4.x libraries. (For Ubuntu systems, these can be
installed through the `libgeos++-dev` and `libproj-dev` packages,
respectively.)
- [h5py](http://www.h5py.org/) >= 2.6.0
- [Keras](https://keras.io/) 1.2.2 [(documentation)](https://faroit.github.io/keras-docs/1.2.2/);
also tested with Keras >= 2.0.2
- [Numpy](http://www.numpy.org/) >= 1.12
- [OpenCV](https://pypi.python.org/pypi/opencv-python) >= 3.2.0.6
- [*pandas*](https://pandas.pydata.org/) >= 0.19.1
- [Pillow](https://python-pillow.org/) >= 3.1.2
- [PyTables](http://www.pytables.org/) >=3.4.2
- [TensorFlow](https://www.tensorflow.org/) 0.10.0rc0, also tested with
TensorFlow >= 1.0

This list can also be found in the `requirements.txt`.

### Data Sources
Our train, validation and test datasets, global DEM, post-processed
crater distribution on the test set, best model, and sample output
images can be found [on Zenodo](https://doi.org/10.5281/zenodo.1133969).

Examples of how to read these data can be found in the
`docs/Using Zenodo Data.ipynb` IPython notebook.

#### Digital Elevation Maps

We use the [LRO-Kaguya merged 59 m/pixel DEM][lola dem]. The DEM was
downsampled to 118 m/pixel and converted to 16-bit GeoTiff with the USGS
Astrogeology Cloud Processing service, and then rescaled to 8-bit PNG using
the [GDAL](http://www.gdal.org/) library:

```
gdal_translate -of PNG -scale -21138 21138 -co worldfile=no
LunarLROLrocKaguya_118mperpix_int16.tif LunarLROLrocKaguya_118mperpix.png
```

#### Crater Catalogues

For the ground truth longitude / latitude locations and sizes of craters, we
combine the [LROC Craters 5 to 20 km diameter][lroc cat] dataset with the
[Head et al. 2010 >= 20 km diameter][head cat] one ([alternate download
link][head cat2]). The LROC dataset was converted from ESRI shapefile to .csv.
They can be found under the `catalogues` folder of the repo, and have had their
formatting slightly modified to be read into *pandas*.

During initial testing, we also used the [Salamunićcar LU78287GT
catalogue][sala cat].

### Running DeepMoon

Each stage of DeepMoon has a corresponding script: `run_input_data_gen.py` for
generating input data, `run_model_training.py` to build and train the convnet,
and `run_get_unique_craters.py` to validate predictions and generate a crater
atlas. User-defined parameters, and instructions on on how to use each script,
can be found in the scripts themselves.

We recommend copying these scripts into a new working directory (and appending
this repo to your Python path) instead of modifying them in the repo.

Our model with default parameters was trained on a 16GB Tesla P100 GPU, however
12GB GPUs are more standard. Therefore, our default model may not run on many
systems without reducing the batch size, number of filters, etc., which can
affect final model convergence.

### Quick Usage

See `docs/Using Zenodo Data.ipynb` for basic examples on generating sample
datasets, loading a pre-trained CNN and using it to make predictions on
samples.

## Authors

* **Ari Silburt** - convnet architecture, crater extraction and post-processing
[silburt](https://github.com/silburt)
* **Charles Zhu** - input image generation, data ingestion and post-processing
[cczhu](https://github.com/cczhu)

### Contributors

* Mohamad Ali-Dib - [malidib](https://github.com/malidib/)
* Kristen Menou - [kmenou](https://www.kaggle.com/kmenou)
* Alan Jackson

## License

Copyright 2018 Ari Silburt, Charles Zhu and contributors.

DeepMoon is free software made available under the MIT License. For details see
the LICENSE.md file.

[lola dem]: https://astrogeology.usgs.gov/search/map/Moon/LRO/LOLA/Lunar_LRO_LrocKaguya_DEMmerge_60N60S_512ppd
[lroc cat]: http://wms.lroc.asu.edu/lroc/rdr_product_select?filter%5Btext%5D=&filter%5Blat%5D=&filter%5Blon%5D=&filter%5Brad%5D=&filter%5Bwest%5D=&filter%5Beast%5D=&filter%5Bsouth%5D=&filter%5Bnorth%5D=&filter%5Btopographic%5D=either&filter%5Bprefix%5D%5B%5DSHAPEFILE&show_thumbs=0&per_page=100&commit=Search
[head cat]: http://science.sciencemag.org/content/329/5998/1504/tab-figures-data
[head cat2]: http://www.planetary.brown.edu/html_pages/LOLAcraters.html
[sala cat]: https://astrogeology.usgs.gov/search/map/Moon/Research/Craters/GoranSalamuniccar_MoonCraters
https://github.com/silburt/DeepMoon/
1 change: 0 additions & 1 deletion catalogues/HeadCraters.csv

This file was deleted.

Loading