Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation geoips installation #651

Draft
wants to merge 23 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
9c1bd78
Working on installing
biosafetylvl5 Jun 5, 2024
dce320c
finish conda install
biosafetylvl5 Jun 18, 2024
ba1a06b
Update expert install, need feedback about whether or not other packa…
biosafetylvl5 Jun 20, 2024
ed06cad
Update installation docs
biosafetylvl5 Jun 26, 2024
516b271
Merge remote-tracking branch 'origin/main' into documentation-geoips-…
biosafetylvl5 Jun 26, 2024
059f535
Add WSL info
biosafetylvl5 Jun 26, 2024
b14d892
updates to install directions
biosafetylvl5 Jul 2, 2024
c89e13a
Add arm info
biosafetylvl5 Jul 8, 2024
108ed4c
Update index.rst to fix install and add gb for depends (#664)
biosafetylvl5 Jul 12, 2024
0065696
Merge branch 'main' into documentation-geoips-installation
mindyls Jul 13, 2024
f8a9ae4
Pinken
biosafetylvl5 Jul 22, 2024
df0c7db
Merge branch 'main' into documentation-geoips-installation
biosafetylvl5 Jul 22, 2024
f15afb0
Pinken
biosafetylvl5 Jul 22, 2024
0b11853
Remove extra information from install
biosafetylvl5 Jul 22, 2024
30f39c4
Change mamba to conda
biosafetylvl5 Jul 22, 2024
469b585
Add editable mode, install test by default
biosafetylvl5 Jul 22, 2024
4a5e1af
Add fortran by default, remove extra info about it
biosafetylvl5 Jul 22, 2024
cd718af
Pinken
biosafetylvl5 Jul 22, 2024
6e06c24
Change both install methods to `-e`
biosafetylvl5 Jul 22, 2024
95c026b
Merge branch 'main' into documentation-geoips-installation
biosafetylvl5 Jul 22, 2024
9c60e4d
Merge branch 'main' into documentation-geoips-installation
mindyls Jul 23, 2024
c0e51aa
Merge branch 'main' into documentation-geoips-installation
mindyls Aug 5, 2024
8caf778
Merge branch 'main' into documentation-geoips-installation
biosafetylvl5 Oct 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/source/new-docs/getting-started/best-practices/testing.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
:orphan:

Testing
=======

.. note::

Setting up WSL through windows is the easiest method for running GeoIPS on
a Windows system. To do so, follow directions on
`Microsoft WSL Install <https://learn.microsoft.com/en-us/windows/wsl/install>`_.
This requires administrator privileges.
Comment on lines +1 to +11
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the intention for this section of the getting started?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention is to add more information about testing geoips in the future, but that's not needed here

155 changes: 155 additions & 0 deletions docs/source/new-docs/getting-started/installing/advanced-env-setup.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
:orphan:

.. dropdown:: Distribution Statement

| # # # This source code is protected under the license referenced at
| # # # https://github.com/NRLMMD-GEOIPS.

Installing
==========

.. note::

Setting up WSL through windows is the easiest method for running GeoIPS on
a Windows system. To do so, follow directions on
`Microsoft WSL Install <https://learn.microsoft.com/en-us/windows/wsl/install>`_.
This requires administrator privileges.

For all installation methods, you need to setup environmental variables.

The required environmental variables are:

-``GEOIPS_PACKAGES_DIR`` is where GeoIPS looks for itself and its plugin packages
-``GEOIPS_TESTDATA_DIR`` is where GeoIPS looks to find its test datasets
-``GEOIPS_OUTDIRS`` defines where GeoIPS will write output data and imagery

These are fine defaults for most users:

.. code:: bash

export GEOIPS_PACKAGES_DIR=$HOME/geoips
export GEOIPS_TESTDATA_DIR=$GEOIPS_PACKAGES_DIR/test_data
export GEOIPS_OUTDIRS=$GEOIPS_PACKAGES_DIR/outdirs

If desired, the GeoIPS environment variables can be added to your
default shell configuration file (which is typically stored in
Comment on lines +8 to +35
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we going to link to this from anywhere, or just leave it orphaned for now, for later update / completion?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ I think orphaned for now is fine

``$HOME/.bashrc``) by running the following commands or something similar:

.. code:: bash

echo "export GEOIPS_PACKAGES_DIR=$GEOIPS_PACKAGES_DIR" >> ~/.bashrc
echo "export GEOIPS_TESTDATA_DIR=$GEOIPS_TESTDATA_DIR" >> ~/.bashrc
echo "export GEOIPS_OUTDIRS=$GEOIPS_OUTDIRS" >> ~/.bashrc

Install system dependencies directly
------------------------------------

If you are an expert user, you may want to install the dependencies manually. Do not do this if you already installed
dependencies with Conda.

Required
^^^^^^^^

* ``git``
* ``openblas`` (required for scipy pip install)
* ``make`` (required for pypublicdecompwt)
* ``python`` >= 3.9

Optional
^^^^^^^^

* ``gfortran`` (required for plugins including fortran builds)
* ``gcc`` and ``g++`` (required for plugins including fortran or C builds, and ARM machines)
* ``pdflatex`` (optional, for building pdf documentation)
* ``wget`` (required for downloading test data)
* ``make`` (optional, for building packages on ARM machines)
* Test data repos can be installed in `$GEOIPS_TESTDATA_DIR`

Debian
^^^^^^

For example, on Debian machines, you can install the optional and required dependencies like this:

.. code:: bash

sudo apt-get install git make libopenblas-dev python3
sudo apt-get install gfortran gcc g++ gdal-bin libgdal-dev texlive-latex-base wget make

This takes ~1gb of space.

Install GeoIPS
--------------

We can use ``pip`` to install all GeoIPS Python dependencies, and GeoIPS itself.

First, clone the GeoIPS git repository:

.. code:: bash

git clone https://github.com/NRLMMD-GeoIPS/geoips.git $GEOIPS_PACKAGES_DIR/geoips

.. code:: bash

# Ensure geoips python environment enabled before installing geoips
pip install "$GEOIPS_PACKAGES_DIR/geoips"

If you want to install GeoIPS with all optional dependencies, you can use:

.. code:: bash

# Ensure geoips python environment enabled before installing geoips
pip install "$GEOIPS_PACKAGES_DIR/geoips[doc,lint,test,debug]"

The optional dependencies are:

- ``doc``: for building the documentation with Sphinx
(the documentation is also available online at
https://nrlmmd-geoips.github.io/geoips/)
- ``lint``: for linting the code (useful for developers)
- ``test``: for running the tests
- ``debug``: for debugging the code with IPython/jupyter

Development
-----------

The installation steps for developers are the same as for normal and/or expert users, except for one step.

Most developers use Conda installations, but any dependency management solution is fine.

When installing geoips, please install all the extras and install in **editable** mode so that changes to the code are
immediately reflected in the installed package. Eg.

.. code:: bash

# Ensure geoips python environment enabled before installing geoips
pip install -e "$GEOIPS_PACKAGES_DIR/geoips[doc,lint,test,debug]"

See the [ADDING FUNCTIONALITY] page for more details on how to contribute to GeoIPS.

Docker
------

We provide a working Dockerfile that can be used to run GeoIPS in a container.

The Dockerfile can be built into a Docker image by cloning the GeoIPS repository and
running ``docker build``. For example:

.. code:: bash

git clone https://github.com/NRLMMD-GEOIPS/geoips.git geoips
cd geoips
docker build -t geoips .

The Docker image can be run with the following command:

.. code:: bash

docker run -it geoips # Run the container in interactive mode

Right now, the Docker image is only used for development and testing.

We suggest mounting in a data directory so your containers don't get too large.

If you are interested in using the Docker build
for production or plugin development, please reach
out to us via our contact page [CONTACT PAGE] or create an issue on GitHub [ISSUE PAGE].
146 changes: 146 additions & 0 deletions docs/source/new-docs/getting-started/installing/basic-env-setup.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
:orphan:

.. dropdown:: Distribution Statement

| # # # This source code is protected under the license referenced at
| # # # https://github.com/NRLMMD-GEOIPS.

Installing
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is going to be our "official" installatin method that we are sending out for people to use this week, then we will need to update the README.md to point here for installation.

Do we want to update this now?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what I tested, but I'm fine sending out the old instructions and waiting on this if we still have things to discuss

==========

.. note::

Setting up WSL through windows is the easiest method for running GeoIPS on
a Windows system. To do so, follow directions on
`Microsoft WSL Install <https://learn.microsoft.com/en-us/windows/wsl/install>`_.
This requires administrator privileges.

For all installation methods, you need to setup environmental variables.

The required environmental variables are:

- ``GEOIPS_PACKAGES_DIR`` is where GeoIPS looks for itself and its plugin packages
- ``GEOIPS_TESTDATA_DIR`` is where GeoIPS looks to find its test datasets
- ``GEOIPS_OUTDIRS`` defines where GeoIPS will write output data and imagery

These are fine defaults for most users:

.. code:: bash

export GEOIPS_PACKAGES_DIR=$HOME/geoips
export GEOIPS_TESTDATA_DIR=$GEOIPS_PACKAGES_DIR/test_data
export GEOIPS_OUTDIRS=$GEOIPS_PACKAGES_DIR/outdirs

If desired, the GeoIPS environment variables can be added to your
default shell configuration file (which is typically stored in
``$HOME/.bashrc``) by running the following commands or something similar:

.. code:: bash

echo "export GEOIPS_PACKAGES_DIR=$GEOIPS_PACKAGES_DIR" >> ~/.bashrc
echo "export GEOIPS_TESTDATA_DIR=$GEOIPS_TESTDATA_DIR" >> ~/.bashrc
echo "export GEOIPS_OUTDIRS=$GEOIPS_OUTDIRS" >> ~/.bashrc

Install Dependencies with Conda
-------------------------------

Conda is a package and virtual environment manager that can
be used to install the dependencies of GeoIPS.

We recommend using Conda if you are new to GeoIPS, programming or Python.

If you are an expert user, you may want to use the manual
or Docker installation methods.

Get Conda
^^^^^^^^^

- Download the appropriate version of `Conda
<https://www.anaconda.com/download#downloads>`_, `Miniconda
<https://docs.conda.io/en/latest/miniconda.html>`_, or
`Miniforge/Mambaforge <https://github.com/conda-forge/miniforge#download>`_.

For example, for Linux with Intel chips, one of the following might work:

.. code:: bash

wget https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-Linux-x86_64.sh

To find your specific version, visit the `Miniforge/Mambaforge github page
<https://github.com/conda-forge/miniforge/releases/>`_.

- Make the install script executable and run the installer,
following the prompts. Eg.:

.. code:: bash

chmod u+x ./Mambaforge-Linux-x86_64.sh
# After installing, follow instructions regarding conda init / restarting your terminal !
# Make sure you set env vars again if you restart your terminal !
./Mambaforge-Linux-x86_64.sh

.. code:: bash

rm -f ./Mambaforge-Linux-x86_64.sh

Create and activate a conda environment with dependencies
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Next we'll create a conda environment named ``geoips`` that contains all system
requirements for GeoIPS.

.. code:: bash

mamba create -y -n geoips -c conda-forge python=3.10 openblas git gfortran

.. warning::

If you are using an ARM machine, you may need to install ``pyhdf`` with conda.
If you run into ``fatal error: 'hdf.h' file not found`` when installing ``pyhdf``,
you may need to install ``pyhdf`` with conda. You can do this with the following command:

.. code:: bash

mamba create -y -n geoips -c conda-forge python=3.10 openblas git gfortran **pyhdf**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might just comment this line so people don't do this by default. I don't know if it matters either way, but 99% sure people will just copy/paste their way through.


Activate the conda environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

You will need to run ``conda activate geoips`` (or ``mamba activate geoips``)
every time you want to run or work on GeoIPS. Eg.

.. code:: bash

conda activate geoips

Install GeoIPS
--------------

We can use ``pip`` to install all GeoIPS Python dependencies, tests for GeoIPS and GeoIPS itself.

First, clone the GeoIPS git repository:

.. code:: bash

git clone https://github.com/NRLMMD-GeoIPS/geoips.git $GEOIPS_PACKAGES_DIR/geoips

.. code:: bash

# Ensure geoips python environment enabled before installing geoips
pip install -e "$GEOIPS_PACKAGES_DIR/geoips[test]"

If you want to install GeoIPS with all optional dependencies, you can use:

.. code:: bash

# Ensure geoips python environment enabled before installing geoips
pip install -e "$GEOIPS_PACKAGES_DIR/geoips[doc,lint,test,debug]"

The optional dependencies are:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would make a note here:

"These optional dependencies are all required for tests to fully pass"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comments above 🙂


- ``doc``: for building the documentation with Sphinx
(the documentation is also available online at
https://nrlmmd-geoips.github.io/geoips/)
- ``lint``: for linting the code (useful for developers)
- ``test``: for running the tests
- ``debug``: for debugging the code with IPython/jupyter (also useful for developers)
Copy link
Contributor

@mindyls mindyls Jul 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is worth having the base_install.sh and base_test.sh bundled as part of the basic installation - it is worth having immediate feedback that the install was successful. And it's really only 3 extra lines (base_install.sh, create_plugin_registries, and base_test.sh).

Oh - there is no create_plugin_registries included in this installation, so geoips is not fully functional at the end of this document.

6 changes: 0 additions & 6 deletions docs/source/new-docs/getting-started/installing/index.rst

This file was deleted.

33 changes: 33 additions & 0 deletions docs/source/new-docs/getting-started/installing/install-geoips.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
:orphan:

Install GeoIPS
--------------

We can use ``pip`` to install all GeoIPS Python dependencies, and GeoIPS itself.

First, clone the GeoIPS git repository:

.. code:: bash

git clone https://github.com/NRLMMD-GeoIPS/geoips.git $GEOIPS_PACKAGES_DIR/geoips

.. code:: bash

# Ensure geoips python environment enabled before installing geoips
pip install "$GEOIPS_PACKAGES_DIR/geoips"

If you want to install GeoIPS with all optional dependencies, you can use:

.. code:: bash

# Ensure geoips python environment enabled before installing geoips
pip install "$GEOIPS_PACKAGES_DIR/geoips[doc,lint,test,debug]"

The optional dependencies are:

- ``doc``: for building the documentation with Sphinx
(the documentation is also available online at
https://nrlmmd-geoips.github.io/geoips/)
- ``lint``: for linting the code (useful for developers)
- ``test``: for running the tests
- ``debug``: for debugging the code with IPython/jupyter
Comment on lines +3 to +33
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the intention for this section of the installation?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intention is to - later - embed it in the advanced and the basic installation instructions