You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/concepts_skills/linux/building_linux_containers.rst
+1-3Lines changed: 1 addition & 3 deletions
Original file line number
Diff line number
Diff line change
@@ -1,7 +1,5 @@
1
-
.. _building_linux_containers:
2
-
3
1
How to build a Ubuntu Linux image with Docker
4
-
=================================================
2
+
=============================================
5
3
Docker images are configurable, free-standing computing environments that can be used to create and run custom software on top of an operating system. Unlike virtual machines (VM), images do not contain an operating system. Docker images are a convenient way to distribute complicated software programs that have numerous dependencies and complicated configurations. We are using Docker images because CircleCI allows users to use Docker images to customize the environment used to execute each build. This makes it much easier to install programs into the environment used by CircleCI to run our builds.
6
4
7
5
Docker images are built by compiling Dockerfiles which are explicit instructions on how to build the image. Importantly, this makes Docker images very transparent.
Copy file name to clipboardExpand all lines: docs/concepts_skills/software_engineering/continuous_integration.rst
+14-4Lines changed: 14 additions & 4 deletions
Original file line number
Diff line number
Diff line change
@@ -37,7 +37,9 @@ Follow these instructions to use CircleCI to continuously test a GitHub reposito
37
37
#. Click the `Follow Project` button for any repository you want to compile and test on CircleCI
38
38
#. Add a CircleCI configuration file, ``/path/to/repo/.circleci/config.yml``, to the repository to instruct CircleCI what to execute within each build. This includes the following instructions
39
39
40
-
* Which container/virtual machine should be used to run the build. We are using a custom container so that little additional software needs to be installed to test our code. See :numref:`building_linux_containers` for more information about how to create and use custom Linux containers.
40
+
See :ref:`Using the CircleCI cloud-based continuous integration system` please.
41
+
42
+
* Which container/virtual machine should be used to run the build. We are using a custom container so that little additional software needs to be installed to test our code. See :ref:`How to build a Ubuntu Linux image with Docker` for more information about how to create and use custom Linux containers.
41
43
* Which GitHub repository to checkout.
42
44
* How to install any additional packages needed to execute the tests.
43
45
* Instructions on how to run the tests and store the results.
@@ -75,15 +77,23 @@ There are two main mechanisms to decrease the runtime of CircleCI builds by load
75
77
76
78
The Dockerfile for the Docker image that the Karr Lab uses with CircleCI is located at `https://github.com/KarrLab/karr_lab_docker_images/tree/master/build <https://github.com/KarrLab/karr_lab_docker_images/tree/master/build>`_.
77
79
78
-
See the :numref:`building_linux_containers` for more information.
80
+
See :ref:`How to build a Ubuntu Linux image with Docker` for more information.
79
81
80
82
The Karr Lab uses both of these mechanisms.
81
83
82
84
Changing package dependencies for a CircleCI build
Occasionally, you may need to change the dependencies of a repository. The following recipe can be used to update the PyPI dependencies of a repository:
86
+
Occasionally, you may need to change the dependencies of a repository. The following steps can perform this task:
87
+
88
+
#. Update the ``pip`` ``requirements.txt`` files which identify packages that the repository uses. To automate this process, use the commands in `karr_lab_build_utils <https://docs.karrlab.org/karr_lab_build_utils/latest/tutorial_developers.html#managing-dependencies-of-packages>`_ that can obtain a package's dependencies and identify dependencies that may be missing or unnecessary.
89
+
90
+
* ``./requirements.txt`` describes the dependencies of the package. It lists the package's immediate dependencies, i.e., the other packages that are imported, and may constraint which versions are suitable for the package. It should not contain URLs, specify the source which should provide a package, or specify the specific version of a dependency to install. The systems administrator who configures the package's environment, not the programmer, should be responsible for these details.
91
+
* ``./requirements.optional.txt`` describes the package's optional dependencies.
92
+
* ``./tests/requirements.txt`` lists the dependencies of the package's tests.
93
+
* ``./docs/requirements.txt`` describes the dependencies of the software that compiles the package's documentation.
94
+
* ``.circleci/requirements.txt`` tells CircleCI where to obtain dependencies that are not located in PyPI. Dependencies can be identified by GitHub URLs with the format ``git+https://github.com/--account_name--/--package_name--.git#egg=--package_name--``. All dependencies--including transitive dependencies--must be listed. The list must be arranged in dependency order, so that if package `y` depends on package `x` then `x` precedes `y`, as in a `topological sort <https://en.wikipedia.org/wiki/Topological_sorting>`_ of the dependencies. This file works around limitations in pip and PyPI.
95
+
* ``./docs/requirements.rtd.txt`` tells Read the Docs where to obtain dependencies that are not located in PyPI.
85
96
86
-
#. Update the ``pip`` ``requirements.txt`` files which describe the dependencies of the package, its tests, and its documentation. This includes ``./requirements.txt`` which describes the dependencies of the package, ``./requirements.optional.txt`` which describes optional dependencies of the package, ``./tests/requirements.txt`` which describes the dependencies of the package's tests, ``./docs/requirements.txt`` which describes the dependencies of the package's documentation, and ``.circleci/requirements.txt`` and ``./docs/requirements.rtd.txt`` which tell CircleCI and Read the Docs where to obtain dependencies that are not located in PyPI.
87
97
#. Commit the changes to the ``requirements.txt`` files to your code repository.
88
98
89
99
If there are errors in the compilation and/or installation of the new dependencies, you can try rebuilding the build without its cache. As described above, we recommend using CircleCI's cache to avoid repeatedly recompiling dependent packages. The cache avoids recompiling dependent packages by storing them after the first time they are built, and loading them on subsequent builds. You can force CircleCI to create a new cache by incrementing the cache version number ``vXXX`` specified in ``.circleci/config.yml`` and pushing the updated configuration file to your code repository::
Copy file name to clipboardExpand all lines: docs/concepts_skills/software_engineering/distributing_python.rst
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -262,7 +262,7 @@ There are also several online tutorials with more information about how to uploa
262
262
263
263
Distributing containers with Docker Hub
264
264
---------------------------------------
265
-
Docker Hub can be used to distribute virtual machines simply by changing the public/private setting of a repository. See :numref:`building_linux_containers` for more information about using Docker and Docker Hub.
265
+
Docker Hub can be used to distribute virtual machines simply by changing the public/private setting of a repository. See :ref:`How to build a Ubuntu Linux image with Docker` for more information about using Docker and Docker Hub.
Copy file name to clipboardExpand all lines: docs/concepts_skills/software_engineering/structuring_python_projects.rst
+22-15Lines changed: 22 additions & 15 deletions
Original file line number
Diff line number
Diff line change
@@ -2,43 +2,50 @@ Structuring Python projects
2
2
===========================
3
3
We recommend using the following principles to organize Python projects:
4
4
5
-
* Use separate repositories for each project
5
+
* Use a separate repository for each project
6
6
* Store only one package in each repository
7
-
* Structure each package as outlined below following the recommendations provided by `The Hitchhiker's Guide To Python <http://python-guide-pt-br.readthedocs.io/en/latest/writing/structure/>`_:
7
+
* Structure each package as outlined below, following the recommendations provided by `The Hitchhiker's Guide To Python <https://docs.python-guide.org/writing/structure/#structure-of-the-repository>`_:
8
8
9
9
.. code-block :: text
10
10
11
-
repository_name/ # source code directory
12
-
__init__.py # each source code directory must contain an ``__init__.py`` file
11
+
repository_name/ # source code directory (1)
12
+
__init__.py # each source code directory must contain an __init__.py file
13
13
__main__.py # optional, for command line programs
14
14
VERSION # text file with version number
15
15
data/ # directory for data files needed by the code
16
16
tests/ # directory for test code
17
17
fixtures/ # fixtures for tests
18
-
secret/ # git-ignored fixtures that contain usernames, passwords, and tokens
19
-
requirements.txt # list of packages required to run the tests; used by CircleCI
18
+
secret/ # git-ignored fixtures containing usernames, passwords, and tokens
19
+
requirements.txt # list of packages required to run the tests, but not
20
+
# required by the project; used by CircleCI (2, 3)
20
21
docs/ # directory for documentation
21
22
conf.py # documentation configuration
22
23
index.rst # main documentation file
23
-
requirements.txt # list of packages required to compile the documentation
24
-
requirements.rtd.txt # list of packages required to compile the documentation; used by Read the Docs
24
+
requirements.txt # packages required to compile the documentation (2, 3)
25
+
requirements.rtd.txt # list of packages required to compile the documentation;
26
+
# used by Read the Docs (2)
25
27
_build/html/ # directory where compiled documentation is saved
26
-
_static # optional for static files such as .css and .js files needed for the documentation
28
+
_static # optional for static files such as .css and .js files
29
+
# needed for the documentation
27
30
examples/ # (optional) directory for examples of how to use the code
28
31
LICENSE # license file
29
32
MANIFEST.in # list of files that should be distributed with the package
30
33
README.md # Read me file; displayed by GitHub
31
-
requirements.txt # list of requirements
32
-
requirements.optional.txt # list of optional requirements
34
+
requirements.txt # list of required packages (2, 3)
35
+
requirements.optional.txt # list of optional requirements (2, 3)
33
36
setup.cfg # options for the installation script
34
37
setup.py # installation script
35
38
.circleci/ # directory for CircleCI configuration
36
39
config.yml # CircleCI configuration
37
-
requirements.txt # list of locations of requirements not in PyPI
38
-
downstream_dependencies.yml # List of downstream dependencies in YAML format
40
+
requirements.txt # list of locations of requirements not in PyPI (2, 3)
41
+
downstream_dependencies.yml # List of downstream dependencies in YAML format (3)
39
42
.gitignore # list of file paths and extensions that Git should ignore
40
43
.readthedocs.yml # Read the Docs configuration
41
44
42
-
*Note: the name of the source code directory should be the same as that of the repository*
45
+
\(1\) The name of the source code directory should be the same as that of the repository.
43
46
44
-
* Separate code which is useful on its own, distinct from the project, into their own packages and repositories
47
+
\(2\) For details about `requirements.txt` files see the section about :ref:`Changing package dependencies for a CircleCI build`.
48
+
49
+
\(3\) These dependencies can be determined automatically by `karr_lab_build_utils <https://docs.karrlab.org/karr_lab_build_utils/latest/tutorial_developers.html#managing-dependencies-of-packages>`_.
50
+
51
+
* Separate code that is useful on its own---distinct from the project, into independent packages and repositories.
0 commit comments