Merge pull request jupyter#519 from GeorgianaElena/architecture-section

docs/source/architecture/content-architecture.rst

@@ -1,8 +1,137 @@
+============
 Architecture
 ============
 
-.. toctree::
-   :maxdepth: 1
+**A high level visual overview of project relationships**
+
+
+.. image:: ../_static/_images/repos_map.png
+   :width: 75%
+   :alt: Architecture diagram of project relationships
+
+
+
+How IPython and Jupyter Notebook work
+=====================================
+
+.. contents:: Contents
+   :local:
+
+Abstract
+--------
+This section focuses on IPython and Jupyter notebook and how they interact.
+When we discuss ``IPython``, we talk about two fundamental roles:
+
+- Terminal IPython as the familiar REPL
+- The IPython kernel that provides computation and communication with the
+  frontend interfaces, like the notebook
+
+Jupyter Notebook and its flexible interface extends the notebook beyond code
+to visualization, multimedia, collaboration, and more.
+
+Terminal IPython
+----------------
+
+When you type ``ipython``, you get the original IPython interface, running in
+the terminal. It does something like this::
+
+    while True:
+        code = input(">>> ")
+        exec(code)
+
+Of course, it's much more complex, because it has to deal with multi-line
+code, tab completion using :mod:`readline`, magic commands, and so on. But the
+model is like code example: prompt the user for some code, and when they've
+entered it, execute it in the same process. This model is often called a
+:term:`REPL`, or Read-Eval-Print-Loop.
+
+The IPython Kernel
+------------------
+
+All the other interfaces —- the Notebook, the Qt console, ``ipython console``
+in the terminal, and third party interfaces —- use the IPython Kernel. The
+IPython Kernel is a separate process which is responsible for running user
+code, and things like computing possible completions. Frontends, like the
+notebook or the Qt console, communicate with the IPython Kernel using JSON
+messages sent over `ZeroMQ <http://zeromq.org/>`_ sockets; the protocol used
+between the frontends and the IPython Kernel is described in
+:ref:`jupyterclient:messaging`.
+
+The core execution machinery for the kernel is shared with terminal IPython:
+
+.. image:: figs/ipy_kernel_and_terminal.png
+
+A kernel process can be connected to more than one frontend simultaneously. In
+this case, the different frontends will have access to the same variables.
+
+.. TODO: Diagram illustrating this?
+
+This design was intended to allow easy development of different frontends
+based on the same kernel, but it also made it possible to support new
+languages in the same frontends, by developing kernels in those languages, and
+we are refining IPython to make that more practical.
+
+Today, there are two ways to develop a kernel for another language. Wrapper
+kernels reuse the communications machinery from IPython, and implement only
+the core execution part. Native kernels implement execution and communications
+in the target language:
+
+.. image:: figs/other_kernels.png
+
+Wrapper kernels are easier to write quickly for languages that have good
+Python wrappers, like `octave_kernel <https://pypi.python.org/pypi/octave_kernel>`_,
+or languages where it's impractical to implement the communications machinery,
+like `bash_kernel <https://pypi.python.org/pypi/bash_kernel>`_. Native kernels
+are likely to be better maintained by the community using them, like
+`IJulia <https://github.com/JuliaLang/IJulia.jl>`_ or
+`IHaskell <https://github.com/gibiansky/IHaskell>`_.
+
+.. seealso::
+
+   :ref:`jupyterclient:kernels`
+
+   :ref:`Kernels <kernels-langs>`
+
+Notebooks
+---------
+
+The Notebook frontend does something extra. In addition to running your code,
+it stores code and output, together with markdown notes, in an editable
+document called a notebook. When you save it, this is sent from your browser
+to the notebook server, which saves it on disk as a JSON file with a
+``.ipynb`` extension.
+
+.. image:: figs/notebook_components.png
+
+The notebook server, not the kernel, is responsible for saving and loading
+notebooks, so you can edit notebooks even if you don't have the kernel for
+that language—you just won't be able to run code. The kernel doesn't know
+anything about the notebook document: it just gets sent cells of code to
+execute when the user runs them.
+
+Exporting notebooks to other formats
+------------------------------------
+
+The Nbconvert tool in Jupyter converts notebook files to other formats, such
+as HTML, LaTeX, or reStructuredText. This conversion goes through a series of
+steps:
+
+.. image:: figs/nbconvert.png
+
+1. Preprocessors modify the notebook in memory. E.g. ExecutePreprocessor runs
+   the code in the notebook and updates the output.
+2. An exporter converts the notebook to another file format. Most of the
+   exporters use templates for this.
+3. Postprocessors work on the file produced by exporting.
+
+The `nbviewer <http://nbviewer.jupyter.org/>`_ website uses nbconvert with the
+HTML exporter. When you give it a URL, it fetches the notebook from that URL,
+converts it to HTML, and serves that HTML to you.
+
+IPython.parallel
+----------------
 
-   how_jupyter_ipython_work
-   visual_overview
+IPython also includes a parallel computing framework,
+`IPython.parallel <https://ipyparallel.readthedocs.io/en/latest/>`_. This
+allows you to control many individual engines, which are an extended version
+of the IPython kernel described above.

docs/source/contributing/dev-contributions/contrib_guide.rst

@@ -120,7 +120,7 @@ including partial or in-progress work, is appreciated.
 
 Contribution Workflow
 ---------------------
-.. image:: ../_static/_images/contribution_workflow.png
+.. image:: ../../_static/_images/contribution_workflow.png
 
 
 

docs/source/contributing/dev-contributions/index.rst

@@ -9,7 +9,6 @@ Developer Guide
 .. toctree::
    :maxdepth: 1
 
-   contrib_first_time.rst
    contrib_guide.rst
    jupyter_enhancement_proposals.rst
    releasing.rst

docs/source/index.rst

@@ -29,15 +29,15 @@ Start Here
 
        How to contribute to the projects
 
-   * - :doc:`Narratives and Use Cases <use-cases/content-user>`
+   * - :doc:`Narratives and Use Cases <projects/use-cases/content-user>`
 
        Narratives of common deployment scenarios
 
      - :doc:`Release Notes <releases>`
 
        New features, upgrades, deprecation notes, and bug fixes
 
-   * - :doc:`IPython <ipython/content-ipython>`
+   * - :doc:`IPython <reference/ipython>`
 
        An interactive Python kernel and REPL
 
@@ -49,7 +49,7 @@ Start Here
 
        Documentation for users
 
-     - :doc:`Advanced <migrating>`
+     - :doc:`Advanced <projects/advanced/content-advanced>`
 
        Documentation for advanced use-cases
 
@@ -140,20 +140,6 @@ Table of Contents
 
    architecture/content-architecture.rst
 
-.. _about-docs:
-
-.. toctree::
-   :maxdepth: 2
-
-   use-cases/content-user.rst
-
-.. _about-ipython:
-
-.. toctree::
-   :maxdepth: 2
-
-   ipython/content-ipython.rst
-
 .. _jupyter-subprojects:
 
 .. toctree::

docs/source/projects/advanced/content-advanced.rst

@@ -0,0 +1,6 @@
+Advanced Use Cases
+==================
+.. toctree::
+   :maxdepth: 2
+
+   migrating

docs/source/migrating.rst → docs/source/projects/advanced/migrating.rst

@@ -1,5 +1,3 @@
-.. _migrating:
-
 ===============================
 Migrating from IPython Notebook
 ===============================

docs/source/projects/content-projects.rst

@@ -16,6 +16,8 @@ Using Jupyter
    incubator
    kernels
    doc-proj-categories
+   advanced/content-advanced
+   use-cases/content-user
 
 Installation
 ------------

docs/source/use-cases/content-user.rst → docs/source/projects/use-cases/content-user.rst

@@ -1,14 +1,6 @@
 Narratives and Use Cases
 ========================
 
-What are Narratives?
---------------------
-Narratives are *collaborative*, *shareable*, *publishable*, and
-*reproducible*. We believe that Narratives help both yourself and other
-researchers by sharing your use of Jupyter projects, technical specifics of
-your deployment, and installation and configuration tips so that others can
-learn from your experiences.
-
 .. toctree::
    :maxdepth: 2
 
@@ -19,3 +11,11 @@ learn from your experiences.
    science
    education
    enterprise
+
+What are Narratives?
+--------------------
+Narratives are *collaborative*, *shareable*, *publishable*, and
+*reproducible*. We believe that Narratives help both yourself and other
+researchers by sharing your use of Jupyter projects, technical specifics of
+your deployment, and installation and configuration tips so that others can
+learn from your experiences.

docs/source/reference/content-reference.rst

@@ -5,6 +5,7 @@ Reference
    :maxdepth: 1
 
    mimetype.md
+   ./ipython
    ../glossary