Merge pull request robertmartin8#279 from robertmartin8/v1.4.0

v1.4.0

.github/workflows/main.yml

@@ -10,7 +10,12 @@ jobs:
         runs-on: ubuntu-latest
         strategy:
             matrix:
+                os: [ubuntu-latest]
                 python-version: [3.6, 3.7, 3.8]
+                include:
+                    - os: [windows-latest, macos-latest]
+                      python-version: 3.8
+
         steps:
             - uses: actions/checkout@v2
             - name: Set up Python ${{ matrix.python-version }}

Dockerfile

@@ -1,8 +1,5 @@
 FROM python:3.7.7-slim-stretch as builder
 
-# File Author / Maintainer
-# MAINTAINER
-
 # this will be user root regardless whether home/beakerx is not
 COPY . /tmp/pypfopt
 

README.md

@@ -8,7 +8,7 @@
         <img src="https://img.shields.io/badge/python-v3-brightgreen.svg"
             alt="python"></a> &nbsp;
     <a href="https://pypi.org/project/PyPortfolioOpt/">
-        <img src="https://img.shields.io/badge/pypi-v1.3.1-brightgreen.svg"
+        <img src="https://img.shields.io/badge/pypi-v1.4.0-brightgreen.svg"
             alt="pypi"></a> &nbsp;
     <a href="https://opensource.org/licenses/MIT">
         <img src="https://img.shields.io/badge/license-MIT-brightgreen.svg"
@@ -68,6 +68,8 @@ Head over to the [documentation on ReadTheDocs](https://pyportfolioopt.readthedo
 If you would like to play with PyPortfolioOpt interactively in your browser, you may launch Binder [here](https://mybinder.org/v2/gh/robertmartin8/pyportfolioopt/master). It takes a
 while to set up, but it lets you try out the cookbook recipes without having to deal with all of the requirements.
 
+*Note: macOS users will need to install [Command Line Tools](https://osxdaily.com/2014/02/12/install-command-line-tools-mac-os-x/).*
+
 *Note: if you are on windows, you first need to installl C++. ([download](https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools&rel=16), [install instructions](https://drive.google.com/file/d/0B4GsMXCRaSSIOWpYQkstajlYZ0tPVkNQSElmTWh1dXFaYkJr/view))*
 
 This project is available on PyPI, meaning that you can just:
@@ -395,9 +397,12 @@ Special shout-outs to:
 - Aditya Bhutra
 - Thomas Schmelzer
 - Rich Caputo
+- Nicolas Knudde
 
 ## Getting in touch
 
-If you are having a problem with PyPortfolioOpt, please raise an issue.
+If you are having a problem with PyPortfolioOpt, please raise a GitHub issue. For anything else, you can reach me at:
 
-For anything else, you can contact me via the [form](https://reasonabledeviations.com/about/) on my website.
+<center>
+<img src="https://github.com/robertmartin8/ReasonableDeviations/blob/gh-pages/assets/images/contact.png" style="width:75%;"/>
+</center>

docs/BlackLitterman.rst

@@ -5,7 +5,7 @@ Black-Litterman Allocation
 ##########################
 
 The Black-Litterman (BL) model [1]_ takes a Bayesian approach to asset allocation.
-Specifically, it combines a **prior** estimate of returns (canonically, the market-implied
+Specifically, it combines a **prior** estimate of returns (for example, the market-implied
 returns) with **views** on certain assets, to produce a **posterior** estimate of expected
 returns. The advantages of this are:
 
@@ -52,11 +52,10 @@ I'd like to thank  `Felipe Schneider <https://github.com/schneiderfelipe>`_ for
 contributions to the Black-Litterman implementation. A full example of its usage, including the acquistion
 of market cap data for free, please refer to the `cookbook recipe <https://github.com/robertmartin8/PyPortfolioOpt/blob/master/cookbook/4-Black-Litterman-Allocation.ipynb>`_.
 
-.. caution:: 
+.. tip:: 
 
-    Our implementation of Black-Litterman makes frequent use of the fact that python 3.6+ dictionaries
-    remain ordered. It is still possible to use python 3.5 but you will have to construct the BL inputs
-    explicitly (``Q``, ``P``, ``omega``).
+    Thomas Kirschenmann has built a neat interactive `Black-Litterman tool <https://github.com/thk3421-models/cardiel>`_
+    on top of PyPortfolioOpt, which allows you to visualise BL outputs and compare optimisation objectives.
 
 Priors
 ======

docs/EfficientFrontier.rst

@@ -20,7 +20,7 @@ class of problems, which happen to be incredibly useful for finance. A convex pr
 
 where :math:`\mathbf{x} \in \mathbb{R}^n`, and :math:`f(\mathbf{x}), g_i(\mathbf{x})` are convex functions. [1]_
 
-Fortunately, portfolio optimisation problems (with standard and objective constraints) are convex. This
+Fortunately, portfolio optimisation problems (with standard objectives and constraints) are convex. This
 allows us to immediately apply the vast body of theory as well as the refined solving routines -- accordingly,
 the main difficulty is inputting our specific problem into a solver.
 
@@ -239,7 +239,7 @@ frontier. Note  that some of the parent methods, like :py:func:`max_sharpe` and
 are not applicable to mean-semivariance portfolios, so calling them returns an error.
 
 :py:class:`EfficientSemivariance` has a slightly different API to :py:class:`EfficientFrontier`. Instead of passing
-in a covariance matrix, you should past in a dataframe of historical returns (this can be constructed
+in a covariance matrix, you should past in a dataframe of historical/simulated returns (this can be constructed
 from your price dataframe using the helper method :py:func:`expected_returns.returns_from_prices`). Here
 is a full example, in which we seek the portfolio that minimises the semivariance for a target
 annual return of 20%::
@@ -279,6 +279,59 @@ implementation is based on Markowitz et al (2019) [3]_.
     :members:
     :exclude-members: max_sharpe, min_volatility
 
+Efficient CVaR
+==============
+
+The **conditional value-at-risk** (a.k.a **expected shortfall**) is a popular measure of tail risk. The CVaR can be
+thought of as the average of losses that occur on "very bad days", where "very bad" is quantified by the parameter
+:math:`\beta`.
+
+For example, if we calculate the CVaR to be 10% for :math:`\beta = 0.95`, we can be 95% confident that the worst-case
+average daily loss will be 3%. Put differently, the CVaR is the average of all losses so severe that they only occur
+:math:`(1-\beta)\%` of the time. 
+
+While CVaR is quite an intuitive concept, a lot of new notation is required to formulate it mathematically (see
+the `wiki page <https://en.wikipedia.org/wiki/Expected_shortfall>`_ for more details). We will adopt the following
+notation: 
+
+- *w* for the vector of portfolio weights
+- *r* for a vector of asset returns (daily), with probability distribution :math:`p(r)`.
+- :math:`L(w, r) = - w^T r` for the loss of the portfolio
+- :math:`\alpha` for the portfolio value-at-risk (VaR) with confidence :math:`\beta`.
+
+The CVaR can then be written as:
+
+.. math::
+    CVaR(w, \beta) = \frac{1}{1-\beta} \int_{L(w, r) \geq \alpha (w)} L(w, r) p(r)dr.
+
+This is a nasty expression to optimise because we are essentially integrating over VaR values. The key insight
+of Rockafellar and Uryasev (2001) [4]_ is that we can can equivalently optimise the following convex function:
+
+.. math:: 
+    F_\beta (w, \alpha) = \alpha + \frac{1}{1-\beta} \int [-w^T r - \alpha]^+ p(r) dr,
+
+where :math:`[x]^+ = \max(x, 0)`. The authors prove that minimising :math:`F_\beta(w, \alpha)` over all
+:math:`w, \alpha` minimises the CVaR. Suppose we have a sample of *T* daily returns (these
+can either be historical or simulated). The integral in the expression becomes a sum, so the CVaR optimisation
+problem reduces to a linear program:
+
+.. math::
+
+    \begin{equation*}
+    \begin{aligned}
+    & \underset{w, \alpha}{\text{minimise}} & & \alpha + \frac{1}{1-\beta} \frac 1 T \sum_{i=1}^T u_i \\
+    & \text{subject to} & & u_i \geq 0  \\
+    &&&  u_i \geq -w^T r_i - \alpha. \\
+    \end{aligned}
+    \end{equation*}
+
+This formulation introduces a new variable for each datapoint (similar to Efficient Semivariance), so
+you may run into performance issues for long returns dataframes. At the same time, you should aim to
+provide a sample of data that is large enough to include tail events. 
+
+.. autoclass:: pypfopt.efficient_frontier.EfficientCVaR
+    :members:
+    :exclude-members: max_sharpe, min_volatility, max_quadratic_utility
 
 
 .. _custom-optimisation:
@@ -295,7 +348,7 @@ The :py:class:`EfficientFrontier` class inherits from the ``BaseConvexOptimizer`
 define your own optimisation problem. You can either optimise some generic ``convex_objective``
 (which *must* be built using ``cvxpy`` atomic functions -- see `here <https://www.cvxpy.org/tutorial/functions/index.html>`_)
 or a ``nonconvex_objective``, which uses ``scipy.optimize`` as the backend and thus has a completely
-different API. For examples, check out this `cookbook recipe
+different API. For more examples, check out this `cookbook recipe
 <https://github.com/robertmartin8/PyPortfolioOpt/blob/master/cookbook/3-Advanced-Mean-Variance-Optimisation.ipynb>`_.
 
     .. class:: pypfopt.base_optimizer.BaseConvexOptimizer
@@ -311,3 +364,4 @@ References
 .. [1] Boyd, S.; Vandenberghe, L. (2004). `Convex Optimization <https://web.stanford.edu/~boyd/cvxbook/>`_.
 .. [2] Estrada, J (2007). `Mean-Semivariance Optimization: A Heuristic Approach <https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1028206>`_.
 .. [3] Markowitz, H.; Starer, D.; Fram, H.; Gerber, S. (2019). `Avoiding the Downside <https://www.hudsonbaycapital.com/documents/FG/hudsonbay/research/599440_paper.pdf>`_.
+.. [4] Rockafellar, R.; Uryasev, D. (2001). `Optimization of conditional value-at-risk <https://www.ise.ufl.edu/uryasev/files/2011/11/CVaR1_JOR.pdf>`_

docs/Plotting.rst

@@ -22,18 +22,60 @@ and add constraints like you normally would, but *before* calling an optimisatio
     ef.add_constraint(lambda w: w[2] == 0.15)
     ef.add_constraint(lambda w: w[3] + w[4] <= 0.10)
 
-    # 100 portfolios with risks between 0.10 and 0.30
-    risk_range = np.linspace(0.10, 0.40, 100)
-    ax = plotting.plot_efficient_frontier(ef, ef_param="risk", ef_param_range=risk_range,
-                               show_assets=True, showfig=True)
+    fig, ax = plt.subplots()
+    plotting.plot_efficient_frontier(ef, ax=ax, show_assets=True)
+    plt.show()
 
-This produces the following plot -- you can set attributes using the returned ``ax`` object: 
+This produces the following plot:
 
     .. image:: ../media/ef_plot.png
         :width: 80%
         :align: center
         :alt: the Efficient Frontier
 
+You can explicitly pass a range of parameters (risk, utility, or returns) to generate a frontier:: 
+
+    # 100 portfolios with risks between 0.10 and 0.30
+    risk_range = np.linspace(0.10, 0.40, 100)
+    plotting.plot_efficient_frontier(ef, ef_param="risk", ef_param_range=risk_range,
+                                    show_assets=True, showfig=True)
+
+
+We can easily generate more complex plots. The following script plots both the efficient frontier and
+randomly generated (suboptimal) portfolios, coloured by the Sharpe ratio::
+
+    fig, ax = plt.subplots()
+    plotting.plot_efficient_frontier(ef, ax=ax, show_assets=False)
+
+    # Find the tangency portfolio
+    ef.max_sharpe()
+    ret_tangent, std_tangent, _ = ef.portfolio_performance()
+    ax.scatter(std_tangent, ret_tangent, marker="*", s=100, c="r", label="Max Sharpe")
+
+    # Generate random portfolios
+    n_samples = 10000
+    w = np.random.dirichlet(np.ones(len(mu)), n_samples)
+    rets = w.dot(mu)
+    stds = np.sqrt(np.diag(w @ S @ w.T))
+    sharpes = rets / stds
+    ax.scatter(stds, rets, marker=".", c=sharpes, cmap="viridis_r")
+
+    # Output
+    ax.set_title("Efficient Frontier with random portfolios")
+    ax.legend()
+    plt.tight_layout()
+    plt.savefig("ef_scatter.png", dpi=200)
+    plt.show()
+
+This is the result:
+
+    .. image:: ../media/ef_scatter.png
+        :width: 80%
+        :align: center
+        :alt: the Efficient Frontier with random portfolios
+
+Documentation reference
+=======================
 
 .. automodule:: pypfopt.plotting
 

docs/Roadmap.rst

@@ -13,7 +13,6 @@ are interested in implementing one of these, raise an issue or send me an email
 discuss. If you have any other feature requests, please raise them using GitHub
 `issues <https://github.com/robertmartin8/PyPortfolioOpt/issues>`_
 
-- Improved plotting
 - Open-source backtests using either `Backtrader <https://www.backtrader.com/>`_ or
   `Zipline <https://github.com/quantopian/zipline>`_.
 - Optimising for higher moments (i.e skew and kurtosis)
@@ -22,6 +21,17 @@ discuss. If you have any other feature requests, please raise them using GitHub
 - Monte Carlo optimisation with custom distributions
 - Further support for different risk/return models
 
+1.4.0
+=====
+
+- Finally implemented CVaR optimisation! This has been one of the most requested features. Many thanks
+  to `Nicolas Knudde <https://github.com/nknudde>`_ for the initial draft. 
+- Re-architected plotting so users can pass an ax, allowing for complex plots (see cookbook).
+- Helper method to compute the max-return portfolio (thanks to `Philipp Schiele <https://github.com/phschiele>`_)
+  for the suggestion). 
+- Several bug fixes and test improvements (thanks to `Carl Peasnell <https://github.com/SeaPea1>`_).
+
+
 1.3.0
 =====
 

docs/conf.py

@@ -56,9 +56,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = "1.3"
+version = "1.4"
 # The full version, including alpha/beta/rc tags.
-release = "1.3.1"
+release = "1.4.0"
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/index.rst

@@ -15,7 +15,7 @@
                 <img src="https://img.shields.io/badge/python-v3-brightgreen.svg"
                     alt="python"></a> &nbsp;
             <a href="https://pypi.org/project/PyPortfolioOpt/">
-                <img src="https://img.shields.io/badge/pypi-v1.3.1-brightgreen.svg"
+                <img src="https://img.shields.io/badge/pypi-v1.4.0-brightgreen.svg"
                     alt="python"></a> &nbsp;
             <a href="https://opensource.org/licenses/MIT">
                 <img src="https://img.shields.io/badge/license-MIT-brightgreen.svg"
@@ -43,14 +43,16 @@ in a risk-efficient way.
 Installation
 ============
 
-Installation on macOS or linux is as simple as::
+Prior to installing PyPortfolioOpt, you need to install C++. On macOS, this means that you need
+to install XCode Command Line Tools (see `here <https://osxdaily.com/2014/02/12/install-command-line-tools-mac-os-x/>`__).
 
-    pip install PyPortfolioOpt
-
-Windows users need to go through the additional step of downloading C++ (for ``cvxpy``). You can
-download this `here <https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools&rel=16>`__,
+For Windows users, download Visual Studio `here <https://visualstudio.microsoft.com/thank-you-downloading-visual-studio/?sku=BuildTools&rel=16>`__,
 with additional instructions `here <https://drive.google.com/file/d/0B4GsMXCRaSSIOWpYQkstajlYZ0tPVkNQSElmTWh1dXFaYkJr/view>`__.
 
+Installation can then be done via pip::
+
+    pip install PyPortfolioOpt
+
 For the sake of best practice, it is good to do this with a dependency manager. I suggest you
 set yourself up with `poetry <https://github.com/sdispater/poetry>`_, then within a new poetry project
 run:
@@ -73,23 +75,22 @@ Thanks to Thomas Schmelzer, PyPortfolioOpt now supports Docker (requires
 
 .. note::
     If any of these methods don't work, please `raise an issue
-    <https://github.com/robertmartin8/PyPortfolioOpt/issues>`_  on GitHub
-
+    <https://github.com/robertmartin8/PyPortfolioOpt/issues>`_ with the 'packaging' label on GitHub
 
 
 
 For developers
 --------------
 
 If you are planning on using PyPortfolioOpt as a starting template for significant
-modifications, it probably makes sense to clone this repository and to just use the
+modifications, it probably makes sense to clone the repository and to just use the
 source code
 
 .. code-block:: text
 
     git clone https://github.com/robertmartin8/PyPortfolioOpt
 
-Alternatively, if you still want the convenience of ``from pypfopt import x``,
+Alternatively, if you still want the convenience of a global ``from pypfopt import x``,
 you should try
 
 .. code-block:: text
@@ -158,25 +159,13 @@ Contents
     Plotting
 
 .. toctree::
+    :maxdepth: 1
     :caption: Other information
 
     Roadmap
     Contributing
     About
 
-Advantages over existing implementations
-========================================
-
-- Includes both classical methods (Markowitz 1952 and Black-Litterman), suggested best practices
-  (e.g covariance shrinkage), along with many recent developments and novel
-  features, like L2 regularisation, shrunk covariance, hierarchical risk parity.
-- Native support for pandas dataframes: easily input your daily prices data.
-- Extensive practical tests, which use real-life data.
-- Easy to combine with your proprietary strategies and models.
-- Robust to missing data, and price-series of different lengths (e.g FB data
-  only goes back to 2012 whereas AAPL data goes back to 1980).
-
-
 Project principles and design decisions
 =======================================
 
@@ -193,10 +182,24 @@ Project principles and design decisions
   <https://github.com/ambv/black>`_.
 
 
+Advantages over existing implementations
+========================================
+
+- Includes both classical methods (Markowitz 1952 and Black-Litterman), suggested best practices
+  (e.g covariance shrinkage), along with many recent developments and novel
+  features, like L2 regularisation, exponential covariance, hierarchical risk parity.
+- Native support for pandas dataframes: easily input your daily prices data.
+- Extensive practical tests, which use real-life data.
+- Easy to combine with your proprietary strategies and models.
+- Robust to missing data, and price-series of different lengths (e.g FB data
+  only goes back to 2012 whereas AAPL data goes back to 1980).
+
+
 Contributors
 =============
 
-This is a non-exhaustive unordered list of contributors:
+This is a non-exhaustive unordered list of contributors. I am sincerely grateful for all
+of your efforts!
 
 - Philipp Schiele
 - Carl Peasnell
@@ -206,6 +209,7 @@ This is a non-exhaustive unordered list of contributors:
 - Aditya Bhutra
 - Thomas Schmelzer
 - Rich Caputo
+- Nicolas Knudde 
 
 
 Indices and tables