Merge remote-tracking branch 'origin/series/3.4'

Signed-off-by: Roberto Rosario <roberto.rosario@mayan-edms.com>

HISTORY_3_x.rst

@@ -285,10 +285,25 @@
 - Default ``DEFAULT_CELERY_BROKER_URL`` to ``'memory://'``.
   This ensures operation even when there is no broker available.
 
-3.4.20 (2020-XX-XX)
+3.4.20 (2020-11-11)
 ===================
 - Fix REST API chapter formatting.
 - Add search documentation chapter.
+- Remove extra space from link label.
+- Add keyword arguments to .acquire_lock().
+- Add keyword arguments to shutil library usage
+- Merge c18d145c4ea1d5cfb23dc8cd517bc8ddd4149782 "Generate only one CSRF
+  token per HTML form" from Version 4.0.
+- Merge fb3f0d3c35bf7c0880a8a4b4b650f7767ee089a7 "Merge URL and form data
+  in a smarter way" from Version 4.0.
+
+  Use URI class to merge the URL and the query string for the
+  form fields in a smart way instead of just concatenating using
+  a '?'.
+
+  Closes GitLab issue #706. Thanks to Matthias Urlichs (@smurfix)
+  for the report.
+- Add keyword arguments to sh.Command().
 
 3.4.19 (2020-10-26)
 ===================

docs/releases/3.4.19.txt

@@ -200,6 +200,7 @@ Backward incompatible changes
 Issues closed
 -------------
 
+- :gitlab-issue:`877` Lost Swagger and ReDoc interface when upgrade docker image version (from 3.4.12 to 3.4.17)
 - :gitlab-issue:`885` Error accessing document indexing API
 
 .. _PyPI: https://pypi.python.org/pypi/mayan-edms/

docs/releases/3.4.20.txt

@@ -0,0 +1,227 @@
+Version 3.4.20
+==============
+
+Released: November 11, 2020
+
+Changes
+-------
+
+Code style
+^^^^^^^^^^
+
+As part of the continuous effort to keep the code maintainable and to comply
+with MERC 0005 (https://docs.mayan-edms.com/mercs/0005-explicit-arguments.html)
+keyword arguments were added to the ``.acquire_lock()`` function, uses
+of the library ``shutil``, as well as uses of ``Command`` class from the
+``sh`` library.
+
+
+Documentation
+^^^^^^^^^^^^^
+
+A chapter devoted to the search system was added. Minor formatting error
+in the REST API chapter were fixed.
+
+
+User interface
+^^^^^^^^^^^^^^
+
+Two commits from the version 4.0 development branch were merged back.
+The first, makes sure that only one CSRF token is generated per form. There
+are no functional changes and this only concern the sources app form.
+
+The second commit fixes an edge case where URLs with a query string were
+being incorrectly modified when a form was submitted in such a way it caused
+a reload, as is the case when a required field was left empty.
+
+
+Upgrading process
+-----------------
+
+
+Upgrading from Mayan EDMS 3.2.x or earlier
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+#. Stop supervisord::
+
+    sudo systemctl stop supervisor
+
+#. Update the Redis configuration:
+
+   Configure Redis to discard data when it runs out of memory, not save its
+   database, and only keep 2 database:
+
+   .. code-block:: bash
+
+       echo "maxmemory-policy allkeys-lru" | sudo tee -a /etc/redis/redis.conf
+       echo "save \"\"" | sudo tee -a /etc/redis/redis.conf
+       echo "databases 2" | sudo tee -a /etc/redis/redis.conf
+       echo "requirepass |DEFAULT_REDIS_PASSWORD|" | sudo tee -a /etc/redis/redis.conf
+       sudo systemctl restart redis
+
+
+#. Install the Python 3 development OS package:
+
+   .. code-block:: bash
+
+    sudo apt-get install python3-dev
+
+
+#. Update the virtualenv to use Python 3:
+
+   .. code-block:: bash
+
+    sudo -u |DEFAULT_OS_USERNAME| virtualenv --clear /opt/mayan-edms -p /usr/bin/python3
+
+
+#. Create a home directory for the Mayan EDMS system user:
+
+   .. code-block:: bash
+
+    mkdir /home/mayan
+
+
+#. Grant ownership to the Mayan EDMS system user:
+
+   .. code-block:: bash
+
+    chown mayan:mayan /home/mayan
+
+#. Reinstall the Python client for PostgreSQL and Redis:
+
+   .. code-block:: bash
+
+       sudo -u |DEFAULT_OS_USERNAME| |MAYAN_PIP_BIN| install psycopg2==|PYTHON_PSYCOPG2_VERSION| redis==|PYTHON_REDIS_VERSION|
+
+   .. note::
+
+       Platforms with the ARM CPU might also need additional requirements:
+
+       .. code-block:: bash
+
+           sudo -u |DEFAULT_OS_USERNAME| |MAYAN_PIP_BIN| install psutil==|PYTHON_PSUTIL_VERSION|
+
+
+#. Reinstall the Python client for RabbitMQ if you are using RabbitMQ as a broker:
+
+   .. code-block:: bash
+
+       sudo -u |DEFAULT_OS_USERNAME| |MAYAN_PIP_BIN| install amqp==|PYTHON_AMQP_VERSION|
+
+
+Upgrading from Mayan EDMS 3.4.x or 3.3.x
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+#. Stop supervisord::
+
+    sudo systemctl stop supervisor
+
+
+#. Upgrade to the latest pip version:
+
+   .. code-block:: bash
+
+       sudo -u |DEFAULT_OS_USERNAME| |MAYAN_PIP_BIN| install -U pip
+
+
+#. Update the Redis configuration to enable password protection:
+
+   .. code-block:: bash
+
+       echo "requirepass mayanredispassword" | sudo tee -a /etc/redis/redis.conf
+       sudo systemctl restart redis
+
+
+#. Remove deprecated requirements:
+
+   .. code-block:: bash
+
+    sudo -u |DEFAULT_OS_USERNAME| curl |SOURCE_CODE_REPOSITORY|raw/master/removals.txt -o /tmp/removals.txt \
+    && sudo -u |DEFAULT_OS_USERNAME| |MAYAN_PIP_BIN| uninstall -y -r /tmp/removals.txt
+
+
+#. Update the Mayan EDMS Python package:
+
+   .. code-block:: bash
+
+    sudo -u |DEFAULT_OS_USERNAME| |MAYAN_PIP_BIN| install mayan-edms==3.4.20
+
+   the requirements will also be updated automatically.
+
+
+#. Make a backup of your supervisord file:
+
+   .. code-block:: bash
+
+       sudo cp |MAYAN_SUPERVISOR_CONF| |MAYAN_SUPERVISOR_CONF|.bck
+
+
+#. Update the supervisord configuration file. Replace the environment
+   variables values show here with your respective settings. This step will refresh
+   the supervisord configuration file with the new queues and the latest
+   recommended layout:
+
+   .. code-block:: bash
+
+       sudo -u |DEFAULT_OS_USERNAME| MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=|DEFAULT_DATABASE_NAME| \
+       MAYAN_DATABASE_PASSWORD=|DEFAULT_DATABASE_PASSWORD| MAYAN_DATABASE_USER=|DEFAULT_DATABASE_USER| \
+       MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \
+       |MAYAN_BIN| platformtemplate supervisord | sudo sh -c "cat > |MAYAN_SUPERVISOR_CONF|"
+
+   or:
+
+   .. code-block:: bash
+
+       sudo -u |DEFAULT_OS_USERNAME| MAYAN_DATABASES=\"{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'|DEFAULT_DATABASE_NAME|','PASSWORD':'|DEFAULT_DATABASE_PASSWORD|','USER':'|DEFAULT_DATABASE_USER|','HOST':'127.0.0.1'}}\" \
+       MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \
+       |MAYAN_BIN| platformtemplate supervisord | sudo sh -c "cat > |MAYAN_SUPERVISOR_CONF|"
+
+
+#. Edit the supervisord configuration file and update any setting specific to your installation:
+
+   .. code-block:: bash
+
+       sudo vi |MAYAN_SUPERVISOR_CONF|
+
+
+#. Migrate existing database schema and static media files with:
+
+   .. code-block:: bash
+
+       sudo -u |DEFAULT_OS_USERNAME| MAYAN_DATABASE_ENGINE=django.db.backends.postgresql MAYAN_DATABASE_NAME=|DEFAULT_DATABASE_NAME| \
+       MAYAN_DATABASE_PASSWORD=|DEFAULT_DATABASE_PASSWORD| MAYAN_DATABASE_USER=|DEFAULT_DATABASE_USER| \
+       MAYAN_DATABASE_HOST=127.0.0.1 MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \
+       |MAYAN_BIN| performupgrade
+
+   or:
+
+   .. code-block:: bash
+
+       sudo -u |DEFAULT_OS_USERNAME| MAYAN_DATABASES="{'default':{'ENGINE':'django.db.backends.postgresql','NAME':'|DEFAULT_DATABASE_NAME|','PASSWORD':'|DEFAULT_DATABASE_PASSWORD|','USER':'|DEFAULT_DATABASE_USER|','HOST':'127.0.0.1'}}" \
+       MAYAN_MEDIA_ROOT=|MAYAN_MEDIA_ROOT| \
+      |MAYAN_BIN| performupgrade
+
+
+#. Start supervisord:
+
+   .. code-block:: bash
+
+       sudo systemctl start supervisor
+
+#. Clear the browser cache to avoid loading old web assets.
+
+The upgrade procedure is now complete.
+
+
+Backward incompatible changes
+-----------------------------
+
+- None
+
+
+Issues closed
+-------------
+
+- :gitlab-issue:`706` Error due to wrong URL in POST request
+
+.. _PyPI: https://pypi.python.org/pypi/mayan-edms/

docs/releases/index.txt

@@ -32,6 +32,7 @@ versions of the documentation contain the release notes for any later releases.
    3.5.2
    3.5.1
    3.5
+   3.4.20
    3.4.19
    3.4.18
    3.4.17

mayan/apps/appearance/static/appearance/js/partial_navigation.js

@@ -246,17 +246,32 @@ class PartialNavigation {
                 var uri = new URI(location);
                 var uriFragment = uri.fragment();
                 var url = $form.attr('action') || uriFragment;
+                var finalUrl = new URI(url);
+                var formQueryString = new URLSearchParams(decodeURIComponent($form.serialize()));
 
                 options.url = url;
-                lastAjaxFormData.url = url + '?' + decodeURIComponent($form.serialize());
+
+                // Merge the URL and the form values in a smart way instead
+                // of just blindly adding a '?' between them.
+                formQueryString.forEach(function(value, key) {
+                    finalUrl.addQuery(key, value);
+                });
+
+                lastAjaxFormData.url = finalUrl.toString();
 
                 if ($form.attr('target') == '_blank') {
                     // If the form has a target attribute we emulate it by
                     // opening a new window and passing the form serialized
                     // data as the query.
-                    window.open(
-                        $form.attr('action') + '?' + decodeURIComponent($form.serialize())
-                    );
+                    var finalUrl = new URI($form.attr('action'));
+                    var formQueryString = new URLSearchParams(decodeURIComponent($form.serialize()));
+
+                    // Merge the URL and the form values in a smart way instead
+                    // of just blindly adding a '?' between them.
+                    formQueryString.forEach(function(value, key) {
+                        finalUrl.addQuery(key, value);
+                    });
+                    window.open(finalUrl.toString());
 
                     return false;
                 }

mayan/apps/appearance/templates/appearance/generic_multiform_subtemplate.html

@@ -7,11 +7,11 @@
             <form action="{{ form_action }}" class="{{ form_css_classes|default:'' }}" id="{{ form_id|default:'' }}" method="{{ submit_method|default:'post' }}">
         {% endif %}
 
-        {% for form_name, form in forms.items %}
-            {% if submit_method != 'GET' and submit_method != 'get' %}
-                {% csrf_token %}
-            {% endif %}
+        {% if submit_method != 'GET' and submit_method != 'get' %}
+            {% csrf_token %}
+        {% endif %}
 
+        {% for form_name, form in forms.items %}
             {% if next %}
                 <input name="next" type="hidden" value="{{ next }}" />
             {% endif %}

mayan/apps/converter/backends/python.py

@@ -27,7 +27,7 @@
 )
 
 try:
-    pdftoppm = sh.Command(pdftoppm_path)
+    pdftoppm = sh.Command(path=pdftoppm_path)
 except sh.CommandNotFound:
     pdftoppm = None
 else:
@@ -50,7 +50,7 @@
 )
 
 try:
-    pdfinfo = sh.Command(pdfinfo_path)
+    pdfinfo = sh.Command(path=pdfinfo_path)
 except sh.CommandNotFound:
     pdfinfo = None
 

mayan/apps/converter/classes.py

@@ -59,7 +59,7 @@ def __init__(self, file_object, mime_type=None):
         self.soffice_file = None
         Image.init()
         try:
-            self.command_libreoffice = sh.Command(libreoffice_path).bake(
+            self.command_libreoffice = sh.Command(path=libreoffice_path).bake(
                 '--headless', '--convert-to', 'pdf:writer_pdf_Export'
             )
         except sh.CommandNotFound:

mayan/apps/dependencies/classes.py

@@ -488,11 +488,12 @@ def extract(self, replace_list=None):
         # We do a copy and delete instead of move because os.rename doesn't
         # support renames across filesystems.
         path_uncompressed_package = Path(temporary_directory, 'package')
-        shutil.rmtree(force_text(s=path_install))
+        shutil.rmtree(path=force_text(s=path_install))
         shutil.copytree(
-            force_text(s=path_uncompressed_package), force_text(s=path_install)
+            src=force_text(s=path_uncompressed_package),
+            dst=force_text(s=path_install)
         )
-        shutil.rmtree(force_text(s=path_uncompressed_package))
+        shutil.rmtree(path=force_text(s=path_uncompressed_package))
 
         # Clean up temporary directory used for download
         shutil.rmtree(path=temporary_directory, ignore_errors=True)
@@ -745,9 +746,9 @@ def extract(self, replace_list=None):
         shutil.rmtree(path=force_text(s=path_install), ignore_errors=True)
 
         shutil.copytree(
-            force_text(s=self.path_cache), force_text(s=path_install)
+            src=force_text(s=self.path_cache), dst=force_text(s=path_install)
         )
-        shutil.rmtree(force_text(s=self.path_cache), ignore_errors=True)
+        shutil.rmtree(path=force_text(s=self.path_cache), ignore_errors=True)
 
     def get_install_path(self):
         app = apps.get_app_config(app_label=self.app_label)

mayan/apps/dependencies/tests/test_classes.py

@@ -29,7 +29,7 @@ def setUp(self):
 
     def tearDown(self):
         super().tearDown()
-        shutil.rmtree(self.temporary_directory, ignore_errors=True)
+        shutil.rmtree(path=self.temporary_directory, ignore_errors=True)
 
     def _patch_test_file(self):
         replace_list = [

mayan/apps/django_gpg/backends/python_gnupg.py

@@ -84,7 +84,7 @@ def gpg_command(self, function, **kwargs):
 
         result = function(gpg=gpg, **kwargs)
 
-        shutil.rmtree(temporary_directory)
+        shutil.rmtree(path=temporary_directory)
 
         return result