Merge pull request #333 from adobe-apiplatform/v2

prepare for 2.3rc4 release.

This fixes #89.

Author: adobeDan

README.md

@@ -58,57 +58,78 @@ While that looks simple, the `make pex` command will try to download and install
 We pre-package releases on Ubuntu, so the advice here is definitely accurate for that platform, but something similar should work for most other Debian variants.
 
 * Make sure you have Python 2.7.5 or above; earlier versions will give you InsecurePlatformWarning messages due to older SSL components.
+* For Ubuntu 16.04, an easy way to get python 3.6 is from the Jonathan F archive, do:
+    ```bash
+	sudo add-apt-repository ppa:jonathonf/python-3.6
+	sudo apt-get update
+	sudo pat-get install -y python3.6
+	```
 * Take all security updates before you start.
 * You don't need a GUI on the platform, as the entire build can be done from the command line.  Server variants are fine.
 * You will need a standard C development environment to build a variety of the modules that use native extensions.  Use this command to get one:
     ```bash
-    sudo apt-get install build-essential
+    sudo apt-get install -y build-essential
     ```
 * Make sure you use the system package manager to install the following packages (and their dependencies):
-    * python-dev (or python3-dev if you are doing python3 builds)
-    * python-pip (not needed for python3)
-	* python-virtualenv (not needed for python3)
+    * python-dev (or python3.6-dev if you are doing python3 builds)
+    * python-pip (or python3-pip if you are doing python3 builds)
+	* python-virtualenv (or python3-virtualenv if you are doing python3 builds)
 	* pkg-config
 	* libssl-dev
 	* libldap2-dev
 	* libsasl2-dev
-	* libdbus-glib-1-dev
+	* python-dbus-devel
 	* libffi-dev
-* For convenience, you can copy and paste this command:
+* For convenience, you can copy and paste these commands (choosing from the first two based on your python version):
     ```bash
-    sudo apt-get install python-dev python-pip python-virtualenv pkg-config libssl-dev libldap2-dev libsasl2-dev libdbus-glib-1-dev libffi-dev
+    sudo apt-get install -y python-dev python-pip python-virtualenv
+    sudo apt-get install -y python3.6-dev
+    sudo apt-get install -y pkg-config libssl-dev libldap2-dev libsasl2-dev dbus-glib-devel python-dbus libffi-dev
     ```
-* You don't need the python-dbus package to _build_ user-sync, but you will need it to run user-sync if you use the dbus secure store for your credentials.
+* You need the python-dbus package to _build_ user-sync, but you don't need it to *run* user-sync unless you use the dbus secure store for your credentials.
 
 ### CentOS and other RedHat variants
 
 We pre-package releases on CentOS, so the advice here is definitely accurate for that platform, but something similar should work for most other RedHat variants.
 
 * Make sure you have Python 2.7.5 or above; earlier versions will give you InsecurePlatformWarning messages due to older SSL components.
+* You cannot build on CentOS 6, because user-sync uses python-dbus for the keyring, and python-dbus requires dbus 1.6 or greater which is not available on CentOS 6.  However, you can run a CentOS 7 build on CentOS 6 as long as you are running the same build of python 3.6 or later.
+  * For centos6, to run builds, you will need to install python 3.6, which you can do as follows:
+	  ```bash
+      sudo yum install -y https://centos6.iuscommunity.org/ius-release.rpm # installs inline-upstream-stable pkg repo
+	  sudo yum install -y python36u-devel python36u-pip python36u-virtualenv
+	  ```
+  * For centos7, to install python 3.6, do the following:
+      ```bash
+      sudo yum install -y https://centos7.iuscommunity.org/ius-release.rpm # installs inline-upstream-stable pkg repo
+	  sudo yum install -y python36u-devel python36u-pip python36u-virtualenv
+	  ```
 * Take all security updates before you start.
 * You don't need a GUI on the platform, as the entire build can be done from the command line.  Server variants are fine.
 * You will need a standard C development environment to build a variety of the modules that use native extensions.  Use this command to get one:
     ```bash
-    sudo yum group install "Development Tools"
+    sudo yum groupinstall -y "Development Tools"
     ```
 * Your OS may not know about `pip` by default, although it will know about `virtualenv`.  Rather than installing `pip` manually, we recommend telling your OS about the Red Hat "Extra Package for Enterprise Linux (EPEL)" package library, which on CentOS you can do with:
     ```bash
-    sudo yum install epel-release
+    sudo yum install -y epel-release
     ```
 * Make sure you use the system package manager to install the following packages (and their dependencies):
-    * python-devel (or python3-devel, if you are doing python3 builds)
-    * python-pip (not needed for python3)
-	* python-virtualenv (not needed for python3)
+    * python-devel (or python36u-devel, if you are doing python3 builds)
+    * python-pip (or python36u-pip, if you are doing python3 builds)
+	* python-virtualenv (or python36u-virtualenv, if you are doing python3 builds)
 	* pkgconfig
 	* openssl-devel
 	* openldap-devel (includes sasl)
 	* dbus-glib-devel
+	* dbus-python (version 1.6 or greater, not available on CentOS 6)
 	* libffi-devel
-* For convenience, you can copy and paste this command:
+* For convenience, you can copy and paste these commands (pick among the first two based on your python version):
     ```bash
-    sudo yum install python-devel python-pip python-virtualenv pkgconfig openssl-devel openldap-devel dbus-glib-devel libffi-devel
+	sudo yum install -y python-devel python-pip python-virtualenv
+	sudo yum install -y python36u-devel python36u-pip python36u-virtualenv
+    sudo yum install -y pkgconfig openssl-devel openldap-devel dbus-glib-devel dbus-python libffi-devel
     ```
-* You don't need the python-dbus package to _build_ user-sync, but you will need it to run user-sync if you use the dbus secure store for your credentials.
 
 ### Mac OS X
 

RELEASE_NOTES.md

@@ -1,6 +1,6 @@
 # Release Notes for User Sync Tool Version 2.3
 
-These notes apply to v2.3rc3 of 2017-12-10.
+These notes apply to v2.3rc4 of 2018-01-29.
 
 ## New Features
 
@@ -12,16 +12,26 @@ There is a new command-line argument `--connector` for specifying whether to get
 
 [#299](https://github.com/adobe-apiplatform/user-sync.py/issues/299) You can now use an `invocation_defaults` section to specify desired values for command-line arguments in the main configuration file.  This can make it a lot easier to repeat runs with a stable set of arguments, even when running interactively rather than from a script.  The sample main configuration file specifies the configuration parameters to use as well as the syntax for specifying values.  See [the docs](https://adobe-apiplatform.github.io/user-sync.py/en/user-manual/command_parameters.html) for full details.
 
+[#322](https://github.com/adobe-apiplatform/user-sync.py/issues/322), [#319](https://github.com/adobe-apiplatform/user-sync.py/issues/319) As it has been with email, you can now use formatted combinations of ldap/okta attributes for the Adobe-side first name, last name, and country.  (See the sample configuration files for details.)  You can also specify the country code in lower case.
+
 ## Bug Fixes
 
 [#305](https://github.com/adobe-apiplatform/user-sync.py/issues/305) General issues with Okta connector.
 
 [#306](https://github.com/adobe-apiplatform/user-sync.py/issues/306) v2.2.2 crashes if country code not specified.
 
+[#308](https://github.com/adobe-apiplatform/user-sync.py/issues/308) docs are unclear about how to set PEX_ROOT.
+
 [#314](https://github.com/adobe-apiplatform/user-sync.py/issues/314) invocation_defaults section should be optional.
 
 [#315](https://github.com/adobe-apiplatform/user-sync.py/issues/315) Can't specify --user-filter or other string-valued args.
 
+[#318](https://github.com/adobe-apiplatform/user-sync.py/issues/318) Fix the README build instructions regarding dbus.
+
+[#324](https://github.com/adobe-apiplatform/user-sync.py/issues/324) Handle LDAP servers with no support for PagedResults.
+
+[#325](https://github.com/adobe-apiplatform/user-sync.py/issues/325) Adding '--process-groups' doesn't override the default.
+
 ## Compatibility with Prior Versions
 
 All configuration and command-line arguments accepted in prior releases work in this release.  The `--users file` argument is still accepted, and is equivalent to (although more limited than) specifying `--connector csv`.

examples/config files - basic/3 connector-ldap.yml

@@ -133,6 +133,36 @@ user_email_format: "{mail}"
 # produce a unique username for each user.
 #user_username_format: "{department}_{user_id}"
 
+# (optional) user_given_name_format (default value given below)
+# user_given_name_format specifies how to construct a user's given name by
+# combining constant strings with the values of specific directory attributes.
+# Any names in curly braces are taken as attribute names, and everything including
+# the braces will be replaced on a per-user basis with the values of the attributes.
+# The default value used here is simple, and suitable for OpenLDAP systems.
+# NOTE: for this and every format setting, the constant strings must be in
+# the encoding specified by the string_encoding setting, above.
+#user_given_name_format: "{givenName}"
+
+# (optional) user_surname_format (default value given below)
+# user_surname_format specifies how to construct a user's surname by
+# combining constant strings with the values of specific directory attributes.
+# Any names in curly braces are taken as attribute names, and everything including
+# the braces will be replaced on a per-user basis with the values of the attributes.
+# The default value used here is simple, and suitable for OpenLDAP systems.
+# NOTE: for this and every format setting, the constant strings must be in
+# the encoding specified by the string_encoding setting, above.
+#user_surname_format: "{sn}"
+
+# (optional) user_country_code_format (default value given below)
+# user_country_code_format specifies how to construct a user's country code by
+# combining constant strings with the values of specific directory attributes.
+# Any names in curly braces are taken as attribute names, and everything including
+# the braces will be replaced on a per-user basis with the values of the attributes.
+# The default value used here is simple, and suitable for OpenLDAP systems.
+# NOTE: for this and every format setting, the constant strings must be in
+# the encoding specified by the string_encoding setting, above.
+#user_country_code_format: "{c}"
+
 # Some additional info about LDAP connectors:
 #
 # Unlike the CSV connector, the LDAP connector does not have custom specifications

examples/config files - basic/5 connector-okta.yml

@@ -34,3 +34,83 @@ all_users_filter: 'user.status == "ACTIVE"'
 # the valid values are: enterpriseID, federatedID
 # If not specified, the default identity type from the main config file is used.
 # user_identity_type: federatedID
+
+# (optional) string_encoding (default value given below)
+# string_encoding specifies the Unicode string encoding used by the directory.
+# All values retrieved from the directory are converted to Unicode before being
+# sent to or compared with values on the Adobe side, to avoid encoding issues.
+# The value must be a Python codec name or alias, such as 'latin1' or 'big5'.
+# See https://docs.python.org/2/library/codecs.html#standard-encodings for details.
+#string_encoding: utf8
+
+# (optional) user_identity_type_format (no default)
+# user_identity_type_format specifies how to construct a user's desired identity
+# type on the Adobe side by combining constant strings with attribute values.
+# Any names in curly braces are take as attribute names, and everything including
+# the braces will be replaced on a per-user basis with the values of the attributes.
+# There is no default value for this setting, because most directories don't contain
+# users with different identity types (so setting the default identity type suffices).
+# If your directory contains users of different identity types, you should define
+# this field to look at the value of an appropriate attribute in your okta user profile.
+# For example, if your Okta user profile attribute "idType" had one of the values
+# adobe, enterprise, or federated in it for each user, you could use:
+#user_identity_type_format: "{idType}ID"
+
+# (optional) user_email_format (default value given below)
+# user_email_format specifies how to construct a user's email address by
+# combining constant strings with the values of specific Okta profile attributes.
+# Any names in curly braces are taken as attribute names, and everything including
+# the braces will be replaced on a per-user basis with the values of the attributes.
+# The default value is from "email" field in Okta user profile.
+# NOTE: for this and every format setting, the constant strings must be in
+# the encoding specified by the string_encoding setting, above.
+user_email_format: "{email}"
+
+# (optional) user_domain_format (no default value)
+# user_domain_format is analogous to user_email_format in syntax, but it
+# is used to discover the domain for a given user.  If not specified, the
+# domain is taken from the domain part of the user's email address.
+#user_domain_format: "{domain}"
+
+# (optional) user_username_format (no default value)
+# user_username_format specifies how to construct a user's username on the
+# Adobe side by combining constant strings with attribute values.
+# Any names in curly braces are taken as attribute names, and everything including
+# the braces will be replaced on a per-user basis with the values of the attributes.
+# This setting should only be used when you are using federatedID and your
+# federation configuration specifies username-based login.  In all other cases,
+# make sure this is not set or returns an empty value, and the user's username
+# will be taken from the user's email.
+# This example supposes that the department and user_id are concatenated to
+# produce a unique username for each user.
+#user_username_format: "{department}_{user_id}"
+
+# (optional) user_given_name_format (default value given below)
+# user_given_name_format specifies how to construct a user's given name by
+# combining constant strings with the values of specific Okta profile attributes.
+# Any names in curly braces are taken as attribute names, and everything including
+# the braces will be replaced on a per-user basis with the values of the attributes.
+# The default value is from "firstName" field in Okta user profile..
+# NOTE: for this and every format setting, the constant strings must be in
+# the encoding specified by the string_encoding setting, above.
+#user_given_name_format: "{firstName}"
+
+# (optional) user_surname_format (default value given below)
+# user_surname_format specifies how to construct a user's surname by
+# combining constant strings with the values of specific Okta profile attributes.
+# Any names in curly braces are taken as attribute names, and everything including
+# the braces will be replaced on a per-user basis with the values of the attributes.
+# The default value is from "lastName" field in Okta user profile.
+# NOTE: for this and every format setting, the constant strings must be in
+# the encoding specified by the string_encoding setting, above.
+#user_surname_format: "{lastName}"
+
+# (optional) user_country_code_format (default value given below)
+# user_country_code_format specifies how to construct a user's country code by
+# combining constant strings with the values of specific Okta profile attributes.
+# Any names in curly braces are taken as attribute names, and everything including
+# the braces will be replaced on a per-user basis with the values of the attributes.
+# The default value is from "countryCode" field in Okta user profile.
+# NOTE: for this and every format setting, the constant strings must be in
+# the encoding specified by the string_encoding setting, above.
+#user_country_code_format: "{countryCode}"

tests/connector/directory_okta_test.py

@@ -250,17 +250,12 @@ def __init__(self, status_code, data):
                 self.links = {}
 
         self.mock_response = MockResponse
-        self.orig_directory_init = OktaDirectoryConnector.__init__
-        OktaDirectoryConnector.__init__ = mock.Mock(return_value=None)
-        directory = OktaDirectoryConnector({})
+        directory = OktaDirectoryConnector({'host': 'okta-test.com', 'api_token': 'abcdefghijklmnopqrstuvwxyz'})
         directory.logger = mock.create_autospec(logging.Logger)
         directory.groups_client = okta.UserGroupsClient('example.com', 'xyz')
         directory.user_identity_type = 'enterpriseID'
         self.directory = directory
 
-    def tearDown(self):
-        OktaDirectoryConnector.__init__ = self.orig_directory_init
-
     @mock.patch('user_sync.connector.directory_okta.OktaDirectoryConnector.find_group')
     @mock.patch('okta.framework.ApiClient.requests')
     def test_success_extended_attribute_key(self, mock_requests, mock_find_group):
@@ -281,7 +276,6 @@ def test_success_extended_attribute_key(self, mock_requests, mock_find_group):
         mock_find_group.return_value = mockID
 
         directory = self.directory
-        directory.options = {'all_users_filter': 'user.status == "ACTIVE"', 'group_filter_format': '{group}'}
         extended_attributes = ['firstName', 'lastName', 'login', 'email', 'countryCode', 'additionalTest']
         iterGroupResponse = directory.iter_group_members("testGroup",directory.options['all_users_filter'], extended_attributes)
         temp_var = list(iterGroupResponse)
@@ -308,7 +302,6 @@ def test_success_extended_attribute_value(self, mock_requests, mock_find_group):
         mock_find_group.return_value = mockID
 
         directory = self.directory
-        directory.options = {'all_users_filter': 'user.status == "ACTIVE"', 'group_filter_format': '{group}'}
         extended_attributes = ['firstName', 'lastName', 'login', 'email', 'countryCode', 'additionalTest']
         iterGroupResponse = directory.iter_group_members("testGroup", directory.options['all_users_filter'],
                                                          extended_attributes)
@@ -336,7 +329,6 @@ def test_non_existence_extended_attribute_key(self, mock_requests, mock_find_gro
         mock_find_group.return_value = mockID
 
         directory = self.directory
-        directory.options = {'all_users_filter': 'user.status == "ACTIVE"', 'group_filter_format': '{group}'}
         extended_attributes = ['firstName', 'lastName', 'login', 'email', 'countryCode', 'badattribute']
         iterGroupResponse = directory.iter_group_members("testGroup", directory.options['all_users_filter'],
                                                          extended_attributes)
@@ -364,7 +356,6 @@ def test_non_existence_extended_attribute_value(self, mock_requests, mock_find_g
         mock_find_group.return_value = mockID
 
         directory = self.directory
-        directory.options = {'all_users_filter': 'user.status == "ACTIVE"', 'group_filter_format': '{group}'}
         extended_attributes = ['firstName', 'lastName', 'login', 'email', 'countryCode', 'badattribute']
         iterGroupResponse = directory.iter_group_members("testGroup", directory.options['all_users_filter'],
                                                          extended_attributes)
@@ -391,7 +382,6 @@ def test_invalid_missing_profile_user(self, mock_requests, mock_find_group):
         mock_find_group.return_value = mockID
 
         directory = self.directory
-        directory.options = {'all_users_filter': 'user.status == "ACTIVE"', 'group_filter_format': '{group}'}
         extended_attributes = ['firstName', 'lastName', 'login', 'email', 'countryCode', 'badattribute']
         iterGroupResponse = directory.iter_group_members("testGroup", directory.options['all_users_filter'],
                                                          extended_attributes)