implements an IDNA-encoded version of publicsuffix2.

includes additional functionality for strict checks, ignoring
wildcards, and finding eTLD only. maps main function of
get_public_suffix() to get_sld() for clarity.

Author: KnitCode

MANIFEST.in

@@ -2,5 +2,6 @@ graft src
 
 include CHANGELOG.rst
 include README.rst
+include publicsuffix2.LICENSE
 
 global-exclude *.py[co] __pycache__ *.so *.pyd

README.rst

@@ -1,21 +1,37 @@
 Public Suffix List module for Python
 ====================================
 
-This module allows you to get the public suffix of a domain name using the
-Public Suffix List from http://publicsuffix.org
+This module allows you to get the public suffix, as well as the registrable domain,
+ of a domain name using the Public Suffix List from http://publicsuffix.org
 
 A public suffix is a domain suffix under which you can register domain
-names. Some examples of public suffixes are ".com", ".co.uk" and "pvt.k12.wy.us".
+names. It is sometimes referred to as the extended TLD (eTLD).
+Some examples of public suffixes are ".com", ".co.uk" and "pvt.k12.wy.us".
 Accurately knowing the public suffix of a domain is useful when handling
 web browser cookies, highlighting the most important part of a domain name
 in a user interface or sorting URLs by web site.
 
+This module builds the public suffix list as a Trie structure, making it more efficient
+than other string-based modules available for the same purpose. It can be used
+effectively in large-scale distributed environments, such as PySpark.
+
 This Python module includes with a copy of the Public Suffix List so that it is
 usable out of the box. Newer versions try to provide reasonably fresh copies of
 this list. It also includes a convenience method to fetch the latest list.
 
-The code is a fork of the publicsuffix package and uses the same base API.
-You just need to import publicsuffix2 instead
+The code is a fork of the publicsuffix2 package and includes the same base API. In
+addition, it contains a few variants useful for certain use cases, such as the option to
+ignore wildcards or return only the extended TLD (eTLD).
+Publicsuffix2 is a an extension of publicsuffix, and uses the same base API.
+You just need to import publicsuffix2 instead.
+
+The public suffix list is now provided in UTF-8 format. To correctly process
+IDNA-encoded domains, either the query or the list must be converted. This module
+contains the option to IDNA-encode the public suffix list upon creating the Trie; this
+is set to happen by default. If your use case includes UTF-8 domains, e.g., '食狮.com.cn',
+you'll need to set the IDNA-encoding flag to False on instantiation (see examples below).
+Failure to use the correct encoding for your use case can lead to incorrect results for
+domains that utilize unicode characters.
 
 The code is MIT-licensed and the publicsuffix data list is MPL-2.0-licensed.
 
@@ -31,6 +47,10 @@ The code is MIT-licensed and the publicsuffix data list is MPL-2.0-licensed.
 Usage
 -----
 
+To install from source, first build the package file:
+    python setup.py build sdist
+and then pip install from the dist directory.
+
 Install with::
 
     pip install publicsuffix2
@@ -103,6 +123,63 @@ You can use it this way::
 Note that the once loaded, the data file is cached and therefore fetched only
 once.
 
+If using this library in large-scale pyspark processing, you should instantiate the class as
+a global variable, not within a user function. The class methods can then be used within user
+functions for distributed processing.
+
+Changes in this Fork
+--------------------
+
+This fork of publicsuffix2 addresses a change in the format to the standard public suffix list,
+which was previously IDNA-encoded and now is in UTF-8 format, as well as some additional
+functionality useful to certain use cases. These additions include the ability to ignore
+wildcards and to require strict adherence to the TLDs included in the list. Lastly, we include
+some convenience functions for obtaining only the extended TLD (eTLD) rather than the
+registrable domain (SLD). These are outlined below.
+
+IDNA-encoding. The public suffix list is now provided in UTF-8 format. For those use cases that
+include IDNA-encoded domains, the module will not return accurate results unless the list is
+converted. In this fork, IDNA encoding is included as a parameter in the class and is on by
+default.::
+
+    >>> from publicsuffix2 import PublicSuffixList
+    >>> psl = PublicSuffixList(idna=True)  # on by default
+    >>> psl.get_public_suffix('www.google.com')
+    'google.com'
+    >>> psl = PublicSuffixList(idna=False)  # use UTF-8 encodings
+    >>> psl.get_public_suffix('食狮.com.cn')
+    '食狮.com.cn'
+
+Ignore wildcards. In some use cases, particularly those related to large-scale domain processing,
+the user might want to ignore wildcards to create more aggregation. This is possible by setting
+the parameter wildcard=False.
+
+Require valid eTLDs (strict). In the publicsuffix2 module, a domain with an invalid TLD will still return
+a public suffix, e.g,::
+
+    >>> psl.get_public_suffix('www.mine.local')
+    'mine.local'
+
+
+This is useful for many use cases, while in others, we want to ensure that the domain includes a
+valid eTLD. In this case, the boolean parameter strict provides a solution. If this flag is set,
+an invalid TLD will return None.::
+
+    >>> psl.get_public_suffix('www.mine.local', strict=True) is None
+    True
+
+Return eTLD only. The standard use case for publicsuffix2 is to return the registrable domain
+according to the public suffix list. In some cases, however, we only wish to find the eTLD
+itself. In this fork, this is available via the get_tld() method.::
+
+    >>> psl.get_tld('www.google.com')
+    'com'
+
+All of the methods and functions include the wildcard and strict parameters.
+
+For convenience, the public method get_sld() is available. This is identical to the method
+get_public_suffix() and is intended to clarify the output for some users.
+
 
 Source
 ------
@@ -116,7 +193,11 @@ branch::
 
 History
 -------
-This code is forked from Tomaž Šolc's fork of David Wilson's code originally at:
+This code is forked from NexB's fork of Tomaž Šolc's fork of David Wilson's code.
+
+The original publicsuffix2 code is Copyright (c) 2015 nexB Inc.
+
+David Wilson's code originally at:
 
 https://www.tablix.org/~avian/git/publicsuffix.git
 
@@ -138,6 +219,7 @@ License
 The code is MIT-licensed. 
 The vendored public suffix list data from Mozilla is under the MPL-2.0.
 
+Copyright (c) 2019 Renée Burton
 
 Copyright (c) 2015 nexB Inc.
 

publicsuffix2.LICENSE

@@ -1,3 +1,7 @@
+Copyright (c) 2019 Renée Burton
+This code is based on nexB Inc. code found at:
+https://www.github.com/nexB/python-publicsuffix2
+
 Copyright (c) 2015 nexB Inc.
 This code is based on Tomaž Šolc fork of David Wilson code originally at
 https://www.tablix.org/~avian/git/publicsuffix.git

setup.py

@@ -81,14 +81,14 @@ def run(self):
 
 setup(
     name='publicsuffix2',
-    version='2.20190205',
+    version='2.20190328',
     license='MIT and MPL-2.0',
     description='Get a public suffix for a domain name using the Public Suffix '
         'List. Forked from and using the same API as the publicsuffix package.',
     long_description='%s\n%s' % (read('README.rst'), read('CHANGELOG.rst')),
-    author='nexB Inc., Tomaz Solc and David Wilson',
-    author_email='info@nexb.com',
-    url='https://github.com/nexB/python-publicsuffix2',
+    author='Renée Burton, nexB Inc., Tomaz Solc and David Wilson',
+    author_email='',
+    url='https://github.com/KnitCode/python-publicsuffix2',
     packages=find_packages('src'),
     package_dir={'': 'src'},
     py_modules=[splitext(basename(path))[0] for path in glob('src/*.py')],