Error in INVALID output

[spirillen]

Bug description

Found a few misses in the INVALID/list after testing https://raw.githubusercontent.com/Clefspeare13/pornhosts/master/domains%20to%20check.txt

cat output/domains/INVALID/list
# File generated by PyFunceble (v2.45.0.dev) / https://github.com/funilrys/PyFunceble
# Date of generation: 2019-11-18T01:14:07.846084

a
added
and
checked
collectionofbestporn
command,
curl
did
double
file
hosts
list
needed.
not
of
properly
respond
the
which
will
www.adultcashtraffic.com
www.blog.gfrevenge.com
www.hqporner.comstudiowow-girls
www.largehdtube.comen
www.media.the-adult-company.com
www.pornblade.comcategoryanal-porn
www.porndoepremium.comcategories
www.sexdating
www.spankbang
yallainternethotnig

The errors I see is:

www.adultcashtraffic.com
www.blog.gfrevenge.com
www.media.the-adult-company.com

gfrevenge.com
   Domain Name: GFREVENGE.COM
   Registry Domain ID: 1312074137_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.eurodns.com
   Registrar URL: http://www.EuroDNS.com
   Updated Date: 2019-10-27T04:31:37Z
   Creation Date: 2007-11-02T00:31:15Z
   Registry Expiry Date: 2020-11-02T00:31:15Z
   Registrar: EuroDNS S.A.
   Registrar IANA ID: 1052
   Registrar Abuse Contact Email: legal@eurodns.com
   Registrar Abuse Contact Phone: +352.27220150
   Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
   Name Server: DNS1.P03.NSONE.NET
   Name Server: DNS2.P03.NSONE.NET
   Name Server: DNS3.P03.NSONE.NET
   Name Server: DNS4.P03.NSONE.NET
   Name Server: SDNS3.ULTRADNS.BIZ
   Name Server: SDNS3.ULTRADNS.COM
   Name Server: SDNS3.ULTRADNS.NET
   Name Server: SDNS3.ULTRADNS.ORG
   DNSSEC: unsigned

whois adultcashtraffic.com
Expired 

whois the-adult-company.com

   Domain Name: THE-ADULT-COMPANY.COM
   Registry Domain ID: 1370101655_DOMAIN_COM-VRSN
   Registrar WHOIS Server: whois.safenames.net
   Registrar URL: http://www.safenames.net
   Updated Date: 2019-08-04T05:05:18Z
   Creation Date: 2008-01-03T13:29:55Z
   Registry Expiry Date: 2021-01-03T13:29:55Z
   Registrar: SafeNames Ltd.
   Registrar IANA ID: 447
   Registrar Abuse Contact Email: abuse@safenames.net
   Registrar Abuse Contact Phone: +44.1908200022
   Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited
   Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
   Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited
   Name Server: NS1.XMODELS-LIVE.CH
   Name Server: NS2.XMODELS-LIVE.CH
   Name Server: NS3.XMODELS-LIVE.CH
   DNSSEC: unsigned

This means two out of 3 should have been added to the ACTIVE/list

Reproduction

pyfunceble --plain -h -m -p 4 -db --database-type mariadb -f 'https://raw.githubusercontent.com/Clefspeare13/pornhosts/master/domains%20to%20check.txt'

Expected behavior**

Only invalid entries, or invalid tld's are added to invalid/list

Versions

OS: Disco
Python Version: 3.7.3
PyFunceble Version: pyfunceble -v pyfunceble 2.45.0.dev (Green Galago: Skitterbug)

Additional context

Have you seen issue at gitlab?

[spirillen]

printf "www.adultcashtraffic.com\nwww.media.the-adult-company.com\n" >> output/domains/ACTIVE/list 
printf "www.adultcashtraffic.com\n" >> output/domains/INACTIVE/list

[funilrys]

Look at example.org 😉 I refuse to do the work of the Registry. That's why I don't necessarily ensure that the date in the past/future if it is not archived in the database.

Indeed, we both know that example.org is a special case but who knows how many "special cases" are there in the wild?

Therefor I'm not going to improve that. It will stay like that since I consider that if a Registry still provides some expiration date, it implicitly means that the subject of the record may be active.

About www.blog.gfrevenge.com, I don't bother with the WHOIS lookup as it is a subdomain. It's like that since D1 I'm not sure I'm ready to put some time into that at it will make things slower than they actually are... But I'm open to implementing that as a feature/option later.

(Could you imagine how many sub-domains are tested per hours by PyFunceble in @dead-hosts @Ultimate-Hosts-Blacklist @Import-External-Sources ?)

What is strange is that I don't get the same results 🤔

+ cat issues/61.list
a
added
and
checked
collectionofbestporn
command,
curl
did
double
file
hosts
list
needed.
not
of
properly
respond
the
which
will
www.adultcashtraffic.com
www.blog.gfrevenge.com
www.hqporner.comstudiowow-girls
www.largehdtube.comen
www.media.the-adult-company.com
www.pornblade.comcategoryanal-porn
www.porndoepremium.comcategories
www.sexdating
www.spankbang
yallainternethotnig
+ PyFunceble --version
PyFunceble 2.46.0.dev (Green Galago: Skitterbug)
+ PyFunceble --plain -m -db --database-type=mariadb -f issues/61.list

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝

Your version is more recent!
You should really think about sharing your changes with the community!
Your version: 2.46.0.dev (Green Galago: Skitterbug)
Upstream version: 2.45.0.dev (Green Galago: Skitterbug)

🌟 🎉 🍰 We now support IPv6! 🍰 🎉 🌟


Subject                                                                                              Status      HTTP Code 
---------------------------------------------------------------------------------------------------- ----------- ----------
added                                                                                                INVALID     XXX       
command,                                                                                             INVALID     XXX       
did                                                                                                  INVALID     XXX       
curl                                                                                                 INVALID     XXX       
checked                                                                                              INVALID     XXX       
a                                                                                                    INVALID     XXX       
and                                                                                                  INVALID     XXX       
needed.                                                                                              INVALID     XXX       
file                                                                                                 INVALID     XXX       
collectionofbestporn                                                                                 INVALID     XXX       
double                                                                                               INVALID     XXX       
list                                                                                                 INVALID     XXX       
hosts                                                                                                INVALID     XXX       
not                                                                                                  INVALID     XXX       
the                                                                                                  INVALID     XXX       
of                                                                                                   INVALID     XXX       
properly                                                                                             INVALID     XXX       
which                                                                                                INVALID     XXX       
will                                                                                                 INVALID     XXX       
www.adultcashtraffic.com                                                                             INACTIVE    XXX       
www.largehdtube.comen                                                                                INVALID     XXX       
www.hqporner.comstudiowow-girls                                                                      INVALID     XXX       
respond                                                                                              INVALID     XXX       
www.sexdating                                                                                        INVALID     XXX       
www.porndoepremium.comcategories                                                                     INVALID     XXX       
www.pornblade.comcategoryanal-porn                                                                   INVALID     XXX       
www.media.the-adult-company.com                                                                      INACTIVE    XXX       
www.blog.gfrevenge.com                                                                               INACTIVE    XXX       
www.spankbang                                                                                        INVALID     XXX       
yallainternethotnig                                                                                  INVALID     XXX       


Status      Percentage   Numbers     
----------- ------------ ------------
ACTIVE      0%           0           
INACTIVE    10%          3           
INVALID     90%          27          

██████╗ ██╗   ██╗███████╗██╗   ██╗███╗   ██╗ ██████╗███████╗██████╗ ██╗     ███████╗
██╔══██╗╚██╗ ██╔╝██╔════╝██║   ██║████╗  ██║██╔════╝██╔════╝██╔══██╗██║     ██╔════╝
██████╔╝ ╚████╔╝ █████╗  ██║   ██║██╔██╗ ██║██║     █████╗  ██████╔╝██║     █████╗
██╔═══╝   ╚██╔╝  ██╔══╝  ██║   ██║██║╚██╗██║██║     ██╔══╝  ██╔══██╗██║     ██╔══╝
██║        ██║   ██║     ╚██████╔╝██║ ╚████║╚██████╗███████╗██████╔╝███████╗███████╗
╚═╝        ╚═╝   ╚═╝      ╚═════╝ ╚═╝  ╚═══╝ ╚═════╝╚══════╝╚═════╝ ╚══════╝╚══════╝


Thanks for using PyFunceble!
+ cat output/domains/INVALID/list
# File generated by PyFunceble (v2.46.0.dev) / https://github.com/funilrys/PyFunceble
# Date of generation: 2019-11-20T00:44:19.410784

a
added
and
checked
collectionofbestporn
command,
curl
did
double
file
hosts
list
needed.
not
of
properly
respond
the
which
will
www.hqporner.comstudiowow-girls
www.largehdtube.comen
www.pornblade.comcategoryanal-porn
www.porndoepremium.comcategories
www.sexdating
www.spankbang
yallainternethotnig
+ cat output/domains/INACTIVE/list
# File generated by PyFunceble (v2.46.0.dev) / https://github.com/funilrys/PyFunceble
# Date of generation: 2019-11-20T00:44:19.406948

www.adultcashtraffic.com
www.blog.gfrevenge.com
www.media.the-adult-company.com
+ cat output/domains/ACTIVE/list
cat: output/domains/ACTIVE/list: Datei oder Verzeichnis nicht gefunden

P.S: For those who came late to the party: example.org have - at the time I write this - an expiration date set to 2010-08-30T04:00:00Z

[dnmTX]

@funilrys this is not a bug. That's why we testing each domain on whom www. is added as many are none existing and therefore the complaint here is irrelevant:

Just ignore that issue....or better yet...CLOSE IT 😈

[spirillen]

@funilrys I see your point... That's the good part of open thought, one remember what another didn't think of 🤔

Building a list of exceptions, that would be quit a work.....

I put put my head in work to see if something usefully strikes me...

(Could you imagine how many sub-domains are tested per hours by PyFunceble in)

hrm hrm tooooo 🙊 🙄 🤣

Retest

retest results

# File generated by PyFunceble (v2.45.0.dev) / https://github.com/funilrys/PyFunceble
# Date of generation: 2019-11-20T01:08:02.246762

a
added
and
checked
collectionofbestporn
command,
curl
did
double
file
hosts
list
needed.
not
of
properly
respond
the
which
will
www.hqporner.comstudiowow-girls
www.largehdtube.comen
www.pornblade.comcategoryanal-porn
www.porndoepremium.comcategories
www.sexdating
www.spankbang
www.spankbanglistaaa

Same as your on 3rd run....