Audit?

[fishcharlie]

I don't want to stifle discussion, but after some recent scalability problems, I don't see this happening anytime soon. I'm locking this discussion for the time being. Long term, this is something I think we should revisit. However, I don't believe the project is in a position to look at this currently.

---

Wanted to start a discussion about an idea I had regarding @WordsOfMe's comments on #350.

Quality is critical for a project like this. Part of that is always coming up with ideas and methods to work to increase quality of the lists. So, I have no idea if this is even a remotely good idea. But brainstorming new ideas to increase quality I think benefits the project.

So. Do we think doing a full audit of every domain on every list would be beneficial to the project? Basically verifying that every one is legitimate and should exist on the list.

I'm not quite sure how this would work. Maybe it can be assisted by automation? Possibly a system can create audit issues based on if domains have recently be transferred owners or changed IP addresses that the domain points to? Then as those issues get marked as closed the system continues to expand and create more issues to end up encapsulating every domain? Maybe there can even be tools to give additional information in those issues that would be beneficial to determine if the domain should exist on the blocklist or not (screenshots of domain content maybe?)?

This would be a massive undertaking. And I'm not sure we are prepared for a project like this currently. I think it would also require some more concrete guidelines about what domains should be included vs not.

Just thought it'd be worth discussing to see if this is something we should work towards.

[WordsOfMe]

Unfortunately I cannot contribute with code, just some thoughts:

• Manpower / brains is most valueable, so if a domain had to be removed once manually it should be avoided to generate work once more (hence the whitelist proposal)
• Domains that were taken over some time ago / historically would be a lot of work to review, and until no one complains it would probably save time to just keep them for now
• What I would see critical is the mechanism / workflow new domains are added to the list, which I must confess do not know and as I understood is meant to kept confidential. But the crucial point here will be to review how a false positive made it to the list and what can be learned from this. This question has been asked several times: "Is there a way to determine why it was added in the first place?"

Some thoughts on automatic checking before adding:

• Check if the domain is actually registered
• Think about if domains that only resolve a MX should be added, I guess a domain should at least resolve to an A / AAAA / CNAME.
• One possibility to check the benevolence of a domain would be querying a number of DNSBL used for blocking spam. When I lookup the last false positives I had at https://multirbl.valli.org, a1.net is on 2 whitelists, and gmail-smtp-in.l.google.com is on 3 whitelists
• Git is probably not ideal for managing add / remove requests. Most other blocklists (e.g. spamhaus) feature a web interface that will make it easier for the maintainers to modify the lists in an automatic way.

[iam-py-test]

I think this is a good idea.
We could (automatically or not) check these lists against other reliable blocklists, and focus on the ones not in other lists.
For malware domains, we could check against @DandelionSprout’s antimalware list (there are also domains and HOSTs versions) or the URLHaus blocklist (and related).
I know there are tons of other lists (and tools) which could help (including some I created), but many have old entries or FPs.

For the porn list, we could automate testing using McAfee (if their rating system & use of a unofficial api are ok) and review manually the ones not detected. We also can test those in a VM, but that will take a long time & is subject to website downtimes, ip blocking, and other problems.

We also could use VirusTotal, either via the web interface or the API.

[fishcharlie]

@iam-py-test I think we have to be careful about using those other systems. If you want a DNS blocker based on those APIs and lists, go use them. We have to bring something unique here.

I'm also not a lawyer, but I've voiced concerns previously about license conflicts and such with using those APIs.

Who knows, maybe we'll get to the point where we can build systems to use ML to detect those things automatically.

---

I did end up locking this discussion. And the scalability issues I mentioned in the updated post, especially apply here.