New Feature: Dashboard: Add DNS top permitted domains - pihole Object

[dbouwer]

New Feature: Can you please add "DNS Top Permitted Domains" object?

Currently, the DNS top domains object includes all domains and there is no way of only showing the permitted domains.

[nin9s]

actually you can easily apply a filter or KQL filter to achieve this. You can also safe that filter for later use.
You can do this in the dashboard layer for all visualizations (see screenshot) or you can do this by editing the permitted domain viz and apply the filter there.

Its a little bit exaggerated in my oppionen to create a whole new visualization for just this change.
However you can even do this easily. Just click the edit button of the permitted domain viz, add your filter and safe it under a new name. This way you are able to easily add the new one to the dashboard

Hope that helps :)

[nin9s]

while I thought a little more about this it would maybe be better to just replace the viz to just show the permitted domain. I guess it would be more logical to have a viz showing the blocked domains and another showing only permitted ones.
For now you can edit the viz yourself in the above mentioned way

Thanks for pointing me in this direction

[dbouwer]

while I thought a little more about this it would maybe be better to just replace the viz to just show the permitted domain. I guess it would be more logical to have a viz showing the blocked domains and another showing only permitted ones.
For now you can edit the viz yourself in the above mentioned way

Thanks for pointing me in this direction

Is what I was thinking as well....same concept on the pihole's native dashboard

I'll play around with the viz for now....

[dbouwer]

"not blocked_domain :*" filter does not work. All it does is remove the piholed results, but the top Domains results still show all the results. On the Dashboard and Viz views...

[nin9s]

Not here. If I apply this filter every event containing a blocked_domain field is excluded on every viz and the top blocked domains vis is empty.

Do I misunderstand you ?

BTW the Screenshot above also shows 0 pinholed Domains in the diagram as we excluded the blocked field -> you only see domains which were accepted and not blocked

[dbouwer]

At first I also thought it worked great, until I started verifying the data.

With the filter on, the piholed data is removed, but the top domain still included piholed URL's

I then built a new dashboard with the 2 viz's . Screenshots attached... as you can see the piholed result in top domains is still present after the filter.

[nin9s]

You are right thanks. I'm on mobile right now but I guess I know why

Top domains is counting the requests and at this time there is no actual information about the result (blocked or not)
Will change that logic asap

[dbouwer]

No rush.....appreciate the effort. :-)

[nin9s]

this gets more complicated than I thought.

what do you think about: message: *is* and not blocked_domain :* and (ip_response:* or domain_response:*)

"is" should grab all reponse events (regardless of type) but I'm not sure if I miss something

//e
I think we need a new field "domain_response_or_ip_response" and copy the values from "domain_response" and "ip_response" into this new field to correctly count both in one viz

[dbouwer]

Busy testing your suggestions....getting a lot of inconsistencies. I'll feedback once I know more...

[nin9s]

@dbouwer I've made some progress. I've implemented a new field "ip_or_domain_response" which we can use in the permitted domain viz. This way, we can easily show the results which are NOT piholed.
Previously it was indeed the case that I evaluated (incorrectly) the requests where we didnt know if it is going to be blocked or not - now we are evaluating the responses

However it would be great if you some spare time to test it because right now I'm not 100% sure the counting works 100% correctly.

Advice would be to create a new viz with the new field next to the old one to have a better comparization.

updated: grok logic: #30
please update the output section to your liking

by the way: there maybe some strange differences between the domains being displayed betwenn the old and the new viz but I suspect this sources from the fact pihole is NOT blocking CNAME until now: https://discourse.pi-hole.net/t/apply-pi-hole-blocking-to-cnames/25445/89
example:

19:00:28 dnsmasq[1067]: 4716531 192.168.254.106/62623 query[A] d179kwmlpc4o47.cloudfront.net from 192.168.254.106
19:00:28 dnsmasq[1067]: 4716531 192.168.254.106/62623 /etc/pihole/gravity.list d179kwmlpc4o47.cloudfront.net is 0.0.0.0

19:23:25 dnsmasq[1067]: 4774867 192.168.254.100/46770 query[A] cdn.samsungcloudsolution.com from 192.168.254.100
19:23:25 dnsmasq[1067]: 4774867 192.168.254.100/46770 forwarded cdn.samsungcloudsolution.com to 127.0.0.100
19:23:25 dnsmasq[1067]: 4774867 192.168.254.100/46770 reply cdn.samsungcloudsolution.com is <CNAME>
19:23:25 dnsmasq[1067]: 4774867 192.168.254.100/46770 reply d179kwmlpc4o47.cloudfront.net is 99.86.245.166
19:23:25 dnsmasq[1067]: 4774867 192.168.254.100/46770 reply d179kwmlpc4o47.cloudfront.net is 99.86.245.117
19:23:25 dnsmasq[1067]: 4774867 192.168.254.100/46770 reply d179kwmlpc4o47.cloudfront.net is 99.86.245.11
19:23:25 dnsmasq[1067]: 4774867 192.168.254.100/46770 reply d179kwmlpc4o47.cloudfront.net is 99.86.245.115

got the idea?

[nin9s]

there is another issue we face since we are evaluating the responses because if the response is different from the actual domain query if the answer is a CNAME

see here:

domain answer not equal domain reponse because of cname (imap-mail.outlook.com vs VIE-efz.ms-acdc.office.com)

dig **imap-mail.outlook.com**

; <<>> DiG 9.10.3-P4-Debian <<>> imap-mail.outlook.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36541
;; flags: qr rd ra; QUERY: 1, ANSWER: 7, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;imap-mail.outlook.com.         IN      A

;; ANSWER SECTION:
imap-mail.outlook.com.  886     IN      CNAME   outlook.office365.com.
outlook.office365.com.  658     IN      CNAME   outlook.ms-acdc.office.com.
outlook.ms-acdc.office.com. 658 IN      CNAME   VIE-efz.ms-acdc.office.com.
**VIE-efz.ms-acdc.office.com**. 658 IN      A       52.97.181.66
**VIE-efz.ms-acdc.office.com**. 658 IN      A       40.101.55.114
**VIE-efz.ms-acdc.office.com**. 658 IN      A       52.97.141.82
**VIE-efz.ms-acdc.office.com**. 658 IN      A       40.101.55.210

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Nov 23 20:07:14 CET 2019
;; MSG SIZE  rcvd: 229

answer equal domain (heise.de vs heise.de)

dig **heise.de**

; <<>> DiG 9.10.3-P4-Debian <<>> heise.de
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39314
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;heise.de.                      IN      A

;; ANSWER SECTION:
**heise.de.**               86301   IN      A       193.99.144.80

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat Nov 23 20:07:22 CET 2019
;; MSG SIZE  rcvd: 53

any ideas?

[dbouwer]

Hi nin9s

I tested the new field and the results are not there just yet.
I included screenshots from the pihole's web dashboard and 3 tables from elk-hole.

Top Blocked and piholed is exactly the same.
Top Permitted and ip_or_domain_response filtered is way off. (I created a viz with the ip_or_domain_response filter applied)
elk-hole is counting way too many hits compared to pihole.

[nin9s]

could it be related to: https://discourse.pi-hole.net/t/total-queries-in-dashboard/9786
?

[dbouwer]

""We’re counting the queries from inside the resolver (FTLDNS), so it might be different from a program looking at the log.""

Ok so we shouldn't really compare them directly.

Only issue I see then is that the ip_or_domain_response filter is removing valid permitted domains if you just look at the following 2 URL's from the previous screenshot...

They don't even show up in the ip_or_domain_response results
connectivitycheck.gstatic.com
mtalk.google.com

[nin9s]

for mtalk.google.com this would fall under the CNAME issue mentioned above. ip_or_domain_response is populated by the answer, not the request! So if we query for mtalk.google.com the response would be a cname mobile-gtalk.l.google.com. But to be fair, mobile-gtalk.l.google.com isnt showing up either :(

; <<>> DiG 9.10.3-P4-Debian <<>> mtalk.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39218
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;mtalk.google.com.              IN      A

;; ANSWER SECTION:
mtalk.google.com.       20043   IN      CNAME   mobile-gtalk.l.google.com.
------------------------------->   mobile-gtalk.l.google.com. 472  IN      A       74.125.140.188

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Nov 24 11:44:05 CET 2019
;; MSG SIZE  rcvd: 100

but the count of the permitted domains in general or way to low in arent they? There is still something we might miss here ...

[dbouwer]

I'm still investigating to see if I can find an easier pattern in the pihole.log file...

[dbouwer]

and what happens with replies like this?? 1 query, 10 replies

query[A] za-odc.samsungapps.com from 172.21.0.99
forwarded za-odc.samsungapps.com to 208.67.220.220
reply za-odc.samsungapps.com is
reply za-odc.gw.samsungapps.com is
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 54.171.254.244
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 54.229.195.1
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 176.34.139.133
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 52.30.24.226
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 52.49.42.36
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 54.76.32.100
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 54.76.97.87
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 54.76.214.56

[nin9s]

and what happens with replies like this?? 1 query, 10 replies

what happens is the same as with the other CNAME records

dig  za-odc.samsungapps.com

; <<>> DiG 9.10.3-P4-Debian <<>> za-odc.samsungapps.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50873
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;za-odc.samsungapps.com.                IN      A

;; ANSWER SECTION:
za-odc.samsungapps.com. 899     IN      CNAME   za-odc.gw.samsungapps.com.
za-odc.gw.samsungapps.com. 900  IN      CNAME   fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com.
fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com. 900 IN A 54.76.97.87
fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com. 900 IN A 54.76.214.56
fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com. 900 IN A 52.30.24.226
fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com. 900 IN A 52.49.42.36
fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com. 900 IN A 54.171.9.135
fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com. 900 IN A 54.171.254.244
fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com. 900 IN A 54.229.195.1
fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com. 900 IN A 176.34.139.133

;; AUTHORITY SECTION:
eu-west-1.elb.amazonaws.com. 591 IN     NS      ns-1053.awsdns-03.org.
eu-west-1.elb.amazonaws.com. 591 IN     NS      ns-2023.awsdns-60.co.uk.
eu-west-1.elb.amazonaws.com. 591 IN     NS      ns-341.awsdns-42.com.
eu-west-1.elb.amazonaws.com. 591 IN     NS      ns-739.awsdns-28.net.

;; Query time: 1494 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Nov 24 16:14:19 CET 2019
;; MSG SIZE  rcvd: 417

the old behavior was to count the actual request e.g. za-odc.samsungapps.com but the new behavior is to count the response to decide if it a blocked domain or not.
Unfortunately the CNAME responses all count as a single (response) event

reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 54.171.254.244
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 54.229.195.1
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 176.34.139.133
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 52.30.24.226
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 52.49.42.36
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 54.76.32.100
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 54.76.97.87
reply fe-pew1-ext-samsungapps4-lb-1819241093.eu-west-1.elb.amazonaws.com is 54.76.214.56

and they look the same as if it wasnt a CNAME reply but a [A] response. and thats also the reason why the count is actually so different
See what I mean?

Currently I'm thinking of a way to query the pihole api concerning its blocked domain status und only populate "domain_request" if its not a blocked domain.

http://pi.hole/admin/api.php?getAllQueries&domain=google.at&auth=<token>

getAllQueries
Fifth column: Answer type (1 = blocked by gravity.list , 2 = forwarded to upstream server, 3 = answered by local cache, 4 = blocked by wildcard blocking)