Enrichment

[krmaxwell]

For each IP address, get the ASN and hostnames (if enrichment is enabled).

@alexcpsec: How do we want to handle multiple names for an IP address?

[alexcpsec]

I have a script here:

• If you have an IPv4 entry, just get the ASnumber, ASname, country, and try to get the reverse DNS (PTR record of the domain) from DNSDB. If multiple, get the one with the most hits. If none, set as empty string. Do not gather their hostnames it will be too many, and will probably not serve much purpose
• Now, if the entity is a FQDN extract the IP addresses from it using an A query on DNSDB for the reference date and "duplicate" that FQDN entry, transforming into multiple IPv4 entries with the original FQDN as the "host" field and now you can do the ASN, country and rhost enrichments on it

[krmaxwell]

Okay, first stab at pseudocode

for each in IPv4_addresses:
    (ASN, AS_name, country) = cymru(each)
    hostname = dnsdb(each, "PTR")
    each.enrichment = (ASN, AS_name, country, hostname)

for each in FQDN:
    for everyone in dnsdb(each, "A"):
       harvest += everyone
       enrich(everyone)

[krmaxwell]

~s/cymru/maxmind/

[alexcpsec]

More like this:

for each in IPv4_addresses:
    (ASN, AS_name, country) = maxmind(each)
    hostname = max_hits(dnsdb(each, "PTR"))
    each.enrichment = (ASN, AS_name, country, hostname)

for each in FQDN:
    for everyone in dnsdb(each, "A"):
       harvest += as.IPv4(everyone, host=FQDN.value)
       enrich(everyone)

[alexcpsec]

For MaxMindASN, I use this database:

.mmasn.url = "http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum2.zip"

For MaxMindGeo, I use these (not sure which one encodes the country):

.mmgeoBlocksLocations.url = "http://geolite.maxmind.com/download/geoip/database/GeoLiteCity_CSV/GeoLiteCity-latest.zip"
.mmgeoRegions.url = "http://dev.maxmind.com/static/csv/codes/maxmind/region.csv"

[krmaxwell]

I'm having a little trouble understanding harvest += as.IPv4(everyone, host=FQDN.value). Can you elaborate a bit pl0x?

[alexcpsec]

I am trying to encode this if the entity is a FQDN extract the IP addresses from it using an A query on DNSDB for the reference date and "duplicate" that FQDN entry, transforming into multiple IPv4 entries with the original FQDN as the "host" field.

everyone has the IP addresses, and the "original" FQDN entry, right? Make them into IPv4 entries with the original FQDN entity (the domain you resolved in DNSDB) in the "host" field

[krmaxwell]

Ah hah, okay, you're referring to the enrichment schema. We have a first pass at it - is that the order you want things in for CSV?

[alexcpsec]

Yes. This is what I am currently exporting out for reference:

aperture-2:R alexcp$ cd -
/Users/alexcp/src/tiq-test/data/enriched/public_outbound
aperture-2:public_outbound alexcp$ head 20140403.csv
"entity","type","direction","source","notes","date","asnumber","asname","country","host","rhost"
"1.234.23.28","IPv4","outbound","alienvault","MLSec-Export","2014-04-03","9318","Hanaro Telecom Inc.","KR",,
"1.234.35.198","IPv4","outbound","alienvault","MLSec-Export","2014-04-03","9318","Hanaro Telecom Inc.","KR",,
"1.25.36.76","IPv4","outbound","alienvault","MLSec-Export","2014-04-03","4837","CNCGROUP China169 Backbone","CN",,
"1.93.1.162","IPv4","outbound","alienvault","MLSec-Export","2014-04-03","4808","CNCGROUP IP network China169 Beijing Province Network","CN",,
"1.93.44.147","IPv4","outbound","alienvault","MLSec-Export","2014-04-03","4808","CNCGROUP IP network China169 Beijing Province Network","CN",,
"100.42.218.250","IPv4","outbound","alienvault","MLSec-Export","2014-04-03","18450","WebNX, Inc.","US",,"100-42-218-250.static.webnx.com"
"100.42.55.2","IPv4","outbound","alienvault","MLSec-Export","2014-04-03","36351","SoftLayer Technologies Inc.","US",,"stats.wren.arvixe.com"
"100.42.55.220","IPv4","outbound","alienvault","MLSec-Export","2014-04-03","36351","SoftLayer Technologies Inc.","US",,"stats.warthog.arvixe.com"
"100.42.58.137","IPv4","outbound","alienvault","MLSec-Export","2014-04-03","36351","SoftLayer Technologies Inc.","US",,"100.42.58.137-static.reverse.mysitehosted.com"

[krmaxwell]

Gotcha. Okay, this also means we have a dependency on #24 which I'm hereby documenting.

[krmaxwell]

Derp, https://docs.python.org/dev/library/ipaddress.html#networks-as-containers-of-addresses will also help (plus the properties is_private and is_reserved).