http_events table and JA3 support

[ppatel-uptycs]

Description: HTTP events and JA3 support (only for Linux platform)

• Support for new events table http_events along with JA3 fingerprinting.
• The table consists of various HTTP headers information, source IP and ports, remote IP and ports, and if more headers are available, it populates the other_headers columns.
• The table also extracts some of the TLS Client Hello frame information(SSLVersion, Cipher, SSLExtension, EllipticCurve, EllipticCurvePointFormat) and constructs JA3 fingerprinting.

About JA3:

• JA3 is a method of TLS fingerprinting, JA3 is a much more effective way to detect malicious activity over SSL than IP or domain-based IOCs. Since JA3 detects the client application, it doesn’t matter if malware uses DGA (Domain Generation Algorithms), or different IPs for each C2 host, or even if the malware uses Twitter for C2, JA3 can detect the malware itself based on how it communicates rather than what it communicates to.

• JA3 gathers the decimal values of the bytes for the following fields in the Client Hello packet-
  SSL Version, Accepted Ciphers, List of Extensions, Elliptic Curves, and Elliptic Curve Formats.
  It then concatenates those values together in order, using a "," to delimit each field and a "-" to delimit each value in each field.

• The field order is as follows:
  SSLVersion,Cipher,SSLExtension,EllipticCurve,EllipticCurvePointFormat
  Example:
  769,47-53-5-10-49161-49162-49171-49172-50-56-19-4,0-10-11,23-24-25,0
  If there are no SSL Extensions in the Client Hello, the fields are left empty.

Example:
769,4-5-10-9-100-98-3-6-19-18-99,,,
These strings are then MD5 hashed to produce an easily consumable and shareable 32 character fingerprint. This is the JA3 SSL Client Fingerprint.
769,47-53-5-10-49161-49162-49171-49172-50-56-19-4,0-10-11,23-24-25,0 --> ada70206e40642a3e4461f35503241d5
769,4-5-10-9-100-98-3-6-19-18-99,,, --> de350869b8c85de67a350c8d186f11e6

More information about JA3 fingerprinting can be found here - https://github.com/salesforce/ja3

Flags introduced:

• To enable or disable the http_events capturing, there is a flag called --enable_http_lookups, which is a boolean flag and by default, it is set to false, set true to enable http events capturing.
• By default other_headers column won't display any value of available headers keys, to populate it in the column, there is a flag --include_http_headers which accepts a comma-separated list of headers.
• The flag --disable_http_event_filters is introduced to control the kind of http events that must be filtered out by osquery. This flag is used to eliminate low-value http events and decrease the volume of http_events generated. One of the options that can be provided is private_ip_events, this option will eliminate all http_events to and from private IP addresses (as specified in RFC 1918).
  For example, the following setting will eliminate http_events from/to addresses specified in RFC 1918:--disable_http_event_filters=private_ip_events
  Currently, the only option supported is "private_ip_events". In the future, other categories of http_events can be added and the option provided as a comma-separated list
• Private IPs list can be found here - https://en.wikipedia.org/wiki/Reserved_IP_addresses

Table Schema:
table_name : http_events
description: Tracks HTTP request headers, and provide JA3 fingerprint.
schema([
Column("time", BIGINT, "Time of HTTP event"),
Column("method", TEXT, "HTTP request method GET/POST/PUT/DELETE/OPTIONS"),
Column("protocol", TEXT, "SSL protocol used in communication(only for HTTPS/TLS)"),
Column("local", TEXT, "IP address of the local interface"),
Column("remote", TEXT, "IP address of the HTTP responder"),
Column("s_port", BIGINT, "Source port"),
Column("d_port", BIGINT, "Destination port"),
Column("host", TEXT, "Host name available in HTTP header"),
Column("port", INTEGER, "Port number from HTTP URL"),
Column("uri", TEXT, "URI added to HTTP host name(only for HTTP)"),
Column("content_type", TEXT, "Content type of HTTP request(only for HTTP)"),
Column("user_agent", TEXT, "Client making HTTP request(only for HTTP)"),
Column("ja3", TEXT, "Parameters responsible for JA3 calculation(only for HTTPS/TLS)"),
Column("ja3_fingerprint", TEXT, "MD5 hash of ja3 string(only for HTTPS/TLS)"),
Column("other_headers", TEXT, "Others header available in the HTTP packets(only for HTTP)")
])

NOTE: Since HTTPS headers are encrypted, http_events can not capture HTTPS header information, so it is expected that the http_events table will not have all the columns populated for HTTPS or HTTP traffic at the same time.

Using HTTP parser files from - https://github.com/nekipelov/httpparser