[meta] Add support for email in ECS

[webmat]

We'd like to add support for email in ECS. Email is a big topic, so this issue isn't meant to be exhaustive 😂

This could cover:

• Email activity streams from email server logs or events published by email SaaS solutions
  • delivery events, engagement events
• DNS records around email like SPF, DMARC

Past discussions

#411, #534, #593, #815

Beats: elastic/beats#13466

Related: #749 (mime types)

[mbudge]

I've been putting a lot of email data from logs/phishing automation into elastic recently.

Here's an summary of the structure we use in elastic

email.subject: keyword
email.from.address: string array - some log sources like Mimecast generate multiple from addresses. These include the original from address and the postmaster address. This happens when the email solution has rules configured to forward emails to other internal accounts. Logs with the same unique ID (like Mimecast aCode) are generated for the original email and the forwarded email triggered by a rule. Some users might use painless scripting to merge those logs into 1 document so they are easier to use.
email.mailfrom.address: The mailfrom address in the email header.
email.to.address: string array
email.date: timestamp for email logs and keyword for phishing. If some users extract the Date: string from the envelope Date field in the email body, this might not be a valid timestamp format accepted by Elasticsearch. Examples - 22 July 2019 11:13, Thursday, 17 October 2019 7:15 PM, 2020-08-03T15:26:21, 10/8/19 9:16 AM (GMT-04:00) or a malformed string as this field can be set to anything.
email.direction: keyword: Inbound, Outbound, Internal
email.action:
destination.ip
source.ip

The email.to, email.from and email.mailfrom also have the following fields

• domain
• registered_domain
• subdomain
• top_level_domain

email.message_id: Message ID from the email header
email.process.action: Accept, Reject, On Hold
email.process.file.extension: string array of file extensions
email.process.file.name: string array of file names
email.process.timestamp: Timestamp
email.receipt.action: Accept, Reject, On Hold
email.receipt.timestamp: timestamp
email.receipt.tls.cipher: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
email.receipt.tls.version: TLSv1.2
mimecast.acode: Mimecast ID but the field name could be more generic.

The receipt/process fields in our case make merging mimecast logs clearer to the end user.
https://www.mimecast.com/tech-connect/documentation/tutorials/understanding-siem-logs/

[webmat]

Thanks for dropping all of these details here! This will be very useful ❤️

[nicpenning]

@mbudge How are you handling the header fields?

We too ingest emails but more specifically reported emails into Elastic.

We use:
reportphish.attachment_names
reportphish.email.body_plain_text
reportphish.email.body_raw
reportphish.email.from
reportphish.email.header.Authentication-Results
reportphish.email.header.Content-Type
reportphish.email.header.Date
reportphish.email.header.From
reportphish.email.header.MIME-Version
reportphish.email.header.Message-ID
reportphish.email.header.Received
reportphish.email.header.Received-SPF
reportphish.email.header.Reply-To
reportphish.email.header.Return-Path
reportphish.email.header.Subject
reportphish.email.header.To
reportphish.email.header.X-CrossPremisesHeadersFiltered
reportphish.email.header.X-CrossPremisesHeadersPromoted
reportphish.email.header.X-EOPAttributedMessage
reportphish.email.header.X-Forefront-Antispam-Report
reportphish.email.header.X-MS-Exchange-CrossTenant-AuthAs
reportphish.email.header.X-MS-Exchange-CrossTenant-AuthSource
reportphish.email.header.X-MS-Exchange-CrossTenant-FromEntityHeader
reportphish.email.header.X-MS-Exchange-CrossTenant-Id
reportphish.email.header.X-MS-Exchange-CrossTenant-Network-Message-Id
reportphish.email.header.X-MS-Exchange-CrossTenant-OriginalArrivalTime
reportphish.email.header.X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp
reportphish.email.header.X-MS-Exchange-Organization-AuthAs
reportphish.email.header.X-MS-Exchange-Organization-AuthSource
reportphish.email.header.X-MS-Exchange-Organization-ExpirationInterval
reportphish.email.header.X-MS-Exchange-Organization-ExpirationIntervalReason
reportphish.email.header.X-MS-Exchange-Organization-ExpirationStartTime
reportphish.email.header.X-MS-Exchange-Organization-ExpirationStartTimeReason
reportphish.email.header.X-MS-Exchange-Organization-MessageDirectionality
reportphish.email.header.X-MS-Exchange-Organization-Network-Message-Id
reportphish.email.header.X-MS-Exchange-Organization-SCL
reportphish.email.header.X-MS-Exchange-Processed-By-BccFoldering
reportphish.email.header.X-MS-Exchange-Transport-CrossTenantHeadersStamped
reportphish.email.header.X-MS-Exchange-Transport-EndToEndLatency
reportphish.email.header.X-MS-Office365-Filtering-Correlation-Id
reportphish.email.header.X-MS-Oob-TLC-OOBClassifiers
reportphish.email.header.X-MS-PublicTrafficType
reportphish.email.header.X-MS-TrafficTypeDiagnostic
reportphish.email.header.X-Microsoft-Antispam
reportphish.email.header.X-Microsoft-Antispam-Mailbox-Delivery
reportphish.email.header.X-Microsoft-Antispam-Message-Info
reportphish.email.header.X-OrganizationHeadersPreserved
reportphish.email.header.X-OriginalArrivalTime
reportphish.email.header.X-OriginatorOrg
reportphish.email.header.X-SEG-SpamProfiler-Analysis
reportphish.email.header.X-SEG-SpamProfiler-Score
reportphish.email.received
reportphish.email.reported
reportphish.email.sender
reportphish.email.subject
reportphish.email.to

The header is dynamically ingested into a field via a PowerShell script so header data may vary. The sample above is just the filed names from 1 email.

[AydinChavez]

I'd rather recommend to put the email fields as subfields to the already existing root level fields like source and/or destination like it is already proposed in the documentation for the user fields.
Also do not rely on the email field names (like from or sender, recipient or to, ..) but using abstract names like address:

Example:

source.email.address
destination.email.address

[mbudge]

A combination of the following would work well.

source.email.address
destination.email.address
email.from.address
email.to.address
email.cc.address
email.bcc.address
email.reply_to.address

These fields would be more generic so email addresses are easier to search.
source.email.address
destination.email.address

These fields would keep a record of how each address was extracted from the email.
email.from.address
email.to.address
email.mailfrom.address
email.cc.address
email.bcc.address
email.reply_to.address

[BenB196]

Just a thought, but would it be better to not do:

source.email.address
destination.email.address

And instead use:

source.user.email
destination.user.email

[jamiehynds]

I've submitted an RFC for an email fieldset in ECS. Would love to get some thoughts on the fields and any additional fields you'd like to see included.

#999

[the-pixel-hunter]

Is there any chance for this to be added would be very beneficial for a lot use cases

[djptek]

Hi @the-pixel-hunter there is an RFC open for an email fieldset - you can view that here: https://github.com/elastic/ecs/blob/master/rfcs/text/0010-email.md

[jamiehynds]

Thanks @djptek! @the-pixel-hunter as the email RFC advances, we're seeking as much feedback as possible to ensure the proposed fields meet the use cases our users have in mind. If there's some e-mail fields you feel that we're missing in the RFC, feel free to chime in :)