[ECS] Upgrade modules to 1.8

[andrewstucki]

This is to track changes needed to upgrade modules to ECS 1.8:

• After 1.8 is released, update ecs dependency to 1.8

Carry-over from 1.7 upgrade:

• network.direction Filebeat cisco umbrella (need CIDR matching processors/painless support in elasticsearch) - can leverage Network direction processor elasticsearch#66644
• network.direction in Filebeat rsa2elk modules - Add network.direction classification to rsa2elk modules #23114

Add os.type field:

• add_host_metadata processor (@adriansr) Add os.type field from ECS 1.8 #23513
• other places that fill in os information (@adriansr) Add os.type field from ECS 1.8 #23513

New event.category value registry:

• Winlogbeat (@marc-gr) [ECS] Winlogbeat ecs 1.8 changes #23563

New event.category value session:

• Winlogbeat (@marc-gr) [ECS] Winlogbeat ecs 1.8 changes #23563
• Auditbeat (@adriansr)
  • beats Update Auditbeat auditd module to ECS 1.8 #23594
  • integrations (auditd ingest pipeline)
• Filebeat auditd fileset (@adriansr)
  • beats Update filebeat auditd module to ECS 1.8 #23723
  • integrations Sync auditd package with beats integrations#715

Multiple users in an event elastic/ecs#914:

• Auditbeat (sudo, iam events, AUDIT_CH*_ID events, file ownership syscalls, set/get*id syscalls, all auid values) (@adriansr) Update Auditbeat auditd module to ECS 1.8 #23594
• Journalbeat (maybe?, sudo & iam events) (@marc-gr) [Journalbeat][ecs] Journalbeat ecs 1.8 #23737
• Packetbeat (@marc-gr) [ECS] Packetbeat ecs 1.8 #23783
• Winlogbeat (Run As, iam events) (@marc-gr)
  • beats [ECS] Winlogbeat ecs 1.8 changes #23563
  • integrations [Windows] Sync with winlogbeat module changes integrations#685
• Filebeat auditd (same as auditbeat) (@adriansr)
  • beats Update filebeat auditd module to ECS 1.8 #23723
  • integrations Sync auditd package with beats integrations#715
• Filebeat rsa2elk modules (@adriansr) (no changes)
• Filebeat checkpoint firewall (@marc-gr) (no changes)
• Filebeat cisco asa (@marc-gr)
  • beats [ECS] Upgrade cisco modules to ecs 1.8 #23819
  • integrations [cisco] Sync ftd and asa data streams with beats integrations#686
• Filebeat cef (@marc-gr)
  • beats [ecs] Upgrade cef to 1.8.0 #23832
  • integrations [cef] Sync cef data streams with beats integrations#687
• Filebeat cisco ftd (@marc-gr)
  • beats [ECS] Upgrade cisco modules to ecs 1.8 #23819
  • integrations [cisco] Sync ftd and asa data streams with beats integrations#686
• Filebeat cisco umbrella (@marc-gr)
  • beats [ECS] Upgrade cisco modules to ecs 1.8 #23819
    - [ ] integrations
• Filebeat crowdstrike falcon (@marc-gr)
  • beats Crowdstrike: Move logic to ingest pipeline and upgrade ECS to 1.8.0 #23875
  • integrations [crowdstrike] Sync crowdstrike data streams with beats integrations#688
• Filebeat fortinet firewall (@marc-gr)
  • beats [ecs] Upgrade fortinet/firewall to ECS 1.8 #23902
  • integrations [fortinet] Sync fortinet firewall data stream with beats integrations#689
• Filebeat googlecloud audit (iam events) (@adriansr) (no changes, discuss)
  • beats
  • integrations
• Filebeat microsoft (@adriansr)
  • beats Update Microsoft module to ECS 1.8 #23897
  • integrations Add Microsoft Defender ATP integrations#468
• Filebeat elasticsearch/audit (maybe?) (@marc-gr)
  • beats [ecs] Upgrade elasticsearch/audit to ECS 1.8 #24000
    - [ ] integrations
• Filebeat Gsuite/Workspace (@marc-gr)
  • beats [ECS][Filebeat] Gsuite/Google Workspace ECS 1.8 #23709
  • integrations [google_workspace] Sync google_workspace data streams with beats integrations#690
• Filebeat o365 (Actors, iam events) (@adriansr)
  • beats Update o365 module to ECS 1.8 #23896
  • integrations Sync o365 package with beats integrations#716
• Filebeat zoom (@adriansr)
  • beats Update zoom module to ECS 1.8 #23904
  • integrations Sync zoom package with beats integrations#719
• Filebeat okta (maybe? actors, iam events, targets) (@marc-gr)
  • beats [ecs] Upgrade okta to ecs 1.8.0 and move js processor to ingest pipeline #23929
  • integrations [okta] Sync okta package with beats integrations#691
• Filebeat aws cloudtrail (assumed role, iam events) (@adriansr)
  • beats Filebeat: Update aws/cloudtrail dataset to ECS 1.8 #23911
  • integrations Sync aws cloudtrail and s3access with beats integrations#721
• Filebeat aws s3access (assumed role) (@adriansr)
  • beats Update Filebeat aws/s3access dataset to ECS 1.8 #23920
  • integrations Sync aws cloudtrail and s3access with beats integrations#721
• Filebeat azure (@adriansr) (not a lot of test data, discuss)
  • beats Update Filebeat azure module to ECS 1.8 #23927
  • integrations Sync azure package with beats integrations#722
• Filebeat juniper/srx (@marc-gr)
  • beats [ecs] Upgrade juniper/srx to ecs 1.8.0 #23936
  • integrations [juniper] Sync juniper src data stream with beats integrations#692
• Filebeat panw (@marc-gr)
  • beats [ecs] Upgrade panw module to ecs 1.8 #23931
  • integrations [panw] Sync panw package with beats integrations#694
• Filebeat sophos/xg (@adriansr)
  • beats Update sophos/xg to ECS 1.8 #23967
  • integrations Add sophos xg integration integrations#479
• Filebeat system/auth (sudo) (@marc-gr)
  • beats [ecs] Upgrade system/auth to ecs 1.8 #23961
  • integrations [system] Sync system auth data stream with beats integrations#695
• Filebeat mysql/mysqlenterprise (@adriansr)
  • beats Update mysqlenterprise module to ECS 1.8 #23978
    - [ ] integrations
• Filebeat zeek (@marc-gr)
  • beats [ECS] Zeek upgrade to ecs 1.8.0 #23847
  • integrations [zeel] Sync zeek package with beats integrations#696
• Review other modules to make sure none missing

---

• Make all Beats and modules report ECS 1.8.0 Update all Beats to report ECS version 1.8.0 #23992

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[navilg]

Hi,

Below are Python2 related vulnerabilities in filebeat v7.10.1. I suppose that can not be fixed unless we completely move from Python2 to Python3. Are we expecting these to be fixed in 7.11 or 7.12.0

Thanks

CVE-2014-4650
CVE-2016-5636
CVE-2017-1000158
CVE-2019-9636
CVE-2019-9948
CVE-2013-1753
CVE-2014-1912
CVE-2015-5652
CVE-2017-17522
CVE-2018-1060
CVE-2018-1061
CVE-2018-14647
CVE-2019-13404
CVE-2019-16056
CVE-2019-20907
CVE-2019-5010
CVE-2019-9674

[andrewstucki]

@navilg thanks for your interest in beats. A couple of things.

1. AFAIK we migrated all of our tooling to Python 3 some months back, just prior to Python 2 being EOL, so I don't believe this is a problem.
2. This issue is unrelated to to Python and has to do with schema changes from Elastic Common Schema being incorporated into the project.
3. If, on the off chance you wish to disclose any security issues, please follow the guide at https://www.elastic.co/community/security and email security@elastic.co for responsible disclosure

Thank you!