Merge branch 'master' into check-for-dupes-on-bulk

docs/settings/alert-action-settings.asciidoc

@@ -0,0 +1,47 @@
+[role="xpack"]
+[[alert-action-settings-kb]]
+=== Alerting and action settings in Kibana
+++++
+<titleabbrev>Alerting and action settings</titleabbrev>
+++++
+
+Alerts and actions are enabled by default in {kib}, but require you configure the following in order to use them: 
+
+. <<using-kibana-with-security,Set up {kib} to work with {stack} {security-features}>>.
+. <<configuring-tls-kib-es,Set up TLS encryption between {kib} and {es}>>.
+. <<general-alert-action-settings,Specify a value for `xpack.encrypted_saved_objects.encryptionKey`>>.
+
+You can configure the following settings in the `kibana.yml` file.
+
+
+[float]
+[[general-alert-action-settings]]
+==== General settings
+
+`xpack.encrypted_saved_objects.encryptionKey`::
+
+A string of 32 or more characters used to encrypt sensitive properties on alerts and actions before they're stored in {es}. Third party credentials &mdash; such as the username and password used to connect to an SMTP service &mdash; are an example of encrypted properties.  
++
+If not set, {kib} will generate a random key on startup, but all alert and action functions will be blocked. Generated keys are not allowed for alerts and actions because when a new key is generated on restart, existing encrypted data becomes inaccessible. For the same reason, alerts and actions in high-availability deployments of {kib} will behave unexpectedly if the key isn't the same on all instances of {kib}.
++
+Although the key can be specified in clear text in `kibana.yml`, it's recommended to store this key securely in the <<secure-settings,{kib} Keystore>>.
+
+[float]
+[[alert-settings]]
+==== Action settings
+
+`xpack.actions.whitelistedHosts`::
+A list of hostnames that {kib} is allowed to connect to when built-in actions are triggered. It defaults to `[*]`, allowing any host, but keep in mind the potential for SSRF attacks when hosts are not explicitly whitelisted. An empty list `[]` can be used to block built-in actions from making any external connections.
++
+Note that hosts associated with built-in actions, such as Slack and PagerDuty, are not automatically whitelisted. If you are not using the default `[*]` setting, you must ensure that the corresponding endpoints are whitelisted as well.
+
+`xpack.actions.enabledActionTypes`::
+A list of action types that are enabled. It defaults to `[*]`, enabling all types. The names for built-in {kib} action types are prefixed with a `.` and include: `.server-log`, `.slack`, `.email`, `.index`, `.pagerduty`, and `.webhook`. An empty list `[]` will disable all action types.
++
+Disabled action types will not appear as an option when creating new connectors, but existing connectors and actions of that type will remain in {kib} and will not function.  
+
+[float]
+[[action-settings]]
+==== Alert settings
+
+You do not need to configure any additional settings to use alerting in {kib}.

docs/settings/settings-xkb.asciidoc

@@ -10,6 +10,7 @@ include::{asciidoc-dir}/../../shared/settings.asciidoc[]
 
 For more {kib} configuration settings, see <<settings>>.
 
+include::alert-action-settings.asciidoc[]
 include::apm-settings.asciidoc[]
 include::dev-settings.asciidoc[]
 include::graph-settings.asciidoc[]

docs/setup/settings.asciidoc

@@ -457,16 +457,7 @@ Rollup user interface.
 
 `i18n.locale`:: *Default: en* Set this value to change the Kibana interface language. Valid locales are: `en`, `zh-CN`, `ja-JP`.
 
-`xpack.actions.enabledActionTypes:`:: *Default: +[ {asterisk} ]+* Set this value
-to an array of action types that are enabled.  An element of `*` indicates all
-action types registered are enabled.  The action types provided by Kibana are:
-`.server-log`, `.slack`, `.email`, `.index`, `.pagerduty`, `.webhook`.
-
-`xpack.actions.whitelistedHosts:`:: *Default: +[ {asterisk} ]+* Set this value
-to an array of host names which actions such as email, slack, pagerduty, and
-webhook can connect to.  An element of `*` indicates any host can be connected
-to.  An empty array indicates no hosts can be connected to.
-
+include::{docdir}/settings/alert-action-settings.asciidoc[]
 include::{docdir}/settings/apm-settings.asciidoc[]
 include::{docdir}/settings/dev-settings.asciidoc[]
 include::{docdir}/settings/graph-settings.asciidoc[]

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_dns_directly_to_the_internet.json

@@ -4,10 +4,7 @@
     "Exclude DNS servers from this rule as this is expected behavior. Endpoints usually query local DNS servers defined in their DHCP scopes, but this may be overridden if a user configures their endpoint to use a remote DNS server. This is uncommon in managed enterprise networks because it could break intranet name resolution when split horizon DNS is utilized. Some consumer VPN services and browser plug-ins may send DNS traffic to remote Internet destinations. In that case, such devices or networks can be excluded from this rule when this is expected behavior."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -42,5 +39,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_ftp_file_transfer_protocol_activity_to_the_internet.json

@@ -4,10 +4,7 @@
     "FTP servers should be excluded from this rule as this is expected behavior. Some business workflows may use FTP for data exchange. These workflows often have expected characteristics such as users, sources, and destinations. FTP activity involving an unusual source or destination may be more suspicious. FTP activity involving a production server that has no known associated FTP workflow or business requirement is often suspicious."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -53,5 +50,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_irc_internet_relay_chat_protocol_activity_to_the_internet.json

@@ -4,10 +4,7 @@
     "IRC activity may be normal behavior for developers and engineers but is unusual for non-engineering end users. IRC activity involving an unusual source or destination may be more suspicious. IRC activity involving a production server is often suspicious. Because these ports are in the ephemeral range, this rule may false under certain conditions, such as when a NAT-ed web server replies to a client which has used a port in the range by coincidence. In this case, these servers can be excluded. Some legacy applications may use these ports, but this is very uncommon and usually only appears in local traffic using private IPs, which does not match this rule's conditions."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -53,5 +50,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_nat_traversal_port_activity.json

@@ -4,10 +4,7 @@
     "Some networks may utilize these protocols but usage that is unfamiliar to local network administrators can be unexpected and suspicious. Because this port is in the ephemeral range, this rule may false under certain conditions, such as when an application server with a public IP address replies to a client which has used a UDP port in the range by coincidence. This is uncommon but such servers can be excluded."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -38,5 +35,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_port_26_activity.json

@@ -4,10 +4,7 @@
     "Servers that process email traffic may cause false positives and should be excluded from this rule as this is expected behavior."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -57,5 +54,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_port_8000_activity_to_the_internet.json

@@ -4,10 +4,7 @@
     "Because this port is in the ephemeral range, this rule may false under certain conditions, such as when a NATed web server replies to a client which has used a port in the range by coincidence. In this case, such servers can be excluded. Some applications may use this port but this is very uncommon and usually appears in local traffic using private IPs, which this rule does not match. Some cloud environments, particularly development environments, may use this port when VPNs or direct connects are not in use and cloud instances are accessed across the Internet."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -38,5 +35,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_pptp_point_to_point_tunneling_protocol_activity.json

@@ -4,10 +4,7 @@
     "Some networks may utilize PPTP protocols but this is uncommon as more modern VPN technologies are available. Usage that is unfamiliar to local network administrators can be unexpected and suspicious. Torrenting applications may use this port. Because this port is in the ephemeral range, this rule may false under certain conditions, such as when an application server replies to a client that used this port by coincidence. This is uncommon but such servers can be excluded."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -38,5 +35,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_proxy_port_activity_to_the_internet.json

@@ -4,10 +4,7 @@
     "Some proxied applications may use these ports but this usually occurs in local traffic using private IPs\n    which this rule does not match. Proxies are widely used as a security technology but in enterprise environments\n    this is usually local traffic which this rule does not match. Internet proxy services using these ports can be\n    white-listed if desired. Some screen recording applications may use these ports. Proxy port activity involving\n    an unusual source or destination may be more suspicious. Some cloud environments may use this port when VPNs or\n    direct connects are not in use and cloud instances are accessed across the Internet. Because these ports are in\n    the ephemeral range, this rule may false under certain conditions such as when a NATed web server replies to a\n    client which has used a port in the range by coincidence. In this case, such servers can be excluded if desired."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -38,5 +35,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_rdp_remote_desktop_protocol_from_the_internet.json

@@ -4,10 +4,7 @@
     "  Some network security policies allow RDP directly from the Internet but usage that is unfamiliar to\n  server or network owners can be unexpected and suspicious. RDP services may be exposed directly to the\n  Internet in some networks such as cloud environments. In such cases, only RDP gateways, bastions or jump\n  servers may be expected expose RDP directly to the Internet and can be exempted from this rule. RDP may\n  be required by some work-flows such as remote access and support for specialized software products and\n  servers. Such work-flows are usually known and not unexpected."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -68,5 +65,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_rdp_remote_desktop_protocol_to_the_internet.json

@@ -4,10 +4,7 @@
     "RDP connections may be made directly to Internet destinations in order to access\n    Windows cloud server instances but such connections are usually made only by engineers.\n    In such cases, only RDP gateways, bastions or jump servers may be expected Internet\n    destinations and can be exempted from this rule. RDP may be required by some work-flows\n    such as remote access and support for specialized software products and servers. Such\n    work-flows are usually known and not unexpected. Usage that is unfamiliar to server or\n    network owners can be unexpected and suspicious."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -53,5 +50,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_rpc_remote_procedure_call_from_the_internet.json

@@ -1,10 +1,7 @@
 {
   "description": "This rule detects network events that may indicate the use of RPC traffic\nfrom the Internet. RPC is commonly used by system administrators to remotely\ncontrol a system for maintenance or to use shared resources. It should almost\nnever be directly exposed to the Internet, as it is frequently targeted and\nexploited by threat actors as an initial access or back-door vector.\n",
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -35,5 +32,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_rpc_remote_procedure_call_to_the_internet.json

@@ -1,10 +1,7 @@
 {
   "description": "This rule detects network events that may indicate the use of RPC traffic\nto the Internet. RPC is commonly used by system administrators to remotely\ncontrol a system for maintenance or to use shared resources. It should almost\nnever be directly exposed to the Internet, as it is frequently targeted and\nexploited by threat actors as an initial access or back-door vector.\n",
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -35,5 +32,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_smb_windows_file_sharing_activity_to_the_internet.json

@@ -1,10 +1,7 @@
 {
   "description": "This rule detects network events that may indicate the use of Windows\nfile sharing (also called SMB or CIFS) traffic to the Internet. SMB is commonly\nused within networks to share files, printers, and other system resources amongst\ntrusted systems. It should almost never be directly exposed to the Internet, as\nit is frequently targeted and exploited by threat actors as an initial access\nor back-door vector or for data exfiltration.\n",
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -50,5 +47,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_smtp_to_the_internet.json

@@ -4,10 +4,7 @@
     "NATed servers that process email traffic may false and should be excluded from this rule as this is expected behavior for them. Consumer and  personal devices may send email traffic to remote Internet destinations. In this case, such devices or networks can be excluded from this rule if this is expected behavior."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -53,5 +50,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_sql_server_port_activity_to_the_internet.json

@@ -4,10 +4,7 @@
     "Because these ports are in the ephemeral range, this rule may false under certain conditions\n    such as when a NATed web server replies to a client which has used a port in the range by\n    coincidence. In this case, such servers can be excluded if desired. Some cloud environments may\n    use this port when VPNs or direct connects are not in use and database instances are accessed\n    directly across the Internet."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -38,5 +35,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_ssh_secure_shell_from_the_internet.json

@@ -4,10 +4,7 @@
     "Some network security policies allow SSH directly from the Internet but usage that is\n    unfamiliar to server or network owners can be unexpected and suspicious. SSH services may\n    be exposed directly to the Internet in some networks such as cloud environments. In such\n    cases, only SSH gateways, bastions or jump servers may be expected expose SSH directly to\n    the Internet and can be exempted from this rule. SSH may be required by some work-flows\n    such as remote access and support for specialized software products and servers. Such\n    work-flows are usually known and not unexpected."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -68,5 +65,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }

x-pack/legacy/plugins/siem/server/lib/detection_engine/rules/prepackaged_rules/network_ssh_secure_shell_to_the_internet.json

@@ -4,10 +4,7 @@
     "SSH connections may be made directly to Internet destinations in order to access Linux\n    cloud server instances but such connections are usually made only by engineers. In such cases,\n    only SSH gateways, bastions or jump servers may be expected Internet destinations and can be\n    exempted from this rule. SSH may be required by some work-flows such as remote access and support\n    for specialized software products and servers. Such work-flows are usually known and not unexpected.\n    Usage that is unfamiliar to server or network owners can be unexpected and suspicious."
   ],
   "index": [
-    "auditbeat-*",
-    "filebeat-*",
-    "packetbeat-*",
-    "winlogbeat-*"
+    "filebeat-*"
   ],
   "language": "kuery",
   "max_signals": 100,
@@ -38,5 +35,5 @@
     }
   ],
   "type": "query",
-  "version": 1
+  "version": 2
 }