Skip to content

Commit

Permalink
Document ephemeral storage (tech preview in 3.10)
Browse files Browse the repository at this point in the history
  • Loading branch information
RobertKrawitz committed Jul 10, 2018
1 parent 314df53 commit a251d4a
Show file tree
Hide file tree
Showing 12 changed files with 543 additions and 24 deletions.
12 changes: 11 additions & 1 deletion _topic_map.yml
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,9 @@ Topics:
- Name: Persistent Storage
File: storage
Distros: openshift-*
- Name: Ephemeral Storage
File: ephemeral-storage
Distros: openshift-*
- Name: Source Control Management
File: scm
Distros: openshift-*
Expand Down Expand Up @@ -467,6 +470,9 @@ Topics:
File: storage_classes_legacy
- Name: Configuring Azure Blob Storage for Integrated Docker Registry
File: azure_blob_docker_registry_example
- Name: Configuring Ephemeral Storage
File: configuring_ephemeral
Distros: openshift-origin,openshift-enterprise
- Name: Working with HTTP Proxies
File: http_proxies
Distros: openshift-origin,openshift-enterprise
Expand Down Expand Up @@ -704,8 +710,10 @@ Topics:
File: host_practices
- Name: Optimizing Compute Resources
File: optimizing_compute_resources
- Name: Optimizing Storage
- Name: Optimizing Persistent Storage
File: optimizing_storage
- Name: Optimizing Ephemeral Storage
File: optimizing_ephemeral_storage
- Name: Network Optimization
File: network_optimization
- Name: Routing Optimization
Expand Down Expand Up @@ -935,6 +943,8 @@ Topics:
Distros: openshift-enterprise,openshift-origin,openshift-dedicated
- Name: Application memory sizing
File: application_memory_sizing
- Name: Application ephemeral storage sizing
File: application_ephemeral_storage_sizing
- Name: Revision History
File: revhistory_dev_guide
Distros: openshift-enterprise,openshift-dedicated
Expand Down
9 changes: 7 additions & 2 deletions admin_guide/allocating_node_resources.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,14 @@ kubeletArguments:
----
====

Currently, the `*cpu*` and `*memory*` resource types are supported. For `*cpu*`,
Currently, the `*cpu*` and `*memory*` resource types are supported. If the ephemeral storage tech preview feature is enabled, the `*ephemeral-resource*` resource type is supported as well. For `*cpu*`,
the resource quantity is specified in units of cores (e.g., 200m, 0.5, 1).
For `*memory*`, it is specified in units of bytes (e.g., 200Ki, 50Mi, 5Gi).
For `*memory*` and `*ephemeral-storage*`, it is specified in units of bytes (e.g., 200Ki, 50Mi, 5Gi).

_Ephemeral storage management is disabled by default; to enable this
feature, please see
xref:../install_config/configuring_ephemeral.adoc[Configuring for
Ephemeral Storage]._

See xref:../dev_guide/compute_resources.adoc#dev-guide-compute-resources[Compute Resources] for more
details.
Expand Down
15 changes: 15 additions & 0 deletions admin_guide/limits.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,12 @@ All resource create and modification requests are evaluated against each
enumerated constraints, then the resource is rejected. If the resource does not
set an explicit value, and if the constraint supports a default value, then the
default value is applied to the resource.

NOTE: As of {product-title} 3.10, it is possible to specify limits and
requests for ephemeral storage, via the ephemeral storage tech
preview. This is disabled by default; to enable this feature, please
see xref:../install_config/configuring_ephemeral.adoc[Configuring for
Ephemeral Storage].
// end::admin_limits_overview[]


Expand Down Expand Up @@ -117,10 +123,19 @@ spec:
max:
openshift.io/image-tags: 20 <2>
openshift.io/images: 30 <3>
- type: "Pod"
max:
cpu: "2" <4>
memory: "1Gi" <5>
ephemeral-storage: "1Gi" <6>
----
<1> The maximum size of an image that can be pushed to an internal registry.
<2> The maximum number of unique image tags per image stream's spec.
<3> The maximum number of unique image references per image stream's status.
<4> The maximum amount of CPU that a pod can request on a node across all containers
<5> The maximum amount of memory that a pod can request on a node across all containers
<6> The maximum amount of ephemeral storage that a pod can request on a node across all containers (if you have enabled the Ephemeral Storage tech preview in {product-title} 3.10)

// end::admin_limits_sample_definitions_2[]

Expand Down
20 changes: 20 additions & 0 deletions admin_guide/overcommit.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,26 @@ system tasks or daemons need more memory than was accounted for in the node's
resource reservation. If a container specifies a limit on memory, it is
immediately killed if it exceeds the limit amount.

[[overcommit-ephemeral-storage]]
=== Ephemeral Storage

This applies only if you have enabled the Ephemeral Storage tech
preview in {product-title} 3.10. This is disabled by default; to
enable this feature, please see
xref:../install_config/configuring_ephemeral.adoc[Configuring for
Ephemeral Storage].

A container is guaranteed the amount of ephemeral storage it
requests. A container may use more ephemeral storage than requested,
but once it exceeds its requested amount, it could be killed in if the
available ephemeral disk space gets too low.

If a container uses less ephemeral storage than requested, it will not
be killed unless system tasks or daemons need more local ephemeral
storage than was accounted for in the node's resource reservation. If
a container specifies a limit on ephemeral storage, it is immediately
killed if it exceeds the limit amount.

[[qos-classes]]
== Quality of Service Classes

Expand Down
52 changes: 43 additions & 9 deletions admin_guide/quota.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,14 @@ interchangeably.
exceed this value. `*memory*` and `*requests.memory*` are the same value and can
be used interchangeably.

|`*ephemeral-storage*`
|The sum of local ephemeral storage requests across all pods in a
non-terminal state cannot exceed this value. `*ephemeral-storage*` and
`*requests.ephemeral-storage*` are the same value and can be used
interchangeably. _This applies only if you have enabled the Ephemeral Storage tech preview in {product-title} 3.10. This is disabled by default; to enable this feature, please
see xref:../install_config/configuring_ephemeral.adoc[Configuring for
Ephemeral Storage]._

|`*requests.cpu*`
|The sum of CPU requests across all pods in a non-terminal state cannot exceed
this value. `*cpu*` and `*requests.cpu*` are the same value and can be used
Expand All @@ -66,6 +74,12 @@ interchangeably.
exceed this value. `*memory*` and `*requests.memory*` are the same value and can
be used interchangeably.

|`*requests.ephemeral-storage*`
|The sum of ephemeral storage requests across all pods in a non-terminal state
cannot exceed this value. `*ephemeral-storage*` and
`*requests.ephemeral-storage*` are the same value and can be used
interchangeably. *This applies only if you have enabled the Ephemeral Storage tech preview in {product-title} 3.10.*


|`*limits.cpu*`
|The sum of CPU limits across all pods in a non-terminal state cannot exceed
Expand All @@ -74,6 +88,10 @@ this value.
|`*limits.memory*`
|The sum of memory limits across all pods in a non-terminal state cannot exceed
this value.

|`*limits.ephemeral-storage*`
|The sum of ephemeral storage limits across all pods in a non-terminal state cannot exceed
this value. *This applies only if you have enabled the Ephemeral Storage tech preview in {product-title} 3.10.*
|===


Expand Down Expand Up @@ -187,6 +205,11 @@ to tracking the following resources:
- `*cpu*`
- `*requests.cpu*`
- `*limits.cpu*`
- `*ephemeral-storage*`
- `*requests.ephemeral-storage*`
- `*limits.ephemeral-storage*`

Note that the ephemeral-storage requests and limits applies only if you have enabled the Ephemeral Storage tech preview in {product-title} 3.10.
// end::admin_quota_scopes[]

[[quota-enforcement]]
Expand Down Expand Up @@ -222,7 +245,7 @@ the system.
When allocating
xref:../dev_guide/compute_resources.adoc#dev-compute-resources[compute
resources], each container may specify a request and a limit value each for
CPU and memory. Quotas can restrict any of these values.
CPU, memory, and ephemeral storage. Quotas can restrict any of these values.

If the quota has a value specified for `*requests.cpu*` or `*requests.memory*`,
then it requires that every incoming container make an explicit request for
Expand Down Expand Up @@ -297,19 +320,25 @@ spec:
pods: "4" <1>
requests.cpu: "1" <2>
requests.memory: 1Gi <3>
limits.cpu: "2" <4>
limits.memory: 2Gi <5>
requests.ephemeral-storage: 2Gi <4>
limits.cpu: "2" <5>
limits.memory: 2Gi <6>
limits.ephemeral-storage: 4Gi <7>
----
<1> The total number of pods in a non-terminal state that can exist in the
project.
<2> Across all pods in a non-terminal state, the sum of CPU requests cannot
exceed 1 core.
<3> Across all pods in a non-terminal state, the sum of memory requests cannot
exceed 1Gi.
<4> Across all pods in a non-terminal state, the sum of CPU limits cannot exceed
<4> Across all pods in a non-terminal state, the sum of ephemeral storage requests cannot
exceed 2Gi.
<5> Across all pods in a non-terminal state, the sum of CPU limits cannot exceed
2 cores.
<5> Across all pods in a non-terminal state, the sum of memory limits cannot
<6> Across all pods in a non-terminal state, the sum of memory limits cannot
exceed 2Gi.
<7> Across all pods in a non-terminal state, the sum of ephemeral storage limits cannot
exceed 4Gi.


// end::admin_quota_compute_resources[]
Expand Down Expand Up @@ -346,15 +375,18 @@ spec:
pods: "4" <1>
limits.cpu: "4" <2>
limits.memory: "2Gi" <3>
limits.ephemeral-storage: "4Gi" <4>
scopes:
- NotTerminating <4>
- NotTerminating <5>
----
<1> The total number of pods in a non-terminal state.
<2> Across all pods in a non-terminal state, the sum of CPU limits cannot exceed
this value.
<3> Across all pods in a non-terminal state, the sum of memory limits cannot exceed
this value.
<4> Restricts the quota to only matching pods where `spec.activeDeadlineSeconds` is
<4> Across all pods in a non-terminal state, the sum of ephemeral storage limits cannot exceed
this value.
<5> Restricts the quota to only matching pods where `spec.activeDeadlineSeconds` is
set to `nil`. Build pods will fall under `NotTerminating` unless the
`RestartNever` policy is applied.

Expand All @@ -371,13 +403,15 @@ spec:
pods: "2" <1>
limits.cpu: "1" <2>
limits.memory: "1Gi" <3>
limits.ephemeral-storage: "1Gi" <4>
scopes:
- Terminating <4>
- Terminating <5>
----
<1> The total number of pods in a non-terminal state.
<2> Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value.
<3> Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value.
<4> Restricts the quota to only matching pods where `spec.activeDeadlineSeconds >=0`. For example,
<4> Across all pods in a non-terminal state, the sum of ephemeral storage limits cannot exceed this value.
<5> Restricts the quota to only matching pods where `spec.activeDeadlineSeconds >=0`. For example,
this quota would charge for build or deployer pods, but not long running pods like a web server or database.


Expand Down
84 changes: 84 additions & 0 deletions architecture/additional_concepts/ephemeral-storage.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
[[architecture-additional-concepts-ephemeral-storage]]
= Ephemeral Local Storage
{product-author}
{product-version}
:data-uri:
:icons:
:experimental:
:toc: macro
:toc-title:
:prewrap!:

toc::[]

*Note: this only applies if the ephemeral-storage tech preview in
{product-title} 3.10 has been enabled. This is disabled by default;
to enable this feature, please see
xref:../../install_config/configuring_ephemeral.adoc[Configuring for
Ephemeral Storage].*

== Overview

In addition to persistent storage, pods and containers may require
ephemeral (transient) local storage for their operation. The lifetime
of this ephemeral storage does not extend beyond the life of the
individual pod, and this ephemeral storage cannot be shared across
pods.

Prior to {product-title} 3.10, ephemeral local storage was exposed to
pods via the container’s writable layer, logs directory, and EmptyDir
volumes. Pods use ephemeral local storage for scratch space, caching
and logs. There are many issues related to the lack of local storage
accounting and isolation, including:

- Pods do not know how much local storage is available to them.

- Pods cannot request "guaranteed" local storage.

- Local storage is a "best effort" resource.

- Pods can get evicted due to other pods filling up the local storage,
after which no new pods will be admitted until sufficient storage
has been reclaimed.

Unlike persistent volumes, ephemeral storage is unstructured and
shared (the space, not the actual data) between all pods running on a
node, in addition to other uses by the system, the container runtime,
and {product-title}. The ephemeral storage framework allows pods to
specify their transient local storage needs, and {product-title} to
schedule pods where appropriate and protect the node against excessive
use of local storage.

While the ephemeral storage framework allows administrators and
developers to better manage this local storage, it does not provide
any promises related to I/O throughput and latency.

== Types of Ephemeral Storage

Ephemeral local storage is always made available in the primary
partition. There are two basic ways of creating the primary
partition.

=== Root

This partition holds the kubelet’s root directory (`/var/lib/origin`
by default) and `/var/log` directory. This partition may be shared
between user pods, OS and Kubernetes system daemons. This partition
can be consumed by pods via EmptyDir volumes, container logs, image
layers and container writable layers. Kubelet will manage shared
access and isolation of this partition. This partition is "ephemeral"
and applications cannot expect any performance SLAs (Disk IOPS for
example) from this partition.

=== Runtime

This is an optional partition which runtimes can use for overlay
filesystems. {product-title} will attempt to identify and provide
shared access along with isolation to this partition. Container image
layers and writable layers are stored here. If the runtime partition
exists, `root` partition will not hold any image layer or other writable storage.

_Note that when using devicemapper to provide runtime storage,
containers' copy-on-write layer is not accounted for in ephemeral
storage management. Using Overlay storage will allow this use of
ephemeral storage to be monitored.
Loading

0 comments on commit a251d4a

Please sign in to comment.