Skip to content

Commit ef63377

Browse files
pacevedompacevedom
committed
USHIFT-5072: MicroShift router logging options
1 parent 995b620 commit ef63377

File tree

1 file changed

+253
-0
lines changed

1 file changed

+253
-0
lines changed

‎enhancements/microshift/microshift-router-configuration-errors-logging.md‎

Lines changed: 253 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,253 @@
1+
---
2+
title: microshift-router-configuration-errors-logging
3+
authors:
4+
- "@pacevedom"
5+
reviewers:
6+
- "@eslutsky"
7+
- "@copejon"
8+
- "@ggiguash"
9+
- "@pmtk"
10+
- "@pliurh"
11+
- "@Miciah"
12+
approvers:
13+
- "@jerpeter1"
14+
api-approvers:
15+
- None
16+
creation-date: 2025-04-24
17+
last-updated: 2025-04-24
18+
tracking-link:
19+
- https://issues.redhat.com/browse/USHIFT-4092
20+
---
21+
22+
# MicroShift router errors and logging configuration options
23+
24+
## Summary
25+
MicroShift's default router is created as part of the platform, but does not
26+
allow configuring some of its specific parameters. For example, you cannot
27+
configure custom behavior with error pages, or whether headers and cookies are
28+
captured in the access logs.
29+
30+
In order to allow these operations and more, a set of configuration options is
31+
proposed.
32+
33+
## Motivation
34+
Microshift Customers need a way to override the default Ingress Controller
35+
logging configuration similar as OpenShift does.
36+
37+
### User Stories
38+
* As a MicroShift admin, I want to configure custom error code pages in the
39+
router.
40+
* As a MicroShift admin, I want to enable/disable access logging in the
41+
router.
42+
* As a MicroShift admin, I want to configure which HTTP headers are captured
43+
in the access logs.
44+
* As a MicroShift admin, I want to configure which HTTP cookies are captured
45+
in the access logs.
46+
47+
### Goals
48+
Allow users to configure the additional Router customization parameters.
49+
50+
### Non-Goals
51+
N/A
52+
53+
## Proposal
54+
Microshift doesnt use [ingress operator](https://github.com/openshift/cluster-ingress-operator),
55+
which means all the customization is performed through the configuration file.
56+
The configuration will propagate to the router deployment [manifest](https://github.com/openshift/microshift/blob/aea40ae1ee66dc697996c309268be1939b018f56/assets/components/openshift-router/deployment.yaml) through environment variables, just like what the ingress operator does.
57+
58+
See the API Extensions section to check the details.
59+
60+
See full [OpenShift Router](https://docs.openshift.com/container-platform/4.18/networking/ingress-operator.html)
61+
configuration reference for more information.
62+
63+
### Workflow Description
64+
***configuring errors and logging options***
65+
1. The cluster admin adds specific configuration for the router prior to
66+
MicroShift's start.
67+
2. After MicroShift starts, the system will read the configuration and setup
68+
the router according to it.
69+
70+
### API Extensions
71+
As mentioned in the proposal, there is an entire new section in the configuration:
72+
```yaml
73+
ingress:
74+
httpErrorCodePages: <string>
75+
accessLogging: <Enable/Disable>
76+
httpCaptureHeaders:
77+
request:
78+
- maxLength: <integer. Min 1>
79+
name: <string regex: ^[-!#$%&'*+.0-9A-Z^_`a-z|~]*$. Must comply with RFC 2616 section 4.2>
80+
response:
81+
- maxLength: <integer. Min 1>
82+
name: <string regex: ^[-!#$%&'*+.0-9A-Z^_`a-z|~]*$. Must comply with RFC 2616 section 4.2>
83+
httpCaptureCookies:
84+
- matchType: <string. Can be Exact or Prefix>
85+
maxLength: <integer. Min 1, Max 1024>
86+
name: <string regex: ^[-!#$%&'*+.0-9A-Z^_`a-z|~]+$. Must comply with RFC 6265 section 4.1>
87+
namePrefix: <string regex: ^[-!#$%&'*+.0-9A-Z^_`a-z|~]+$. Must comply with RFC 6265 section 4.1>
88+
```
89+
90+
For more information check each individual section.
91+
92+
#### Hypershift / Hosted Control Planes
93+
N/A
94+
### Topology Considerations
95+
N/A
96+
97+
#### Standalone Clusters
98+
N/A
99+
100+
#### Single-node Deployments or MicroShift
101+
Enhancement is solely intended for MicroShift.
102+
103+
### Implementation Details/Notes/Constraints
104+
The default router is composed of a bunch of assets that are embedded as part
105+
of the MicroShift binary. These assets come from the rebase, copied from the
106+
original router in [cluster-ingress-operator](https://github.com/openshift/cluster-ingress-operator).
107+
108+
Based on the configuration parameters, the manifest for the router pod will
109+
mutate to translate all the new options.
110+
111+
#### Enabling access logging
112+
HAProxy allows configuration for access logging. This happens through rsyslog,
113+
which requires another container in the router. This is the equivalent to the
114+
`Container` approach in OpenShift Router access logging.
115+
116+
The second container (named `access-log`) will print through stdout all the
117+
logs from the router.
118+
119+
This approach does not require configuring rsyslogd in the host and is self
120+
contained, not dedicating any resources in case it is not enabled.
121+
122+
Configuring either of `ingress.httpCaptureHeaders` or
123+
`ingress.httpCaptureCookies` will also enable `ingress.accessLogging`.
124+
125+
`ingress.accessLogging` defaults to `Disable`.
126+
127+
#### Configuring custom error code pages
128+
To configure custom error code pages the user needs to specify a configmap name
129+
in `httpErrorCodePages`. This configmap must be in the `openshift-config`
130+
namespace and should have keys in the format of `error-page-<error code>.http`
131+
where `<error code>` is an HTTP status code.
132+
133+
Each value in the configmap should be the full response, including HTTP
134+
headers.
135+
136+
As of today, only errors for 503 and 404 can be customized.
137+
138+
`ingress.httpErrorCodePages` defaults to empty.
139+
140+
#### Capturing headers
141+
To configure specific HTTP header capture so they are included in the access
142+
logs the user needs to create entries in `ingress.httpCaptureHeaders`. This
143+
field is a list and allows capturing request and response headers
144+
independently. Each of the entries in the list has different parameters that
145+
follow. If the list is empty (which is the default value) no headers will be
146+
captured.
147+
148+
This option only applies to cleartext HTTP or reencrypt connections. Headers
149+
can not be captured for TLS passthrough connections.
150+
151+
Each element of the list includes:
152+
* `request`. Specifies which HTTP request headers to capture. If this field is
153+
empty, no request headers are captured.
154+
* `response`. Specifies which HTTP response headers to capture. If this field
155+
is empty, no request headers are captured.
156+
157+
Both elements have the same fields:
158+
* `maxLength`. Specifies a maximum length for the header value. If a header
159+
value exceeds this length, the value will be truncated in the log message.
160+
* `name`. Specifies a header name. Its value must be a valid HTTP header name
161+
as defined in RFC 2616 section 4.2.
162+
163+
If configured, it is mandatory to include at least `maxLength` and `name`.
164+
165+
`ingress.httpCaptureHeaders` defaults to an empty list.
166+
167+
#### Capturing cookies
168+
To configure specific HTTP cookie capture so they are included in the access
169+
logs the user needs to create an entry in `ingress.httpCaptureCookies`. This
170+
field is a list (limited to 1 element) which includes information on which
171+
cookie to capture. If the list is empty (which is the default value) no cookies
172+
will be captured.
173+
174+
In each element of the list we find:
175+
* `matchType`. Specifies the type of match to perform against the cookie name.
176+
Allowed values are `Exact` and `Prefix`.
177+
* `maxLength`. Specifies a maximum length of the string that will be logged,
178+
which includes the cookie name, cookie value, and one-character delimiter.
179+
If the log entry exceeds this length, the value will be truncated in the log
180+
message.
181+
* `name`. Specifies a cookie name. It must be a valid HTTP cookie name as
182+
defined in RFC 6265 section 4.1.
183+
* `namePrefix`. Specifies a cookie name prefix. It must be a valid HTTP cookie
184+
name as defined in RFC 6265 section 4.1.
185+
186+
If configured, it is mandatory to include at least `matchType` and `maxLength`.
187+
188+
`ingress.httpCaptureCookies` defaults to an empty list.
189+
190+
#### How config options change manifests
191+
Each of the configuration options described above has a direct effect on the
192+
manifests that MicroShift will apply after starting.
193+
See the full Implementation details in the [router-configuration](microshift-router-configuration.md)
194+
enhancement.
195+
196+
### Risks and Mitigations
197+
* Not configuring custom error pages will return the default ones, which are
198+
usually empty and only return the http status code.
199+
* Not configuring capture of http headers and/or cookies will not include them
200+
in the access logs of the router.
201+
202+
### Drawbacks
203+
N/A
204+
205+
## Open Questions
206+
* Do we also allow configuring the access log format?
207+
208+
## Test Plan
209+
All configuration changes will be included in already existing e2e router
210+
tests. Testing router functionality is out of scope of this enhancement.
211+
212+
213+
## Graduation Criteria
214+
Not applicable
215+
216+
### Dev Preview -> Tech Preview
217+
- Ability to utilize the enhancement end to end
218+
- End user documentation, relative API stability
219+
- Sufficient test coverage
220+
221+
### Tech Preview -> GA
222+
- More testing (upgrade, downgrade)
223+
- Sufficient time for feedback
224+
- Available by default
225+
- User facing documentation created in [openshift-docs](https://github.com/openshift/openshift-docs/)
226+
227+
### Removing a deprecated feature
228+
N/A
229+
230+
## Upgrade / Downgrade Strategy
231+
When upgrading from 4.19 or earlier the new configuration fields will remain
232+
unset, causing the existing defaults to be used.
233+
234+
When downgrading from 4.20 to earlier versions the new parameters will be
235+
ignored.
236+
237+
## Version Skew Strategy
238+
N/A
239+
240+
## Operational Aspects of API Extensions
241+
242+
### Failure Modes
243+
N/A
244+
245+
## Support Procedures
246+
Access logging, if enabled, will be part of the logs of the openshift-router
247+
logs. Logs from this pod are captured in the already existing sos report
248+
procedure available for MicroShift.
249+
250+
## Implementation History
251+
Implementation [PR](https://github.com/openshift/microshift/pull/4474/) for Micorshift
252+
## Alternatives (Not Implemented)
253+
N/A

0 commit comments

Comments
 (0)