[MD] Logging and Auditing #1986
Description
Task breakdown
- Research notes
- [MD]Enable data source audit log to file #2215
- [MD] Audit log query related operations #2228
Research Notes
Some questions we need to answer
- What's the different between logging and auditing?
- Auditing is used to answer the question "Who did what?" and possibly why.
- Logging is more focussed on what's happening.
- How's the current logging and auditing of OSD visualization/search/dashboards?
- Logging: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/core/server/logging/README.md
- Auditing: https://opensearch.org/docs/latest/security-plugin/audit-logs/index/
- See below section for detail
- What's the requirement for logging and auditing for data source?
- which user
- which datasource
- what query
- at what time
- Error
Logging
Data source logging will log datasource, query, time, and error, with correct logging setting and client settings in osd.yml
Similar to what we currently have with default single opensearch cluster. It makes use of the event emitter provided by opensearch-js client lib, that hook into internal events, such as request and response. Doc Reference
Current logging
Auditing
Security Plugin Audit Log feature
- Auditing in Opensearch is achieved by opensearch security plugin audit log. By correct configuration, it can monitor any REST/transport API request with info of user, request params/body, and timestamp. See below for an example of audit log generated by opening a visualization from OSD.
- As for CRUD operation around datasource and credential manager, it will also be logged because they are all saved objects.
- There's some limitation that we can't get all info in one audit log line, because by connecting to datasource, it sends request to external opensearch endpoints. Currently security plugin only logs API request to the default single cluster. We don't want to make changes to security plugin at this stage. But it can be an enhancement in the future.
- Summary: We'll consider other option - OSD audit service, for data source logging and auditing.
[Proposed Solution] OSD Audit Service + Logging service
- Core has an audit trail service, Plugins can get scoped Auditor from the core service to add events to introspect. Plugin can register some audit trail clients that implements the
Auditinterfaces in core, and make use of the logging service to write output to file by configuring "logging -> custom appender". The audit service can get the authenticated user info, then we can enrich that with datasouce, timestamp, query, error to create single audit log line, and saved to some file on disk
core - audit service
// @public
export interface AuditableEvent {
// (undocumented)
message: string;
// (undocumented)
type: string;
}
// @public
export interface Auditor {
add(event: AuditableEvent): void;
withAuditScope(name: string): void;
}
// @public
export interface AuditorFactory {
// (undocumented)
asScoped(request: OSDRequest): Auditor;
}
// Warning: (ae-missing-release-tag) "AuditTrailSetup" is exported by the package, but it is missing a release tag (@alpha, @beta, @public, or @internal)
//
// @public (undocumented)
export interface AuditTrailSetup {
register(auditor: AuditorFactory): void;
}
data source plugin -> audit trail client
export class AuditTrailClient implements Auditor {
private scope?: string;
constructor(
private readonly request: OSDRequest,
private readonly event$: Subject<AuditEvent>,
private readonly deps: Deps
) {}
public withAuditScope(name: string) {
if (this.scope !== undefined) {
throw new Error(`Audit scope is already set to: ${this.scope}`);
}
this.scope = name;
}
public add(event: AuditableEvent) {
const user = this.deps.getCurrentUser(this.request);
// doesn't use getSpace since it's async operation calling ES
const spaceId = this.deps.getSpaceId ? this.deps.getSpaceId(this.request) : undefined;
this.event$.next({
message: event.message,
type: event.type,
user: user?.username,
space: spaceId,
scope: this.scope,
data source plugin -> plugin.ts
const configSchema = schema.object({
enabled: schema.boolean({ defaultValue: false }),
appender: schema.maybe(coreConfig.logging.appenders),
logger: schema.object({
enabled: schema.boolean({ defaultValue: false }),
}),
});
export type AuditTrailConfigType = TypeOf<typeof configSchema>;
export class DataSource implements Plugin {
private readonly logger: Logger;
private readonly config$: Observable<AuditTrailConfigType>;
private readonly event$ = new Subject<AuditEvent>();
constructor(private readonly context: PluginInitializerContext) {
this.logger = this.context.logger.get();
this.config$ = this.context.config.create();
}
public setup(core: CoreSetup, deps: DepsSetup) {
const depsApi = {
getCurrentUser:
};
core.auditTrail.register({
asScoped: (request: OSDRequest) => {
return new AuditTrailClient(request, this.event$, depsApi);
},
});
core.logging.configure(
this.config$.pipe<LoggerContextConfigInput>(
map((config) => ({
appenders: {
auditTrailAppender: this.getAppender(config),
},
loggers: [
{
// plugins.auditTrail prepended automatically
context: '',
level: config.logger.enabled ? 'debug' : 'off',
appenders: ['auditTrailAppender'],
},
],
}))
)
);
}