Enforce reusable enum component requirement for direct enum payloads

[bsayli]

Describe the bug

Direct enum generic payloads were accepted by response type introspection without verifying that the enum was published as a reusable OpenAPI schema component.

This affected response shapes where the enum itself is the generic payload type, for example:

ServiceResponse<CoverageStatus>

and custom BYOE envelopes such as:

ApiResponse<CoverageStatus>

It also affected supported nested container shapes in the default envelope path, for example:

ServiceResponse<Page<CoverageStatus>>

For these shapes, openapi-generics must generate a wrapper class that directly references the enum as the generic type parameter:

public class ServiceResponseCoverageStatus
    extends ServiceResponse<CoverageStatus> {
}

or, for BYOE:

public class ApiResponseCoverageStatus
    extends ApiResponse<CoverageStatus> {
}

This requires CoverageStatus to exist as a stable, reusable OpenAPI schema component.

Previously, ResponseTypeIntrospector accepted any resolved enum class in the same way as DTOs, scalars, or value types. That meant a descriptor could be produced even when Springdoc would emit the enum inline instead of as a reusable component.

Inline enum payloads are not supported by openapi-generics and are not intended to be supported.

The project does not extract inline schemas into components, synthesize model names, or reshape the OpenAPI document. It only adds generic reconstruction metadata on top of schemas that already have stable OpenAPI identities.

Steps to reproduce

1. Define an enum without enumAsRef:

public enum CoverageStatus {
  ACTIVE,
  PASSIVE,
  EXPERIMENTAL
}

2. Return the enum directly as a generic payload:

@GetMapping("/enum")
public ResponseEntity<ServiceResponse<CoverageStatus>> enumValue() {
  return ResponseEntity.ok(ServiceResponse.of(CoverageStatus.EXPERIMENTAL));
}

3. Or return it through a BYOE envelope:

@GetMapping("/enum")
public ResponseEntity<ApiResponse<CoverageStatus>> enumValue() {
  return ResponseEntity.ok(ApiResponse.of(CoverageStatus.EXPERIMENTAL));
}

4. Or return it as the item type of a supported default container shape:

@GetMapping("/statuses")
public ResponseEntity<ServiceResponse<Page<CoverageStatus>>> statuses() {
  return ResponseEntity.ok(
      ServiceResponse.of(
          Page.of(
              List.of(
                  CoverageStatus.ACTIVE,
                  CoverageStatus.PASSIVE,
                  CoverageStatus.EXPERIMENTAL),
              0,
              3,
              3)));
}

5. Generate the OpenAPI contract and client.

6. Observe that response type introspection may produce generic wrapper metadata for the enum even though the enum was not guaranteed to exist as a reusable OpenAPI component.

Expected behavior

Direct enum generic payloads should be supported only when the enum is explicitly published as a reusable OpenAPI schema component.

For Springdoc users, this means:

import io.swagger.v3.oas.annotations.media.Schema;

@Schema(enumAsRef = true)
public enum CoverageStatus {
  ACTIVE,
  PASSIVE,
  EXPERIMENTAL
}

With this declaration, the following shapes are valid:

ServiceResponse<CoverageStatus>
ApiResponse<CoverageStatus>
ServiceResponse<Page<CoverageStatus>>

Without @Schema(enumAsRef = true), direct enum generic payloads should be ignored by response type introspection and should not produce generic wrapper descriptors.

DTO payloads containing enum fields are not affected.

For example, this remains supported:

ServiceResponse<TypeProfileDto>

where TypeProfileDto contains:

private CoverageStatus status;

because the generic payload is TypeProfileDto, not CoverageStatus.

Enum fields inside DTOs are handled by the regular OpenAPI toolchain.

Affected area

• server starter
• response type introspection
• generic payload discovery
• enum payload support boundary
• BYOE direct payload reconstruction
• generated wrapper metadata correctness

Additional context

This issue enforces the intended support boundary for enum payloads.

The problem was not that inline enum support was missing.

Inline enum payloads are intentionally unsupported.

The problem was that ResponseTypeIntrospector did not previously enforce this boundary.

Before the fix, simple payload descriptors were produced for any non-generic resolved type:

if (!dataType.hasGenerics()) {
  return Optional.of(
      ResponseTypeDescriptor.simple(
          envelopeType,
          payloadPropertyName,
          raw.getSimpleName()));
}

and container descriptors were produced as long as the item type resolved:

Class<?> itemRaw = itemType.resolve();

if (itemRaw == null) {
  return Optional.empty();
}

return Optional.of(
    ResponseTypeDescriptor.container(
        envelopeType,
        payloadPropertyName,
        containerType.getSimpleName(),
        itemRaw.getSimpleName()));

This treated enum payloads as if they always had stable OpenAPI component identities.

The fix adds explicit payload support validation:

private boolean isSupportedPayloadType(Class<?> type) {
  if (type == null) {
    return false;
  }

  if (!type.isEnum()) {
    return true;
  }

  return isEnumAsRefEnabled(type);
}

Enum payloads are now accepted only when @Schema(enumAsRef = true) is present:

private boolean isEnumAsRefEnabled(Class<?> enumType) {
  for (Annotation annotation : enumType.getAnnotations()) {
    if (!SCHEMA_ANNOTATION.equals(annotation.annotationType().getName())) {
      continue;
    }

    try {
      Method enumAsRef = annotation.annotationType().getMethod("enumAsRef");
      Object value = enumAsRef.invoke(annotation);
      return Boolean.TRUE.equals(value);
    } catch (ReflectiveOperationException ignored) {
      return false;
    }
  }

  return false;
}

The descriptor creation paths now use this validation for both simple payloads and container item payloads.

This keeps enum support aligned with the project’s core design rule:

openapi-generics does not modify or normalize the OpenAPI schema.
It only adds generic semantics when the projected schema already has a stable identity.

Fixed on main and currently available in 1.0.3-SNAPSHOT.
Planned for the next public release.

Implementation reference:
a2b769c

[bsayli]

Fixed on main and included in 1.0.3.

Direct enum generic payloads are now accepted only when published as reusable OpenAPI schema components via @Schema(enumAsRef = true).

Implementation:
a2b769c