reopened: BREAKING CHANGE from 8.7.2: lean query return full mongoose object type for populated doc instead of FlattenMaps

[nikzanda]

Prerequisites

• I have written a descriptive issue title
• I have searched existing issues to ensure the bug has not already been reported

Mongoose version

8.9.0

Node.js version

20.10.0

MongoDB server version

6.0.2

Typescript version (if applicable)

5.5.4

Description

I'm reopening this issue because it was closed without resolution. You can find the closed issue at this link.
I'll try to use a simpler example to explain what the bug is:

// child.ts
import {
  Connection,
  HydratedDocument,
  Model,
  Schema,
  SchemaTypes,
  Types,
} from 'mongoose';

export interface IChild {
  _id: Types.ObjectId;
  name: string;
}

type ChildDocumentOverrides = {};

export interface IChildVirtuals {
  id: string;
}

export type ChildInstance = HydratedDocument<
  IChild,
  ChildDocumentOverrides & IChildVirtuals
>;

type ChildModelType = Model<
  IChild,
  {},
  ChildDocumentOverrides,
  IChildVirtuals,
  ChildInstance
>;

export default (connection: Connection) => {
  const childSchema = new Schema<IChild, ChildModelType>(
    {
      name: {
        type: SchemaTypes.String,
        required: true,
        trim: true,
      },
    },
    {},
  );

  return connection.model<IChild, ChildModelType>('Child', childSchema);
};

// parent.ts
import {
  Connection,
  Document,
  HydratedDocument,
  Model,
  PopulatedDoc,
  Schema,
  SchemaTypes,
  Types,
} from 'mongoose';
import { IChild } from './child';

export interface IParent {
  _id: Types.ObjectId;
  name: string;
  child: PopulatedDoc<Document<Types.ObjectId> & IChild>;
  createdAt: Date;
  updatedAt: Date;
}

type ParentDocumentOverrides = {};

export interface IParentVirtuals {
  id: string;
}

export type ParentInstance = HydratedDocument<
  IParent,
  ParentDocumentOverrides & IParentVirtuals
>;

type ParentModelType = Model<
  IParent,
  {},
  ParentDocumentOverrides,
  IParentVirtuals,
  ParentInstance
>;

export default (connection: Connection) => {
  const parentSchema = new Schema<IParent, ParentModelType>(
    {
      name: {
        type: SchemaTypes.String,
        required: true,
        trim: true,
      },
      child: {
        type: SchemaTypes.ObjectId,
        ref: 'Child',
        required: true,
      },
    },
    {},
  );

  return connection.model<IParent, ParentModelType>('Parent', parentSchema);
};

My query:

const parent = await Parent.findOne().lean();

If I execute the query in versions prior to 8.7.2, the returned type is FlattenMaps<IParent> ✅:

From version 8.7.2 onwards, the type within the parent for child is Types.ObjectId or an instance of child ❌. However, this is incorrect because I executed a lean query, so child should be Types.ObjectId | FlattenMaps<IChild>:

In your comment, you suggested updating the code and explicitly setting the type of the variable to be populated. However, if I do not want to populate the variable and use a lean query, the returned type should be FlattenMaps for all PopulatedDoc.
Please, review this situation and the example I provided.

Steps to Reproduce

Execute a lean query (without using "populate") on a model that contains variables of type PopulatedDoc.

Expected Behavior

No response

[nikzanda]

Here is another example to demonstrate that this is a bug: if I declare a function that accepts a parameter of type IParent, and then execute a lean query and pass the query result as a parameter to this function, there should be no compilation errors. However, after updating to version 8.7.2 and later, the error appears.

const test = (p: IParent) => {
  // TODO
};

const parent = await Parent.findOne().lean();
if (parent) {
  test(parent);
}

The compilation error:

Argument of type '{ _id: ObjectId; name: string; child: ObjectId | { _id: ObjectId; $assertPopulated: <Paths = {}>(path: string | string[], values?: Partial<Paths> | undefined) => Omit<...> & Paths; ... 56 more ...; name: string; } | undefined; createdAt: Date; updatedAt: Date; } & Required<...>' is not assignable to parameter of type 'IParent'.
  Types of property 'child' are incompatible.
    Type 'ObjectId | { _id: ObjectId; $assertPopulated: <Paths = {}>(path: string | string[], values?: Partial<Paths> | undefined) => Omit<Document<ObjectId, any, any> & IChild, keyof Paths> & Paths; ... 56 more ...; name: string; } | undefined' is not assignable to type 'PopulatedDoc<Document<ObjectId, any, any> & IChild>'.
      Type '{ _id: ObjectId; $assertPopulated: <Paths = {}>(path: string | string[], values?: Partial<Paths> | undefined) => Omit<Document<ObjectId, any, any> & IChild, keyof Paths> & Paths; ... 56 more ...; name: string; }' is not assignable to type 'PopulatedDoc<Document<ObjectId, any, any> & IChild>'.
        Type '{ _id: ObjectId; $assertPopulated: <Paths = {}>(path: string | string[], values?: Partial<Paths> | undefined) => Omit<Document<ObjectId, any, any> & IChild, keyof Paths> & Paths; ... 56 more ...; name: string; }' is not assignable to type 'Document<ObjectId, any, any> & IChild'.
          Type '{ _id: ObjectId; $assertPopulated: <Paths = {}>(path: string | string[], values?: Partial<Paths> | undefined) => Omit<Document<ObjectId, any, any> & IChild, keyof Paths> & Paths; ... 56 more ...; name: string; }' is not assignable to type 'Document<ObjectId, any, any>'.
            The types of 'schema.paths' are incompatible between these types.
              Type '{ [x: string]: { OptionsConstructor: { [x: string]: any; type?: any; alias?: string | string[] | undefined; validate?: Function | RegExp | [RegExp, string] | [Function, string] | ... 4 more ... | undefined; ... 28 more ...; maxlength?: number | ... 2 more ... | undefined; }; ... 22 more ...; defaultOptions?: { ...; ...' is not assignable to type '{ [key: string]: SchemaType<any, any>; }'.
                'string' index signatures are incompatible.
                  Type '{ OptionsConstructor: { [x: string]: any; type?: any; alias?: string | string[] | undefined; validate?: Function | RegExp | [RegExp, string] | [Function, string] | ... 4 more ... | undefined; ... 28 more ...; maxlength?: number | ... 2 more ... | undefined; }; ... 22 more ...; defaultOptions?: { ...; } | undefined; }' is not assignable to type 'SchemaType<any, any>'.
                    The types of 'OptionsConstructor.index' are incompatible between these types.
                      Type 'boolean | IndexDirection | { expires?: string | number | undefined; weights?: { [x: string]: number; } | undefined; background?: boolean | undefined; unique?: boolean | undefined; ... 40 more ...; explain?: ExplainVerbosityLike | undefined; } | undefined' is not assignable to type 'boolean | IndexDirection | IndexOptions | undefined'.
                        Type '{ expires?: string | number | undefined; weights?: { [x: string]: number; } | undefined; background?: boolean | undefined; unique?: boolean | undefined; name?: string | undefined; partialFilterExpression?: { ...; } | undefined; ... 38 more ...; explain?: ExplainVerbosityLike | undefined; }' is not assignable to type 'boolean | IndexDirection | IndexOptions | undefined'.
                          Type '{ expires?: string | number | undefined; weights?: { [x: string]: number; } | undefined; background?: boolean | undefined; unique?: boolean | undefined; name?: string | undefined; partialFilterExpression?: { ...; } | undefined; ... 38 more ...; explain?: ExplainVerbosityLike | undefined; }' is not assignable to type 'IndexOptions'.
                            Types of property 'session' are incompatible.
                              Type '{ hasEnded: boolean; clientOptions?: { appName?: string | undefined; hosts: { host: string | undefined; port: number | undefined; socketPath: string | undefined; isIPv6: boolean; inspect: () => string; toString: () => string; toHostPort: () => { ...; }; }[]; ... 70 more ...; lookup?: LookupFunction | undefined; } | ...' is not assignable to type 'ClientSession | undefined'.
                                Type '{ hasEnded: boolean; clientOptions?: { appName?: string | undefined; hosts: { host: string | undefined; port: number | undefined; socketPath: string | undefined; isIPv6: boolean; inspect: () => string; toString: () => string; toHostPort: () => { ...; }; }[]; ... 70 more ...; lookup?: LookupFunction | undefined; } | ...' is not assignable to type 'ClientSession'.
                                  Types of property 'clientOptions' are incompatible.
                                    Type '{ appName?: string | undefined; hosts: { host: string | undefined; port: number | undefined; socketPath: string | undefined; isIPv6: boolean; inspect: () => string; toString: () => string; toHostPort: () => { ...; }; }[]; ... 70 more ...; lookup?: LookupFunction | undefined; } | undefined' is not assignable to type 'MongoOptions | undefined'.
                                      Type '{ appName?: string | undefined; hosts: { host: string | undefined; port: number | undefined; socketPath: string | undefined; isIPv6: boolean; inspect: () => string; toString: () => string; toHostPort: () => { ...; }; }[]; ... 70 more ...; lookup?: LookupFunction | undefined; }' is not assignable to type 'MongoOptions'.
                                        The types of 'autoEncryption.keyVaultClient' are incompatible between these types.
                                          Type '{ readonly options: { readonly appName?: string | undefined; readonly hosts: { host: string | undefined; port: number | undefined; socketPath: string | undefined; isIPv6: boolean; inspect: () => string; toString: () => string; toHostPort: () => { ...; }; }[]; ... 70 more ...; readonly lookup?: LookupFunction | undef...' is not assignable to type 'MongoClient | undefined'.
                                            Type '{ readonly options: { readonly appName?: string | undefined; readonly hosts: { host: string | undefined; port: number | undefined; socketPath: string | undefined; isIPv6: boolean; inspect: () => string; toString: () => string; toHostPort: () => { ...; }; }[]; ... 70 more ...; readonly lookup?: LookupFunction | undef...' is not assignable to type 'MongoClient'.
                                              The types of 'options.session' are incompatible between these types.
                                                Type 'Binary | null | undefined' is not assignable to type 'Buffer | undefined'.
                                                  Type 'null' is not assignable to type 'Buffer | undefined'.ts(2345)
const parent: {
    _id: Types.ObjectId;
    name: string;
    child: Types.ObjectId | {
        _id: Types.ObjectId;
        $assertPopulated: <Paths = {}>(path: string | string[], values?: Partial<Paths> | undefined) => Omit<...> & Paths;
        ... 56 more ...;
        name: string;
    } | undefined;
    createdAt: Date;
    updatedAt: Date;
} & Required<...>

Please check this behavior, because before version 8.7.2, my repository did not return compilation errors for functions that accepted a parameter of type IParent when passing the results of lean queries.

Thank you for your patience.

[vkarpov15]

Ah ok, the compilation error you provided is due to #15057 and would be fixed by #15072. The error message has nothing to do with Mongoose not leaning out PopulatedDoc by default - Mongoose has returned the full hydrated document for child in the following script since at least 7.0.0.

// child.ts
import {
  connection,
  Connection,
  HydratedDocument,
  Model,
  Schema,
  SchemaTypes,
  Types,
  PopulatedDoc,
  Document
} from 'mongoose';

export interface IChild {
  _id: Types.ObjectId;
  name: string;
}

type ChildDocumentOverrides = {};

export interface IChildVirtuals {
  id: string;
}

export type ChildInstance = HydratedDocument<
  IChild,
  ChildDocumentOverrides & IChildVirtuals
>;

type ChildModelType = Model<
  IChild,
  {},
  ChildDocumentOverrides,
  IChildVirtuals,
  ChildInstance
>;

export default (connection: Connection) => {
  const childSchema = new Schema<IChild, ChildModelType>(
    {
      name: {
        type: SchemaTypes.String,
        required: true,
        trim: true,
      },
    },
    {},
  );

  return connection.model<IChild, ChildModelType>('Child', childSchema);
};

export interface IParent {
  _id: Types.ObjectId;
  name: string;
  child: PopulatedDoc<Document<Types.ObjectId> & IChild>;
  createdAt: Date;
  updatedAt: Date;
}

type ParentDocumentOverrides = {};

export interface IParentVirtuals {
  id: string;
}

export type ParentInstance = HydratedDocument<
  IParent,
  ParentDocumentOverrides & IParentVirtuals
>;

type ParentModelType = Model<
  IParent,
  {},
  ParentDocumentOverrides,
  IParentVirtuals,
  ParentInstance
>;

function createParentModel (connection: Connection) {
  const parentSchema = new Schema<IParent, ParentModelType>(
    {
      name: {
        type: SchemaTypes.String,
        required: true,
        trim: true,
      },
      child: {
        type: SchemaTypes.ObjectId,
        ref: 'Child',
        required: true,
      },
    },
    {},
  );

  return connection.model<IParent, ParentModelType>('Parent', parentSchema);
};

const Parent = createParentModel(connection);

const test = (p: IParent) => {
  // TODO
};

(async function() {
  test(parent);
})();

You can confirm by using the following:

  const parent = await Parent.findOne().lean().orFail();
  // No compiler error in Mongoose 8.7.1, 8.5.0, 7.8.3, 7.0.0
  if (parent.child != null && !(parent.child instanceof Types.ObjectId)) {
    parent.child.$parent();
  }

We are working on improving #15072 and will hopefully have a fix for that in the next couple of days

[vkarpov15]

Fixed by #15103