Notes on/for v3 / v4

[tunnckoCore]

workspaces/

• formidable
• formidable-plugins?
• formidable-helpers
• formidable-mini - an experimental minimal & modern idea
• formidable-utils
• formidable-storage
  • memory (VolatileFile)
  • disk (PersistentFile)
  • serverless (formidable-serverless)

misc

• minimize dep weight
• monorepo (turborepo /nx.dev + lerna ^5.1 + yarn workspaces) - Hela v6
• written in typescript, drop / deprecated external packages
• kodiak - merge automation & better PRs and commits (https://kodiakhq.com/)
• Dependabot, make it auto-update and auto-merge all versions and branches (latest-2, latest, latest-3, next-2, next-3)
• GitHub CodeQL action
• go async/await and promises; possibly drop callback APIs entirely
• document Release Process, Lines, and dist-tags & branches
  • latest-1, latest-2, next-2, latest-3, latest-4, next-4);
  • latest as master
• better / real benchmarks
• analyze old tests & rethink and probably drop them
• some benchmarks here at busboy v1 - Changes in v1.0.0 mscdex/busboy#266
• standard Web API File

[GrosSacASac]

go async/await and promises; possibly drop callback APIs entirely

Maybe do a mix in v3 and in v4 drop callback

[tunnckoCore]

New FormidableFile, based on fetch-blob and Web API File. Plus, we'll always make temporary files on tmp dir, for the time the process is running. When the process ends, they are cleared (all that is handled by fetch-blob). The user will still be able to to save on disk or to do that file whatever it wants. Something like opts.save on Formidable Options, will mean that we will save the data of that temporary created file to the disk on the user-chosen (or default) upload dir.

It handles:

• random filename generation (default) with cuid
• file rename function opts.rename
• filename sanitization, ALWAYS, with filenamifyPath (with opts.replacement: '-' instead of the default !)
  • whether or not you use custom rename function, the result is always passed to sanitization
  • not yet sure if we should do that when renamer function is passed

/**
 * @typedef {import('filenamify').Options} FilenamifyOptions
 * @typedef {import('node:path').ParsedPath} ParsedPath
 *
 * @typedef FormidableFileOptions
 * @property {string} [type] - mime type from header
 * @property {number} [lastModified] - file mtimeMs, default `Date.now()`
 * @property {number} [size] - file size in bytes
 * @property {boolean} [randomName] - generate random filename, default `true` using `cuid`
 * @property {FilenamifyOptions} [filenamify] - options passed to `filenamify`
 * @property {RenamerFn} [rename] - filename renaming function
 * @property {{[key: string]: unknown}} [data] - any additional data or way of storing things
 *
 * @typedef {(filepath: string, contents: unknown, options: FormidableFileOptions) => string} RenamerFn
 * must be synchronous function
 */

import path from 'node:path';
import cuid from 'cuid';
import { filenamifyPath } from 'filenamify';

/**
 * @typedef {{type?: string, name: string, lastModified: number}} File
 * standard Web API File
 */
// @ts-ignore
import { File } from 'fetch-blob/file.js';

export class FormidableFile extends File {
  /**
   *
   * @param {string} filepath - filename from header
   * @param {unknown} contents - any file content value
   * @param {FormidableFileOptions} [options]
   */
  constructor(filepath, contents, options) {
    const { file, opts, fp } = normalize(filepath, contents, options);

    super(contents, file.base, {
      // @ts-ignore
      type: opts.mimetype || opts.mime || opts.type,
      lastModified: opts.lastModified || Date.now(),
    });

    /**
     * @type {string} - full filepath
     */
    // we assign this here, because to not mess with ParsedPath `file`
    this.path = fp;

    // eslint-disable-next-line no-underscore-dangle
    this._updateProps(file, contents, opts);
  }

  /**
   *
   * @param {ParsedPath} file - custom file
   * @param {unknown} contents - any file content value
   * @param {FormidableFileOptions} [options]
   */
  // eslint-disable-next-line no-underscore-dangle
  _updateProps(file, contents, options) {
    // it's already normalized
    // const opts = normalizeOptions(options);

    this.basename = file.base;
    this.dirname = file.dir;
    this.extname = file.ext;

    this.stem = file.name; // this.stem == this.name == file.name == basename without extname
    this.mime = options.type;

    if (options.size) {
      // @ts-ignore
      this.size = options.size;
    }

    Reflect.defineProperty(this, 'contents', { value: contents });
    Reflect.defineProperty(this, 'options', { value: options });
    Reflect.defineProperty(this, 'base', { value: file.base });
    Reflect.defineProperty(this, 'dir', { value: file.dir });
    Reflect.defineProperty(this, 'ext', { value: file.ext });
    Reflect.defineProperty(this, 'data', { value: options.data });
  }
}

/**
 *
 * @param {string} filepath - filename from header
 * @param {unknown} contents - any file content value
 * @param {FormidableFileOptions} [options]
 *
 */
function normalize(filepath, contents, options) {
  const opts = normalizeOptions(options);

  /**
   * @type {string} fp
   */
  const fp = filenamifyPath(
    // @ts-ignore
    opts.rename(filepath, contents, opts),
    opts.filenamify,
  );

  /**
   *
   * @type {ParsedPath} file
   */
  const file = path.parse(fp);

  return { file, opts, fp };
}

/**
 * @param {{[key: string]: unknown} & FormidableFileOptions} [options]
 * @returns {FormidableFileOptions}
 */
function normalizeOptions(options = {}) {
  const opts = {
    filenamify: { replacement: '-', ...options.filenamify },
    randomName: true,
    data: { ...options.data },
    ...options,
  };

  /**
   * @property {RenamerFn}
   */
  opts.rename = (f, c, o) => {
    if (typeof opts.rename === 'function') {
      return opts.rename(f, c, o);
    }
    return opts.randomName ? cuid() : f;
  };

  return opts;
}

/**
 * examples
 */

const virtualFile = new FormidableFile(
  './foo/bar/qu_sas<mail@at.com>zab.gz<script onchange="alert(1)"></script>sax.png',
  await fs.readFile('./examples/10k.json'),
  {
    type: 'application/json',
    randomName: true,
    data: { sasa: 1 },
  },
);

console.log(virtualFile);

Ideally it will be called from part.on('data') in the future, but for now we can add it at _newFile, then we can use createTemporaryBlob to make that temp file and pass the return to the user.

Something like that:

part.on('data', async () => {
  let file = new FormidableFile(nameFromHeader, dataFromParser, {
    type: contentTypeFromParser,
    rename: formidableOptions.rename,
  });
  const blob = await createTemporaryBlob(
    file.contents,
    file.type,
    file.options.signal,
  );

  file.blob = blob;
  this.emit('file', file, blob);

  if (formidableOptions.save) {
    await fs.writeFile(file.path, file.contents);
  }
});

We are using createTemporaryBlob instead of createTemporaryFile because we already made a file instance of FormidableFile which extends File.

The reason of having our own File implementation on top, is that it's for useful properties that we need to pass to the users. Plus I always seeing it that way since I got in here in v1. The properties of it is based on enough battle-tested vfile (remark, rehype, unified) and vinyl (gulp), plus of course the minimalism of File.

Currently text(), stream() and those methods, are missing, but we probably should implement them too.

The whole thing can be introduced in v3 (and require node 14) and backport to v2 (without the temp file creation thing and without fetch-blob).

The rest of the interesting stuff like iterators and new APIs goes to v4.

[tunnckoCore]

Okay, so the @jimmywarting's cleaned Formidable parser, which is on the node-fetch is almost 2x faster than the current one.

Based on the busboys benchmark bench-multipart-files-100mb-big the old one is avg 3.7s, now it's 1.7s

[jimmywarting]

wow, didn't know that mine was 2x faster. did you come to any conclusion of why that is?

[tunnckoCore]

Not yet. I'm not in parsers and that parser looks like magic to me LOL.

Btw, to make a note: it's faster when use just the class, but not the toFormData. And when you start putting things on top of all that it gets slower. Busboy v1 really does a great job at the very low level.

I'll commit updated benchmarks soon.

[jimmywarting]

fetch-blob@3.2.0 is publised, you can now use createTemporaryBlob or createTemporaryFile if you like

[jimmywarting]

btw, formdata is built into NodeJS core now... maybe you can do a conditional import like use some formdata package like formdata-polyfill (used by node-fetch) or formdata-node (both are spec compatible) if it isn't avalible on global namespace

so something like const FormData = globalThis.FormData || await import('formdata-polyfill/esm.min.js')
just so happen to know that NodeJS (undici's FormData imp accepts 3th party blobs like fetch-blob as well - not just only instances of globalThis.Blob)

fetch-blob is still going to stick around until nodejs/node#37340 has been resolved by nodejs itself

[jimmywarting]

Was a super long time ago since i even used gulp and i have never heard of vinyl or filev before... honestly just wished they followed Web File IDE structure instead...

vinyl should also impl text, arrayBuffer and stream() 👍

[tunnckoCore]

they followed Web File IDE structure instead

Well, yea. But they predated everything i think, that's why.

I built some formidable-mini on top of your fetch-blob and formdata-polyfill, you can check it here, it's pretty clean, small and a beauty, haha. There FormidableFile is just a Web File, or anyone can just pass the return stadnard FormData to their desired place :P

ah.. what a times..

I'm open for ideas and feedback. Not sure how to use these createTemporary* yet..