Handling relationships

[RiftLurker]

Are there any examples on how relationships between Collections are supposed to be handled? The configs prepare hook briefly mentions them but doesn't give any more details.
It's definitely possible to add them in there but this sounds like it would get cumbersome and be prone to errors.

Ideally I would like to define them directly in the schema, similar to Astro's content references to easily ensure data integrity.

[zce]

First of all, Velite does not want to intrude into the user's runtime code. Its purpose is only the intermediate process from content to data layer. Velite will not interfere how users use the data generated by Velite.

e.g.

const authors = {
  name: 'Author',
  pattern: 'authors/index.yml',
  schema: s
    .object({
      name: s.unique('authors'),
      slug: slug,
      avatar: s.image().optional()
    })
}

const posts = {
  name: 'Post',
  pattern: 'posts/**/*.md',
  schema: s
    .object({
      title: title,
      slug: s.slug('post'),
      description: paragraph.optional(),
      author: s.string(), // <= author.name
      content: s.markdown()
    })
}

in your app

import { posts, authors } from '.velite'

const hydratedPosts = posts.map(item => ({ ... posts, author: authors.find(a => a.name === item.author) })

You can query the data in a functional way, and you can even let Velite help you write it to the database and then use SQL to query the data.

Secondly, from the perspective of the data source, dealing with related data is indeed a common requirement. I am also considering adding a s.reference() schema to fulfill this need.

[zce]

here is my previous helper for access generated data

import { authors, categories, pages, plans, posts, tags } from '#/.velite'

import type { Author, Category, Page, Plan, Post, Tag } from '#/.velite'

type Taxonomy = {
  authors: { [P in 'name' | 'slug' | 'email' | 'avatar' | 'bio' | 'permalink']: Author[P] }[]
  categories: { [P in 'name' | 'slug' | 'permalink']: Category[P] }[]
  tags: { [P in 'name' | 'slug' | 'permalink']: Tag[P] }[]
}

type Filter<T> = (value: T, index: number, array: T[]) => boolean
type Sorter<T> = (a: T, b: T) => number

const available = (item: { draft: boolean; private: boolean }) => process.env.NODE_ENV !== 'production' || (!item.draft && !item.private)

export const filters = {
  none: (): boolean => true,
  featured: (item: { featured: boolean }) => item.featured
}

export const sorters = {
  dateAsc: <I extends { date: string }>(a: I, b: I): number => (a.date > b.date ? 1 : -1),
  dateDesc: <I extends { date: string }>(a: I, b: I): number => (a.date > b.date ? -1 : 1),
  nameAsc: <I extends { name: string }>(a: I, b: I): number => (a.name > b.name ? 1 : -1),
  nameDesc: <I extends { name: string }>(a: I, b: I): number => (a.name > b.name ? -1 : 1),
  priceAsc: <I extends { prices: { yearly: number } }>(a: I, b: I): number => (a.prices.yearly > b.prices.yearly ? 1 : -1),
  priceDesc: <I extends { prices: { yearly: number } }>(a: I, b: I): number => (a.prices.yearly > b.prices.yearly ? -1 : 1),
  countAsc: <I extends { count: { total: number } }>(a: I, b: I): number => (a.count.total > b.count.total ? 1 : -1),
  countDesc: <I extends { count: { total: number } }>(a: I, b: I): number => (a.count.total > b.count.total ? -1 : 1),
  titleAsc: <I extends { title: string }>(a: I, b: I): number => (a.title > b.title ? 1 : -1),
  titleDesc: <I extends { title: string }>(a: I, b: I): number => (a.title > b.title ? -1 : 1)
}

const pick = <T extends object, K extends keyof T>(obj: T, keys?: K[]): { [P in K]: T[P] } => {
  if (keys == null) return obj
  return Object.fromEntries(keys.map(k => [k, obj[k]])) as { [P in K]: T[P] }
}

const include = async <I extends keyof Taxonomy = never>(data: { [P in keyof Taxonomy]: string[] }, includes?: I[]): Promise<{ [P in I]: Taxonomy[P] }> => {
  if (includes == null) return {} as { [P in I]: Taxonomy[P] }
  const entities = await Promise.all(
    includes.map(async include => {
      if (include === 'authors') {
        return [
          include,
          (await getAuthors(['name', 'slug', 'email', 'avatar', 'bio', 'permalink'], i => data.authors.includes(i.name))) satisfies Taxonomy['authors']
        ]
      } else if (include === 'categories') {
        return [include, (await getCategories(['name', 'slug', 'permalink'], i => data.categories.includes(i.name))) satisfies Taxonomy['categories']]
      } else if (include === 'tags') {
        return [include, (await getTags(['name', 'slug', 'permalink'], i => data.tags.includes(i.name))) satisfies Taxonomy['tags']]
      }
      return [include, []]
    })
  )
  return Object.fromEntries(entities)
}

export const getAuthors = async <F extends keyof Author>(
  fields?: F[],
  filter: Filter<Author> = filters.none,
  limit: number = Infinity,
  offset: number = 0
): Promise<{ [P in F]: Author[P] }[]> => {
  return authors
    .filter(filter)
    .sort((a, b) => (a.count.total > b.count.total ? -1 : 1))
    .slice(offset, offset + limit)
    .map(author => pick(author, fields))
}

export const getAuthorsCount = async (filter: Filter<Author> = filters.none): Promise<number> => {
  return authors.filter(filter).length
}

export const getAuthor = async <F extends keyof Author>(filter: Filter<Author>, fields?: F[]): Promise<{ [P in F]: Author[P] } | undefined> => {
  const author = authors.find(filter)
  return author && pick(author, fields)
}

export const getAuthorByName = async <F extends keyof Author>(name: string, fields?: F[]): Promise<{ [P in F]: Author[P] } | undefined> => {
  return getAuthor(i => i.name === name, fields)
}

export const getAuthorBySlug = async <F extends keyof Author>(slug: string, fields?: F[]): Promise<{ [P in F]: Author[P] } | undefined> => {
  return getAuthor(i => i.slug === slug, fields)
}

export const getCategories = async <F extends keyof Category>(
  fields?: F[],
  filter: Filter<Category> = filters.none,
  limit: number = Infinity,
  offset: number = 0
): Promise<{ [P in F]: Category[P] }[]> => {
  return categories
    .filter(filter)
    .sort((a, b) => (a.count.total > b.count.total ? -1 : 1))
    .slice(offset, offset + limit)
    .map(author => pick(author, fields))
}

export const getCategoriesCount = async (filter: Filter<Category> = filters.none): Promise<number> => {
  return categories.filter(filter).length
}

export const getCategory = async <F extends keyof Category>(filter: Filter<Category>, fields?: F[]): Promise<{ [P in F]: Category[P] } | undefined> => {
  const category = categories.find(filter)
  return category && pick(category, fields)
}

export const getCategoryByName = async <F extends keyof Category>(name: string, fields?: F[]): Promise<{ [P in F]: Category[P] } | undefined> => {
  return getCategory(i => i.name === name, fields)
}

export const getCategoryBySlug = async <F extends keyof Category>(slug: string, fields?: F[]): Promise<{ [P in F]: Category[P] } | undefined> => {
  return getCategory(i => i.slug === slug, fields)
}

export const getTags = async <F extends keyof Tag>(
  fields?: F[],
  filter: Filter<Tag> = filters.none,
  limit: number = Infinity,
  offset: number = 0
): Promise<{ [P in F]: Tag[P] }[]> => {
  return tags
    .filter(filter)
    .sort((a, b) => (a.count.total > b.count.total ? -1 : 1))
    .slice(offset, offset + limit)
    .map(tag => pick(tag, fields))
}

export const getTagsCount = async (filter: Filter<Tag> = filters.none): Promise<number> => {
  return tags.filter(filter).length
}

export const getTag = async <F extends keyof Tag>(filter: Filter<Tag>, fields?: F[]): Promise<{ [P in F]: Tag[P] } | undefined> => {
  const tag = tags.find(filter)
  return tag && pick(tag, fields)
}

export const getTagByName = async <F extends keyof Tag>(name: string, fields?: F[]): Promise<{ [P in F]: Tag[P] } | undefined> => {
  return getTag(i => i.name === name, fields)
}

export const getTagBySlug = async <F extends keyof Tag>(slug: string, fields?: F[]): Promise<{ [P in F]: Tag[P] } | undefined> => {
  return getTag(i => i.slug === slug, fields)
}

export const getPages = async <F extends keyof Page>(
  fields?: F[],
  filter: Filter<Page> = filters.none,
  sorter: Sorter<Page> = sorters.titleAsc,
  limit: number = Infinity,
  offset: number = 0
): Promise<{ [P in F]: Page[P] }[]> => {
  return pages
    .filter(available)
    .filter(filter)
    .sort(sorter)
    .slice(offset, offset + limit)
    .map(page => pick(page, fields))
}

export const getPagesCount = async (filter: Filter<Page> = filters.none): Promise<number> => {
  return pages.filter(available).filter(filter).length
}

export const getPage = async <F extends keyof Page>(filter: Filter<Page>, fields?: F[]): Promise<{ [P in F]: Page[P] } | undefined> => {
  const page = pages.find(filter)
  return page && pick(page, fields)
}

export const getPageBySlug = async <F extends keyof Page>(slug: string, fields?: F[]): Promise<{ [P in F]: Page[P] } | undefined> => {
  return getPage(i => i.slug === slug, fields)
}

export const getPlans = async <F extends keyof Plan>(
  fields?: F[],
  filter: Filter<Plan> = filters.none,
  sorter: Sorter<Plan> = sorters.priceAsc,
  limit: number = Infinity,
  offset: number = 0
): Promise<{ [P in F]: Plan[P] }[]> => {
  return plans
    .filter(available)
    .filter(filter)
    .sort(sorter)
    .slice(offset, offset + limit)
    .map(plan => pick(plan, fields))
}

export const getPlansCount = async (filter: Filter<Plan> = filters.none): Promise<number> => {
  return plans.filter(available).filter(filter).length
}

export const getPlan = async <F extends keyof Plan>(filter: Filter<Plan>, fields?: F[]): Promise<{ [P in F]: Plan[P] } | undefined> => {
  const plan = plans.find(filter)
  return plan && pick(plan, fields)
}

export const getPlanBySlug = async <F extends keyof Plan>(slug: string, fields?: F[]): Promise<{ [P in F]: Plan[P] } | undefined> => {
  return getPlan(i => i.slug === slug, fields)
}

export const getPosts = async <F extends keyof Omit<Post, I>, I extends keyof Taxonomy = never>(
  fields?: F[],
  includes?: I[],
  filter: Filter<Post> = filters.none,
  sorter: Sorter<Post> = sorters.dateDesc,
  limit: number = Infinity,
  offset: number = 0
): Promise<({ [P in F]: Post[P] } & { [P in I]: Taxonomy[P] })[]> => {
  return Promise.all(
    posts
      .filter(available)
      .filter(filter)
      .sort(sorter)
      .slice(offset, offset + limit)
      .map(async post => ({ ...pick(post, fields), ...(await include(post, includes)) }))
  )
}

export const getPostsCount = async (filter: Filter<Post> = filters.none): Promise<number> => {
  return posts.filter(available).filter(filter).length
}

export const getPost = async <F extends keyof Omit<Post, I>, I extends keyof Taxonomy = never>(
  filter: Filter<Post>,
  fields?: F[],
  includes?: I[]
): Promise<({ [P in F]: Post[P] } & { [P in I]: Taxonomy[P] }) | undefined> => {
  const post = posts.find(filter)
  return post && { ...pick(post, fields), ...(await include(post, includes)) }
}

export const getPostBySlug = async <F extends keyof Omit<Post, I>, I extends keyof Taxonomy = never>(
  slug: string,
  fields?: F[],
  includes?: I[]
): Promise<({ [P in F]: Post[P] } & { [P in I]: Taxonomy[P] }) | undefined> => {
  return getPost(i => i.slug === slug, fields, includes)
}

[RiftLurker]

I realize I wasn't quite clear in my initial message as I'm definitely most interested in the s.reference schema you mentioned. My brief attempts at implementing it myself didn't quite work out as well as I wanted due to cyclical type references.
Thank you for the snippet though.

[RiftLurker]

I've got a somewhat working solution, it is still very far from perfect but at least it'll let me start working on my project without having to worry about messed up data integrity.

import { ZodType, logger } from "velite";

let CACHE: Map<string, string> | null = null;

const PREFIX_REFERENCE_KEY = "references:key";
const PREFIX_REFERENCE_VALUE = "references:value";

export default function createReference(
  reference: string,
  key: ZodType,
  ref: ZodType,
): [ZodType, ZodType] {
  return [
    key.superRefine((value, { meta }) => {
      const { cache } = meta.config;
      cache.set(`${PREFIX_REFERENCE_KEY}:${reference}:${value}`, meta.path);

      // cache is not accessible in prepare hook
      if (CACHE === null) {
        CACHE = cache;
      }
    }),
    ref.superRefine((value, { meta }) => {
      const { cache } = meta.config;
      cache.set(`${PREFIX_REFERENCE_VALUE}:${reference}:${value}`, meta.path);

      // cache is not accessible in prepare hook
      if (CACHE === null) {
        CACHE = cache;
      }
    }),
  ];
}

export function validateReferences() {
  let error = false;

  for (const [key, file] of CACHE?.entries() ?? []) {
    if (!key.startsWith(PREFIX_REFERENCE_VALUE)) {
      continue;
    }

    const [_0, _1, reference, value] = key.split(":");
    if (!CACHE?.get(`${PREFIX_REFERENCE_KEY}:${reference}:${value}`)) {
      logger.error(
        `Referenced key ${value} does not exist for reference ${reference} (file: ${file})`,
      );
      error = true;
    }
  }

  if (error) {
    return false;
  }
}

And it's then being used like this:

const [authorKey, authorRef] = createReference(
  'authors',
  s.slug('authors'), // type of the key
  s.string(), // type of the reference
)

const authors = {
  name: 'Author',
  pattern: 'authors/index.yml',
  schema: s
    .object({
      name: s.unique('authors'),
      slug: authorKey,
      avatar: s.image().optional(),
    }),
};

const posts = {
  name: 'Post',
  pattern: 'posts/**/*.md',
  schema: s
    .object({
      title: title,
      slug: s.slug('post'),
      description: paragraph.optional(),
      author: authorRef,
      content: s.markdown(),
    }),
};

validateReferences is then being called as the prepare hook in the config:

export default defineConfig({
  // [...collections]
  prepare: validateReferences,
});

I'm still looking forward to an official solution on how to handle references!