Domo

Section

⚠️ This library generates code for structures that can bring suboptimal compilation times increased to approximately 20%

🔗 JSON parsing and validation example is in https://github.com/IvanRublev/contentful-elixir-parse-example-nestru-domo repo.

🔗 Commanded + Domo combo used in Event Sourcing and CQRS example app is in https://github.com/IvanRublev/bank-commanded-domo repo.

---

A library to ensure the consistency of structs modelling a business domain via their t() types and associated precondition functions.

Used in a struct's module, the library adds constructor, validation, and reflection functions. Constructor and validation functions guarantee the following at call time:

• A complex struct conforms to its t() type.
• Structs are validated to be consistent to follow given business rules by precondition functions associated with struct types.

If the conditions described above are not met, the constructor and validation functions return an error.

Because precondition function associates with type the validation can be shared across all structs referencing the type.

In terms of Domain Driven Design the invariants relating structs to each other can be defined with types and associated precondition functions.

---

Mix.install([:domo], force: true)

---

Let's say that we have a LineItem and PurchaseOrder structs with relating invariant that is the sum of line item amounts should be less then order's approved limit. That can be expressed like the following:

defmodule LineItem do
  use Domo

  defstruct amount: 0

  @type t :: %__MODULE__{amount: non_neg_integer()}
end

defmodule PurchaseOrder do
  use Domo

  defstruct id: 1000, approved_limit: 200, items: []

  @type id :: non_neg_integer()
  precond(id: &(1000 <= &1 and &1 <= 5000))

  @type t :: %__MODULE__{
          id: id(),
          approved_limit: pos_integer(),
          items: [LineItem.t()]
        }
  precond(t: &validate_invariants/1)

  defp validate_invariants(po) do
    amounts = po.items |> Enum.map(& &1.amount) |> Enum.sum()
    
    if amounts <= po.approved_limit do
      :ok
    else
      {:error, "Sum of line item amounts (#{amounts}) should be <= to approved limit (#{po.approved_limit})."}
    end
  end
end

Then PurchaseOrder struct can be constructed consistently with functions generated by Domo like the following:

PurchaseOrder.new()

{:ok, %PurchaseOrder{approved_limit: 200, id: 1000, items: []}}

The constructor function takes any Enumerable as the input value:

{:ok, po} = PurchaseOrder.new(%{approved_limit: 250})

{:ok, %PurchaseOrder{approved_limit: 250, id: 1000, items: []}}

It returns the descriptive keyword list if there is an error in input arguments. And it validates nested structs automatically:

PurchaseOrder.new(id: 500, items: [%LineItem{amount: -5}])

{:error,
 [
   items: "Invalid value [%LineItem{amount: -5}] for field :items of %PurchaseOrder{}. 
    Expected the value matching the [%LineItem{}] type.
    Underlying errors:
       - The element at index 0 has value %LineItem{amount: -5} that is invalid.
       - Value of field :amount is invalid due to Invalid value -5 for field :amount 
         of %LineItem{}. Expected the value matching the non_neg_integer() type.",
   id: "Invalid value 500 for field :id of %PurchaseOrder{}. Expected the 
    value matching the non_neg_integer() type. And a true value from 
    the precondition function \"&(1000 <= &1 and &1 <= 5000)\" 
    defined for PurchaseOrder.id() type."
 ]}

The returned errors are verbose and are intended for debugging purposes. See the User facing error messages section below for more options.

And manually updated struct can be validated like the following:

po 
|> Map.put(:items, [LineItem.new!(amount: 150)]) 
|> PurchaseOrder.ensure_type()

{:ok, %PurchaseOrder{approved_limit: 200, id: 1000, items: [%LineItem{amount: 150}]}}

Domo returns the error if the precondition function attached to the t() type that validates invariants for the struct as a whole fails:

updated_po = %{po | items: [LineItem.new!(amount: 180), LineItem.new!(amount: 100)]}
PurchaseOrder.ensure_type(updated_po)

{:error, [t: "Sum of line item amounts should be <= to approved limit"]}

Getting the list of the required fields of the struct that have type other then nil or any is like that:

PurchaseOrder.required_fields()

[:approved_limit, :id, :items]

See the Callbacks section for more details about functions added to the struct.

User facing error messages

It's possible to attach error messages to types with the precond macro to display them later to the user. To filter such kinds of messages, pass the maybe_filter_precond_errors: true option to Domo generated functions like that:

defmodule Book do
  use Domo

  defstruct [:title, :pages]

  @type title :: String.t()
  precond title: &(if String.length(&1) > 1, do: :ok, else: {:error, "Book title is required."})

  @type pages :: pos_integer()
  precond pages: &(if &1 > 2, do: :ok, else: {:error, "Book should have more then 3 pages. Given (#{&1})."})
  
  @type t :: %__MODULE__{title: nil | title(), pages: nil | pages()}
end

defmodule Shelf do
  use Domo
  
  defstruct books: []
  
  @type t :: %__MODULE__{books: [Book.t()]}
end

defmodule PublicLibrary do
  use Domo
  
  defstruct shelves: []
  
  @type t :: %__MODULE__{shelves: [Shelf.t()]}
end

library = struct!(PublicLibrary, %{shelves: [struct!(Shelf, %{books: [struct!(Book, %{title: "", pages: 1})]})]})

PublicLibrary.ensure_type(library, maybe_filter_precond_errors: true)

{:error,
 [
   shelves: [
     "Book title is required.",
     "Book should have more then 3 pages. Given (1)."
   ]
]}

That output contains only a flattened list of precondition error messages from the deeply nested structure.

Integration with Ecto

Ecto schema changeset can be automatically validated to conform to t() type and associated preconditions. Then the changeset function can be like the following:

defmodule Customer do
  use Ecto.Schema
  use Domo, ensure_struct_defaults: false

  import Ecto.Changeset
  import Domo.Changeset

  schema "customers" do
    field :first_name, :string
    field :last_name, :string
    field :birth_date, :date

    timestamps()
  end

  @type t :: %__MODULE__{
          first_name: String.t(),
          last_name: String.t(),
          birth_date: Date.t()
        }

  def changeset(changeset, attrs) do
    changeset
    |> cast(attrs, typed_fields())
    |> validate_required(required_fields())
    |> validate_type()
  end

  # Domo adds typed_fields/0, required_fields/0 funcitons to the schema.
  # Domo.Changeset defines validate_type/1 function.
end

See typed_fields/0, required_fields/0, and Domo.Changeset module documentation for details.

See detailed example is in the ./example_avialia project.

Integration with libraries generating t() type for a struct

Domo is compatible with most libraries that generate t() type for a struct or an Ecto schema. Just use Domo in the module, and that's it.

An advanced example is in the ./example_typed_integrations project.

Compile-time and Run-time validations

At the project's compile-time, Domo can perform the following checks:

• It automatically validates that the default values given with defstruct/1 conform to struct's type and fulfill preconditions.

• It ensures that the struct using Domo built with new!/1 function to be a function's default argument or a struct field's default value matches its type and preconditions.

Domo validates struct type conformance with appropriate TypeEnsurer modules built during the project's compilation at the application's run-time. These modules rely on guards and pattern matchings. See __using__/1 for more details.

Depending types tracking

Suppose the given structure field's type depends on a type defined in another module. When the latter type or its precondition changes, Domo recompiles the former module automatically to update its TypeEnsurer to keep type validation in current state.

That works similarly for any number of intermediate modules between module defining the struct's field and module defining the field's final type.

Setup

To use Domo in a project, add the following line to mix.exs dependencies:

{:domo, "~> 1.2.0"}

And the following line to the compilers:

compilers: Mix.compilers() ++ [:domo_compiler]

To avoid mix format putting extra parentheses around precond/1 macro call, add the following import to the .formatter.exs:

[
  import_deps: [:domo]
]

Setup for Phoenix hot reload

To enable hot reload for type changes in structs using Domo, add the following line to the endpoint's configuration in the config.exs file:

config :my_app, MyApp.Endpoint,
  reloadable_compilers: [:phoenix] ++ Mix.compilers() ++ [:domo_compiler]

Callbacks

Constructor, validation, and reflection functions added to the struct module using Domo.

new!/1/0

Creates a struct validating type conformance and preconditions.

The argument is any Enumerable that emits two-element tuples (key-value pairs) during enumeration.

Returns the instance of the struct built from the given enumerable. Does so only if struct's field values conform to its t() type and all field's type and struct's type precondition functions return ok.

Raises an ArgumentError if conditions described above are not fulfilled.

This function will check if every given key-value belongs to the struct and raise KeyError otherwise.

new/2/1/0

Creates a struct validating type conformance and preconditions.

The argument is any Enumerable that emits two-element tuples (key-value pairs) during enumeration.

Returns the instance of the struct built from the given enumerable in the shape of {:ok, struct_value}. Does so only if struct's field values conform to its t() type and all field's type and struct's type precondition functions return ok.

If conditions described above are not fulfilled, the function returns an appropriate error in the shape of {:error, message_by_field}. message_by_field is a keyword list where the key is the name of the field and value is the string with the error message.

Keys in the enumerable that don't exist in the struct are automatically discarded.

Options

• maybe_filter_precond_errors - when set to true, the values in message_by_field instead of string become a list of error messages from precondition functions. If there are no error messages from precondition functions for a field's type, then all errors are returned unfiltered. Helpful in taking one of the custom errors after executing precondition functions in a deeply nested type to communicate back to the user. F.e. when the field's type is another struct. Default is false.

ensure_type!/1

Ensures that struct conforms to its t() type and all preconditions are fulfilled.

Returns struct when it's valid. Raises an ArgumentError otherwise.

Useful for struct validation when its fields changed with map syntax or with Map module functions.

ensure_type/2/1

Ensures that struct conforms to its t() type and all preconditions are fulfilled.

Returns struct when it's valid in the shape of {:ok, struct}. Otherwise returns the error in the shape of {:error, message_by_field}.

Useful for struct validation when its fields changed with map syntax or with Map module functions.

typed_fields/1/0

Returns the list of struct's fields defined with explicit types in its t() type spec.

Does not return meta fields with __underscored__ names and fields having any() type by default.

Includes fields that have nil type into the return list.

Options

• :include_any_typed - when set to true, adds fields with any() type to the return list. Default is false.

• :include_meta - when set to true, adds fields with __underscored__ names to the return list. Default is false.

required_fields/1/0

Returns the list of struct's fields having type others then nil or any().

Does not return meta fields with __underscored__ names.

Useful for validation of the required fields for emptiness. F.e. with validate_required/2 call in the Ecto changeset.

Options

• :include_meta - when set to true, adds fields with __underscored__ names to the return list. Default is false.

Limitations

The recursive types like @type t :: :end | {integer, t()} are not supported. Because of that types like Macro.t() or Path.t() are not supported.

Parametrized types are not supported. Library returns {:type_not_found, :key} error for @type dict(key, value) :: [{key, value}] type definition. Domo returns error for type referencing parametrized type like @type field :: container(integer()).

Generated submodule with TypedStruct's :module option is not supported.

Migration

To complete the migration to a new version of Domo, please, clean and recompile the project with mix clean --deps && mix compile command.

Adoption

It's possible to adopt Domo library in the project having user-defined constructor functions as the following:

1. Add :domo dependency to the project, configure compilers as described in the setup section
2. Set the name of the Domo generated constructor function by adding config :domo, :name_of_new_function, :constructor_name option into the confix.exs file, to prevent conflict with original constructor function names if any
3. Add use Domo to existing struct
4. Change the calls to build the struct for Domo generated constructor function with name set on step 3 and remove original constructor function
5. Repeat for each struct in the project

Performance 🐢

On the average, the current version of the library makes struct operations about 20% sower what may seem plodding. And it may look like non-performant to run in production.

It's not that. The library ensures the correctness of data types at runtime and it comes with the price of computation. As the result users get the application with correct states at every update that is valid in many business contexts.

The output of mix benchmark is following.

Generate 10000 inputs, may take a while.
=========================================

Construction of a struct
=========================================
Operating System: macOS
CPU Information: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.13.0
Erlang 24.1.5

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 14 s

Benchmarking __MODULE__.new!(arg)...
Benchmarking struct!(__MODULE__, arg)...

Name                               ips        average  deviation         median         99th %
struct!(__MODULE__, arg)       13.72 K       72.88 μs    ±64.17%          73 μs         168 μs
__MODULE__.new!(arg)           11.33 K       88.24 μs    ±50.65%          91 μs         177 μs

Comparison: 
struct!(__MODULE__, arg)       13.72 K
__MODULE__.new!(arg)           11.33 K - 1.21x slower +15.36 μs

A struct's field modification
=========================================
Operating System: macOS
CPU Information: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
Number of Available Cores: 8
Available memory: 16 GB
Elixir 1.13.0
Erlang 24.1.5

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 14 s

Benchmarking %{tweet | user: arg} |> __MODULE__.ensure_type!()...
Benchmarking struct!(tweet, user: arg)...

Name                                                        ips        average  deviation         median         99th %
struct!(tweet, user: arg)                               15.16 K       65.96 μs    ±65.51%          66 μs         148 μs
%{tweet | user: arg} |> __MODULE__.ensure_type!()       13.26 K       75.44 μs    ±59.84%          76 μs         163 μs

Comparison: 
struct!(tweet, user: arg)                               15.16 K
%{tweet | user: arg} |> __MODULE__.ensure_type!()       13.26 K - 1.14x slower +9.48 μs

Contributing

1. Fork the repository and make a feature branch

2. After implementing of the feature format the code with:

   mix format
   

   run linter and tests to ensure that all works as expected with:

   mix check || mix check --failed
   

3. Make a PR to this repository

Changelog

1.5.1

• Fix to detect mix compile with more reliable Code.can_await_module_compilation?

• Fix to make benchmark run again as sub-project

• Make :maybe_filter_precond_errors option to lift precondition error messages from the nested structs

1.5.0

• Fix bug to return explicit file read error message during the compile time

• Completely replace apply() with . for validation function calls to run faster

• Link planner server to mix process for better state handling

• Support of the interactive use in iex and live book

Breaking change:

• Improve compilation speed by starting resolve planner only once in Domo mix task. To migrate, please, put the :domo_compiler before :elixir in mix.exs. And do the same for reloadable_compilers key in config file if configured for Phoenix endpoint.

1.4.1

• Improve compatibility with Elixir v1.13

• Format string representations of an anonymous function passed to precond/1 macro error message

1.4.0

• Fix bug to detect runtime mode correctly when launched under test.

• Add support for @opaque types.

Breaking changes:

• Change new_ok constructor function name to new that is more convenient. Search and replace new_ok( -> new( in all files of the project using Domo to migrate.

• Constructor function name generation procedure changes to adding ! to the value of :name_of_new_function option. The defaults are new and new!.

1.3.4

• Make error messages to be more informative

• Improve compatibility with Ecto 3.7.x

• Explicitly define :ecto and :decimal as optional dependencies

• Fix bug to pass :remote_types_as_any option with use Domo

• Explicitly define that MapSet should be validated with precond function for custom user type, because parametrized t(value) types are not supported

• Replace apply() with Module.function calls to run faster

1.3.3

• Support validation of Decimal.t()

• Fix bug to define precondition function for user type referencing any() or term()

1.3.2

• Support remote types in erlang modules like :inet.port_number()

• Shorten the invalid value output in the error message

• Increase validation speed by skipping fields that are not in t() type spec or have the any() type

• Fix bug to skip validation of struct's enforced keys default value because they are ignored during the construction anyway

• Increase validation speed by generating TypeEnsurer modules for Date, Date.Range, DateTime, File.Stat, File.Stream, GenEvent.Stream, IO.Stream, Macro.Env, NaiveDateTime, Range, Regex, Task, Time, URI, and Version structs from the standard library at the first project compilation

• Fix bug to call the precond function of the user type pointing to a struct

• Increase validation speed by encouraging to use Domo or to make a precond function for struct referenced by a user type

• Add Domo.has_type_ensurer?/1 that checks whether a TypeEnsurer module was generated for the given struct.

• Add example of parsing with validating of the Contentful JSON reply via Jason + ExJSONPath + Domo

1.3.1

• Fix bug to validate defaults having | nil type.

1.3.0

• Change the default name of the constructor function to new! to follow Elixir naming convention. You can always change the name with the config :domo, :name_of_new_function, :new_func_name_here app configuration.

• Fix bug to validate defaults for every required field in a struct except __underscored__ fields at compile-time.

• Check whether the precondition function associated with t() type returns true at compile time regarding defaults correctness check.

• Add examples of integrations with TypedStruct and TypedEctoSchema.

1.2.9

• Fix bug to acknowledge that type has been changed after a failed compilation.

• Fix bug to match structs not using Domo with a field of any() type with and without precondition.

• Add typed_fields/1 and required_fields/1 functions.

• Add maybe_filter_precond_errors: true option that filters errors from precondition functions for better output for the user.

1.2.8

• Add Domo.Changeset.validate_type/* functions to validate Echo.Changeset field changes matching the t() type.

• Fix the bug to return custom error from precondition function as underlying error for :| types.

1.2.7

• Fix the bug to make recompilation occur when fixing alias for remote type.

• Support custom errors to be returned from functions defined with precond/1.

1.2.6

• Validates type conformance of default values given with defstruct/1 to the struct's t() type at compile-time.

• Includes only the most matching type error into the error message.

1.2.5

• Add remote_types_as_any option to disable validation of specified complex remote types. What can be replaced by precondition for wrapping user-defined type.

1.2.4

• Speedup resolving of struct types
• Limit the number of allowed fields types combinations to 4096
• Support Range.t() and MapSet.t()
• Keep type ensurers source code after compiling umbrella project
• Remove preconditions manifest file on mix clean command
• List processed structs giving mix --verbose option

1.2.3

• Support struct's attribute introduced in Elixir 1.12.0 for error checking
• Add user-defined precondition functions to check the allowed range of values with precond/1 macro

1.2.2

• Add support for new/1 calls at compile time f.e. to specify default values

1.2.1

• Domo compiler is renamed to :domo_compiler
• Compile TypeEnsurer modules only if struct changes or dependency type changes
• Phoenix hot-reload with :reloadable_compilers option is fully supported

1.2.0

• Resolve all types at compile time and build TypeEnsurer modules for all structs
• Make Domo library work with Elixir 1.11.x and take it as the required minimum version
• Introduce ---/2 operator to make tag chains with Domo.TaggedTuple module

0.0.x - 1.0.x

• MVP like releases, resolving types at runtime. Adds new constructor to a struct

Roadmap

Roadmap

• Check if the field values passed as an argument to the new/1, and put/3 matches the field types defined in typedstruct/1.

• Support the keyword list as a possible argument for the new/1.

• Add module option to put a warning in the console instead of raising of the ArgumentError exception on value type mismatch.

• Make global environment configuration options to turn errors into warnings that are equivalent to module ones.

• Move type resolving to the compile time.

• Keep only bare minimum of generated functions that are new/1, ensure_type!/1 and their _ok versions.

• Make the new/1 and ensure_type!/1 speed to be less or equal to 1.5 times of the struct!/2 speed.

• Support new/1 calls in macros to specify default values f.e. in other structures. That is to check if default value matches type at compile time.

• Support precond/1 macro to specify a struct field value's contract with a boolean function.

• Support types referencing itself for tree structures.

• Evaluate full recompilation time for 1000 structs using Domo.

• Add use option to specify names of the generated functions.

• Add documentation to the generated for new(_ok)/1, and ensure_type!(_ok)/1 functions in a struct.

License

Copyright © 2021 Ivan Rublev

This project is licensed under the MIT license.