[10.x] Add insertWithCasts() to Eloquent Builder

[cosmastech]

The Problem

The method is designed to cast properties, set default values if not supplied, and set the models timestamps where applicable.

The presenting problem is spelled out more fully here in #46791, though it only mentions the pain point of created_at/updated_at timestamps. I have personally either had to write code like this within multiple projects or suffered the consequences of forgetting that Model@insert() doesn't do any handy casting or setting of defaults.

How it would be done today

For context let's consider this model (and associated enum).

// App/Enums/Genre
enum Genre: string
{
    case Rock = 'rock';

    case HipHop = 'hip-hop';
}

// App/Models/Album
use Illuminate\Database\Eloquent\Model;

class Album extends Model
{
    public $timestamps = true;

    protected $fillable = [
        'artist',
        'title',
        'genre',
        'release_date',
        'stats',
    ];

    protected $casts = [
        'genre' => Genre::class,
        'release_date' => 'datetime',
        'stats' => 'json',
    ];
    
    protected $attributes = [
        'artist' => 'unknown',
    ];
}

Say we have an action that can accept data from a form or elsewhere.

use App\Enums\Genre;
use App\Models\Album;
use Carbon\Carbon;

$inputData = [
    ['artist' => 'Built to Spill', 'title' => 'Keep It Like a Secret', 'genre' => Genre::Rock, 'release_date' => '1999-02-02', 'stats' => []],
    ['artist' => 'Tom Petty', 'title' => 'Highway Companion', 'genre' => 'rock', 'release_date' => new Carbon('2006-07-25'), 'stats' => ['producer' => 'Jeff Lynne']],
    ['title' => 'Ill Communication', 'genre' => Genre::HipHop, 'release_date' => null, 'stats' => new stdClass]
];
$values = collect($inputData)->map(function(array $values) {
    // make sure we have a valid enum case
    if (! $values['genre'] instanceof Genre) {
        $values['genre'] = Genre::tryFrom($values['genre']);
    }
    $values['genre'] = $values['genre']?->value;

    // make sure the date is formatted for DB layer
    if ($values['release_date'] instanceof \DateTime) {
        $values['release_date'] = $values['release_date']->format('Y-m-d');
    }
    
    // make sure that the stats column is properly encoded to a string
    if (!is_string($values['stats'])) {
        $values['stats'] = json_encode($values['stats']);
    }
    
    // set the model's default value defined in Artist::$attributes
    if (!isset($values['artist'])) {
        $values['artist'] = \App\Models\Album::newModelInstance()->artist; // get the default value from the model
    }

    return array_merge($values, [
        'created_at' => now(),
        'updated_at' => now()
    ]);
});

\App\Models\Album::insert($values->toArray());

Pretty messy for something that feels standard. We can just create them one by one, but then we have repeated writes to the database. For a large database with many connections or if this whole Action is happening within a request lifecycle, that is not an ideal solution.

Solution

We add an Eloquent\Builder@insertWithCasts() method that makes the above much less painful.

The above example relying on insertWithCasts()

use App\Models\Album;
use App\Enums\Genre;

Album::insertWithCasts([
    ['artist' => 'Built to Spill', 'title' => 'Keep It Like a Secret', 'genre' => Genre::Rock, 'release_date' => '1999-02-02', 'stats' => []],
    ['artist' => 'Tom Petty', 'title' => 'Highway Companion', 'genre' => 'rock', 'release_date' => new Carbon('2006-07-25'), 'stats' => ['producer' => 'Jeff Lynne']],
    ['title' => 'Ill Communication', 'genre' => Genre::HipHop, 'release_date' => null, 'stats' => new stdClass]
]);

$albums = Album::get();
$albums[0]->release_date->format('m/d/Y'); // '02/02/1999'
$albums[1]->stats['producer']; // 'Jeff Lynne'
$albums[2]->artist; // 'unknown'

Final Thoughts

I think there's probably a better name for this method. I would prefer to call it createMany(), but as @bert-w pointed out, then there's the expectation it will emit model events.

[bert-w]

What about the following setup:

$models = collect();
foreach ($values as $key => $attributes) {
    $model = $this->newModelInstance($attributes);
    if ($model->usesTimestamps()) {
        $model->updateTimestamps();
    }
    $models[] = $model;
}
// Trigger pre-events
$this->toBase()->insert($models->map(fn ($m) => $m->getAttributes());
// Trigger post-events

This uses more of the Eloquent methods and is more of a createMany function (it isn't that much different from your code actually). Also getAttributes() is used internally to create insert statements:

framework/src/Illuminate/Database/Eloquent/Model.php

Line 1297 in 997218b

$attributes = $this->getAttributesForInsert();

)
(which routes to the public fn getAttributes())

There are some remaining questions with triggering the events though, e.g. what happens when one event returns false? In current situations, the update fails so it might be wise to use the same thing here.

Also the unguarded() wrapper can probably be removed since I believe its expected to be guarded by default. You can wrap it yourself in the outer function:

User::unguarded(function() {
    User::createMany($values);
});

Just spouting some ideas here

[cosmastech]

@bert-w thanks for the thoughtful comments!

Timestamps

I wanted to avoid creating a new set of timestamps for something that would essentially be static across all of the models being inserted. Not a make or break, and would be amenable to making that change. Just seemed like unnecessary overhead.

Use of unguarded()

This is here so that the call to Model@fill() doesn't discard attributes that are being passed into newModelInstance(), even if they are guarded. If we were to change it to createMany then that'd be great and would happily implement your suggestion. 👍

The way Builder@insert() works is that it let's you pass anything to be inserted, guarded or not. I was just keeping with that convention.

An aside

There's also an issue I never noticed before, which is if you pass two arrays that have different keys, MySQL throws an exception.

Album::insert([
    ['title' => 'Kind of Blue', 'genre' => 'jazz'],
    ['title' => 'Highway 61 Revisited', 'artist' => 'Bob Dylan', 'genre' => 'rock']
]);

Illuminate\Database\QueryException  SQLSTATE[21S01]: Insert value list does not match column list: 1136 Column count doesn't match value count at row 2 (Connection: mysql, SQL: insert into `albums` (`genre`, `title`) values (jazz, Kind of Blue), (Bob Dylan, rock, Highway 61 Revisited)).

(I feel there should be a PR for adding a DEFAULT value to an insert/update/etc statement, but well beyond the scope of this PR).

I guess we'll wait to see what the framework maintainers say. I think this could also be improved by letting the caller pass an array or collection of models that need persisted (in case they need to set the state with their own Model methods or something).

[cosmastech]

I just realized, I don't think you can get the last inserted ID for a for multiple records inserted in a single INSERT statement.

So we wouldn't be able to write createMany() (that emits events) unless we did them one by one.

[bert-w]

there's actually an interesting workaround: you can use last-inserted-id to get the id of the first row of the last batch. Since the batch is one atomic query, the ids are bound to be sequential (assuming mysql+innodb) so the other rows can be simply calculated. But this might introduce inconsistencies across other db drivers since im not sure what happens there (however the integration test should cover it since it runs for all supported db drivers).

[cosmastech]

That's a bit too presumptuous I'm afraid. Within MySQL, you can actually change your auto-increment increment value to something other than 1.

I don't know a lot about PostgreSQL, but I think that engine allows for a SERIAL column type for specifying how you might want to auto-increment.

[cosmastech]

I tried to shorten the "How to do it today" example to this:

$inputData = [
    ['artist' => 'Built to Spill', 'title' => 'Keep It Like a Secret', 'genre' => Genre::Rock, 'release_date' => '1999-02-02', 'stats' => []],
    ['artist' => 'Tom Petty', 'title' => 'Highway Companion', 'genre' => 'rock', 'release_date' => new Carbon('2006-07-25'), 'stats' => ['producer' => 'Jeff Lynne']],
    ['title' => 'Ill Communication', 'genre' => Genre::HipHop, 'release_date' => null, 'stats' => new stdClass]
];

$values = Album::unguarded(fn() => collect($inputData)->mapInto(Album::class));
Album::insert($values->toArray());

but it raises a QueryException:

   Illuminate\Database\QueryException  SQLSTATE[22007]: Invalid datetime format: 1292 Incorrect date value: '1999-02-02T00:00:00.000000Z' for column 'release_date' at row 1 (Connection: mysql, SQL: insert into `albums` (`artist`, `genre`, `release_date`, `stats`, `title`) values (Built to Spill, rock, 1999-02-02T00:00:00.000000Z, Array, Keep It Like a Secret), (Tom Petty, rock, 2006-07-25T00:00:00.000000Z, Array, Highway Companion), (unknown, hip-hop, ?, Array, Ill Communication)).

and it's also not setting the created_at, updated_at fields 😿

[taylorotwell]

You can add a method like this to your own project. Thanks!