Updating unique template field does not remove unique record

[AlexStansfield]

Describe the bug

Our user records have email as a unique field. However we do not hard delete user records, instead we soft delete them. We do this by setting the deletedAt field to a date.

So for this reason rather than setting the email field as unique we have an extra uniqueEmail field that uses a template. If there is no deletedAt then we want this field to have the email, and if there is a deletedAt we want this field to be removed. This should trigger the removal of the unique record for this field.

As this can't be done with a string template we used a function instead.

We're pretty sure this was working before but yesterday I found it no longer did when I went to delete a user and recreate it with the same email.

I did some investigation and found that in Model.update when it works out the hasUniqueProperties the value template function has not been run, so it doesn't know that uniqueEmail is now being set to null and so updateUnique should be called instead of updateItem.

To Reproduce

I've created a branch on my fork of this repository - https://github.com/AlexStansfield/dynamodb-onetable/tree/issue/unique-update-with-value-function

Here I added:

• two new fields to the schema, deletedAt and uniqueEmail - here
• a function to create the value for uniqueEmail (same as email if no deletedAt, null if there is deletedAt) - here
• an extra test to unique.ts with the title "Soft delete user and create with same email". - here

You will see that it fails. It should be expected that the unique record should have been removed so first it fails where it expects the number of items returned in the scan is one less (i.e the unique record is gone). If you remove this expectation you will see the create then fails because it finds the unique record.

The User record however is updated and the uniqueEmail field has been removed.

Here is a console.log of the items that it gets from the scan after the "soft delete"

[
  {
    sk: { S: '_unique#' },
    _type: { S: '_Unique' },
    pk: { S: '_unique#User#interpolated#Judy Smith#judy-a@example.com' }
  },
  {
    interpolated: { S: 'Peter Smith#peter@example.com' },
    deletedAt: { N: '1700121064087' },
    _type: { S: 'User' },
    name: { S: 'Peter Smith' },
    sk: { S: 'User#' },
    pk: { S: 'User#Peter Smith' },
    email: { S: 'peter@example.com' }
  },
  {
    interpolated: { S: 'Judy Smith#judy-a@example.com' },
    uniqueEmail: { S: 'judy-a@example.com' },
    _type: { S: 'User' },
    name: { S: 'Judy Smith' },
    sk: { S: 'User#' },
    pk: { S: 'User#Judy Smith' },
    age: { N: '15' },
    email: { S: 'judy-a@example.com' }
  },
  {
    sk: { S: '_unique#' },
    _type: { S: '_Unique' },
    pk: { S: '_unique#User#interpolated#Peter Smith#peter@example.com' }
  },
  {
    sk: { S: '_unique#' },
    _type: { S: '_Unique' },
    pk: { S: '_unique#User#email#peter@example.com' }
  },
  {
    sk: { S: '_unique#' },
    _type: { S: '_Unique' },
    pk: { S: '_unique#User#uniqueEmail#peter@example.com' }
  },
  {
    sk: { S: '_unique#' },
    _type: { S: '_Unique' },
    pk: { S: '_unique#User#email#judy-a@example.com' }
  },
  {
    sk: { S: '_unique#' },
    _type: { S: '_Unique' },
    pk: { S: '_unique#User#uniqueEmail#judy-a@example.com' }
  }
]

As you can see the Peter Smith record has a deletedAt and no uniqueEmail, however this record remains:

  {
    sk: { S: '_unique#' },
    _type: { S: '_Unique' },
    pk: { S: '_unique#User#uniqueEmail#peter@example.com' }
  },

[AlexStansfield]

So until this point I'd assumed it was just the function causing the issue. However i've just added another new test with a standard string template.

I've added:

• code field that is not unique, uniqueCode template that uses string template '${code}' this is unique. - here
• new test "Unique Code is updated" that creates a user with code 12345678, updates the user's code to 87654321, creates another user with code 12345678. - here

This also fails. When the update of the code from 12345678 to 87654321 is done it should remove the original Unique record and create a new one with the new uniqueCode value. However it doesn't do it.

It seems that unique records are only updated for template fields if there is also a non-template field that is unique being updated.

If the only unique field to be updated is generated from a template or a function, then it's ignored and a standard updateItem is done. Leaving the original unique record in place.

[mobsense]

Alex, I've dug into this, but it is pretty complex.

You said:

I did some investigation and found that in Model.update when it works out the [hasUniqueProperties](https://github.com/sensedeep/dynamodb-onetable/blob/028a644711ad7bd42671531c965a2ff8b2861e37/src/Model.js#L800) the value template function has not been run

This is because you did not provide the unique field "email" to the update. If you provide that, then the update will call updateUnique. You must provide the actual unique field.

Can you create a test case without the value generator? I tried to use your repo: but the entire unique.ts now uses the value generator.

Can you extract just a debug.ts with only your test case and without a value generator?