Intermittent WSOD when generating first FITS derivative

[dannylamb]

After uploading an Original File, when going back to view another page, you can intermittently get a whitescreen with something along the lines of

Drupal\Core\Field\FieldException: Attempt to create a field fits_exiftool_compression_scheme that does not exist on entity type media. in Drupal\field\Entity\FieldConfig->getFieldStorageDefinition() (line 312 of /var/www/html/drupal/web/core/modules/field/src/Entity/FieldConfig.php).

This seems to be from auto generating all the fields for FITS technical metadata, but further analysis is required. Since this is potentially due to one of the new features in the upcoming 1.1.0 release, I'm adding this to the milestone.

[dannylamb]

Linking to Islandora-Devops/islandora-playbook#159, which is where this was first discovered.

[dannylamb]

I'm vagrant up'ing to get a fresh box. I'll snapshot it and see if i can reproduce/debug.

[dannylamb]

I've spent the better part of my day trying to reproduce this and can't. I have run into it before, so I don't believe this is gone. I just can't get it to (mis)behave right now 🤷‍♂️

[dannylamb]

...and just ran into this when testing master of islandora-playbook 😭

[Mon Mar 30 13:44:53.751343 2020] [php7:notice] [pid 3422] [client 10.0.2.2:42102] Uncaught PHP Exception Drupal\\Core\\Field\\FieldException: "Attempt to create a field fits_jhove_size that does not exist on entity type media." at /var/www/html/drupal/web/core/modules/field/src/Entity/FieldConfig.php line 312, referer: http://localhost:8000/media/1

[dannylamb]

After reloading the page, everything comes up fine. One thing I'm noticing just now from the error message is that the referrer is media 1. The FITS file, media 4, totally has field_fits_jhove_size.

But media 1 is the original file and a different media type. So it looks like somewhere the wrong media is referenced.

[dannylamb]

I am consistently getting

PDOException: SQLSTATE[42000]: Syntax error or access violation: 1305 SAVEPOINT savepoint_1 does not exist: RELEASE SAVEPOINT savepoint_1

now from master. This is triggered when the media finally gets saved after all the fields have been created. Oddly enough, it is consistent, but if you view the media once the file's been uploaded but before all the fields have been dynamically generated... things look totally fine. It'll even cache that totally fine version to continue to fool you.

From what research I've done, I'm starting to think this interplay is either a bug in how Drupal handles nested transactions, or it's because mysql does not have transactional capabilities when it comes to structural stuff like creating tables, adding columns, etc... So adding those fields in a transaction causes big problems.

I can potentially solve this problem a few different ways, neither are so nice.

1. Don't do transactions at all in the media source controller. I've done this and it does indeed fix the error.
   1. But then we run the risk of leaving dangling files if errors happen during media creation. We'd need to make new routes and change how camel interacts with them, to create the file and media as two separate steps instead of a one-shot like we have now. That way they can at least be sanely handled in the back end with retries and such instead of just riding transaction-free.
2. Catch that particular PDOException here: https://github.com/Islandora/islandora/blob/8.x-1.x/src/Controller/MediaSourceController.php#L185-L192, then squash it and not rollback. (I have not tried this one yet).

A good next step would be to try and reproduce this with Postgres instead of MySQL.

But maybe for the time being it's easiest to just pull islandora_fits from the build on master until we can sort all of this out.

[mjordan]

@dannylamb or maybe leaving FITS in the release but just not enabling Islandora FITS by default? Simpler than pulling the whole tooth plus root out?

[seth-shaw-unlv]

Keeping it out of the release seems the safest option. Besides, we can't tag a version on the FITS work anyway since it is still based on a repo external to Islandora. Those who want it are still able to add it after-the-fact.

[dannylamb]

It's occurring to me now that the way the 8.x-1.x branch works would sidestep this problem altogether, since it does not auto-generate the fields for you. You have to name the field and when the fits gets generated, if a matching field is found its populated.

Let's slice it out, then issue a PR to get 8.x-1.x islandora_fits as a submodule of islandora and we'll go from there.

[ajstanley]

Another approach would be to include it without indexing each of the fields separately. They're not needed for anything other than fielded search, which is not going to happen very often anyway. (We had a special case for it her at UPEI). If we yank all the presave stuff out then it saves FITS as a file, and the formatter gives us the nice display.

…

On Wed, Apr 8, 2020 at 8:21 AM dannylamb ***@***.***> wrote: It's occurring to me now that the way the 8.x-1.x branch works would sidestep this problem altogether, since it does not auto-generate the fields for you. You have to name the field and when the fits gets generated, if a matching field is found its populated. Let's slice it out, then issue a PR to get 8.x-1.x islandora_fits as a submodule of islandora and we'll go from there. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1451 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADY2J4XJBWJJLQAWTXQZU3RLRM2PANCNFSM4LCNCLZQ> .

-- Alan Stanley Developer and Training Specialist Agile Humanities

[dannylamb]

@ajstanley 👍 I like still being able to deliver the feature and leaving the indexing for later.