Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Single table for file attachments #7420

Merged
merged 96 commits into from
Jun 19, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
0bb25b2
Add basic model for handling generic attachments
SchrodingersGat Jun 4, 2024
3fb7d18
Refactor migration
SchrodingersGat Jun 8, 2024
ae5c38c
Data migration to convert old files across
SchrodingersGat Jun 8, 2024
79d5bc1
Admin updates
SchrodingersGat Jun 8, 2024
9e34b67
Merge remote-tracking branch 'origin/master' into attachments-refactor
SchrodingersGat Jun 8, 2024
5250508
Increase comment field max_length
SchrodingersGat Jun 9, 2024
8031a49
Adjust field name
SchrodingersGat Jun 9, 2024
e50ab6e
Remove legacy serializer classes / endpoints
SchrodingersGat Jun 9, 2024
a4f4280
Expose new model to API
SchrodingersGat Jun 9, 2024
4ca4638
Admin site list filters
SchrodingersGat Jun 9, 2024
66dcee4
Remove legacy attachment models
SchrodingersGat Jun 9, 2024
3cc7dd0
Update data migration
SchrodingersGat Jun 9, 2024
b589ef0
Add migrations to remove legacy attachment tables
SchrodingersGat Jun 9, 2024
353f1dc
Fix for "rename_attachment" callback
SchrodingersGat Jun 9, 2024
04e978e
Refactor model_type field
SchrodingersGat Jun 9, 2024
1f6d4b0
Set allowed options for admin
SchrodingersGat Jun 9, 2024
def177d
Update model verbose names
SchrodingersGat Jun 9, 2024
ee59ca3
Fix logic for file upload
SchrodingersGat Jun 9, 2024
6c8f36b
Add choices for serializer
SchrodingersGat Jun 9, 2024
f390780
Add API filtering
SchrodingersGat Jun 9, 2024
8f3414f
Fix for API filter
SchrodingersGat Jun 9, 2024
5afc259
Fix for attachment tables in PUI
SchrodingersGat Jun 9, 2024
548dd0d
Bump API version
SchrodingersGat Jun 9, 2024
78da986
Record user when uploading attachment via API
SchrodingersGat Jun 9, 2024
0316241
Refactor <AttachmentTable /> for PUI
SchrodingersGat Jun 9, 2024
5ebb9dd
Display 'file_size' in PUI attachment table
SchrodingersGat Jun 9, 2024
b6f5398
Fix company migrations
SchrodingersGat Jun 9, 2024
d193945
Include permission informtion in roles API endpoint
SchrodingersGat Jun 9, 2024
bf82144
Read user permissions in PUI
SchrodingersGat Jun 9, 2024
a3460c3
Simplify permission checks for <AttachmentTable />
SchrodingersGat Jun 9, 2024
8127ba2
Automatically clean up old content types
SchrodingersGat Jun 9, 2024
daa5249
Cleanup PUI
SchrodingersGat Jun 9, 2024
6bdf372
Fix typo in data migration
SchrodingersGat Jun 9, 2024
7c4c004
Add reverse data migration
SchrodingersGat Jun 9, 2024
3b8aec5
Merge branch 'master' into attachments-refactor
SchrodingersGat Jun 9, 2024
35e6aad
Update unit tests
SchrodingersGat Jun 9, 2024
a426162
Use InMemoryStorage for media files in test mode
SchrodingersGat Jun 10, 2024
98f8c77
Data migration unit test
SchrodingersGat Jun 10, 2024
d2c0eae
Fix "model_type" field
SchrodingersGat Jun 10, 2024
9f09512
Add permission check for serializer
SchrodingersGat Jun 10, 2024
add38f4
Fix permission check for CUI
SchrodingersGat Jun 10, 2024
ddbffd8
Merge branch 'master' into attachments-refactor
SchrodingersGat Jun 10, 2024
241ec4d
Fix PUI import
SchrodingersGat Jun 10, 2024
a0cccaf
Test python lib against specific branch
SchrodingersGat Jun 10, 2024
471f17a
Revert STORAGES setting
SchrodingersGat Jun 10, 2024
0c27705
Fix part unit test
SchrodingersGat Jun 11, 2024
af79d26
Fix unit test for sales order
SchrodingersGat Jun 11, 2024
8e593fa
Merge branch 'master' into attachments-refactor
SchrodingersGat Jun 13, 2024
89336fe
Merge remote-tracking branch 'origin/master' into attachments-refactor
SchrodingersGat Jun 14, 2024
6b748fe
Use 'get_global_setting'
SchrodingersGat Jun 14, 2024
a587c85
Use 'get_global_setting'
SchrodingersGat Jun 14, 2024
4803751
Update setting getter
SchrodingersGat Jun 14, 2024
1788f96
Unit tests
SchrodingersGat Jun 14, 2024
a69713a
Tweaks
SchrodingersGat Jun 14, 2024
2389d70
Revert change to settings.py
SchrodingersGat Jun 14, 2024
5a6f9f5
More updates for get_global_setting
SchrodingersGat Jun 14, 2024
ac1acb0
Merge branch 'master' into attachments-refactor
SchrodingersGat Jun 14, 2024
0d73fbb
Relax API query count requirement
SchrodingersGat Jun 15, 2024
9064e49
Merge branch 'attachments-refactor' of github.com:SchrodingersGat/Inv…
SchrodingersGat Jun 15, 2024
5517b34
remove illegal chars and add unit tests
SchrodingersGat Jun 15, 2024
2decad3
Fix unit tests
SchrodingersGat Jun 15, 2024
80a93ea
Fix frontend unit tests
SchrodingersGat Jun 15, 2024
1c615a0
settings management updates
SchrodingersGat Jun 15, 2024
eb2716e
Prevent db write under more conditions
SchrodingersGat Jun 15, 2024
a46374f
Simplify settings code
SchrodingersGat Jun 15, 2024
f28f24a
Pop values before creating filters
SchrodingersGat Jun 15, 2024
c621b94
Prevent settings write under certain conditions
SchrodingersGat Jun 16, 2024
a89c2dc
Add debug msg
SchrodingersGat Jun 16, 2024
e882bf4
Clear db on record import
SchrodingersGat Jun 16, 2024
c8fb905
Refactor permissions checks
SchrodingersGat Jun 16, 2024
e3f34f6
Unit test updates
SchrodingersGat Jun 16, 2024
1c73b70
Merge branch 'master' into attachments-refactor
SchrodingersGat Jun 16, 2024
bc4d219
Prevent delete of attachment without correct permissions
SchrodingersGat Jun 16, 2024
80098cc
Adjust odcker.yaml
SchrodingersGat Jun 16, 2024
b833da1
Cleanup data migrations
SchrodingersGat Jun 16, 2024
4e30078
Tweak migration tests for build app
SchrodingersGat Jun 16, 2024
e3f22ef
Merge branch 'master' into attachments-refactor
SchrodingersGat Jun 16, 2024
5450e73
Update data migration
SchrodingersGat Jun 16, 2024
c6c06d6
Prevent debug shell in TESTING mode
SchrodingersGat Jun 17, 2024
f93d898
Merge branch 'master' into attachments-refactor
SchrodingersGat Jun 17, 2024
da258d9
Update migration dependencies
SchrodingersGat Jun 17, 2024
e164cd3
add file size test
SchrodingersGat Jun 17, 2024
80f06ad
Update migration tests
SchrodingersGat Jun 17, 2024
e09394d
Merge branch 'master' into attachments-refactor
SchrodingersGat Jun 17, 2024
0f867dd
Revert some settings caching changes
SchrodingersGat Jun 17, 2024
e1c424c
Fix incorrect logic in migration
SchrodingersGat Jun 17, 2024
b4767e9
Update unit tests
SchrodingersGat Jun 17, 2024
3d31689
Merge branch 'attachments-refactor' of github.com:SchrodingersGat/Inv…
SchrodingersGat Jun 17, 2024
b713d3d
Prevent create on CURRENCY_CODES
SchrodingersGat Jun 18, 2024
28d5ade
Fix unit test
SchrodingersGat Jun 18, 2024
a17caa6
Some refactoring
SchrodingersGat Jun 18, 2024
18e86ca
Fix typo
SchrodingersGat Jun 18, 2024
7dc5403
Revert change
SchrodingersGat Jun 18, 2024
90bad53
Add "tags" and "metadata"
SchrodingersGat Jun 19, 2024
fda2381
Include "tags" field in API serializer
SchrodingersGat Jun 19, 2024
c94d901
add "metadata" endpoint for attachments
SchrodingersGat Jun 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 24 additions & 6 deletions src/backend/InvenTree/common/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,24 @@
import common.models


@admin.register(common.models.Attachment)
class AttachmentAdmin(admin.ModelAdmin):
"""Admin interface for Attachment objects."""

list_display = (
'model_type',
'model_id',
'attachment',
'link',
'user',
'upload_date',
)

readonly_fields = ['file_size', 'upload_date', 'user']

search_fields = ('content_type', 'comment')


@admin.register(common.models.ProjectCode)
class ProjectCodeAdmin(ImportExportModelAdmin):
"""Admin settings for ProjectCode."""
Expand All @@ -16,6 +34,7 @@ class ProjectCodeAdmin(ImportExportModelAdmin):
search_fields = ('code', 'description')


@admin.register(common.models.InvenTreeSetting)
class SettingsAdmin(ImportExportModelAdmin):
"""Admin settings for InvenTreeSetting."""

Expand All @@ -28,6 +47,7 @@ def get_readonly_fields(self, request, obj=None): # pragma: no cover
return []


@admin.register(common.models.InvenTreeUserSetting)
class UserSettingsAdmin(ImportExportModelAdmin):
"""Admin settings for InvenTreeUserSetting."""

Expand All @@ -40,18 +60,21 @@ def get_readonly_fields(self, request, obj=None): # pragma: no cover
return []


@admin.register(common.models.WebhookEndpoint)
class WebhookAdmin(ImportExportModelAdmin):
"""Admin settings for Webhook."""

list_display = ('endpoint_id', 'name', 'active', 'user')


@admin.register(common.models.NotificationEntry)
class NotificationEntryAdmin(admin.ModelAdmin):
"""Admin settings for NotificationEntry."""

list_display = ('key', 'uid', 'updated')


@admin.register(common.models.NotificationMessage)
class NotificationMessageAdmin(admin.ModelAdmin):
"""Admin settings for NotificationMessage."""

Expand All @@ -70,16 +93,11 @@ class NotificationMessageAdmin(admin.ModelAdmin):
search_fields = ('name', 'category', 'message')


@admin.register(common.models.NewsFeedEntry)
class NewsFeedEntryAdmin(admin.ModelAdmin):
"""Admin settings for NewsFeedEntry."""

list_display = ('title', 'author', 'published', 'summary')


admin.site.register(common.models.InvenTreeSetting, SettingsAdmin)
admin.site.register(common.models.InvenTreeUserSetting, UserSettingsAdmin)
admin.site.register(common.models.WebhookEndpoint, WebhookAdmin)
admin.site.register(common.models.WebhookMessage, ImportExportModelAdmin)
admin.site.register(common.models.NotificationEntry, NotificationEntryAdmin)
admin.site.register(common.models.NotificationMessage, NotificationMessageAdmin)
admin.site.register(common.models.NewsFeedEntry, NewsFeedEntryAdmin)
35 changes: 35 additions & 0 deletions src/backend/InvenTree/common/migrations/0025_attachment.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Generated by Django 4.2.12 on 2024-06-08 12:37

import InvenTree.fields
import InvenTree.models
import common.models
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion


class Migration(migrations.Migration):

dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
('contenttypes', '0002_remove_content_type_name'),
('common', '0024_notesimage_model_id_notesimage_model_type'),
]

operations = [
migrations.CreateModel(
name='Attachment',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('model_id', models.PositiveIntegerField()),
('attachment', models.FileField(blank=True, help_text='Select file to attach', null=True, upload_to=common.models.rename_attachment, verbose_name='Attachment')),
('link', InvenTree.fields.InvenTreeURLField(blank=True, help_text='Link to external URL', null=True, verbose_name='Link')),
('comment', models.CharField(blank=True, help_text='Attachment comment', max_length=100, verbose_name='Comment')),
('upload_date', models.DateField(auto_now_add=True, help_text='Date the file was uploaded', null=True, verbose_name='Upload date')),
('file_size', models.PositiveIntegerField(default=0, help_text='File size in bytes', verbose_name='File size')),
('model_type', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='contenttypes.contenttype')),
('user', models.ForeignKey(blank=True, help_text='User', null=True, on_delete=django.db.models.deletion.SET_NULL, to=settings.AUTH_USER_MODEL, verbose_name='User')),
],
bases=(InvenTree.models.PluginValidationMixin, models.Model),
),
]
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Generated by Django 4.2.12 on 2024-06-08 12:38

from django.db import migrations
from django.core.files.storage import default_storage


def update_attachments(apps, schema_editor):
"""Migrate any existing attachment models to the new attachment table."""

Attachment = apps.get_model('common', 'attachment')

# Legacy attachment types to convert:
# app_label, table name, target model, model ref
legacy_models = [
('build', 'BuildOrderAttachment', 'build', 'build'),
('company', 'CompanyAttachment', 'company', 'company'),
('company', 'ManufacturerPartAttachment', 'manufacturerpart', 'manufacturer_part'),
('order', 'PurchaseOrderAttachment', 'purchaseorder', 'order'),
('order', 'SalesOrderAttachment', 'salesorder', 'order'),
('order', 'ReturnOrderAttachment', 'order', 'order'),
('part', 'PartAttachment', 'part', 'part'),
('stock', 'StockItemAttachment', 'stockitem', 'stock_item')
]

# Get the "ContentType" model
ContentType = apps.get_model('contenttypes', 'ContentType')

for app, model, target_model, model_ref in legacy_models:
LegacyAttachmentModel = apps.get_model(app, model)

if LegacyAttachmentModel.objects.count() == 0:
continue

# Find the ContentType model which matches the target table
content_type = ContentType.objects.get(app_label=app, model=target_model)

to_create = []

for attachment in LegacyAttachmentModel.objects.all():

# Find the size of the file (if exists)
if attachment.attachment and default_storage.exists(attachment.attachment.name):
try:
file_size = default_storage.size(attachment.attachment.name)
except NotImplementedError:
file_size = 0
else:
file_size = 0

to_create.append(
Attachment(
model_type=content_type,
model_id=getattr(attachment, model_ref).pk,
attachment=attachment.attachment,
link=attachment.link,
comment=attachment.comment,
user=attachment.user,
upload_date=attachment.upload_date,
file_size=file_size
)
)

if len(to_create) > 0:
print(f"Migrating {len(to_create)} attachments for the legacy '{model}' model.")
Attachment.objects.bulk_create(to_create)


def delete_attachments(apps, schema_editor):
"""Reverse data migration removes any Attachment objects."""

Attachment = apps.get_model('common', 'attachment')

if n := Attachment.objects.count():
Attachment.objects.all().delete()
print(f"Deleted {n} Attachments in reverse migration")


class Migration(migrations.Migration):

dependencies = [
('common', '0025_attachment'),
]

operations = [
migrations.RunPython(update_attachments, reverse_code=delete_attachments)
]
113 changes: 113 additions & 0 deletions src/backend/InvenTree/common/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -3062,3 +3062,116 @@
from InvenTree.conversion import reload_unit_registry

reload_unit_registry()


def rename_attachment(instance, filename):
"""Callback function to rename an uploaded attachment file.

Arguments:
- instance: The Attachment instance
- filename: The original filename of the uploaded file

Returns:
- The new filename for the uploaded file, e.g. 'attachments/<content_type>/<object_id>/<filename>'
"""
filename = os.path.basename(filename)

Check warning on line 3077 in src/backend/InvenTree/common/models.py

View check run for this annotation

Codecov / codecov/patch

src/backend/InvenTree/common/models.py#L3077

Added line #L3077 was not covered by tests

# Generate a new filename for the attachment
return os.path.join(

Check warning on line 3080 in src/backend/InvenTree/common/models.py

View check run for this annotation

Codecov / codecov/patch

src/backend/InvenTree/common/models.py#L3080

Added line #L3080 was not covered by tests
'attachments',
str(instance.content_type.model),
str(instance.object_id),
filename,
)


class Attachment(InvenTree.models.InvenTreeModel):
"""Class which represents an uploaded file attachment.

An attachment can be either an uploaded file, or an external URL.

Attributes:
attachment: The uploaded file
url: An external URL
comment: A comment or description for the attachment
user: The user who uploaded the attachment
upload_date: The date the attachment was uploaded
file_size: The size of the uploaded file
"""

class Meta:
"""Metaclass options."""

...

model_type = models.ForeignKey(ContentType, on_delete=models.CASCADE)
model_id = models.PositiveIntegerField()

content_object = GenericForeignKey('model_type', 'model_id')

attachment = models.FileField(
upload_to=rename_attachment,
verbose_name=_('Attachment'),
help_text=_('Select file to attach'),
blank=True,
null=True,
)

link = InvenTree.fields.InvenTreeURLField(
blank=True,
null=True,
verbose_name=_('Link'),
help_text=_('Link to external URL'),
)

comment = models.CharField(
blank=True,
max_length=100,
SchrodingersGat marked this conversation as resolved.
Show resolved Hide resolved
verbose_name=_('Comment'),
help_text=_('Attachment comment'),
)

user = models.ForeignKey(
SchrodingersGat marked this conversation as resolved.
Show resolved Hide resolved
User,
on_delete=models.SET_NULL,
blank=True,
null=True,
verbose_name=_('User'),
help_text=_('User'),
)

upload_date = models.DateField(
auto_now_add=True,
null=True,
blank=True,
verbose_name=_('Upload date'),
help_text=_('Date the file was uploaded'),
)

file_size = models.PositiveIntegerField(
SchrodingersGat marked this conversation as resolved.
Show resolved Hide resolved
default=0, verbose_name=_('File size'), help_text=_('File size in bytes')
)

@property
def basename(self):
"""Base name/path for attachment."""
if self.attachment:
return os.path.basename(self.attachment.name)
return None

Check warning on line 3160 in src/backend/InvenTree/common/models.py

View check run for this annotation

Codecov / codecov/patch

src/backend/InvenTree/common/models.py#L3158-L3160

Added lines #L3158 - L3160 were not covered by tests

def fully_qualified_url(self):
"""Return a 'fully qualified' URL for this attachment.

- If the attachment is a link to an external resource, return the link
- If the attachment is an uploaded file, return the fully qualified media URL
"""
if self.link:
return self.link

Check warning on line 3169 in src/backend/InvenTree/common/models.py

View check run for this annotation

Codecov / codecov/patch

src/backend/InvenTree/common/models.py#L3168-L3169

Added lines #L3168 - L3169 were not covered by tests

if self.attachment:
import InvenTree.helpers_model

Check warning on line 3172 in src/backend/InvenTree/common/models.py

View check run for this annotation

Codecov / codecov/patch

src/backend/InvenTree/common/models.py#L3171-L3172

Added lines #L3171 - L3172 were not covered by tests

media_url = InvenTree.helpers.getMediaUrl(self.attachment.url)
return InvenTree.helpers_model.construct_absolute_url(media_url)

Check warning on line 3175 in src/backend/InvenTree/common/models.py

View check run for this annotation

Codecov / codecov/patch

src/backend/InvenTree/common/models.py#L3174-L3175

Added lines #L3174 - L3175 were not covered by tests

return ''

Check warning on line 3177 in src/backend/InvenTree/common/models.py

View check run for this annotation

Codecov / codecov/patch

src/backend/InvenTree/common/models.py#L3177

Added line #L3177 was not covered by tests
Loading