Universalize UID support in all resource types

[reduz]

Ensures all resource types support UIDs in a project. This is required to fix:

• Scripts and many other resource types can´t be referenced by UID, only by path. If used in text (in a script) when refactored the references are lost.
• Path export properties can´t use UID for unsupported types and must resort to paths. When refactored this is always lost (the editor can´t feasibly get into this data to change it).
• Refactoring problems when files are moved outside the editor (this PR effectively fixes it).
• Editor properly refresh paths if they changed externally while opened (as example, git update). This needs
  to be addressed in a subsequent PR, but this one effectively sets the prerequisites.
• Slow editor refactoring times (as file paths need to be modified all across the board every time something is renamed). This PR allows always using UIDs, so refactoring paths file by file is no longer needed.

Resource types that do not support UID will get a .uid file appended to them (this includes .gd , .gdshader, .gdextension, etc files).

SCREENSHOT:

2D Platformer demo with uids auto generated:

(Note RESOURCE.md, these are considered text resources by Godot and exported, so the .uid is actually correct.
Edit: This was fixed before merging.)

TODO:

• Editor refactoring not yet supported.
• De-duplication (if two UIDs are the same) not yet supported.
  FAQ:

Why not using metafiles instead to make it more generic?

1. There has not been any need for metafiles. Most resource types contain metadata on their own.
2. The .uid files are only used by the editor and are effectively gone on export, as this information is moved to the uid database. As such they are unsuitable as metafiles.

[Jordyfel]

As someone who spent some time working on the refactoring problems and thinking about the UID system I fully support this approach.

This will need some code in the filesystem dock to move the .uid files along with the main file, much like with .import.

To make sure, refactoring paths file by file will no longer be needed, means that users should be encouraged to hardcode uids instead of paths, right?

What do you mean by "editor refactoring not yet supported"?

Significant issue that will not be fixed by this: #95637

This comes from the problem that paths for extraction from imported 3d files, stored as import options, currently do not support uids at all, nor do they work with the refactoring logic in the filesystem dock. I've though about this a bit but am unsure (maybe store uid as a hidden import option?), but it seems like paths in .import files will need some significant effort to be made to fully work with refactoring.

[lyuma]

it seems like paths in .import files will need some significant effort to be made to fully work with refactoring.

Agree that this is important: hardcoded paths is one of the biggest issues with the advanced importer, especially given that it causes failed imports or even outright engine crashes in some cases if the dependent files are missing.

Allowing .import files to reference by UID is blocking some workflow improvements such as #86430 so it would be good to get this fixed in 4.4

[reduz]

@lyuma this seems to be a separate issue that will need a separate fix

[JoNax97]

This is a great step forward, but I think your assessment about meta files is a bit disingenuous.

Import files already contain editor-relevant metadata, and if in the future there's a need for saving anything else about code files, then you'll be in a situation where you either have to add another secondary file, rename the uid extension, or have a disconnect between the file intended type and it's content.

I think it would be preferable to future proof a bit here, and allow for meta files that are general and flexible enough to avoid needing more as hoc solutions going forward.

[reduz]

@JoNax97 If you ask me and I could redesign the engine from scratch, I would just have used generic metadata files for everything. Because that's not possible and the other formats already contain their own stuff, and their own metadata, this is the simplest solution for that.

[JoNax97]

I understand your point. Maybe this could be the first step in a progressive migration path.

[reduz]

@JoNax97 AFAIK that is probably Godot 5 material, so not likely to happen for a long time. We need to commit to a stable engine before major changes like this happening again.

[Mickeon]

Very surface-level documentation notes.

Why do _get_resource_uid and _has_custom_uid_support have to be two separate methods? Would it not be possible to verify if only _get_resource_uid is overridden, and/or check if it returns 0?

[arkology]

Did I understand correctly that it will be illegal to add uid files to gitignore?

[reduz]

@arkology I would not do it 😆

[wstumpfz]

I too would like to vote to make this more like a generic metadata file. Currently I'm trying to add "pack file tags" to assets to facilitate the pack file exporting workflow in godot (godotengine/godot-proposals#10580). The initial roadblock I've encountered is there's no real way to add metadata to PackedScenes [.tscn files] and GDScripts. As a proof of concept I added the ability for PackedScenes and GDScripts to have .import files to make it possible to save custom metadata for those assets. I had to edit some logic around how godot handles .import files, but it works. Having dedicated metadata files could be a better alternative, so it would be great though if you could expand the scope of this effort to make it a little more generic. It would make it significantly more futureproof.

[lyuma]

@wstumpfz

there's no real way to add metadata to PackedScenes [.tscn files] and GDScripts

The Godot philosophy is that the solution should follow the problem. I think a more palatable approach (less likely to break assumptions) would be to add a place in the .tscn format for metadata to be written. That would solve that specific sub-problem

scripts like .gd and .gdshader are a bit more complicated, because the code is human-readable, but you could imagine attaching something bulky like a texture into the metadata fields. Imagine for example, a plugin that assigns icons to scripts in metadata. If this image were dumped in base64 format into an annotation like @meta("foo", "Resource(bar...)") that could be unexpected.

If you limit script metadata to resource references and text strings, it could still be feasible to add annotations to scripts, but it wouldn't generalizable to 100% of cases, so this could be unexpected.

There's actually a funky example in Unity which is similar to what you describe. Unity shaders can be in text format like in Godot. However, there is a feature where a default texture can be assigned for a given texture slot. Unity actually puts that texture reference into the .shader.meta file. In Godot, it would not be feasible to put a texture into the body of a shader, and I can see the appeal to having a .meta file to store this type of info, as in these other engines.

My in-depth explanation of the problem with using .import for native formats (copied from RocketChat)

I think changing the definition of .import in this way is going to break a lot of assumptions, so you can do this in your own project, but I would consider it unlikely that it will be approved to merge into Godot
what you are asking for is something like generic meta files,

I would read the discussion on the PR I linked, because it basically explains the problem with moving towards generic meta files. In fact, Reduz would be in favor of this type of change, but only for a major version

Sorry I didn't do a good job of explaining the issue with .import - basically, the way Godot thinks of things, assets are natively in resource format (.res / .scn). The purpose of the importer is to take a file in another non-native format and convert it to a native godot resource format, "remapped" to a native resource in .godot/imported

In other words, through this process, there are now two files at the same logical path in the project: there is the original file (such as a .fbx), and there is the .res in .godot/imported that it has been remapped to. For this reason, the original .fbx is read-only (you wouldn't want to overwrite a source file with a godot native resource in a different format.)

Now, separately, there is something called ResourceFormatLoader, which is what allows Godot to read .tscn, .res, .gdshader and so on, without a .import file. In essence, having a ResourceFormatLoader is what makes a format "native".

I think it is a very valid question to ask why loaders and importers are different types of things, but the fact is that they are. I think one answer is that loaders must be fast, since you will go through them every time you open a file, while importers usually are slow and therefore must cache a second native resource in .godot/imported.

For example, in Unity, every asset has a .meta file, and the meta file is the same format for every file type: even native assets have an importer, but it is set to NativeFormatImporter. (Though I wouldn't be surprised if it works similarly to Godot in the sense of reading the file directly and bypassing the import system)

So to circle back to your proposal, the problem here is that we are taking something with a resource loader, and treating it as if it is imported, but somehow without a resource in .godot/imported.

[huwpascoe]

Because that's not possible and the other formats already contain their own stuff, and their own metadata, this is the simplest solution for that.

A future PR will inevitably need to store more data, the simplest solution will add another string after the UID, and eventually the Godot metadata format will become ill-defined text files with .uid extension.

I'm calling it now. 🙂

[akien-mga]

It's easy enough to rename the extension in the future if we need, and have some compatibility code to handle the transition gracefully (keeping support for reading the old format, and adding support for converting to the new format). So there's no need to future proof and bikeshed now.

[reduz]

@huwpascoe Please check the FAQ I wrote, metadata does not make sense because:

• This is only for very few file types, not all.
• These files do not get exported.

If at some point you want to have metadata, it will have to be done in a different way.

[huwpascoe]

That's okay, Akien's response was convincing enough. I probably should've replied with more than an emote!

I do have a question though, sometimes assets are marked as don't import and show up as a little x icon, or maybe they'd be set for importing again later. Does the .uid handle this edge case?

[reduz]

@huwpascoe I think it should be fine, since these is in practice for assets that are not imported

[reduz]

Changes made:

• Proper uid:// syntax saved in uid file for more consistency.
• Unexposed the _has_custom_uid_support and just checked if virtual has been overriden, as suggested above.
• Moving, renaming, duplicating now supported in filesystem dock.

This PR is now ready for review!

[Saul2022]

• Moving, renaming, duplicating now supported in filesystem dock.

Does that mean it no longer slow af to rename files when there’s a lot of files in the project?( 10k, 20k)

[RevoluPowered]

@reduz Would it be worth adding editor functionality to resolve UID issues like this too?
editor/editor_file_system.cpp:1259 - UID duplicate detected between res://textures/special/clip.png and res://textures/shaders/tangent-test.png.

I have a project that I created in 4.3.dev 3 exhibiting this problem with UID's I think the function might be generating duplicates higher than it should be.

[reduz]

@RevoluPowered Do you have an open issue about this? I feel that is unrelated.

[arkology]

Scripts have UIDs now, so if you re-save the .res file, it should reference the script's UID and the "auto-updating" should be working.
If there are cases where some dependencies still break, open an issue if it isn't tracked already.

If it actually updates script path to script's UID automatically, and as Godot 4.3 doesn't have script UIDs, this PR breaks compatibility.
And this PR doesn't have corresponding tag.
Could someone please add break compat tag? Nevermind, I'm wrong

[KoBeWi]

The references are stored both as path and UID. You can see it in tscn files. There is no compatibility breakage.

[scott-lin]

I don't see any documentation changes to go along with the code changes. How can I understand the tradeoffs (benefits/downsides) of this change, the ways I can use it and its limitations?

I don't intend to come across as bitter, but as someone testing the developer releases, I find myself dealing with an influx of new files that clutter my file views, such as OS Explorer and VSCode. This clutters my workspace and disrupts my workflow, making it more difficult to locate the files I need. I can be persuaded the changes are good for me, but I don't understand how yet due to lack of details.

[SpockBauru]

For me it seems like a medicine that is worse than the disease itself

[Anixias]

I'm all for UID files since the alternative is a messy find-and-replace approach, but I really dislike that it produces extra files all over my project. They really need to be placed in a dedicated directory.

If it's having the same filepath that matters, just recreate the relative path from the root project directory inside of a directory like .uid/ or similar. For example, if a resource has path res://a/b.c, it should look for the UID file at uid://a/b.c.uid (where uid:// maps to .uid/).

Edit: Someone else opened a proposal regarding these concerns.

[JoNax97]

The whole point of having the uid files at the same path is that users move both files at the same time

[daveTheOldCoder]

But that doesn't explain why the filenames can't start with a period, so that they can optionally be hidden and not clutter up the view.

Even if the uid file is always visible, that doesn't ensure that the user will remember to copy/move it along with its parent file.

[JoNax97]

that doesn't ensure that the user will remember to copy/move it along with its parent file.

But it makes is trivial. Having it in a separate folder makes it cumbersome and difficult for new user to know they have to do it.

[daveTheOldCoder]

I suggested that the name start with a ".", not place it in a separate folder.

[JoNax97]

Sorry I lost the thread of who said what. In any case, the "." makes all uids bunch together at the top.

The convenience of the current approach is that you can bulk select files and be sure the uids are selected as well

[scott-lin]

As long as the onus is on the human user, there is absolutely no guarantee the user will move uid files where Godot intends them to be regardless of whether they're located in a separate folder or the same folder.

There are a few misguided assumptions in the comments so far regarding how users will interact with uid files.

1. Will all users understand how uid files function and where Godot expects them to be in the first place? Placing the onus on users to manage uid files is bad design. What are the consequences of doing it wrong or forgetting?

2. The "convenience" assumption. I don't find it convenient to remember at all. Furthermore, users may order files on their system differently; for example, if ordered by Type or Last Modified instead of by Name, then the uid files won't be side-by-side with the code file.

I do not want to be in the business of managing uid files manually when I move code files around period. The benefits of the feature do not outweigh the introduced cost in my opinion. The system needs to handle it automatically.

To reiterate my previous comment, I'd like to see documentation regarding how can I understand the tradeoffs (benefits/downsides) of this change, the ways I can use it and its limitations.

[KoBeWi]

Using a separate folder means you have to search for the corresponding .uid file if you want to move it.
Prepending names with dot means the files are hidden on some systems and users may not be aware of their existence.
Using a single UID index file per directory means you have to manually update the entries and is prone to conflicts.
UID comments might be the most convenient solution, but we had it and some people disliked it very strongly, so it was reverted.

Yes, people might not know how to deal with .uid files as they are currently implemented, but it's still better and more intuitive solution than the other suggestions, especially when we already have .import files.

Also, if you really don't want to deal with .uid files, just don't touch them and only move the base files. Obviously your dependencies will break, but it's the same thing we had before .uid files were introduced. The only thing that changes is that you'll get harmless warnings when doing so.

[arkology]

As someone who disliked UID comments very strongly, I ended up with conclusion that if I have to choose between UID files zerg rush and UID comments/annotations inside scripts, comments/annotations are the lesser of two evils.

[JoNax97]

I know comparisons are odious, but Unity's had this exact scheme for decades and it works. Most game devs have at least some degrees of familiarity with it and know the rules for dealing with metadata files.

It's not the most elegant but it gets the job done and version control is really not an issue. I think you might be overreacting a bit here.

[SpockBauru]

@JoNax97

I know comparisons are odious, but Unity's had this exact scheme for decades and it works. Most game devs have at least some degrees of familiarity with it and know the rules for dealing with metadata files.

As a former Unity dev, I must state that the .meta files system is terrible to manage and not having them was one of the many things that I like in Godot.

@KoBeWi

Also, if you really don't want to deal with .uid files, just don't touch them and only move the base files. Obviously your dependencies will break, but it's the same thing we had before .uid files were introduced. The only thing that changes is that you'll get harmless warnings when doing so.

So make a way to disable the .uid generation. As I state before, is a medicine worse than the disease is trying to heal:

[scott-lin]

I agree with @SpockBauru so far. Give me a choice in disabling this feature. I do not want to deal with the system's side effects, warnings, etc. I want to make games.

Has the feature owner / Godot considered letting us generate a uid file on-demand for a specific file? Do we really need a uid for all files? As a user, if I have specific files I need uids for, then I could ask the system to generate a uid for them; in this model, I choose to create a dependency I'm willing to manage and understand how to seemingly.

[Anixias]

The UID files wouldn't be quite as bad if it didn't make one for every file that could potentially be a resource. I have 168 C# files in my project at the moment, and less than 30 of them are attached to a Node in the editor. The rest are libraries and whatnot yet they all have UID files generated for them as well. It should generate them once they are initially assigned to something in the editor, as needed, and automatically deleted when the last reference in the editor is removed.

[Geometror]

Just for reference, there's #99090.

As I stated in #99090 (comment), UIDs for everything are desirable, but having a choice on how they are stored/managed would be great.

[SpockBauru]

Just for reference, there's #99090.

Thanks! I believe this PR is a better approach, let's move the discussion to #99090

[berarma]

It's a pity that extended file attributes aren't more common. Since I don't think they are, at least outside of Linux, I would have preferred metadata to be in the Godot scripts (GDScript and shaders), and use uid files only for scripts not created by Godot.

[markdibarry]

I'm personally not a big fan of the problems this introduces, but most of them are eliminated with a helpful trick for anyone who uses an IDE like VSCode. They usually have a setting for file nesting similar to .min.js or the like.

By adding * and ${capture}.uid any .uid file will be nested in the associated file, so you won't need to remember anything when moving or deleting files.

[JoNax97]

this workaround does not solve the problem of filesystem noise, but just covers it up.

Look, we all know this is a tradeoff (like most everything in tech) and the community at large has agreed it's a worthwhile one.

I don't see what we win by repeatedly bashing the same points over and over.

[daveTheOldCoder]

If the only purpose of the .uid files is to support copying files between projects, a better approach would be to provide an import/export tool (or documented procedure) for copying files between projects, and state clearly in the documentation that the tool (or documented procedure) is the only supported way of doing that.

[JoNax97]

If the only purpose of the .uid files is to support copying files between projects

It's not. The main purpose is to support moving files externally within the same project.

[daveTheOldCoder]

the community at large has agreed it's a worthwhile one.

How do you know that? Isn't this change only in the 4.4-dev snapshots? Most of the community don't use dev snapshots and haven't dealt with this change yet.

[Macksaur]

Most of the community don't use dev snapshots

Definitely. The real conversation around this feature hasn't actually been had yet. I am hoping that these files will be removed or easy to disable at some point once that actually occurs.

UIDs should have been either 1) hidden optional comments in the scripts themselves or, better yet 2) a popup that appears when references are broken, asking to fix dependencies.

Doubling the volume of files and also the subsequent diff noise has not been fun to incorporate into my workflow. Currently UIDs are a burden with very little gains.

[akien-mga]

Thanks everyone for the feedback so far. We understand that it's a change that affects the usual workflow, and may be unpopular with some users.

I would like to redirect feedback to this discussion I just created, instead of having it spread across various issues and PRs:

godotengine/godot-proposals#11574

The "discussion" feature of Godot supports threads, so it's a much better format to properly discuss pros and cons separately, compared to a sequential flow of comments like on issues/PRs.

It's fine to repeat some of the points you have made here or in other issues in the new discussion, as a starting point for further threaded discussion.

And of course please make sure to read https://godotengine.org/article/uid-changes-coming-to-godot-4-4/ where we tried to explain why this solution was chosen, and why other solutions (which some are bringing up again here) may not be suitable for technical reasons.