Parameter for complete import of IFC

[CabeDigicorp]

Description 📝

The default set of IFC items and relations imported into Fragments is relatively small. Since we have absolutely no control on the IFC files that our customers load, it is necessary to be able to import all of the information included in the IFC.

Suggested solution 💡

Add a boolean parameter to the process() function of IfcImporter . If it's true, all the filtering will be skipped.

Alternative ⛕

No response

Additional context ☝️

No response

Validations ✅

• Read the docs.
• Check that there isn't already an issue that requests the same feature to avoid creating a duplicate.

[agviegas]

@CabeDigicorp Are you suggesting to provide a method that fills items and relations with all existing ifc items and relations? I believe that won't work very well because it will result in huge frag files (e.g. including all innefficient implicit geometry representation), but we can try it out.

In order to do that, we will need a full list of elements and relations, which we don't have at the moment. If you provide it, we'll be happy to try it out and see if it works. If you don't have it either, or you don't want to end up with unnecessarily bloated files, happy to explore what your IFC files look like and let you know how to configure the ifc importer to get all the data you need.

[CabeDigicorp]

I am suggesting exactly that. It might be possible to avoid unnecessary geometries and limit the full import to properties/quantities/materials, but they definitely need to be available even for objects that have no own geometry (i.e. containing structures, types, styles...)
We don't have, and will never have, any reliable info about our customers' IFC files - we don't even know how they are created. So we need to import everything, and accuracy comes before performance.

[agviegas]

@CabeDigicorp I see! We can make it, but we need a full list of IFC elements and relations that you expect. If you provide it, we'll do this.

[CabeDigicorp]

I'd say all those listed here.
Tomorrow I'll discuss this topic with my colleagues, so I'll update you in the afternoon (Italian time)

[agviegas]

@CabeDigicorp we will add a function that adds the whole category map and relations to the frag file.

Keep in mind that if this results in bloated .frag files, we won't be able to cherry pick just the items that suit your specific use case. We have hundreds of users and can't provide a one-line function that solves what they need. In that case, you will need to detect what you don't need and take it out from ifcImporter.classes. Once this is published, I'll update this.

[CabeDigicorp]

Sorry for the delay, we had a lengthy discussion (and a public holiday on Monday). The conclusion is that we can't have a aspecific use case - we offer a platform for our customers to manage their own models, on which we can't make reliable assumptions or force limitations. So every bit of info in the IFC might be meaningful and should be accessible.
Yep, that will result in files not being as efficient as in your implementation, but we're serializing properties in a separate file anyways (loading them directly from fragments is just too slow). We'll see in the future if there is any room for further improvements.
Thank you!

[agviegas]

Available from @thatopen/fragments@3.2.12, you can now do this:

serializer.addAllAttributes();
serializer.addAllRelations();

As said, this will make .frag files bigger. If there are things you don't need, you can still delete the specific categories / relations you don't want. Let us know if you find any issues.

Just FYI, fragments can currently retrieve millions of properties and relations really fast. If you are experiencing slowness it's likely because you are not doing it correctly. If you have any specific issues, feel free to let us know. Cheers.

[CabeDigicorp]

Thank you, I'll check it out imediately.
What I am experiencing is that the getItemsData() takes a few seconds when executed on a single localId in our large (~500MB) sample models - that's not an acceptable wait upon item selection.
Preloading all the properties (from fragments) during model loading is comparatively faster, but still sometimes annoying (may take a couple of minutes on some models, but at least, after that initial process, retrieval is immediate). Storing the same properties in a JSON file and preloading them from the JSON takes the time down by an order of magnitude.

[agviegas]

@CabeDigicorp there's some issue with npm tokens today, so it might take a while to have the package available.

I see. getData was never designed to retrieve mass amounts of data, but to get the properties of a bunch of selected elements. In the example I sent you above they are extracting all the properties, but you don't have to do that; you can just fetch the data you need and it should be pretty fast. You can also read the .frag file directly, which has some benefits over json. Additionally, we deduplicate a lot of data in .frag, so if you try to convert an 80 mb .frag file to json, you might end up with a 1gb .json file.

Just mentioning this in case you stumble any bottleneck that we can fix on our end. Otherwise, nevermind!

[CabeDigicorp]

@agviegas Thank you for the update, today I had the chance of trying it. Frag files are a bit larger but still a fraction of the original IFC file, so it's OK for us.
However...
... the getItemsData still requires a properly configured config parameter in order to retrieve all the info, but that's impossible for the exact same reasons that led me to the original request - we can't know what to expect in the IFC. So I guess that getItemsData() should also be modified in order to retrieve everything. Simply omitting the parameter leads to incomplete results (properties of measurement units are not retrieved, for instance).

Regarding properties, I guess we're totally off topic here. In the next few days I'll prepare some sample scenarios and write you a direct message on thatOpen Community about them. Thanks for your help!

[agviegas]

@CabeDigicorp Unerstood! If you create an issue for getItemsData having an "everything" option, we'll take care.