scaling problem on Elements.update_element_dataframe

[Cubewise-JoeCHK]

Describe what did you try to do with TM1py
We are currently facing challenges with synchronizing a large dimension (>800K elements) across TM1 instances. The requirement is to perform a full synchronization of the dimension each time. However, when utilizing the hierarchies.update_element_dataframe method, we encounter an error in the target instance due to the command line limitation of 16K in the TI process.

Error Encountered
The error arises from exceeding the command line length limit when attempting to execute the update.

To reproduce the use case

1. Dimension A: This is the source dimension, which contains over 20K elements and has a complex rollup structure.
2. Dimension B: This is the target dimension that receives updates from Dimension A. It also has a different complex rollup structure, which necessitates adding and removing edges during the update process.
3. Use elements.get_elements_dataframe() to retrieve the element table from Dimension A.
4. Use hierarchies.update_elements_dataframe() to update the element table in Dimension B.

Version

• TM1py: 2.0.4
• TM1 Server Version: 3.12.7

[macsir]

Haven’t used dataframe functions yet but wouldn’t hierarchy get/update functions do the same sync? If attributes values are needed, then you can do it separately with attribute cube. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Joe Chow HK ***@***.***> Sent: Wednesday, October 23, 2024 4:46:22 PM To: cubewise-code/tm1py ***@***.***> Cc: Subscribed ***@***.***> Subject: [cubewise-code/tm1py] scaling problem on Elements.update_element_dataframe (Issue #1184) Describe what did you try to do with TM1py We are currently facing challenges with synchronizing a large dimension (>800K elements) across TM1 instances. The requirement is to perform a full synchronization of the dimension each time. However, when utilizing the hierarchies.update_element_dataframe method, we encounter an error in the target instance due to the command line limitation of 16K in the TI process. image.png (view on web)<https://github.com/user-attachments/assets/2b680ae3-fbcf-4f28-9edd-4c4f3ebf8335> Error Encountered The error arises from exceeding the command line length limit when attempting to execute the update. To reproduce the use case 1. Dimension A: This is the source dimension, which contains over 20K elements and has a complex rollup structure. 2. Dimension B: This is the target dimension that receives updates from Dimension A. It also has a different complex rollup structure, which necessitates adding and removing edges during the update process. 3. Use elements.get_elements_dataframe() to retrieve the element table from Dimension A. 4. Use hierarchies.update_elements_dataframe() to update the element table in Dimension B. Version * TM1py: 2.0.4 * TM1 Server Version: 3.12.7 — Reply to this email directly, view it on GitHub<#1184>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAZQV7NW5NOLYUJY6DEFBK3Z45A35AVCNFSM6AAAAABQOCCG6WVHI2DSMVQWIX3LMV43ASLTON2WKOZSGYYDONBXGEYDCOI>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[MariusWirtz]

@Cubewise-JoeCHK
Thanks for raising this issue.
We need to switch the edge deletion from use_ti to use_blob. Same thing that we did for the write function a while back.

@macsir
the metadata update function with dataframe has some really cool additional features in it.

• never deletes leaves
• optional unwind or consolidation unwind (coming thanks to @Mr-SabyasachiBose)
• edges validation
• alias validation
• the dataframe follows the idea of leveled hierarchy table which is common across systems.