Error caused by using MAX_STATEMENTS_POST_11_8_15

[jakubdziarkowski]

Probably related to #774

Describe the bug
I was trying to use CellService.write_dataframe function with dataframe that [len(df) > Process.MAX_STATEMENTS = 16_380] and with use_ti=True. In previous TM1py version everything was fine, now I got error (in polish: "Błąd: Prolog wiersz procedury (16387): Przekroczono maks. liczbę instrukcji", it would be something like "Error: Prolog row (16387) max number of instructions exceeded" in english).

I believe that error is caused by CellService._execute_write_statements:

        if verify_version(required_version="11.8.015", version=self.version):
            max_statements = Process.MAX_STATEMENTS_POST_11_8_15

verify_version returns True for required_version="11.8.015", version="11.8.00100.13" (Is it expected behavior?)
so max_statements value is set to 100'000 and it doesn't make much sense even if more statements would be allowed (in my case it isn't allowed) because write_through_unbound_process still uses only Process.MAX_STATEMENTS and sends chunks with (2Process.MAX_STATEMENTS+1) or (2Process.MAX_STATEMENTS) statements to _execute_write_statements.

By the way this is another thing that I believe should be changed:
The first chunk that write_through_unbound_process creates has length = (2*Process.MAX_STATEMENTS+1)

To Reproduce
Described above.

Expected behavior
Use CellService.write_dataframe function to save data to TM1.

Version
TM1py 1.10.0
TM1 Server Version: 11.8.00100.13

Additional context

[MariusWirtz]

Thanks for raising the issue. It slipped through the testing.
The verify_version only checks the major and minor version digits e.g. 11 and 7 for 11.7.015.
I will push a new release to PyPI today to make sure the lower digits are considered.

The first chunk that write_through_unbound_process creates has length = (2*Process.MAX_STATEMENTS+1)

This is because the limitation is lines of code per code tab.
We squeeze MAX_STATEMENTS in the prolog and in the epilog.
Does that resolve your concern?

[MariusWirtz]

Hi @dziaro44,

please upgrade to this branch and verify if it resolves the issue:

pip install https://github.com/cubewise-code/tm1py/archive/refs/heads/issue/819.zip --upgrade

[jakubdziarkowski]

Hi @MariusWirtz
It resolves the issue but doesn't resolve my other concerns.

The more important thing is that If I would have version >= 11.8.015 then performance would not improve from

        if verify_version(required_version="11.8.015", version=self.version):
            max_statements = Process.MAX_STATEMENTS_POST_11_8_015

in _execute_write_statements function because write_through_unbound_process still uses Process.MAX_STATEMENTS regardless of version.

Second thing is related to

The first chunk that write_through_unbound_process creates has length = (2*Process.MAX_STATEMENTS+1)

This is because the limitation is lines of code per code tab.
We squeeze MAX_STATEMENTS in the prolog and in the epilog.
Does that resolve your concern?

Maybe I didn't describe it properly last time. I understand that idea is to send from write_through_unbound_process to _execute_write_statements chunk with 2*Process.MAX_STATEMENTS and then Process.MAX_STATEMENTS goes in prolog and Process.MAX_STATEMENTS in epilog. But with loop and if like this:

        for n, statement in enumerate(statements):
            chunk.append(statement)
            if n > 0 and n % (Process.MAX_STATEMENTS * 2) == 0:
                success, status, log_file = self._execute_write_statements(chunk, enable_sandbox, kwargs)
                successes.append(success)

chunk.append(statement) executes (Process.MAX_STATEMENTS * 2 + 1) times before first execution of _execute_write_statements so first chunk in this loop contains (Process.MAX_STATEMENTS * 2 + 1) statements, so _execute_write_statements sends Process.MAX_STATEMENTS statements in prolog and Process.MAX_STATEMENTS +1 statements in epilog.

So, this code is making use of fact that Process.MAX_STATEMENTS < "Real statement limitation" and there would be errors if Process.MAX_STATEMENTS = "Real statement limitation".

[MariusWirtz]

Hi @dziaro44,

thanks for the catch. You are right that we are missing a condition in the write_through_unbound_process function.
I will address that.

The +1 is not an issue, because the curret MAX_STATEMENTS has a small buffer of a few lines.
In reality, the max statements in a code tab is not 16_380 but something like 16_385 or so.
Does that resolve your concern?

Unfortunately, The impact on speed is not drastic as I had hoped. You can find the details here.
#774 (comment)
We are in discussions with IBM to resolve this bottleneck.

[jakubdziarkowski]

Thank you for the fast reply.

I'm closing Issue.

[MariusWirtz]

I appreciate that you raised the issue. I will merge and release to PyPI