High CPU load on js-controller after triggering historical data

[bjoernroos]

After Triggering the historical data eg. day, the cpu load of the js-conroller increase between 60 and 100%.
When i disable the adapter the cpu-load of the js-contoller sink between 4 and 15%.
After enabling the adpater the cpu load stays at this level. But after triggering the cpu load increase again and stay at this level.

I use iobroker 4.0.23
node.js: 16.17.0
npm: 8.11.0
e3dc: 1.0.6
ubuntu: 20.04 LTS

[git-kick]

@bjoernroos, thank you for the report. I can reproduce the behavior you described (I was not alerted so far because the virtual machine running my ioBroker stays at ca. 20% load, not 60-100% as in your environment).

I will check the code.

[git-kick]

The load accumulates, i.e. by triggering DB requests more times, I can raise the CPU load to 90-100% steady state.
This looks like a "garbage collection" issue, like objects (which I meant to be deleted) remaining and using resources.

Strange is that I could not reproduce this on my development ioBroker instance running on Windows.
If this is confirmed, then we see a Linux-specific issue here, probably within ioBroker.

But this is only an intermediate status, work in progress.

[bjoernroos]

Thank you for the update. Do you explicite delete the objects or do you trigger the garbage collector. I know that javascript can get be difficult with memory leaks.

[git-kick]

I am using Adapter.delObject() - no explicit triggering of js garbage collector.

Concerning Windows vs Linux, I can see now that there is also an initial ca. 10% CPU load increase after triggering the DB request. The CPU load increase is much smaller, but this may be due to the fact that my dev instance is running on a real machine, not a virtual machine.
So I think the hypothesis above is wrong and it is probably not a Linux specific issue.

Any hint concerning ioBroker's object lifecycle is very welcome.

I assume the issue is related to this discussion (German).

[git-kick]

I think I have fixed the problem by deleting recursively: delObject( id, {recursive: true} )
(The old code did id with a nested loop, but obviously thas was somehow wrong.)

You could do me a favour by testing from the master branch
If everythig is ok, I will release v1.0.7

[ArnoD15]

Has this possibly also solved the problem that the CPU load increases, if the polling interval is set to s with too many tags?

[bjoernroos]

I tested the version and started several data queries. The CPU load is in the normal range. Everything seems to be working.
Many Thanks.

[git-kick]

Has this possibly also solved the problem that the CPU load increases, if the polling interval is set to s with too many tags?

@ArnoD15, negative. The CPU load remained high while having no more DB requests from the adapter to the E3/DC.
Also, the issue was solved by the following change at a code section which is only called when storing incoming DB data:

Old code:

	// Delete VALUE_x object branches for a certain x, x+1, ...
	deleteValueObjects( count, path ) {
		const id = `DB.${path}VALUE_${count.toString().padStart(2,"0")}`;
		this.getStates( `${id}.*`, (err, states) => {
			if(states) {
				for( const state in states ) {
					this.delObject( state );
				}
				this.delObject( id );
				this.deleteValueObjects( count+1, path );
			}
		});
	}

New code:

	// Delete VALUE_x object branches for a certain x, x+1, ...
	deleteValueObjects( count, path ) {
		const id = `${this.common.name}.${this.instance}.DB.${path}VALUE_${count.toString().padStart(2,"0")}`;
		this.getForeignObject( id, (err, obj) => {
			if(obj) {
				this.delObject( id, {recursive:true} );
				this.deleteValueObjects( count+1, path );
			}
		});
	}

Insights what's wrong with "Old code" are appreciated.
(Anyway, the "New code" is smarter :-))

[git-kick]

I tested the version and started several data queries. The CPU load is in the normal range. Everything seems to be working. Many Thanks.

@bjoernroos, thank you for testing and supporting me in solving the issue. v1.0.7 release will follow soon.