Assertion error on buffer_id for yr_rules_load_stream

[pivotforensics]

I just recently upgraded to 4.0.2 from 3.11.0, and without any other changes I am getting a new assertion error that I wasn't previously getting when loading compiled Yara rules from memory with yr_rules_load_stream with "large" Yara rulesets.

Assertion failed: buffer_id < arena->num_buffers, file ......\libyara\arena.c, line 390

Example code:

YR_STREAM yr_stream_disk;
yr_stream_disk.user_data = &stream_disk;
yr_stream_disk.read = (YR_STREAM_READ_FUNC)read_yara_stream;
result = yr_rules_load_stream(&yr_stream_disk, &rules_disk);

stream_disk is a custom struct that gets passed to my read function read_yara_stream below:

size_t read_yara_stream(void *buffer, size_t size, size_t count, void *user_data) {
	char *buf = (char *)buffer;
	STREAM *stream = (STREAM *)user_data;

	size_t n;
	for (n = 0; n < count; n++) {
		if (size) {
			memcpy(buf, stream->_ptr, size);
			stream->_ptr += size;
			stream->_cnt -= size;
			buf += size;
		}
	}
	return n;
}

Based on my testing I can only reproduce this error when I am loading in rulesets that are larger than roughly ~180KB (non-compiled) and/or ~570KB compiled. This could still be user error on my part, but it seems odd that I can run these compiled rulesets (and some much larger) with the Yara binary, but then those same rulesets loading dynamically through yr_rules_load_stream error out with this assertion error once the rulesets get too large.

[pivotforensics]

I may have traced it back to a potential root cause, but I'm unsure if this is a bug or not...

When using the following two test rulesets (separate rule files):

rule failing {
    strings:
        $s1 = "testing"
        $s2 = "this"

    condition:
        $s1 or $s2
}

rule passing {
    strings:
        $s1 = "this"

    condition:
        $s1
}

I am originally compiling these rules on Ubuntu 20.04 with yara-python 4.0.2 and then running the compiled rules with the yara32.exe binary (also v4.0.2) on a Windows machine. The "passing" rule does not appear to have any issues, but the "failing" rule (presumably due to the multiple conditions?) fails on all of the files that pass to the second condition with "error code 4" (ERROR_COULD_NOT_MAP_FILE).

[plusvic]

I would like to understand the issue better, I have a few questions:

• How do you compile the rules that you load with yr_rules_load_stream?
• The rules are compiled using YARA 4.0.2?
• The rules are compiled in the same machine that later loads them?

[pivotforensics]

So it is very possible that these two issues I described above are unrelated, but I encountered both of them during my initial testing as described below.

First, to answer your questions:

• The rules are compiled using yara-python 4.0.2 on an Ubuntu 20.04.1 host
• Correct, yara-python 4.0.2 on the compiling host (Ubuntu 20.04.1) and 4.0.2 on the Windows host as well
• No, they are two different hosts

The first error (assertion error on rule loading, described in my first comment) is the initial issue I ran into, however during my testing to attempt to narrow down the focus on why I was encountering this error, I started to notice the second issue of getting consistent ERROR_COULD_NOT_MAP_FILE "error code 4" messages.

test.yar

rule failing {
    strings:
        $s1 = "testing"
        $s2 = "this"

    condition:
        $s1 or $s2
}

Compilation on the Linux host

>>> import yara
>>> yara.__version__
'4.0.2'
>>> rules = yara.compile('test.yar')
>>> rules.save('test.yarc')

Execution on Windows host (with matching files in current directory)
yara32.exe -C test.yarc .

I'm hoping that this potential issue with cross-platform compilation is the root cause of both of these problems.

[plusvic]

Ok, I see that you are compiling the rules with yara32.exe, assuming that your Linux host is a 64-bits system (which is probably the case) the issue may be related with compiling with a 32-bits YARA and loading them with a 64-bits. Maybe some regression was introduced in YARA 4.x

[pivotforensics]

I am actually doing the opposite: compiling on the Linux host in 64-bit and then running them on the Windows host with the 32-bit binary. Switching to the 64-bit binary on the Windows host did actually fix that issue for 64-bit -> 64-bit, however, when I compile on the Linux host with 32-bit Yara I am still encountering the same error when executing with the 32-bit binary on Windows.

[plusvic]

I did some tests in a Linux host, rules compiled with a 32-bits YARA loaded correctly with the 64-bits YARA and the other way around. I'll try to test it in Windows host.

[plusvic]

Fixed in 9fb4b8d