preg_match(): compilation failed: regular expression is too large to offset 143690

[lonelyrider44]

This happens with v2.8.0.
2.7.0 works fine

[JohnMirro]

I get this issue too with v 2.8.0

[k00ni]

This happens with v2.8.0. 2.7.0 works fine

What? Where? Versions? I can't see where the problem is and why. Please provide information about your setup, a stack trace of the error (or at least file line) and a PDF/example string which triggers the exception.

CC @GreyWyvern might be relevant to you.

[GreyWyvern]

Likely the result of binary content slipping through to the formatContent() function. The regexp looks for balanced parentheses in document content (where balance is required), but a binary stream can loop into a huge futile regexp.

Any chance you could post the offending PDFs, @lonelyrider44 or @JohnMirro ?

[JohnMirro]

@GreyWyvern sorry, I lost those PDF

[lonelyrider44]

InvoicesMN0052-2320230303093445.pdf

Stack trace:

[2024-01-24 21:30:54] local.ERROR: preg_match(): Compilation failed: regular expression is too large at offset 143690 {"exception":"[object] (ErrorException(code: 0): preg_match(): Compilation failed: regular expression is too large at offset 143690 at C:\\dev\\vendor\\smalot\\pdfparser\\src\\Smalot\\PdfParser\\PDFObject.php:221)
[stacktrace]
#0 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Foundation\\Bootstrap\\HandleExceptions.php(270): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError(2, 'preg_match(): C...', 'C:\\\\dev\\\\bs\\\\city-...', 221)
#1 [internal function]: Illuminate\\Foundation\\Bootstrap\\HandleExceptions->Illuminate\\Foundation\\Bootstrap\\{closure}(2, 'preg_match(): C...', 'C:\\\\dev\\\\bs\\\\city-...', 221)
#2 C:\\dev\\vendor\\smalot\\pdfparser\\src\\Smalot\\PdfParser\\PDFObject.php(221): preg_match('/\\\\(\\\\(\\x0E\\xD2\\xD5`\\\\000\\\\0...', '\\x01\\xFF\\xFF\\xFF\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...', Array)
#3 C:\\dev\\vendor\\smalot\\pdfparser\\src\\Smalot\\PdfParser\\PDFObject.php(345): Smalot\\PdfParser\\PDFObject->formatContent('\\x01\\xFF\\xFF\\xFF\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...')
#4 C:\\dev\\vendor\\smalot\\pdfparser\\src\\Smalot\\PdfParser\\PDFObject.php(598): Smalot\\PdfParser\\PDFObject->getSectionsText('\\x01\\xFF\\xFF\\xFF\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x00...')
#5 C:\\dev\\vendor\\smalot\\pdfparser\\src\\Smalot\\PdfParser\\PDFObject.php(577): Smalot\\PdfParser\\PDFObject->getTextArray(Object(Smalot\\PdfParser\\Page))
#6 C:\\dev\\vendor\\smalot\\pdfparser\\src\\Smalot\\PdfParser\\PDFObject.php(695): Smalot\\PdfParser\\PDFObject->getText(Object(Smalot\\PdfParser\\Page))
#7 C:\\dev\\vendor\\smalot\\pdfparser\\src\\Smalot\\PdfParser\\PDFObject.php(577): Smalot\\PdfParser\\PDFObject->getTextArray(Object(Smalot\\PdfParser\\Page))
#8 C:\\dev\\vendor\\smalot\\pdfparser\\src\\Smalot\\PdfParser\\Page.php(220): Smalot\\PdfParser\\PDFObject->getText(Object(Smalot\\PdfParser\\Page))
#9 C:\\dev\\vendor\\smalot\\pdfparser\\src\\Smalot\\PdfParser\\Document.php(438): Smalot\\PdfParser\\Page->getText()
#10 C:\\dev\\app\\Services\\MoneyOrderPdfService.php(209): Smalot\\PdfParser\\Document->getText()
#11 C:\\dev\\app\\Services\\MoneyOrderPdfService.php(186): App\\Services\\MoneyOrderPdfService->get_pdf_text('2023-02/Invoice...')
#12 C:\\dev\\app\\Services\\MoneyOrderPdfService.php(55): App\\Services\\MoneyOrderPdfService->get_pdf_data('2023-02', '2023-02/Invoice...')
#13 [internal function]: App\\Services\\MoneyOrderPdfService->App\\Services\\{closure}('2023-02/Invoice...', 0)
#14 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Collections\\Arr.php(560): array_map(Object(Closure), Array, Array)
#15 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Collections\\Collection.php(768): Illuminate\\Support\\Arr::map(Array, Object(Closure))
#16 C:\\dev\\app\\Services\\MoneyOrderPdfService.php(53): Illuminate\\Support\\Collection->map(Object(Closure))
#17 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Collections\\Traits\\EnumeratesValues.php(235): App\\Services\\MoneyOrderPdfService->App\\Services\\{closure}(Object(Illuminate\\Support\\Collection), 0)
#18 C:\\dev\\app\\Services\\MoneyOrderPdfService.php(40): Illuminate\\Support\\Collection->each(Object(Closure))
#19 C:\\dev\\app\\Console\\Commands\\SyncPdfFilesCommand.php(50): App\\Services\\MoneyOrderPdfService->process('2023-02', '2023-02/Invoice...')
#20 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Container\\BoundMethod.php(36): App\\Console\\Commands\\SyncPdfFilesCommand->handle()
#21 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Container\\Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#22 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Container\\BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure(Object(Closure))
#23 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Container\\BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod(Object(Illuminate\\Foundation\\Application), Array, Object(Closure))
#24 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Container\\Container.php(661): Illuminate\\Container\\BoundMethod::call(Object(Illuminate\\Foundation\\Application), Array, Array, NULL)
#25 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Console\\Command.php(183): Illuminate\\Container\\Container->call(Array)
#26 C:\\dev\\vendor\\symfony\\console\\Command\\Command.php(326): Illuminate\\Console\\Command->execute(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#27 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Console\\Command.php(152): Symfony\\Component\\Console\\Command\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Illuminate\\Console\\OutputStyle))
#28 C:\\dev\\vendor\\symfony\\console\\Application.php(1078): Illuminate\\Console\\Command->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#29 C:\\dev\\vendor\\symfony\\console\\Application.php(324): Symfony\\Component\\Console\\Application->doRunCommand(Object(App\\Console\\Commands\\SyncPdfFilesCommand), Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#30 C:\\dev\\vendor\\symfony\\console\\Application.php(175): Symfony\\Component\\Console\\Application->doRun(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#31 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Console\\Application.php(102): Symfony\\Component\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#32 C:\\dev\\vendor\\laravel\\framework\\src\\Illuminate\\Foundation\\Console\\Kernel.php(155): Illuminate\\Console\\Application->run(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#33 C:\\dev\\artisan(35): Illuminate\\Foundation\\Console\\Kernel->handle(Object(Symfony\\Component\\Console\\Input\\ArgvInput), Object(Symfony\\Component\\Console\\Output\\ConsoleOutput))
#34 {main}
"}

[cxammar]

Yup, exactly same issue here when uploading payrolls as pdf. Unfortunately cannot provide pdf example as it contains personal data

[GreyWyvern]

InvoicesMN0052-2320230303093445.pdf

Thanks for this!

So yes, this is a matter of binary content (an image, or something else) being passed to formatContent(). formatContent() does a check for binary content, but only after the problematic regexp. Unfortunately it's required to be this way because the check for binary content may interpret string content as binary, which is perfectly valid. This is the whole reason why strings are removed, then put back.

Either there needs to be a more thorough check for binary before formatContent() is called, or we need an alternate way to detect if a content stream is binary before the regexp is called.

One way may be to bulk remove everything from the content stream from the first parenthesis (start of the first supposed string) to the end of the stream, which would require only a minimal regexp. Everything left would have to be plain text abiding by a predictable regexp to show validity. If any binary bytes exist in what's left, the stream could safely be discarded.

If anyone has any other ideas, let me know.

[GreyWyvern]

One way may be to bulk remove everything from the content stream from the first parenthesis (start of the first supposed string) to the end of the stream, which would require only a minimal regexp. Everything left would have to be plain text abiding by a predictable regexp to show validity. If any binary bytes exist in what's left, the stream could safely be discarded.

Something like this, I think, would cover all the bases:

        $testBinary = preg_replace('/\(.*$/s', '', $content);
        if (!preg_match('/^[a-zA-Z0-9 \r\n\/\[\].-]*$/', $testBinary)) {
            return '';
        }

This code resolves the error for this file, but other files may bring up other characters that need to be included in the regexp.

[k00ni]

This code resolves the error for this file, but other files may bring up other characters that need to be included in the regexp.

Does it make sense to establish a switch using our Config class to enable/disable this check? It could be extended so that the developer can not only enable this check but also provides a custom regex-pattern (which overrides the standard one).

WDYT?