TSC runs out of memory when TransportStream videos are present in the path

[Ved-s]

🔎 Search Terms

typescript transportstream

🕗 Version & Regression Information

• This a bug that causes OOM
• This is the behavior in every version I tried
• This behavior exists on previous versions, checked on 4.2.4

🙁 Actual behavior

I had a 300MB TransportStream (video.ts) video in my project dir, this caused tsc to try and process it, which caused an out-of-memory exception.

🙂 Expected behavior

tsc should ignore non-text files with .ts extensions, or blacklist videos or TransportStream specifically.

[jakebailey]

Did you only check 4.2.4, or have you checked the latest version? More recent versions (as of #53667) should be ignoring binary files.

[Ved-s]

@jakebailey I checked on latest that installs with npm -i typescript on node 20

[Ved-s]

node --version: v20.10.0
npm --version: 10.2.3
tsc --version: Version 5.3.2

[nmain]

More recent versions (as of #53667) should be ignoring binary files.

Only checking the first character isn't going to be sufficient for arbitrary binary files. A perfectly random sequence of bytes only has a ~47% chance to fail UTF8 decoding on the first character, and a valid MPEG-TS file has a 0% chance to fail UTF8 decoding there (the first byte is always ASCII "G").

[jakebailey]

Checking https://en.wikipedia.org/wiki/List_of_file_signatures, it's a bit frustrating that .ts files don't have magic besides the intro G. Though, the chart seems to claim that every 188th byte is G, which is a potential signal, but we don't have bytes, we have characters.

[fatcerberus]

I think the typical way to detect binary files is to read the first 80 or so chars and see if they’re all legal text; in the old days of ASCII the check is simply if more than a certain percentage of initial bytes are >127, then it’s probably binary. Such an algorithm seems easily adaptable to Unicode - just check for UTF8 decode failures instead. Probably has fewer false positives than the ASCII-based check, too.

[jakebailey]

@Ved-s @nmain Can you try the build on #57008 (comment) and see if that fixes your issue?

[Ved-s]

video.ts:1:1 - error TS1490: File appears to be binary.


Found 1 error in video.ts:1


real	0m7,533s
user	0m7,626s
sys	0m0,379s

it detects is as binary now, but it takes time to detect it, 8 seconds on just tsc (compile all) and 11s on tsc --watch on the first compile, then it compiles instantly, ignoring the video file

[jakebailey]

Yeah, this pass happens after the entire file is read. Well, at least it's not an OOM, so that's better.

[jakebailey]

Of course, you should just exclude that file from the project 😄