[IMPROVEMENT] Rescue of PR 2106: Intelligent Teletext sentence merging

[Atul-Chahar]

In raising this pull request, I confirm the following (please check boxes):

• I have read and understood the contributors guide.
• I have checked that another pull request for this purpose does not exist (This rescues and replaces Fix fragmented teletext subtitles by delaying flush until sentence completion #2106).
• I have considered, and confirmed that this submission will be valuable to others.
• I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
• I give this submission freely, and claim no ownership to its content.
• I have mentioned this change in the changelog.

My familiarity with the project is as follows (check one):

• I am an active contributor to CCExtractor.

---

Summary

This PR rescues the Teletext sentence-merging feature proposed by @Apprentice2907 in #2106, while addressing the reviewer feedback to split it out from unassociated commits.

I have:

1. Created a clean branch from master containing only the telxcc.c changes to delay flushing subtitles until a sentence completes.
2. Fixed a critical buffer bug in the original PR: The original PR mistakenly used page_buffer_add_string to append the current fragment to itself, which would cause the buffer to be zeroed out and the fragment lost. I replaced this with proper manual reallocation and concatenation of ctx->page_buffer_prev and ctx->ucs2_buffer_prev so the fragments actually merge correctly.

All C and Rust tests pass locally.

Fixes #2106 requirement for a clean, focused PR.

[ccextractor-bot]

CCExtractor CI platform finished running the test files on linux. Below is a summary of the test results, when compared to test for commit 90128d8...:

Report Name	Tests Passed
Broken	10/13
CEA-708	2/14
DVB	2/7
DVD	3/3
DVR-MS	2/2
General	27/27
Hardsubx	1/1
Hauppage	3/3
MP4	3/3
NoCC	10/10
Options	76/86
Teletext	0/21
WTV	13/13
XDS	34/34

Your PR breaks these cases:

• ccextractor --out=srt --latin1 --autoprogram 73d9313d64...
• ccextractor --out=ttxt --latin1 001dd8cdf7...
• ccextractor --out=srt --latin1 4d4e938ef6...
• ccextractor --service 1 --out=txt --no-bom --no-rollup ea83ff7bcb...
• ccextractor --service 1 --out=txt f17524b53f...
• ccextractor --service 1 --out=txt 80848c45f8...
• ccextractor --service 1 --out=txt --no-bom --no-rollup b5d6aad89f...
• ccextractor --service 1[EUC-KR] --out=txt --no-rollup b5d6aad89f...
• ccextractor --service 1 --out=srt da904de35d...
• ccextractor --service 1 --out=sami da904de35d...
• ccextractor --service 1[EUC-KR] b5d6aad89f...
• ccextractor --service 1[EUC-KR] --no-rollup b5d6aad89f...
• ccextractor --service all da904de35d...
• ccextractor --service all[EUC-KR] b5d6aad89f...
• ccextractor --service 1,2[UTF-8],3[EUC-KR],54 --out=txt da904de35d...
• ccextractor --autoprogram --out=srt --latin1 f1422b8bfe...
• ccextractor --datapid 5603 --autoprogram --out=srt --latin1 --teletext 85c7fc1ad7...
• ccextractor --autoprogram --out=srt --latin1 d41b53b504...
• ccextractor --stdout --quiet --no-fontcolor 79a51f3500...
• ccextractor --stdout --quiet --no-fontcolor 767b546f96...
• ccextractor --service 1 c83f765c66...
• ccextractor --myth c83f765c66...
• ccextractor --in=raw fb79021542...
• ccextractor --mp4vidtrack 5df914ce77...
• ccextractor --startcreditsnotafter 2 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9...
• ccextractor --tpage 801 4e56e88ba4...
• ccextractor --tverbose 4e56e88ba4...
• ccextractor --teletext 4e56e88ba4...
• ccextractor --xmltv=3 --out=null 96efd279cf...
• ccextractor --datapid 2310 --autoprogram --out=srt --latin1 e639e54550...
• ccextractor --autoprogram --out=srt --latin1 4e56e88ba4...
• ccextractor --autoprogram --out=ttxt --latin1 c0d2fba8c0...
• ccextractor --autoprogram --out=ttxt --latin1 006fdc391a...
• ccextractor --autoprogram --out=ttxt --latin1 e92a1d4d2a...
• ccextractor --autoprogram --out=ttxt --latin1 b37ce60eb9...
• ccextractor --autoprogram --out=ttxt --latin1 7e4ebf7fd7...
• ccextractor --autoprogram --out=ttxt --latin1 9256a60e4b...
• ccextractor --autoprogram --out=ttxt --latin1 27d7a43dd6...
• ccextractor --autoprogram --out=ttxt --latin1 297a44921a...
• ccextractor --autoprogram --out=ttxt --latin1 efbe129086...
• ccextractor --autoprogram --out=ttxt --latin1 eae0077731...
• ccextractor --autoprogram --out=ttxt --latin1 e2e2b501e0...
• ccextractor --autoprogram --out=ttxt --latin1 8c1615c1a8...
• ccextractor --autoprogram --out=ttxt --latin1 c6407fb294...
• ccextractor --autoprogram --out=ttxt --latin1 --datets dcada745de...
• ccextractor --autoprogram --out=srt --latin1 --tpage 398 5d5838bde9...
• ccextractor --autoprogram --out=srt --latin1 --tpage 299 44c45593fb...
• ccextractor --autoprogram --out=srt --latin1 --tpage 299 b8c55aa2e9...
• ccextractor --autoprogram --out=srt --latin1 --teletext --tpage 398 3b276ad8bf...
• ccextractor --autoprogram --out=ttxt --latin1 b236a0590b...

NOTE: The following tests have been failing on the master branch as well as the PR:

• ccextractor --startcreditsforatmost 2 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed:

  Test 8730

Congratulations: Merging this PR would fix the following tests:

• ccextractor --autoprogram --out=srt --latin1 --quant 0 85271be4d2..., Last passed: Never
• ccextractor --autoprogram --out=ttxt --latin1 --ucla dab1c1bd65..., Last passed: Never
• ccextractor --out=srt --latin1 --autoprogram 29e5ffd34b..., Last passed: Never
• ccextractor --out=spupng c83f765c66..., Last passed: Never
• ccextractor --parsePAT --out=srt c83f765c66..., Last passed: Never
• ccextractor --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never
• ccextractor --startcreditsnotbefore 1 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never
• ccextractor --startcreditsforatleast 1 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never
• ccextractor --endcreditsforatleast 3 --endcreditstext "CCextractor Ends crdit Testing" addf5e2fc9..., Last passed: Never

It seems that not all tests were passed completely. This is an indication that the output of some files is not as expected (but might be according to you).

Check the result page for more info.

[ccextractor-bot]

CCExtractor CI platform finished running the test files on windows. Below is a summary of the test results, when compared to test for commit e4bcade...:

Report Name	Tests Passed
Broken	10/13
CEA-708	2/14
DVB	2/7
DVD	3/3
DVR-MS	2/2
General	27/27
Hardsubx	1/1
Hauppage	3/3
MP4	3/3
NoCC	10/10
Options	78/86
Teletext	0/21
WTV	13/13
XDS	34/34

Your PR breaks these cases:

• ccextractor --autoprogram --out=srt --latin1 f1422b8bfe...
• ccextractor --datapid 5603 --autoprogram --out=srt --latin1 --teletext 85c7fc1ad7...
• ccextractor --tpage 801 4e56e88ba4...
• ccextractor --tverbose 4e56e88ba4...
• ccextractor --teletext 4e56e88ba4...
• ccextractor --autoprogram --out=srt --latin1 4e56e88ba4...
• ccextractor --autoprogram --out=ttxt --latin1 c0d2fba8c0...
• ccextractor --autoprogram --out=ttxt --latin1 006fdc391a...
• ccextractor --autoprogram --out=ttxt --latin1 e92a1d4d2a...
• ccextractor --autoprogram --out=ttxt --latin1 b37ce60eb9...
• ccextractor --autoprogram --out=ttxt --latin1 7e4ebf7fd7...
• ccextractor --autoprogram --out=ttxt --latin1 9256a60e4b...
• ccextractor --autoprogram --out=ttxt --latin1 27d7a43dd6...
• ccextractor --autoprogram --out=ttxt --latin1 297a44921a...
• ccextractor --autoprogram --out=ttxt --latin1 efbe129086...
• ccextractor --autoprogram --out=ttxt --latin1 eae0077731...
• ccextractor --autoprogram --out=ttxt --latin1 e2e2b501e0...
• ccextractor --autoprogram --out=ttxt --latin1 8c1615c1a8...
• ccextractor --autoprogram --out=ttxt --latin1 c6407fb294...
• ccextractor --autoprogram --out=ttxt --latin1 --datets dcada745de...
• ccextractor --autoprogram --out=srt --latin1 --tpage 398 5d5838bde9...
• ccextractor --autoprogram --out=srt --latin1 --tpage 299 44c45593fb...
• ccextractor --autoprogram --out=srt --latin1 --tpage 299 b8c55aa2e9...
• ccextractor --autoprogram --out=srt --latin1 --teletext --tpage 398 3b276ad8bf...
• ccextractor --autoprogram --out=ttxt --latin1 b236a0590b...

NOTE: The following tests have been failing on the master branch as well as the PR:

• ccextractor --out=srt --latin1 --autoprogram 73d9313d64..., Last passed:

  Test 8611

• ccextractor --out=ttxt --latin1 001dd8cdf7..., Last passed:

  Test 8611

• ccextractor --out=srt --latin1 4d4e938ef6..., Last passed:

  Test 8611

• ccextractor --service 1 --out=txt --no-bom --no-rollup ea83ff7bcb..., Last passed:

  Test 8611

• ccextractor --service 1 --out=txt f17524b53f..., Last passed:

  Test 8611

• ccextractor --service 1 --out=txt 80848c45f8..., Last passed:

  Test 8611

• ccextractor --service 1 --out=txt --no-bom --no-rollup b5d6aad89f..., Last passed:

  Test 8611

• ccextractor --service 1[EUC-KR] --out=txt --no-rollup b5d6aad89f..., Last passed:

  Test 8611

• ccextractor --service 1 --out=srt da904de35d..., Last passed:

  Test 8611

• ccextractor --service 1 --out=sami da904de35d..., Last passed:

  Test 8611

• ccextractor --service 1[EUC-KR] b5d6aad89f..., Last passed:

  Test 8611

• ccextractor --service 1[EUC-KR] --no-rollup b5d6aad89f..., Last passed:

  Test 8611

• ccextractor --service all da904de35d..., Last passed:

  Test 8611

• ccextractor --service all[EUC-KR] b5d6aad89f..., Last passed:

  Test 8611

• ccextractor --service 1,2[UTF-8],3[EUC-KR],54 --out=txt da904de35d..., Last passed:

  Test 8611

• ccextractor --autoprogram --out=srt --latin1 d41b53b504..., Last passed:

  Test 8611

• ccextractor --stdout --quiet --no-fontcolor 79a51f3500..., Last passed:

  Test 8611

• ccextractor --stdout --quiet --no-fontcolor 767b546f96..., Last passed:

  Test 8611

• ccextractor --service 1 c83f765c66..., Last passed:

  Test 8611

• ccextractor --myth c83f765c66..., Last passed:

  Test 8611

• ccextractor --in=raw fb79021542..., Last passed:

  Test 8611

• ccextractor --mp4vidtrack 5df914ce77..., Last passed:

  Test 8611

• ccextractor --xmltv=3 --out=null 96efd279cf..., Last passed:

  Test 8611

• ccextractor --datapid 2310 --autoprogram --out=srt --latin1 e639e54550..., Last passed:

  Test 8611

Congratulations: Merging this PR would fix the following tests:

• ccextractor --autoprogram --out=srt --latin1 --quant 0 85271be4d2..., Last passed: Never
• ccextractor --autoprogram --out=ttxt --latin1 --ucla dab1c1bd65..., Last passed: Never
• ccextractor --out=srt --latin1 --autoprogram 29e5ffd34b..., Last passed: Never
• ccextractor --out=spupng c83f765c66..., Last passed: Never
• ccextractor --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never
• ccextractor --startcreditsnotbefore 1 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never
• ccextractor --startcreditsnotafter 2 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never
• ccextractor --startcreditsforatleast 1 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never
• ccextractor --startcreditsforatmost 2 --startcreditstext "CCextractor Start crdit Testing" c4dd893cb9..., Last passed: Never

It seems that not all tests were passed completely. This is an indication that the output of some files is not as expected (but might be according to you).

Check the result page for more info.