TypeScript declarations, RemoveEmptyLines, TrimLines, InsertFinalNewLine

[scripthunter7]

• TypeScript declarations + JSoc
  • Suggestions in VSCode:
    
• RemoveEmptyLines
  • Input:

    rule1
    
    rule2
    
    
    rule3
    

  • Output:

    rule1
    rule2
    rule3
    

• TrimLines
  • Input:

    rule1
      rule2
            rule3
        rule4
    

  • Output:

    rule1
    rule2
    rule3
    rule4
    

• InsertFinalNewLine
  • Input:

    rule1
    rule2
    

  • Output:

    rule1
    rule2
    
    

Test example

Input file test.txt:

        ||ads.geeks.hu^     
||adv.forpsi.hu^
||adv.hu^       
                ||adverbs.napilapcsoport.hu^
||advert*.ultraweb.hu^

! random comment


            ||adverticum.*.hu^
||app.livechatoo.com^
||as*.prog.hu^
! duplicate
||audit.median.hu^
||audit.median.hu^

||avatraffic.com^
||banner.terminal.hu^
||banner.webtar.hu^

Test script: hostlist-compiler-test.ts

import compile from '@adguard/hostlist-compiler';
import { writeFileSync } from 'fs';

(async () => {
    // Compile filters
    const result = await compile({
        name: 'Test Output',
        sources: [
            {
                type: 'adblock',
                source: 'test.txt',
                // Transformations for the imported source
                transformations: ['RemoveComments', 'Validate'],
            },
        ],
        // Transformations for the whole list
        transformations: ['Deduplicate', 'RemoveEmptyLines', 'TrimLines', 'InsertFinalNewLine'],
    });

    // Write to file
    writeFileSync('test-output.txt', result.join('\n'));
})();

Output file test-output.txt:

!
! Title: Test Output
! Last modified: 2022-06-02T22:08:39.598Z
!
! Compiled by @adguard/hostlist-compiler v1.0.14
!
!
! Source: release/test.txt
!
||ads.geeks.hu^
||adv.forpsi.hu^
||adv.hu^
||adverbs.napilapcsoport.hu^
||advert*.ultraweb.hu^
||adverticum.*.hu^
||app.livechatoo.com^
||as*.prog.hu^
||audit.median.hu^
||avatraffic.com^
||banner.terminal.hu^
||banner.webtar.hu^

"Real life" usage

hufilter-dns.ts:

import HostlistCompiler, { IConfiguration as HostlistCompilerConfiguration } from '@adguard/hostlist-compiler';
import { writeFileSync } from 'fs';

;(async () => {
    // Configuration
    const config: HostlistCompilerConfiguration = {
        name: 'hufilter DNS filter',
        description: 'Simplified hufilter version specifically to be better compatible with DNS-level ad blocking.',
        homepage: 'https://github.com/hufilter/hufilter/wiki',
        license: 'https://github.com/hufilter/hufilter/blob/master/LICENSE',
        sources: [
            {
                type: 'adblock',
                source: 'https://raw.githubusercontent.com/hufilter/hufilter/master/hufilter-adguard.txt',
                // Transformations for the imported source
                transformations: ['RemoveComments', 'Validate'],
            },
        ],
        // Transformations for the whole list
        transformations: ['Deduplicate', 'RemoveEmptyLines', 'TrimLines', 'InsertFinalNewLine'],
    };

    // Compile
    const result = await HostlistCompiler(config);

    // Write to file
    writeFileSync('hufilter-dns.txt', result.join('\n'));
})();

Related issues

• [Feature request] Optimization steps #12
• Remove blank lines #30

[ameshkov]

@scripthunter7 thank you for the contribution! @tvinzz will take care of it.

[maximtop]

@scripthunter7 lodash _.trim function for the second argument takes string, not array
with the array, it implicitly trims commas in the lines too, though there are no such cases in the tests

[maximtop]

please fix

[scripthunter7]

please fix

@maximtop Thank you! Fixed, see dbf44ad