remove duplicate lines when importing user filters

[lvmm]

Prerequisites

• I verified that this is not a filter issue
  • Filter issues MUST be reported at filter issue tracker
• This is not a support issue or a question
  • Support issues and questions are handled at /r/uBlockOrigin
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue
  • Your issue may already be reported.
• I tried to reproduce the issue when...
  • uBlock Origin is the only extension
  • uBlock Origin with default lists/settings
  • using a new, unmodified browser profile
• I am running the latest version of uBlock Origin
• I checked the documentation to understand that the issue I report is not a normal behavior

Description

Please provide ability to remove duplicate lines when importing filters the same way it now works for rules.

A specific URL where the issue occurs

[A specific URL is MANDATORY for issue happening on a web page, even if it happens "everywhere"]

Steps to Reproduce

1. Open the ublock dashboard
2. Go to My Filters tab
3. Export current filters to file
4. Import filters you've just exported.

Expected behaviour:

All imported lines should be marked as duplicates and ignored. Don't care what will happen with comments and cosmetic empty lines.

Actual behaviour:

All rules are imported and duplicated.

Your environment

• uBlock Origin version: 1.18.16, firefox-legacy 1.16.4.10
• Browser Name and version: chrome 72.0.3626.121 and seamonkey 2.53 (wg9s nightly)
• Operating System and version: windows 10, ubuntu 16.04

[uBlock-user]

The filters are imported but not saved automatically, you have to click Apply changes to save those duplicated filters. I'm conflicted on this, @gorhill what do you think ?

[lvmm]

Well, yes, I do have to commit, but identical files is just an example. IRL only some lines will be duplicated, sorting it all manually is not at all easy and turning filters into a duplicate-ridden mess is even worse. Actually, I am just trying to keep my filters in sync between different makes of browsers and with existing tools it is not straightforward.

[gwarser]

Actually, I am just trying to keep my filters in sync between different makes of browsers and with existing tools it is not straightforward.

Create one custom filter list and import in all browsers as "Custom" in Filter lists tab.

[lvmm]

Good idea - allows to implement cloud-agnostic filter syncing, albeit with some manual operations. But I immediately ran into a problem: when on windows machine I added file:///c:\googledrive\ublock.filter as a custom filter in firefox-legacy version of ublock it worked like a charm, but the same line doesn't work in the latest version of ublock: constantly out of date and 0 out of 0 loaded. Is it a bug?

update: in chrome granting 'Allow access to file URLs' permission for ublock fixed this problem, but I cannot find anything similar in firefox.

[gwarser]

Firefox bug/limitation https://bugzilla.mozilla.org/show_bug.cgi?id=1266960
gorhill/uBlock#2922