Fix empty urlClassification

[franciscawijaya]

As mentioned in the previous meeting, when I did the first batch of the crawl, all the data are collected except for the Firefox's urlClassification. When I open the site manually without a crawl, I was able to check the sites flagged by Firefox Tracking Protection

Some fundamental things that I checked:

• Opened the protection dashboard while the crawl is ongoing to check if it is a matter of there are first party and third party sites blocked but just not recorded or if it is the problem of Firefox's Enhanced Tracking Protection not flagging and blocking the necessary sites during the crawl. The observation showed that it is the latter.
• Used the code version for June crawl and ran the crawl for one isolated site and it still didn't collect the UrlClassification and the protections dashboard also suggests that 'No Trackers known to Nightly were detected on this page'

Since then, I have tried out different things and ruled out some of the possible causes:

• VPN: I tried with both on and off and different IP locations for the crawl but the issue still persists.
• Firefox version: I made sure the version that I used manually and for the crawl is the same via opening the setting for Firefox when the crawl is ongoing before the crawl exited Firefox to make sure of the version (there is a recent update of Firefox version on July 9)
• Dependabot update on Optmeowt extension: I repackaged the XPI extension to include the dependabot update as well as @Mattm27's edit on the workflow for the extension and checked it manually by dropping it to Firefox extension and open a site. I was able to get the urlClassification when doing it manually. However, when I ran the crawl for just one isolated site with the newly updated xpi file, the urlClassification data is not collected.

[franciscawijaya]

Main problem: there is no urlClassification data collected during the crawl because Firefox's Enhanced Tracking Protection did not detect any sites specifically during the crawl. (When visiting a site manually, opening the protection dashboard showed that Enhanced Tracking Protection detected and blocked first party and third party sites that fall under the relevant categories).

[franciscawijaya]

Ruled out the Firefox Nightly recent update: I downloaded the older nightly version (the one that I used for June crawl) with both the current version of the code and the June crawl version of the code and they both still gave an empty output for the urlClassification.

[franciscawijaya]

Ruled out the possibility of a problem from local device: I tried doing the crawl on my own computer and I also got the same empty output for urlClassification.

[franciscawijaya]

I read more about the Firefox's Enhanced Tracking Protection and I think the problem is coming from the act of crawl or Firefox's response to the crawl.

These are some of the observations that I made from the previous tries:

• Every time I entered a site manually (without using a crawl), the protection banner suggests that Firefox blocked trackers and harmful scripts on the site and I can see the sites that were blocked when clicking this protection banner (ie. the urlClassification that the crawl is looking for)
• When I ran the crawl on the same set of sites for testing, whenever I checked the protection banner while the crawl is ongoing it shows that the Enhanced Protection is turned on the site but Firefox did not block any trackers or scripts (I noted that the Enhanced Tracking Protection is not inactive which implies that it is a sign that the problem is coming from Firefox responding to our crawl)

I also ruled out the possibility of the settings of Enhanced Tracking Protection to be the cause of the problem: while the crawl was ongoing, I made sure all the settings for the Enhanced Tracking Protection is standard and same as its settings when I did it manually.

For now, it seems to me that the functionality of Enhanced Tracking Protection (ETP) during the crawl is the problem; while ETP is on and in the same settings for both conditions (manual and crawl access to the site), it did not successfully give the output of the sites blocked for the crawl.

[franciscawijaya]

As discussed, I have checked several things the past week,

• Dependencies: I went through the versions of all the dependencies for the July codebase and June codebase and they are identical (I also compared them to April's)
• Selenium version: I realized that our crawler has been using the same Selenium version (4.7.1 - released in December 2022 according to this archive).

For July, June, and even April crawl, the codebase is the same regarding the selenium-webcrawler used (see attached). I further confirmed this by checking the "resolved" field in the package-lock.json in our codebase; it is indeed the downloaded 4.7.1 version. So, npm install will install this exact version. The only way for us to update the selenium version when running the crawl is either manually editing the package.json file or running npm update selenium-webdriver. I can be sure that I used the version 4.7.1 for June crawl and presumably that is also the case for the previous crawls.

So, my last experiment is manually editing the package-lock.json file to update the selenium version to the latest one (4.23.0) in issue-122. However, it still gave me no urlClassification. Nevertheless, I don't think we can rule out selenium versions just yet, given that the jump from v4.7.1 to v4.23.0 is pretty big.

[SebastianZimmeck]

I went through the versions of all the dependencies for the July codebase and June codebase and they are identical (I also compared them to April's)

Thanks, @franciscawijaya! You compared the April/June dependency versions against the July dependency versions that you have installed locally on your computer, right? (as opposed to the July dependency versions that are committed here in the remote repo)

[franciscawijaya]

You compared the April/June dependency versions against the July dependency versions that you have installed locally on your computer, right?

Yes, correct.

[SebastianZimmeck]

@franciscawijaya and @Mattm27, there are three places with dependencies:

• Top-level: https://github.com/privacy-tech-lab/gpc-web-crawler/blob/main/package.json and https://github.com/privacy-tech-lab/gpc-web-crawler/blob/main/package-lock.json
• REST API: https://github.com/privacy-tech-lab/gpc-web-crawler/blob/main/rest-api/package.json and https://github.com/privacy-tech-lab/gpc-web-crawler/blob/main/rest-api/package-lock.json
• Crawler: https://github.com/privacy-tech-lab/gpc-web-crawler/blob/main/selenium-optmeowt-crawler/package.json and https://github.com/privacy-tech-lab/gpc-web-crawler/blob/main/selenium-optmeowt-crawler/package-lock.json

When you go back in time to check if an updated dependency is causing the issue, consider that any of these may be relevant.