User Manual for PriceProbe

MFLG Staff - Users

Use the google account found in the tech report for login. The account is the same as the google developer account. For security, we are not putting the credentials here, so contact one of the project members for a copy of the tech report!

MFLG Staff - Admin

Follow the following steps to add more users to PriceProbe.

Step 1: Go to Google Developer Console: https://console.cloud.google.com/cloud-resource-manager

Step 2: Log in to the google developer account. This account can be found in the tech report.

Step 3: Go to the navigation menu > APIs and Services > OAuth consent screen.

Step 4: Scroll down to the Test users section and add or delete authorised users.

Developer

Setting up Supabase

Supabase is the database used for this project. Below are the steps to set up Supabase.

Step 1: create an account on Supabase

Step 2: Log in to your account and go to the Dashboard page. Create a new project by clicking New project

Step 3: Create tables

• Click SQL Editor at the left sidebar.
• Run the following query below. This query creates the scraper schema and the cleaned table. It also adds Row Level Security to ensure that only authenticated users can insert data into the table.

-- Creates customised schema called 'scraper' and allows connection to local computer
create schema if not exists scraper;
GRANT USAGE ON SCHEMA scraper TO anon, authenticated, service_role;
GRANT ALL ON ALL TABLES IN SCHEMA scraper TO anon, authenticated, service_role;
GRANT ALL ON ALL ROUTINES IN SCHEMA scraper TO anon, authenticated, service_role;
GRANT ALL ON ALL SEQUENCES IN SCHEMA scraper TO anon, authenticated, service_role;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA scraper GRANT ALL ON TABLES TO anon, authenticated, service_role;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA scraper GRANT ALL ON ROUTINES TO anon, authenticated, service_role;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres IN SCHEMA scraper GRANT ALL ON SEQUENCES TO anon, authenticated, service_role;


-- Creates the 'cleaned' table in the scraper schema.
CREATE TABLE if not exists  scraper.cleaned(
 id BIGINT,
 title TEXT,
 url TEXT,
 description TEXT,
 original_price DOUBLE PRECISION,
 discounted_price DOUBLE PRECISION,
 category TEXT,
 photo_url TEXT,
 booked_number BIGINT,
 review_score REAL,
 review_number BIGINT,
 tags TEXT,
 is_mflg BOOLEAN
);


-- Add Row Level security
create policy "Enable insert for authenticated users only"
on "scraper"."cleaned"
to authenticated
with check (
 true
);

Step 4: Enable RLS Go to the table editor at the left side bar. Then, go to the table we just created. Next, click the RLS disabled button at the top right, and enable RLS.

Step 5: Get Supabase API keys These keys are necessary to connect your python scripts to Supabase. To get the APIs, go to project settings at the left sidebar. Then, click API under the CONFIGURATION heading. Note down the Project URL and the Project API Keys (The one that is for anon/public). They are located at the red boxes in the image below.

Step 6: Put the Supabase API key and URL in a .env file in your local repository. Under the same directory as the docker-compose.yaml file, create a .env file with the Project URL and Project API Key. Do not push this file to your github/gitlab.

Compute Analytics

This step will take a very long time without a GPU, so we have provided some precalculated analytics results in a separate git repo.The associated pre-scraped product data is also available in that repo as cleaned_rows.csv. To use it, do

cd ./analytics/
mkdir export
cd ./export/
git clone https://github.com/shotnothing/DSA3101-Checkpoint.git`
mv ./DSA3101-Checkpoint/* ./ 
rm -rf ./DSA3101-Checkpoint
# In windows, rm -r -Force ./DSA3101-Checkpoint
cd ../../

Alternatively, if you want to run it yourself (WARNING: It can take >1 hour to run if you don't have a good GPU), do

mkdir ./analytics/export
python ./analytics/analytics.py

Setup

Go to the README for more details!