TNG Statement in CSV

This code builds a CSV table that records all transactions within TNG e-Wallet from its statement.

• How to download TNG Ewallet Statement
  • Download PDF via TNG eWallet App
  • Download PDF via TNG ePortal
• Usage
  • Clone Repo
  • Managing virtual environment
  • Dependencies
  • Run Python & Get your CSV
• Troubleshoot
  • Reversing Entries
  • Money Packets 2023 Balance
  • Missing Direct Credit Entries
  • GO+ Daily Earning showing 2 decimal places

How to download TNG Ewallet Statement

Do not go to Touch'n Go Web Portal, it's unrelated to TNG Ewallet

Download the PDF Statement via TNG eWallet App

Here're the official instruction provided by Touch'n Go in TNG eWallet app.

The instruction above is little bit outdated, a newer flow as of writing time is like:

• go to your "Transactions" in TNG eWallet app
• Click "View All", just need to go to the Transaction History page
• Check top right corner, click "Email"
• You have the option to choose to email Last 90 days transaction history or current search result
• Submit

After wait for awhile you should receive an email from Touch'n Go as shown below:

Download the attached PDF, make sure its name is tng_ewallet_transactions.pdf, and put this file into pdf/ folder of this project.

Download the PDF Statement via TNG ePortal

Here're the instruction to download the newer version of PDF Statement via TNG ePortal:

1. Log in to the TNG ePortal by scaning QR Code using your TNG eWallet APP.
2. Click See More button navigate to the My Transactions page.
3. Click the Download button to download the your statement.
4. Place the pdf statement under the pdf/ folder of this project

Usage

1. Clone this repo to your local.

   git clone https://github.com/Rexpert/TNG_Statement_in_CSV.git
   

2. It is recommended to setup virtual environment before installing the dependencies. You may choose either venv or conda to manage your virtual environment

   Managing virtual environment using venv

   
   

   This guide will help you create a Python virtual environment using venv and install dependencies from a requirements.txt file.

   First, You need to install Python 3

   To create a virtual environment:

   # Create a virtual environment (you can name it '.venv' or something else)
   python -m venv .venv

   Then, activate the virtual environment.

   # On Windows:
   .\.venv\Scripts\activate
   
   # On macOS/Linux:
   source .venv/bin/activate

   Once your virtual environment is activated, install the dependencies listed in requirements.txt by running:

   # Installing required dependencies
   pip install -r requirements.txt

   Alternatively you may install the package manually using the method mentioned in 3. Dependencies

   When you’re done working, you can deactivate the virtual environment by running:

   deactivate

   This will deactivate the current environment and return you to your system’s default Python environment.

   ---

   Managing virtual environment using conda

   
   

   Download & install miniconda from the official website

   To create a new conda environment (cloning the base):

   # # Create a conda environment (you can name it 'tng' or something else)
   conda create -n tng --clone base

   To switch over the tng environment

   # Activating 'tng' environment
   conda activate tng

   Once your conda environment is activated, install the dependencies listed in requirements.txt by running:

   # Installing required dependencies
   pip install -r requirements.txt

   Alternatively you may install the package manually using the method mentioned in 3. Dependencies

   To leave the tng environment

   # Deactivating 'tng' environment
   conda deactivate tng

3. Dependencies

   Recommended setup

   Installation	Version
   python	3.9.12
   camelot-py	0.10.1
   pandas	1.4.3
   numpy	1.26.4
   matplotlib	3.5.2
   opencv-python	4.10.0.84
   PyPDF2	2.12.1

   camelot-py: to read PDF statement

   • Installation via pip

     pip install camelot-py==0.10.1
     

   • or if you're using conda environment

     conda install -c conda-forge camelot-py==0.10.1
     

   • you will need to install its dependencies (TkInter & Ghostscript) before using, please refer installation step in the Manual
   • Detail installation please refer to camelot-py Documentation
   pandas: data manipulation

   • Installation via pip

     pip install pandas==1.4.3
     

   • or if you're using conda environment

     conda install -c conda-forge pandas==1.4.3
     

   • Detail installation please refer to pandas Documentation
   Numpy: Handle binary incompatibility for camelot

   • Installation via pip

     pip install numpy==1.26.4
     

   • or if you're using conda environment

     conda install -c conda-forge numpy==1.26.4
     

   • Detail installation please refer to numpy Documentation
   matplotlib: page visualization (Optional)

   • Installation via pip

     pip install matplotlib==3.5.2
     

   • or if you're using conda environment

     conda install -c conda-forge matplotlib==3.5.2
     

   • Visual Debug on table generation
   • Detail installation please refer to matplotlib Documentation
   opencv-python: Handle missing / no module named cv2

   • Installation via pip

     pip install opencv-python==4.10.0.84
     

   • or if you're using conda environment

     conda install -c conda-forge opencv==4.10.084
     

   PyPDF2: Handle PdfFileReader deprecation

   • Installation via pip

     pip install PyPDF2==2.12.1
     

   • or if you're using conda environment

     conda install -c conda-forge PyPDF2==2.12.1
     

4. Download your TNG statement (which is named tng_ewallet_transactions.pdf) and locating it into a pdf/ folder. By default, the program is designed to feed in the latest pdf with the fuzzy name matching *transactions*.pdf

5. Run the main.py

   python main.py
   

6. Get your transaction table named tng_ewallet_transactions.csv in csv/ folder.

Troubleshoot

Some known bugs happen during the generation of the pdf transaction report by TNG, but the only thing we can do is to manually make correction on the data:

1. Reverse Entry (Found on c5156d7)

   • The latest transaction recorded before an older transaction.
   • This usually happens during the Quick Reload Payment via Go+. In this scenario, the payment is recorded first, then the reload occurs after.
   • I have implemented an autofix in the code to address this.
   • Example:

2. Money Packet Balance (Found on 8fe26a5) (Fixed on 2024)

   • The wallet balance of the money packet entries unexpectedly equals the amount of money packet received.
   • This is happened in the CNY 2023 when the Money Packet Campaign took place.
   • I have implemented an autofix in the code to address this.
   • Example:
   • This bug was fixed on the pdf statement after CNY 2024:

3. Missing Direct Credit entry (Found on 362cc8a)

   • Direct Credit entries are not recorded in the transaction history
   • Some of us might be involved in the Weekly Check-in on the A+ reward. The check-in for 9 Sept 2023, 10 Sept 2023 and 12 Sept 2023 rewards free credits, but the transactions are not recorded in the pdf.
   • However, they can be viewed in the TNG e-Wallet app's history：
     • In the transaction history page, filter the transaction by Others > Direct Credit
       
   • You need to input those transactions manually if you were involved in those rewards, otherwise the ValueError: Some Entry Not Recorded Properly will be raised:
     • Heading to missing_data/missing_data.csv
     • Input the Direct Credit entries that obtained in the TNG eWallet App as the example shown:

       Date	Transaction Type	Description	Amount (RM)
       12/9/2023	Direct Credit	ADVANCED TECHNOLOGIES PTE LTD (EC)	0.5
       ...	...	...	...

     • After that, run main.py as usual:

       python main.py

   • Example:

4. GO+ Daily Earning showing 2 decimal places

   • In the newer version of TNG Statement downloaded via TNG ePortal, the GO+ Daily Earning showing only 2 decimal places instead of 4 decimal places in the TNG Statement downloaded via TNG eWallet APP
   • This discrepancy may cause rounding error when summing up the Wallet Balance.
   • Currently, we do not have fix on this issue.

5. Other Unknown Bugs

   • Any uncaught bug will raise ValueError: Some Entry Not Recorded Properly and exit the code unexpectedly. Please open an issue and attach/screenshot the relevant transaction history pdf if found such case.

Disclaimer

1. I don't work in Touch 'n Go company, and hence do not represent Touch 'n Go.
2. This repository is my work to ease myself in analyzing my own expenses in Touch 'n Go e-Wallet. But you can freely use it and welcome to contribute, you are helping me to make this code more meaningful.
3. Please consider use this code in your own responsibility, although it is not harmful at all.

MIT @Rexpert 2022 - 2024