Update README.md

README.md

@@ -1,10 +1,12 @@
 # HKEx Announcement Classifier
 
-HKEx Announcement Classifier is a project on data exploration, analysis and finally training a recurrent neural network (RNN) to ~93-94% validation accuracy to classify disclosure announcements submitted by listed companies on the Hong Kong Stock Exchange (HKEx). **Jan 2021 Update**: The transformer architecture is unreasonably effective - not only surpassing the validation accuracy of RNN, but also requiring far fewer parameters.
+HKEx Announcement Classifier is a project on data exploration, analysis and finally training a recurrent neural network (RNN) to ~93-94% validation accuracy to classify disclosure announcements submitted by listed companies on the Hong Kong Stock Exchange (HKEx). 
+
+**Jan 2021 Update**: The transformer architecture is unreasonably effective - not only surpassing the validation accuracy of RNN, but also requiring far fewer parameters and far less time to train. 
 
 <img src="/images/prediction.png" width="600">
 
-In this project, I scraped data corresponding to 6 types of announcements, namely:
+In this project, I scraped data from the Hong Kong Stock Exchange corresponding to 6 types of announcements, namely:
 
 (1) Trading Halt; 
 
@@ -25,10 +27,25 @@ For example:
 - Accurately classifying legal texts can help lawyers drastically increase their efficiency in AI-assisted legal due diligence. 
 - The ability to classify important disclosure announcements on different stock exchanges is incredibly useful in news-based quantitative trading algorithms. 
 
-# Future Implementations
-The types of data can, in a future iteration of this project, include different types of contracts such as loan agreements, lease agreements, joint venture agreements, so on and so forth. The scope of jurisdiction can easily extend beyond Hong Kong, such as to include offshore jurisdictions (British Virgin Islands, Cayman Islands etc.) and international stock exchanges (e.g. NYSE, NASDAQ, LSE) for a more powerful legal text classifier. 
+# Training a Recurrent Neural Network 
+I implemented a recurrent neural network with a bidirectional LSTM layer after an embedding layer using pre-trained GloVe embeddings of 300 dimensions. The details of my implementation can be found <a href='HKEx_Announcement_Classifier.ipynb'>here</a>.
+
+Through fine-tuning hyperparameters of the model, I was able to improve on the initial training and validation accuracy of the model (72.3%, 60.0%) to（95.0%, 93.6%).
+
+# Training a Transformer 
+
+The transformer architecture was able to easily reach training accuracy of ~100% and validation accuracy of 95-96% with little fine-tuning. The implementation can be found <a href='Transformers - HKEX Annt Classifier.ipynb'>here</a>.
+
+## Comparison of RNN vs Transformer 
+### RNN 
+![Image of Training Plot](/images/training_plot.png)
 
-# Data Exploration and Analysis
+### Transformer 
+![Image of Transformer Training](/images/transformers_performance.png)
+
+The training of the Transformer model converged much faster than the RNN. The Transformer employed ~ 2.5 million parameters in total while the RNN used ~23 million parameters. After training for just 10 epochs, the Transformer was able to achieve ~95% accuracy on the validation set while the RNN was only able to achieve ~93% validation accuracy after 40 full epochs of training. This comparison illustrates the unreasonable effectiveness of the self-attention mechanism of the Transformer, not only foregoing the need for pre-trained word embeddings, but also vastly saving computational costs and improving accuracy. It comes as no surprise that the Transformer architecture has largely overshadowed or even rendered RNNs obsolete in the realm of natural language processing. 
+
+# What does the data look like ? - Data Exploration and Analysis
 Includes bi-gram analysis and average word count analysis across different types of announcements. The implementation and markdown write-up can be found <a href='Data%20Exploration%20for%20HKEX%20Announcements.ipynb'>here</a>.
 
 ## Bigram Analysis
@@ -69,26 +86,8 @@ Annual results occupy the number 1 spot in terms of length of announcement. This
 
 'Connected Transactions' and 'Notifiable Transactions' have the two highest mean word counts after annual results. This is due to these types of announcements often having to disclose the background of entering into such transactions at length.
 
-# Training a Recurrent Neural Network 
-I implemented a recurrent neural network with a bidirectional LSTM layer after an embedding layer using pre-trained GloVe embeddings of 300 dimensions. The details of my implementation can be found <a href='HKEx_Announcement_Classifier.ipynb'>here</a>.
-
-Through fine-tuning hyperparameters of the model, I was able to improve on the initial training and validation accuracy of the model (72.3%, 60.0%) to（95.0%, 93.6%).
-
-# Training a Transformer 
-
-The transformer architecture was able to easily reach training accuracy of ~100% and validation accuracy of 95-96% with little fine-tuning. The implementation can be found <a href='Transformers - HKEX Annt Classifier.ipynb'>here</a>.
-
-## Comparison of RNN vs Transformer 
-### RNN 
-![Image of Training Plot](/images/training_plot.png)
-
-### Transformer 
-![Image of Transformer Training](/images/transformers_performance.png)
-
-# Improving the Model 
-Further improvements on the model would include more data from different jurisdictions, and from different types of legal documents, so as to create a more general and more accurate legal text classifier.
-
 # Application
 As someone working in the legal industry, a legal text classifier would be immensely helpful to my day-to-day workflow, saving significant amounts of time spent on manually filing different types of legal texts. This type of classifier can save both time and costs in legal due diligence for Initial Public Offerings on Stock Exchanges. 
 
-
+# Future Implementations
+The types of data can, in a future iteration of this project, include different types of contracts such as loan agreements, lease agreements, joint venture agreements, so on and so forth. The scope of jurisdiction can easily extend beyond Hong Kong, such as to include offshore jurisdictions (British Virgin Islands, Cayman Islands etc.) and international stock exchanges (e.g. NYSE, NASDAQ, LSE) for a more powerful legal text classifier.