First commit

Author: Omar

.gitignore

@@ -0,0 +1,14 @@
+exploration
+.env
+data_dumps
+data
+*.png
+*.csv
+sync_db.sh
+.DS_Store
+Makefile
+*.pyc
+__pycache__
+poetry.lock
+containers/scrapper/benchhmark
+demo_nomic.py

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Omar Mohammed
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,17 @@
+# HackerNews analysis - How to use it?
+
+Running the scraping script is very simple. You just need to run the following command:
+
+```shell
+cd ./scrapper
+poetry run python main.py
+```
+
+The analysis is spread across multiple scripts.
+
+```shell
+cd ./analysis
+poetry run python core_analysis.py # Main visualization
+poetry run python thematic_extraction.py # Use GPT-4 to extract the topics of interest
+poetry run python thematic_analysis.py # Analyze the topics extracted and visualize them
+```

analysis/.env_template

@@ -0,0 +1,6 @@
+OPENAI_API_KEY=
+HUGGINGFACE_AUTH_BEARER_TOKEN=
+MONGO_DB_URL=
+OPENAI_ORG_ID=
+DB_NAME=
+NOMIC_AI_KEY=