hi new project 🤩

.gitignore

@@ -0,0 +1,4 @@
+emoji-cdn.code-workspace
+emojis.txt
+bin/
+.emojis-db/

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) [year] [fullname]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,84 @@
+# ⚡️ Emoji CDN
+## a blazing fast emojis CDN that supports 30 emoji sets
+
+![GitHub License](https://img.shields.io/github/license/oddmario/emoji-cdn)
+![GitHub commit activity](https://img.shields.io/github/commit-activity/m/oddmario/emoji-cdn)
+![GitHub Issues or Pull Requests](https://img.shields.io/github/issues/oddmario/emoji-cdn)
+![GitHub Issues or Pull Requests](https://img.shields.io/github/issues-pr/oddmario/emoji-cdn)
+
+<h1 align="center">
+  🤩 🕺 🥳 
+</h1>
+
+-----
+
+## Usage
+Simply, use the base URL for **emoji-cdn** which is `https://emoji-cdn.mqrio.dev/[emoji]?style=[style]`
+
+The CDN will respond with an image containing the emoji for the specified style (emoji set/design)
+
+For example:
+```html
+<img src="https://emoji-cdn.mqrio.dev/🎉?style=google" />
+```
+
+## Supported styles
+- apple
+- google
+- samsung
+- microsoft-3D-fluent
+- microsoft
+- whatsapp
+- twitter
+- facebook
+- huawei
+- joypixels
+- lg
+- telegram
+- animated-noto-color-emoji
+- microsoft-teams
+- skype
+- twitter-emoji-stickers
+- joypixels-animations
+- serenityos
+- toss-face
+- sony
+- noto-emoji
+- openmoji
+- icons8
+- emojidex
+- messenger
+- htc
+- softbank
+- docomo
+- au-kddi
+- mozilla
+
+## Why not [benborgers/emojicdn](https://github.com/benborgers/emojicdn)?
+The emojicdn project by Ben Borgers is definitely a cool project! It actually is what inspired me to create this whole project.
+
+However, I found myself in the need of a little faster serving speed (despite how the benborgers CDN is fast enough).
+
+This project stores and fetches the emojis locally, unlike `benborgers/emojicdn` which behaves as a reverse proxy for a remotely hosted emojis source.
+
+In addition to that, I also needed to use some other styles which were missing from the benborgers project (especially the WhatsApp emoji set).
+
+So in order to make up for all that, this project was born.
+
+## Self Hosting
+You are absolutely more than welcome to self-host your own instance of **emoji-cdn** :)
+
+Here is how:
+
+1. Build the project:
+```
+python3 build.py
+```
+
+2. [Create your own Emojis database](/UPDATE_EMOJIS_DB.md)
+
+3. Run the application to start emoting!
+
+## License
+- Code licensed under the MIT License: http://opensource.org/licenses/MIT
+- All the emojis were obtained from [Emojipedia.org](https://emojipedia.org/) and are owned by their original distributors & designers

UPDATE_EMOJIS_DB.md

@@ -0,0 +1,54 @@
+# 🧐 Update/create the Emojis database
+
+## Stage I. Preparing the `emojis.txt` file
+
+In order to update or create the emojis database, this application requires an `emojis.txt` file containing a list of the emojis that it should scrape from Emojipedia.
+
+The `emojis.txt` file should be placed beside the binary file of the **emoji-cdn** application, and it should contain the emojis one per line.
+
+For example:
+```txt
+🫣
+😞
+```
+
+Below is one way to create an emojis.txt file containing all the emojis available:
+1. Go to https://getemoji.com/
+2. Perform any user interacion activity on the page (e.g. click anywhere on the page for example)
+3. Run the following snippet in the console of your browser:
+```js
+var finalOutput = "";
+
+document.querySelectorAll(".emoji-large").forEach((emoji) => {
+    finalOutput += emoji.textContent + "\n";
+});
+
+navigator.clipboard.writeText(finalOutput);
+```
+4. Voila! You have the emojis list on your clipboard now.
+5. Paste the clipboard into a file named `emojis.txt`
+6. Make sure the `emojis.txt` file is placed beside the binary file of **emoji-cdn**.
+
+## Stage II. Starting the scraper
+To start scraping from Emojipedia, run **emoji-cdn** in the scraping mode:
+```
+./emoji-cdn --update-db
+```
+
+To speed things up, the scraper scrapes in multiple threads. The default is 10 threads.
+
+To scrape faster, try a higher number of threads like 50:
+```
+./emoji-cdn --update-db --threads=50
+```
+
+Kindly note that the scraper may take some time even with a high number of threads. This happens due to the nature of the Emojipedia server response times (they are sometimes not the fastest), and also because some emoji images exceed the size of 500 KB.
+
+If you prefer starting the scraper in the background (for example, inside a `screen` session), feel free to do that :)
+
+### And tada!
+That's it really!
+
+Once the scraper finishes its job, you shall find all the scraped emojis in a directory named `.emojis-db` placed alongside the **emoji-cdn** binary.
+
+**emoji-cdn** will use the data inside that directory to serve its requests.

build.py

@@ -0,0 +1,66 @@
+import subprocess
+import os
+if os.name == 'nt':
+    import pyuac
+
+def patcher(undo = False):
+    patches = [] # we don't need any patches for now
+
+    for patch in patches:
+        filePath = ""
+        if os.name == 'nt':
+            filePath = patch['file_path_win']
+        else:
+            filePath = patch['file_path_lin']
+
+        with open(filePath, 'r', encoding="utf8") as file:
+            filedata = file.read()
+
+        if undo == False:
+            filedata = filedata.replace(patch['find'], patch['replace'])
+        else:
+            filedata = filedata.replace(patch['replace'], patch['find'])
+
+        with open(filePath, 'w', encoding="utf8") as file:
+            file.write(filedata)
+
+def cleanGoBuildCache():
+    subprocess.check_output(["go", "clean", "-cache"])
+
+def build():
+    sys_env = os.environ.copy()
+
+    print("Building for Linux amd64...")
+    sys_env['GOARCH'] = "amd64"
+    sys_env['GOOS'] = "linux"
+    sys_env['CGO_ENABLED'] = "0"
+    subprocess.check_output(["go", "build", "-o", "bin/emoji-cdn_linux_amd64"], env=sys_env)
+
+    print("Building for Linux arm64...")
+    sys_env['GOARCH'] = "arm64"
+    sys_env['GOOS'] = "linux"
+    subprocess.check_output(["go", "build", "-o", "bin/emoji-cdn_linux_arm64"], env=sys_env)
+
+    print("Building for Windows amd64...")
+    sys_env['GOARCH'] = "amd64"
+    sys_env['GOOS'] = "windows"
+    subprocess.check_output(["go", "build", "-o", "bin/emoji-cdn_windows_amd64.exe"], env=sys_env)
+
+def main():
+    patcher()
+    cleanGoBuildCache()
+    build()
+    patcher(True)
+
+    input("Done. Press any key to exit.")
+
+if __name__ == "__main__":
+    if os.name == 'nt':
+        if not pyuac.isUserAdmin():
+            # https://stackoverflow.com/a/19719292/8524395
+            print("Re-launching as admin...")
+            pyuac.runAsAdmin()
+        else:        
+            main()
+    else:
+        main()

cli/updateDb.go

@@ -0,0 +1,153 @@
+package cli
+
+import (
+	"bufio"
+	"errors"
+	"fmt"
+	"io"
+	"log"
+	"mario/emoji-cdn/constants"
+	"mario/emoji-cdn/lock"
+	"mario/emoji-cdn/utils"
+	"os"
+	"path/filepath"
+	"time"
+)
+
+func updateDbThread(emoji string, active_threads_count_lock, emojis_count_lock *lock.ThreadsafeVariable) {
+	active_threads_count_lock.Lock()
+	active_threads_count_lock.Set(int(active_threads_count_lock.Get().(int) + 1))
+	active_threads_count_lock.Unlock()
+
+	fmt.Println("started scraper for", emoji)
+
+	emojisDbPath := filepath.Join(".", ".emojis-db", emoji)
+	err := os.MkdirAll(emojisDbPath, os.ModePerm)
+	if err != nil {
+		active_threads_count_lock.Lock()
+		active_threads_count_lock.Set(int(active_threads_count_lock.Get().(int) - 1))
+		active_threads_count_lock.Unlock()
+
+		emojis_count_lock.Lock()
+		emojis_count_lock.Set(int(emojis_count_lock.Get().(int) + 1))
+		emojis_count_lock.Unlock()
+
+		return
+	}
+
+	for _, platform := range constants.EmojipediaPlatforms {
+		emoji_img_reader, file_ext, err := utils.EmojipediaScraper(emoji, platform)
+
+		if err != nil {
+			fmt.Println("scrape err", emoji, platform, err)
+
+			continue
+		}
+
+		f, err := os.OpenFile(filepath.Join(".", ".emojis-db", emoji, platform+file_ext), os.O_CREATE|os.O_WRONLY, 0777)
+		if err != nil {
+			fmt.Println("error while opening file", err)
+
+			emoji_img_reader.Close()
+
+			continue
+		}
+
+		io.Copy(f, emoji_img_reader)
+
+		f.Close()
+		emoji_img_reader.Close()
+
+		fmt.Println(emoji, platform, "has been scraped")
+	}
+
+	active_threads_count_lock.Lock()
+	active_threads_count_lock.Set(int(active_threads_count_lock.Get().(int) - 1))
+	active_threads_count_lock.Unlock()
+
+	emojis_count_lock.Lock()
+	emojis_count_lock.Set(int(emojis_count_lock.Get().(int) + 1))
+	emojis_count_lock.Unlock()
+}
+
+func UpdateDb(maxThreads int) {
+	emojisFilename := "emojis.txt"
+	path, _ := filepath.Abs("./" + emojisFilename)
+
+	if _, err := os.Stat(path); errors.Is(err, os.ErrNotExist) {
+		fmt.Println("The `emojis.txt` file does not exist. Exiting...")
+
+		return
+	}
+
+	emojisCount := 0
+
+	fileEmojisCount, err := os.Open(path)
+	if err != nil {
+		log.Fatal(err)
+
+		return
+	}
+	emojisCount, _ = utils.LineCounter(fileEmojisCount)
+	fileEmojisCount.Close()
+
+	file, err := os.Open(path)
+	if err != nil {
+		log.Fatal(err)
+
+		return
+	}
+	defer file.Close()
+
+	initEmojisDb := filepath.Join(".", ".emojis-db")
+	err = os.MkdirAll(initEmojisDb, os.ModePerm)
+	if err != nil {
+		log.Fatal(err)
+
+		return
+	}
+
+	var activeThreadsCountLock *lock.ThreadsafeVariable = new(lock.ThreadsafeVariable)
+	var emojisCountLock *lock.ThreadsafeVariable = new(lock.ThreadsafeVariable)
+
+	activeThreadsCountLock.Set(int(0))
+	emojisCountLock.Set(int(0))
+
+	scanner := bufio.NewScanner(file)
+
+	for scanner.Scan() {
+		for {
+			if activeThreadsCountLock.Get().(int) >= maxThreads {
+				time.Sleep(2 * time.Second)
+
+				continue
+			} else {
+				time.Sleep(500 * time.Millisecond)
+
+				if activeThreadsCountLock.Get().(int) >= maxThreads {
+					continue
+				} else {
+					break
+				}
+			}
+		}
+
+		go updateDbThread(scanner.Text(), activeThreadsCountLock, emojisCountLock)
+	}
+
+	for {
+		if emojisCountLock.Get().(int) >= emojisCount {
+			fmt.Println("Done.")
+
+			break
+		} else {
+			time.Sleep(2 * time.Second)
+
+			continue
+		}
+	}
+
+	if err := scanner.Err(); err != nil {
+		log.Fatal(err)
+	}
+}