Skip to content

Commit e916ede

Browse files
SevelantisSevelantis
committed
first
0 parents  commit e916ede

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

54 files changed

+8454
-0
lines changed

‎.gitignore

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
develop-eggs/
13+
dist/
14+
downloads/
15+
eggs/
16+
.eggs/
17+
lib/
18+
lib64/
19+
parts/
20+
sdist/
21+
var/
22+
wheels/
23+
share/python-wheels/
24+
*.egg-info/
25+
.installed.cfg
26+
*.egg
27+
MANIFEST
28+
29+
# PyInstaller
30+
# Usually these files are written by a python script from a template
31+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
32+
*.manifest
33+
*.spec
34+
35+
# Installer logs
36+
pip-log.txt
37+
pip-delete-this-directory.txt
38+
39+
# Unit test / coverage reports
40+
htmlcov/
41+
.tox/
42+
.nox/
43+
.coverage
44+
.coverage.*
45+
.cache
46+
nosetests.xml
47+
coverage.xml
48+
*.cover
49+
*.py,cover
50+
.hypothesis/
51+
.pytest_cache/
52+
cover/
53+
54+
# Translations
55+
*.mo
56+
*.pot
57+
58+
# Django stuff:
59+
*.log
60+
local_settings.py
61+
db.sqlite3
62+
db.sqlite3-journal
63+
64+
# Flask stuff:
65+
instance/
66+
.webassets-cache
67+
68+
# Scrapy stuff:
69+
.scrapy
70+
71+
# Sphinx documentation
72+
docs/_build/
73+
74+
# PyBuilder
75+
.pybuilder/
76+
target/
77+
78+
# Jupyter Notebook
79+
.ipynb_checkpoints
80+
81+
# IPython
82+
profile_default/
83+
ipython_config.py
84+
85+
# pyenv
86+
# For a library or package, you might want to ignore these files since the code is
87+
# intended to run in multiple environments; otherwise, check them in:
88+
# .python-version
89+
90+
# pipenv
91+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
92+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
93+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
94+
# install all needed dependencies.
95+
#Pipfile.lock
96+
97+
# poetry
98+
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
99+
# This is especially recommended for binary packages to ensure reproducibility, and is more
100+
# commonly ignored for libraries.
101+
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
102+
#poetry.lock
103+
104+
# pdm
105+
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
106+
#pdm.lock
107+
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
108+
# in version control.
109+
# https://pdm.fming.dev/#use-with-ide
110+
.pdm.toml
111+
112+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
113+
__pypackages__/
114+
115+
# Celery stuff
116+
celerybeat-schedule
117+
celerybeat.pid
118+
119+
# SageMath parsed files
120+
*.sage.py
121+
122+
# Environments
123+
.env
124+
.venv
125+
env/
126+
venv/
127+
ENV/
128+
env.bak/
129+
venv.bak/
130+
131+
# Spyder project settings
132+
.spyderproject
133+
.spyproject
134+
135+
# Rope project settings
136+
.ropeproject
137+
138+
# mkdocs documentation
139+
/site
140+
141+
# mypy
142+
.mypy_cache/
143+
.dmypy.json
144+
dmypy.json
145+
146+
# Pyre type checker
147+
.pyre/
148+
149+
# pytype static type analyzer
150+
.pytype/
151+
152+
# Cython debug symbols
153+
cython_debug/
154+
155+
# PyCharm
156+
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
157+
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
158+
# and can be added to the global gitignore or merged into this file. For a more nuclear
159+
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
160+
#.idea/
161+
162+
tracks/
163+
out/
164+
**.DS_Store**
165+
upload_storage/
166+
storage_*/
167+
.vscode/

‎CD_report.pdf

1.07 MB
Binary file not shown.

‎LICENSE

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2023 DETI - UA
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎Makefile

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
up:
2+
docker-compose up activemq -d
3+
4+
down:
5+
docker-compose stop activemq
6+
7+
clean:
8+
docker rmi -f $(shell docker images -aq)
9+
docker rm -f $(shell docker ps -aq)
10+
docker ps -a
11+
docker images -a
12+
13+
14+
workers:
15+
source venv/bin/activate && python src/audio_worker/client.py
16+
17+
server:
18+
source venv/bin/activate && python src/main.py

‎README.md

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Goal
2+
University of Aveiro
3+
4+
The goal of this project was to implement a distributed system to process time-costly audio processing jobs, by distributing them across many remote workers.
5+
6+
The end-user doesn't need to wait years, but seconds.
7+
8+
# Prominent project highlights
9+
- **Fanout exchange queue** - allows to send exactly one message to every subscribed client
10+
- **Fault Tolerance** - ACK/NACK messages, adjusting clients' heartbeats
11+
- **Performance** - Identifying bottlenecks is the first thing to consider to tune performance.
12+
13+
## Dependencies
14+
15+
For Ubuntu (and other debian based linux), run the following commands:
16+
17+
```bash
18+
sudo apt install ffmpeg
19+
sudo apt install make
20+
```
21+
22+
## setup local python environment
23+
Run the following commands to setup the environement:
24+
```bash
25+
python3 -m venv venv
26+
source venv/bin/activate
27+
28+
pip install pip --upgrade
29+
python3 -m venv --upgrade venv
30+
pip install -r requirements.txt
31+
```
32+
## setup the broker
33+
- `docker-compose up -d`
34+
- **IMPORTANT!!!** - Create an exchange queue using the broker's UI!
35+
- Go to http://localhost:7777/#/exchanges, login with user: admin, pass: admin
36+
- Add a new exchange
37+
- Name: `reset`
38+
- Type: `fanout`
39+
- Press Add exchange
40+
## Start the server
41+
```
42+
export PYTHONPATH=$PWD
43+
make server
44+
```
45+
## Start workers
46+
- You can start the workers to run on a different machine.
47+
- Before run, since they subscribe to the broker, make sure to set the broker IP in the `src/config.py` file. E.g. `BROKER_HOST = '192.168.0.4'`. By default it is `localhost`, if the server and the workers run on the same machine.
48+
```
49+
export PYTHONPATH=$PWD
50+
make workers
51+
```
52+
53+
# Usage
54+
1. Open the Web Page. Open `templates/index.html` file using browser.
55+
2. Drop an MP3 file, and wait for the progress to go 100% (don't refresh the page while processing, the output will disappear), afterwards you can download your files:
56+
- Separated drums track
57+
- Separated vocals track
58+
- Separated other instruments (mostly guitar + keyboard) track
59+
- Separated bass track
60+
- Mix of chosen instruments (default is drums + bass)
61+
62+
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
63+
64+
## Fault Tolerance
65+
- This system is resillient to worker crashes, please see the `WORKERS_UNHEALTHY` parameter to simulate the crash after `WORKER_SIMULATE_CRASH_AFTER` time.
66+
67+
## Configurations
68+
- in `src/config.py` you can find a number of parameters to be adjusted, e.g.:
69+
```python
70+
### User Conf ###
71+
BROKER_HOST = 'localhost' # broker IP
72+
WORKERS_HEALTHY = 10 # Amount of workers that do not crash
73+
WORKERS_UNHEALTHY = 2 # Amount of workers that crash (their process terminates)
74+
WORKER_SIMULATE_CRASH_AFTER = 20 # After this amount of seconds.
75+
### User Conf ###
76+
```
77+
78+
# Techstack
79+
- Backend: FastAPI
80+
- Middlewares: RabbitMQ
81+
- Frontend: Bootstrap, html, css
82+
83+
# Authors
84+
85+
Student: Miron Oskroba
86+
87+
NMEC: 112169
88+
89+
University of Aveiro
90+
91+
Computação Distribuída
92+
93+

‎api.html

Lines changed: 4635 additions & 0 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)