Skip to content

Commit 57a18cd

Browse files
nikborovetsnikborovets
authored
Merge pull request #1 from nikborovets/dev
Dev
2 parents d5e2fa0 + c8602c1 commit 57a18cd

File tree

21 files changed

+407
-3
lines changed

21 files changed

+407
-3
lines changed

‎.gitignore

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,7 @@
22

33
myenv
44
processed_frames
5-
videos
5+
videos
6+
7+
.env
8+
kafka_postgres_project/videos

‎README.md

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,16 @@
11
# Kafka-Postgres Learning Project
22

3-
Этот проект создан для изучения работы с Apache Kafka и PostgreSQL с использованием Python.
3+
Этот проект создан для изучения работы с Apache Kafka и PostgreSQL с использованием Python и Django.
44

55
## Описание
66

7-
Проект включает три основных компонента:
7+
Проект включает четыре основных компонента:
88

99
1. **Producer**: Захватывает кадры с веб-камеры и отправляет их в Kafka.
1010
2. **Consumer**: Читает кадры из Kafka, обрабатывает их (преобразует в оттенки серого) и сохраняет в PostgreSQL.
1111
3. **Analyzer**: Анализирует кадры, хранящиеся в PostgreSQL, и может создавать видео из этих кадров.
12+
4. **Web Interface**: Веб-интерфейс на Django, который повторяет функциональность **Analyzer**. Также в админской панели можно управлять кадрами (удалять, изменять, добавлять).
13+
1214

1315
## Установка
1416

@@ -22,6 +24,22 @@
2224
./setup.sh
2325
```
2426

27+
### Настройка и запуск
28+
`Django`
29+
30+
```sh
31+
python -m venv myenv
32+
source myenv/bin/activate
33+
pip install -r requirements.txt
34+
```
35+
36+
```sh
37+
python manage.py migrate
38+
python manage.py createsuperuser
39+
python manage.py runserver
40+
```
41+
42+
2543
### Важно!
2644

2745
Для наглядности рекомендуется запускать `producer.py`, `consumer.py` и `analyzer.py` в отдельных терминалах.
@@ -38,3 +56,7 @@ source myenv/bin/activate && python consumer.py
3856
source myenv/bin/activate && python analyzer.py
3957
```
4058

59+
60+
### Использование
61+
1. `http://127.0.0.1:8000/admin/` - доступ к админке `Django`.
62+
2. `http://127.0.0.1:8000/frames/` - функционал `analyzer.py`.

‎consumer.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,4 +45,6 @@ def save_frame(frame_data, timestamp):
4545
timestamp = message.value['timestamp']
4646

4747
save_frame(frame_data, timestamp)
48+
# save_frame(frame_str, timestamp)
49+
4850
print(f"Consumed and saved frame with timestamp: {timestamp}")

‎kafka_postgres_project/frames/__init__.py

Whitespace-only changes.

‎kafka_postgres_project/frames/admin.py

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
from django.contrib import admin
2+
from .models import Frame
3+
4+
class FrameAdmin(admin.ModelAdmin):
5+
list_display = ('frame_id', 'get_formatted_timestamp')
6+
7+
def get_formatted_timestamp(self, obj):
8+
return obj.get_formatted_timestamp()
9+
get_formatted_timestamp.short_description = 'Timestamp'
10+
11+
admin.site.register(Frame, FrameAdmin)

‎kafka_postgres_project/frames/apps.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
from django.apps import AppConfig
2+
3+
4+
class FramesConfig(AppConfig):
5+
default_auto_field = 'django.db.models.BigAutoField'
6+
name = 'frames'

‎kafka_postgres_project/frames/migrations/0001_initial.py

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Generated by Django 4.2.14 on 2024-07-18 17:12
2+
3+
from django.db import migrations, models
4+
5+
6+
class Migration(migrations.Migration):
7+
8+
initial = True
9+
10+
dependencies = [
11+
]
12+
13+
operations = [
14+
migrations.CreateModel(
15+
name='Frame',
16+
fields=[
17+
('frame_id', models.AutoField(primary_key=True, serialize=False)),
18+
('frame_data', models.TextField()),
19+
('timestamp', models.BigIntegerField()),
20+
],
21+
options={
22+
'db_table': 'frames',
23+
},
24+
),
25+
]

‎kafka_postgres_project/frames/migrations/__init__.py

Whitespace-only changes.

‎kafka_postgres_project/frames/models.py

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
from django.db import models
2+
from datetime import datetime
3+
4+
class Frame(models.Model):
5+
frame_id = models.AutoField(primary_key=True)
6+
frame_data = models.TextField()
7+
timestamp = models.BigIntegerField()
8+
9+
class Meta:
10+
db_table = 'frames'
11+
12+
def __str__(self):
13+
return f"Frame {self.frame_id} at {self.get_formatted_timestamp()}"
14+
15+
def get_formatted_timestamp(self):
16+
return datetime.fromtimestamp(self.timestamp).strftime('%Y-%m-%d %H:%M:%S')

‎kafka_postgres_project/frames/templates/frames/frame_detail.html

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
<!DOCTYPE html>
2+
<html>
3+
<head>
4+
<title>Frame Detail</title>
5+
</head>
6+
<body>
7+
<h1>Frame {{ frame.frame_id }}</h1>
8+
<p>Timestamp: {{ frame.get_formatted_timestamp }}</p>
9+
<img src="data:image/jpeg;base64,{{ img_base64 }}" alt="Frame Image"/>
10+
<a href="{% url 'frame_list' %}">Back to Frame List</a>
11+
</body>
12+
</html>

‎kafka_postgres_project/frames/templates/frames/frame_list.html

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
<!DOCTYPE html>
2+
<html>
3+
<head>
4+
<title>Frame List</title>
5+
</head>
6+
<body>
7+
<h1>Frame List</h1>
8+
<ul>
9+
{% for frame in frames %}
10+
<li>
11+
<a href="{% url 'frame_detail' frame.frame_id %}">Frame {{ frame.frame_id }} at {{ frame.get_formatted_timestamp }}</a>
12+
</li>
13+
{% endfor %}
14+
</ul>
15+
<a href="{% url 'create_video' %}">Create Video from Frames</a>
16+
</body>
17+
</html>

‎kafka_postgres_project/frames/templates/frames/video_create.html

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
<!DOCTYPE html>
2+
<html>
3+
<head>
4+
<title>Create Video</title>
5+
</head>
6+
<body>
7+
<h1>Create Video</h1>
8+
<p>{{ message }}</p>
9+
<a href="{% url 'frame_list' %}">Back to Frame List</a>
10+
</body>
11+
</html>

‎kafka_postgres_project/frames/tests.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
from django.test import TestCase
2+
3+
# Create your tests here.

‎kafka_postgres_project/frames/urls.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
from django.urls import path
2+
from .views import FrameListView, FrameDetailView, VideoCreateView
3+
4+
urlpatterns = [
5+
path('', FrameListView.as_view(), name='frame_list'),
6+
path('frame/<int:frame_id>/', FrameDetailView.as_view(), name='frame_detail'),
7+
path('create_video/', VideoCreateView.as_view(), name='create_video'),
8+
]

‎kafka_postgres_project/frames/views.py

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
from django.shortcuts import render, get_object_or_404
2+
from django.views import View
3+
from .models import Frame
4+
import base64
5+
import cv2
6+
import numpy as np
7+
import os
8+
from datetime import datetime
9+
10+
class FrameListView(View):
11+
def get(self, request):
12+
frames = Frame.objects.all()
13+
return render(request, 'frames/frame_list.html', {'frames': frames})
14+
15+
class FrameDetailView(View):
16+
def get(self, request, frame_id):
17+
frame = get_object_or_404(Frame, pk=frame_id)
18+
img_base64 = frame.frame_data
19+
# frame_bytes = base64.b64decode(frame.frame_data)
20+
# nparr = np.frombuffer(frame_bytes, np.uint8)
21+
# img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
22+
# _, img_encoded = cv2.imencode('.jpg', img)
23+
# img_base64 = base64.b64encode(img_encoded).decode('utf-8')
24+
return render(request, 'frames/frame_detail.html', {'frame': frame, 'img_base64': img_base64})
25+
26+
class VideoCreateView(View):
27+
def get(self, request):
28+
frames = Frame.objects.order_by('timestamp').all()
29+
if not frames:
30+
return render(request, 'frames/video_create.html', {'message': 'No frames found in the database.'})
31+
32+
frame_list = []
33+
for frame in frames:
34+
frame_data = frame.frame_data
35+
frame_bytes = base64.b64decode(frame_data)
36+
nparr = np.frombuffer(frame_bytes, np.uint8)
37+
img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
38+
frame_list.append(img)
39+
40+
height, width = frame_list[0].shape[:2]
41+
if not os.path.isdir('videos'):
42+
os.makedirs('videos')
43+
44+
timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
45+
video_path = os.path.join('videos', f'output_video_{timestamp}.mp4')
46+
out = cv2.VideoWriter(video_path, cv2.VideoWriter_fourcc(*'mp4v'), 1, (width, height))
47+
48+
for frame in frame_list:
49+
out.write(frame)
50+
51+
out.release()
52+
return render(request, 'frames/video_create.html', {'message': f'Video created at {video_path}'})

‎kafka_postgres_project/kafka_postgres_project/__init__.py

Whitespace-only changes.

‎kafka_postgres_project/kafka_postgres_project/asgi.py

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
"""
2+
ASGI config for kafka_postgres_project project.
3+
4+
It exposes the ASGI callable as a module-level variable named ``application``.
5+
6+
For more information on this file, see
7+
https://docs.djangoproject.com/en/4.2/howto/deployment/asgi/
8+
"""
9+
10+
import os
11+
12+
from django.core.asgi import get_asgi_application
13+
14+
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'kafka_postgres_project.settings')
15+
16+
application = get_asgi_application()

‎kafka_postgres_project/kafka_postgres_project/settings.py

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
"""
2+
Django settings for kafka_postgres_project project.
3+
4+
Generated by 'django-admin startproject' using Django 4.2.14.
5+
6+
For more information on this file, see
7+
https://docs.djangoproject.com/en/4.2/topics/settings/
8+
9+
For the full list of settings and their values, see
10+
https://docs.djangoproject.com/en/4.2/ref/settings/
11+
"""
12+
13+
import os
14+
from decouple import config
15+
from pathlib import Path
16+
17+
# Build paths inside the project like this: BASE_DIR / 'subdir'.
18+
BASE_DIR = Path(__file__).resolve().parent.parent
19+
20+
21+
# Quick-start development settings - unsuitable for production
22+
# See https://docs.djangoproject.com/en/4.2/howto/deployment/checklist/
23+
24+
# SECURITY WARNING: keep the secret key used in production secret!
25+
SECRET_KEY = config('SECRET_KEY')
26+
27+
# SECURITY WARNING: don't run with debug turned on in production!
28+
DEBUG = True
29+
30+
ALLOWED_HOSTS = []
31+
32+
33+
# Application definition
34+
35+
INSTALLED_APPS = [
36+
'frames',
37+
'django.contrib.admin',
38+
'django.contrib.auth',
39+
'django.contrib.contenttypes',
40+
'django.contrib.sessions',
41+
'django.contrib.messages',
42+
'django.contrib.staticfiles',
43+
]
44+
45+
46+
MIDDLEWARE = [
47+
'django.middleware.security.SecurityMiddleware',
48+
'django.contrib.sessions.middleware.SessionMiddleware',
49+
'django.middleware.common.CommonMiddleware',
50+
'django.middleware.csrf.CsrfViewMiddleware',
51+
'django.contrib.auth.middleware.AuthenticationMiddleware',
52+
'django.contrib.messages.middleware.MessageMiddleware',
53+
'django.middleware.clickjacking.XFrameOptionsMiddleware',
54+
]
55+
56+
ROOT_URLCONF = 'kafka_postgres_project.urls'
57+
58+
TEMPLATES = [
59+
{
60+
'BACKEND': 'django.template.backends.django.DjangoTemplates',
61+
'DIRS': [],
62+
'APP_DIRS': True,
63+
'OPTIONS': {
64+
'context_processors': [
65+
'django.template.context_processors.debug',
66+
'django.template.context_processors.request',
67+
'django.contrib.auth.context_processors.auth',
68+
'django.contrib.messages.context_processors.messages',
69+
],
70+
},
71+
},
72+
]
73+
74+
WSGI_APPLICATION = 'kafka_postgres_project.wsgi.application'
75+
76+
77+
# Database
78+
# https://docs.djangoproject.com/en/4.2/ref/settings/#databases
79+
80+
# DATABASES = {
81+
# 'default': {
82+
# 'ENGINE': 'django.db.backends.sqlite3',
83+
# 'NAME': BASE_DIR / 'db.sqlite3',
84+
# }
85+
# }
86+
DATABASES = {
87+
'default': {
88+
'ENGINE': 'django.db.backends.postgresql',
89+
'NAME': 'orders_db',
90+
'USER': 'user',
91+
'PASSWORD': 'password',
92+
'HOST': 'localhost',
93+
'PORT': '5433',
94+
}
95+
}
96+
97+
98+
99+
# Password validation
100+
# https://docs.djangoproject.com/en/4.2/ref/settings/#auth-password-validators
101+
102+
AUTH_PASSWORD_VALIDATORS = [
103+
{
104+
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
105+
},
106+
{
107+
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
108+
},
109+
{
110+
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
111+
},
112+
{
113+
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
114+
},
115+
]
116+
117+
118+
# Internationalization
119+
# https://docs.djangoproject.com/en/4.2/topics/i18n/
120+
121+
LANGUAGE_CODE = 'en-us'
122+
123+
TIME_ZONE = 'UTC'
124+
125+
USE_I18N = True
126+
127+
USE_TZ = True
128+
129+
130+
# Static files (CSS, JavaScript, Images)
131+
# https://docs.djangoproject.com/en/4.2/howto/static-files/
132+
133+
STATIC_URL = 'static/'
134+
135+
# Default primary key field type
136+
# https://docs.djangoproject.com/en/4.2/ref/settings/#default-auto-field
137+
138+
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'

0 commit comments

Comments
 (0)