Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release/1.23.0 #143

Merged
merged 122 commits into from
Oct 8, 2024
Merged
Changes from 1 commit
Commits
Show all changes
122 commits
Select commit Hold shift + click to select a range
556cd28
Add priorities to tasks published using RabbitMQ
catileptic Sep 29, 2023
369f8b8
Paint it black
catileptic Sep 29, 2023
a308782
Paint it black again
catileptic Sep 29, 2023
e53f8fb
Add priority to Task body
catileptic Oct 2, 2023
2042dc6
Temporary debug commit
catileptic Oct 10, 2023
0839455
Temporary debug commit
catileptic Oct 10, 2023
82daf4e
Fix logging message formatting
catileptic Nov 8, 2023
3d70937
Make RabbitMQ qork without threads
catileptic Nov 20, 2023
f0622a3
Fix linting errors
catileptic Nov 20, 2023
3ddfebe
Fix tests for priority RabbitMQ implementation
catileptic Nov 20, 2023
655706b
Add one thread
catileptic Nov 20, 2023
59ae9db
Add python3-setuptools
catileptic Nov 20, 2023
fca3d4f
Update pip, setuptools and wheel before installing packages
catileptic Nov 20, 2023
f56e36a
Install setuptools in Makefile, rm from Dockerfile
catileptic Nov 21, 2023
1301028
last_updated timestamp for the status API (RabbitMQ impl)
stchris Dec 14, 2023
728dbfa
Merge pull request #112 from alephdata/feature/rabbitmq-priorities
stchris Dec 20, 2023
0333891
Merge pull request #137 from alephdata/feature/last-updated-rabbitmq
stchris Dec 20, 2023
a3f2fd0
Add rc support to bumpversion cfg
stchris Dec 21, 2023
69dc59b
Bump version: 1.22.1 → 1.23.0-rc1
stchris Dec 21, 2023
e757cc1
Refactor the creation of the rabbitMQ connection to ensure it is alwa…
catileptic Jan 23, 2024
5229f3f
Bump version: 1.23.0-rc1 → 1.23.0-rc2
catileptic Jan 23, 2024
f6756bf
Add Redis keys for tracking task stages
catileptic Feb 6, 2024
378f8dd
Fix incorrect call to pipe.incr
catileptic Feb 6, 2024
4b6a9ae
Update call to moto
catileptic Feb 6, 2024
8184e3e
Ensure status have a valid minimum value
catileptic Feb 8, 2024
15ebc8d
Make job_id value an empty string in order to not break the schema va…
catileptic Feb 8, 2024
2cd1938
Bump version: 1.23.0-rc2 → 1.23.0-rc3
catileptic Feb 8, 2024
f0fc18a
Merge branch 'main' into release/1.23.0
catileptic Mar 14, 2024
569729f
Fix call to moto to be compatible with versions <5
catileptic Mar 18, 2024
cf2ccf1
Remove jobs from status API
catileptic Mar 18, 2024
863cacf
Bump version: 1.23.0-rc3 → 1.23.0-rc4
catileptic Mar 21, 2024
bdc70d5
Merge branch 'main' into release/1.23.0
catileptic Apr 22, 2024
a34f38b
Will revert: limit queue size to 10 msgs & drop newer msgs first
catileptic Apr 23, 2024
093b3fc
Set queues to delete newer msgs if the queue fills up
catileptic Apr 23, 2024
d8d8213
Merge branch 'main' into release/1.23.0
stchris May 3, 2024
b7df50c
Bump version: 1.23.0-rc4 → 1.23.0-rc5
stchris May 3, 2024
4de7e00
Log worker hostname when dispatching task
catileptic May 8, 2024
e2fb94a
Fix linter error
catileptic May 8, 2024
d7a0d6f
Fix other linter
catileptic May 8, 2024
838964c
Bump version: 1.23.0-rc5 → 1.23.0-rc6
catileptic May 8, 2024
cbf0ba6
Catch pika.exceptions.StreamLostError
catileptic May 14, 2024
5f32bf1
Fix linter error
catileptic May 14, 2024
2ba898e
Bump version: 1.23.0-rc6 → 1.23.0-rc7
catileptic May 14, 2024
0d6328f
chore: friendlier rabbitmq connection names
stchris May 14, 2024
431fb72
Bump version: 1.23.0-rc7 → 1.23.0-rc8
stchris May 14, 2024
2b058fa
Remove periodic invocation
stchris May 21, 2024
41d70a4
Bump version: 1.23.0-rc8 → 1.23.0-rc9
stchris May 22, 2024
7981c16
Allow workers to define queues and QoS settings dynamically (#171)
catileptic May 22, 2024
2e9247e
Bump version: 1.23.0-rc9 → 1.23.0-rc10
catileptic May 22, 2024
5ef6c25
Sanity check for connection establishment
stchris May 23, 2024
968c366
Expose management UI port for RabbitMQ
stchris May 23, 2024
75f4665
Always re-establish connection after a failed attempt
stchris May 23, 2024
29e1daa
Sanity check connection is alive
stchris May 23, 2024
b235092
Use the newly assigned connection
stchris May 23, 2024
8a57068
Add a note about the unrelated SQLAlchemy warning
stchris May 23, 2024
671570e
Bump version: 1.23.0-rc10 → 1.23.0-rc11
stchris May 23, 2024
2946dc7
Don't log first failed rmq connectiona attempt
stchris May 24, 2024
acb734b
Bump version: 1.23.0-rc11 → 1.23.0-rc12
stchris May 24, 2024
b2cfe45
Update ruff.toml settings
stchris May 24, 2024
7e4ad36
Fix formatting
stchris May 24, 2024
5838b6d
Bump version: 1.23.0-rc12 → 1.23.0-rc13
stchris May 24, 2024
614a28e
More accurate logging of attempt retries
stchris May 24, 2024
71812e1
Revert "Don't log first failed rmq connectiona attempt"
stchris May 24, 2024
ef5bbdb
Bump version: 1.23.0-rc13 → 1.23.0-rc14
stchris May 24, 2024
fa51c23
Run periodic tasks after handling tasks :/
stchris May 24, 2024
799c0b8
Bump version: 1.23.0-rc14 → 1.23.0-rc15
stchris May 24, 2024
e2cd6fb
Run periodic tasks before handling tasks
stchris May 24, 2024
03a9b0f
Bump version: 1.23.0-rc15 → 1.23.0-rc16
stchris May 24, 2024
3c597d5
Add a sentry tag for rmq conn attempt count
stchris May 27, 2024
7179ea0
Don't log an error for the first attempt
stchris May 27, 2024
8e0ef2b
Bump version: 1.23.0-rc16 → 1.23.0-rc17
stchris May 27, 2024
0bd2b41
Revert "Add a sentry tag for rmq conn attempt count"
stchris May 27, 2024
35a7983
Bump version: 1.23.0-rc17 → 1.23.0-rc18
stchris May 27, 2024
79db6a5
Try 30 times to establish whether a task should be executed (#176)
catileptic May 30, 2024
397acbb
Bump version: 1.23.0-rc18 → 1.23.0-rc19
catileptic May 30, 2024
6dc7d4e
Enable Sentry in the RMQ Worker (#182)
catileptic Jun 5, 2024
8c8e531
Bump version: 1.23.0-rc19 → 1.23.0-rc20
catileptic Jun 5, 2024
d6d6129
Enable Prometheus metrics on the RabbitMQ Worker (#181)
catileptic Jun 5, 2024
8bd4082
Modify the on_message function to be usable from AlephWorker
catileptic Jun 5, 2024
8dcfab3
Bump version: 1.23.0-rc20 → 1.23.0-rc21
catileptic Jun 5, 2024
2e826c4
call periodic tasks after handling tasks
stchris Jun 6, 2024
87ffe7e
Bump version: 1.23.0-rc21 → 1.23.0-rc22
stchris Jun 6, 2024
7192299
un-skip test for tasks which shouldn't execute
stchris Jun 12, 2024
551e742
remove unused import
stchris Jun 12, 2024
608b5f8
Properly NACK failed tasks
stchris Jun 17, 2024
695b569
Make sure we don't requeue tasks with excessive retries
stchris Jun 18, 2024
9eab35a
Bump version: 1.23.0-rc22 → 1.23.0-rc23
stchris Jun 18, 2024
c8fb3a6
Add a task to Redis if it is no longer tracked after NACKing
catileptic Jun 18, 2024
53ef724
Removed unused code
stchris Jun 19, 2024
f34fe01
Bump version: 1.23.0-rc23 → 1.23.0-rc24
stchris Jun 19, 2024
b7b1730
Hotfix: call add_task with correct parameters
catileptic Jun 19, 2024
8fa15bc
Bump version: 1.23.0-rc24 → 1.23.0-rc25
catileptic Jun 19, 2024
21f1c8c
Generic queue_task implementation with priority buckets
stchris Jun 25, 2024
386fa09
Add flush_queues
stchris Jun 25, 2024
c1b24f5
Refactor queue_task to take a collection_id
stchris Jun 25, 2024
981dff7
Bump version: 1.23.0-rc25 → 1.23.0-rc26
stchris Jun 25, 2024
46e29db
Add high and low prio lists
stchris Jun 25, 2024
2d279bb
Bump version: 1.23.0-rc26 → 1.23.0-rc27
stchris Jun 25, 2024
2fe5598
Fix rmq channel leak (#189)
stchris Jun 28, 2024
8b522b7
Bump version: 1.23.0-rc27 → 1.23.0-rc28
stchris Jun 28, 2024
6ab363a
Call confirm_delivery only once per channel
stchris Jul 1, 2024
e0f19ed
Bump version: 1.23.0-rc28 → 1.23.0-rc29
stchris Jul 3, 2024
ef315d0
Clear inactive dataset if the last task was executed
catileptic Jul 3, 2024
264f878
Fix formatting
stchris Jul 3, 2024
f12e13e
Do not add failed tasks to redis
catileptic Jul 4, 2024
a92c592
Fix tests now that we clean up finished tasks
stchris Jul 4, 2024
6200817
Bump version: 1.23.0-rc29 → 1.23.0-rc30
stchris Jul 4, 2024
decef6d
Fix Prometheus metrics (#199)
tillprochaska Jul 8, 2024
a7ef3aa
Bump version: 1.23.0-rc30 → 1.23.0-rc31
stchris Jul 8, 2024
8f69579
Remove task from Redis if NACK-ing with retry=False
catileptic Jul 8, 2024
3c1c690
Bump version: 1.23.0-rc31 → 1.23.0-rc32
stchris Jul 9, 2024
ef5027e
Refactor Redis keys in order to ensure status cleanup (#202)
catileptic Jul 19, 2024
eedfb94
Fix dataset timestamps (#190)
tillprochaska Jul 19, 2024
995743f
Bump version: 1.23.0-rc32 → 1.23.0-rc33
catileptic Jul 19, 2024
f97f086
Don't show retry flag when a task succeeded
stchris Sep 3, 2024
8a4f717
s/docker-compose/docker compose
stchris Sep 3, 2024
85d8544
Add _channel attr in test runs, too
catileptic Sep 11, 2024
24c4b7f
Revert "Add _channel attr in test runs, too"
stchris Sep 24, 2024
273496b
Add _channel attr to task obj in tests
catileptic Sep 24, 2024
83c62cb
Bump version: 1.23.0-rc33 → 1.23.0-rc34
catileptic Sep 24, 2024
59629f1
Fix Redis keys (#201)
catileptic Oct 8, 2024
2c32068
Merge branch 'main' into release/1.23.0
catileptic Oct 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update pip, setuptools and wheel before installing packages
  • Loading branch information
catileptic committed Nov 20, 2023
commit fca3d4fc15872b62509590d61bf88358f99f22d1
8 changes: 6 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ ENV DEBIAN_FRONTEND noninteractive

RUN apt-get -qq -y update \
&& apt-get -qq -y install python3-pip \
pkg-config libicu-dev python3-setuptools \
pkg-config libicu-dev \
&& apt-get -qq -y autoremove \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
Expand All @@ -14,7 +14,11 @@ ENV LANG='en_US.UTF-8'

COPY . /opt/servicelayer
WORKDIR /opt/servicelayer
RUN pip3 install -q --no-cache-dir -e /opt/servicelayer[dev]

RUN pip3 install --no-cache-dir --prefer-binary --upgrade pip
RUN pip3 install --no-cache-dir --prefer-binary --upgrade setuptools wheel
RUN pip3 install --no-cache-dir --prefer-binary -r requirements.txt
RUN pip3 install -q --no-cache-dir --config-settings editable_mode=compat --use-pep517 -e /opt/servicelayer[dev]
RUN pip3 install -r requirements.txt

CMD /bin/bash
Loading