Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multithreading via C++ thread pool of clients #125

Merged
merged 95 commits into from
Jun 5, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
a6e238b
switching machines
kentslaney Dec 11, 2023
22b1e47
switch back...
kentslaney Dec 11, 2023
9fe14b3
LockPool
kentslaney Dec 13, 2023
02c941d
ClientPool
kentslaney Dec 13, 2023
3fb5dd7
missed renamed symbol
kentslaney Dec 13, 2023
b7305ca
cppcheck fixes
kentslaney Dec 13, 2023
c936fdf
minor build fixes
kentslaney Dec 13, 2023
a4662c2
std::execution::par[_unseq] requires -fexceptions
kentslaney Dec 14, 2023
f5eca1c
act support
kentslaney Dec 14, 2023
448458b
python bindings
kentslaney Dec 15, 2023
b5410a3
avoiding iota
kentslaney Dec 17, 2023
2324b0f
switching machines
kentslaney Dec 17, 2023
1e9f944
still broken...
kentslaney Dec 20, 2023
6c860c3
...with a different error though
kentslaney Dec 20, 2023
9ce5327
stranger error
kentslaney Dec 21, 2023
07964a8
well that feels silly after the fact
kentslaney Dec 21, 2023
550ef71
redundancies
kentslaney Dec 21, 2023
f099e68
partially fix cython integration
kentslaney Dec 21, 2023
cfdd1b4
reset cmake
kentslaney Dec 21, 2023
e4af40d
python class
kentslaney Dec 21, 2023
f538b6e
remove header...
kentslaney Dec 21, 2023
f0b37f1
__cinit__ semantics
kentslaney Dec 21, 2023
4d7ecdb
split up tests
kentslaney Dec 21, 2023
87b834f
flustered and cygdb can't find debugging logs
kentslaney Dec 21, 2023
df5f3d3
mild QOL patch
kentslaney Dec 21, 2023
5eff5b3
trailing whitespace
kentslaney Dec 22, 2023
6285a81
forgot to init ClientPool
kentslaney Dec 22, 2023
167d123
pre-commit still fails on macos
kentslaney Dec 22, 2023
9416571
made applying debug patch easier
kentslaney Dec 22, 2023
80d19c6
forgot to update method reference
kentslaney Dec 22, 2023
c675ec5
TCP capture
kentslaney Dec 24, 2023
a25a7cd
cython debug script
kentslaney Dec 24, 2023
5f2c427
messed up setup.py
kentslaney Dec 24, 2023
6e40df0
python versioning and resolve symlinks
kentslaney Dec 24, 2023
d440b51
ubuntu compat
kentslaney Dec 26, 2023
e1a5de4
missed _update_servers ref
kentslaney Dec 26, 2023
5369e41
stricter inheritance
kentslaney Dec 26, 2023
4079333
invalid keys in test
kentslaney Jan 13, 2024
2d05017
removed race conditions
kentslaney Jan 14, 2024
1e9c2bc
hung threads illustration
kentslaney Jan 15, 2024
8362719
commit for easier copying to a docker environment with a development …
kentslaney Jan 15, 2024
7afa87e
QoL nit
kentslaney Jan 15, 2024
4673c90
fresh environment fixes
kentslaney Jan 15, 2024
15a705e
threaded traceback bug
kentslaney Jan 17, 2024
03cb651
excepthook behavior
kentslaney Jan 17, 2024
71d44e4
debugging tools
kentslaney Jan 17, 2024
9f3f4ef
ln fails if symlink exists but is broken
kentslaney Jan 17, 2024
3303ad3
venv for both debug and test
kentslaney Jan 17, 2024
0b2346f
support for <3.11
kentslaney Jan 17, 2024
f54eb9b
WIP debug build flags
kentslaney Jan 18, 2024
e367a3f
debug build flags
kentslaney Jan 18, 2024
c3ff9ae
can't assign to python scope variables w/o GIL and underlying functio…
kentslaney Jan 19, 2024
bdaf276
sequential benchmark, but measures overhead
kentslaney Jan 20, 2024
d8d0014
remove debugging statements and shared cython resources
kentslaney Jan 20, 2024
f8d4728
cython threading hangs under load
kentslaney Jan 20, 2024
804a18e
Revert "can't assign to python scope variables w/o GIL and underlying…
kentslaney Jan 23, 2024
340a8e5
revert reverted debug setup removal
kentslaney Jan 23, 2024
f7964e5
debugging conviences
kentslaney Jan 23, 2024
a82f34b
cython gil misunderstanding
kentslaney Jan 23, 2024
aff16cf
benchmark setup
kentslaney Jan 23, 2024
e06cafe
greenify initial pass
kentslaney Jan 23, 2024
1def670
greenlet misunderstanding
kentslaney Jan 23, 2024
ecc5ec7
eventlet unittest
kentslaney Jan 23, 2024
59662fc
eventlet name instead of thread name
kentslaney Jan 23, 2024
723addc
threaded python benchmarks
kentslaney Jan 27, 2024
afa71a0
typo
kentslaney Jan 27, 2024
3787008
missed gevent failure
kentslaney Jan 27, 2024
ad0bd7a
not sure why this one hangs
kentslaney Jan 27, 2024
37eb65c
add py FIFO thread pool
kentslaney Jan 27, 2024
31d8569
pytest stdout
kentslaney Feb 2, 2024
7a92e7b
config typo
kentslaney Feb 4, 2024
f767931
allow pytest stdout for run-test
kentslaney Feb 4, 2024
63ab7f5
remove greenify debug setup and internal threading metrics
kentslaney Feb 6, 2024
5cbc847
switch hash function back
kentslaney Feb 6, 2024
39a87a5
fail fast
kentslaney Feb 6, 2024
299508a
cppcheck lint
kentslaney Feb 6, 2024
2722b56
build fixes
kentslaney Feb 6, 2024
af0240f
remove dead code
kentslaney Feb 6, 2024
d80eff1
add c++ std flags
kentslaney Feb 6, 2024
b1263d1
update cgo CXXFLAGS
kentslaney Feb 6, 2024
36f1644
fix double acquire attempt for m_pool_lock
kentslaney Feb 6, 2024
ccd20a5
explicit irange constructor
kentslaney Feb 7, 2024
7e7dfae
README update
kentslaney Feb 7, 2024
1c8c6bc
link format
kentslaney Feb 7, 2024
399e4e6
typo
kentslaney Feb 7, 2024
728ddca
md -> rst
kentslaney Feb 7, 2024
d561b84
remove dead link
kentslaney Feb 7, 2024
1f8dd88
clarifying statistics
kentslaney Feb 7, 2024
87c4e64
better footnote about traffic stats
kentslaney Feb 7, 2024
78a5431
wording
kentslaney Feb 7, 2024
f8ecc8e
gevent support explanation
kentslaney May 31, 2024
51e627f
version bump
kentslaney May 31, 2024
1327704
remove gevent test for ThreadedClient
kentslaney May 31, 2024
f0263ad
Separation of concerns
kentslaney Jun 4, 2024
081db1d
mistake in benchmark thread spawning
kentslaney Jun 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
act support
  • Loading branch information
kentslaney committed Dec 14, 2023
commit f5eca1c46a56cbd56e3185064f583b24a97931c9
1 change: 1 addition & 0 deletions .github/workflows/golang.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.gover }}
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Start memcached servers
run: ./misc/memcached_server start
- name: Run gotest
Expand Down
89 changes: 89 additions & 0 deletions .github/workflows/manual.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
name: manual

on:
workflow_dispatch:

jobs:
cpptest:
runs-on: ubuntu-latest
strategy:
matrix:
compiler: ["gcc"]
build_type: ["Debug"]

steps:
- uses: actions/checkout@v2
- name: Setup system dependencies
run: |
sudo apt-get update
sudo apt-get -y install memcached g++
- name: Install python dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools future pytest greenify gevent numpy
- name: Run cpptest
run: |
if [[ ${{ matrix.compiler }} = "gcc" ]]; then export CC=gcc CXX=g++; fi
if [[ ${{ matrix.compiler }} = "clang" ]]; then export CC=clang CXX=clang++; fi
./misc/travis/cpptest.sh

pylatest:
runs-on: ubuntu-latest
strategy:
matrix:
pyver: ["3.12"]
compiler: ["gcc"]
build_type: ["Debug"]

steps:
- uses: actions/checkout@v2
- name: Setup system dependencies
run: |
sudo apt-get update
sudo apt-get -y install valgrind memcached g++
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.pyver }}
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install python dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools future pytest greenify gevent numpy
- name: Start memcached servers
run: ./misc/memcached_server startall
- name: Run unittest
run: |
if [[ ${{ matrix.compiler }} = "gcc" ]]; then export CC=gcc CXX=g++; fi
if [[ ${{ matrix.compiler }} = "clang" ]]; then export CC=clang CXX=clang++; fi
./misc/travis/unittest.sh
- name: Stop memcached servers
run: ./misc/memcached_server stopall

goclang:
runs-on: ubuntu-latest
strategy:
matrix:
gover: ["1.15"]
compiler: ["clang"]

steps:
- uses: actions/checkout@v2
- name: Setup system dependencies
run: |
sudo apt-get update
sudo apt-get -y install memcached
- name: Set up Golang
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.gover }}
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Start memcached servers
run: ./misc/memcached_server start
- name: Run gotest
run: |
export CC=clang CXX=clang++
clang++ -v
./misc/travis/gotest.sh
- name: Stop memcached servers
run: ./misc/memcached_server stop
2 changes: 2 additions & 0 deletions .github/workflows/python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ jobs:
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.pyver }}
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install python dependencies
run: |
python -m pip install --upgrade pip
Expand Down Expand Up @@ -55,6 +56,7 @@ jobs:
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.pyver }}
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Install python dependencies
run: |
python -m pip install --upgrade pip
Expand Down
1 change: 1 addition & 0 deletions include/ClientPool.h
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#pragma once

#include <shared_mutex>
#include "Client.h"
#include "LockPool.h"

Expand Down
1 change: 0 additions & 1 deletion src/ClientPool.cpp
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
#include <execution>
#include <thread>

#include "ClientPool.h"

namespace douban {
Expand Down