Skip to content

Commit 046dcc3

Browse files
author
root
committed
initial
0 parents  commit 046dcc3

File tree

14 files changed

+740
-0
lines changed

14 files changed

+740
-0
lines changed

‎.github/workflows/python-package.yml

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
2+
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
3+
4+
name: Python tests
5+
6+
on: [push, pull_request]
7+
8+
jobs:
9+
build:
10+
11+
runs-on: ubuntu-latest
12+
strategy:
13+
fail-fast: false
14+
matrix:
15+
python-version: [3.7, 3.8, 3.9]
16+
17+
steps:
18+
- uses: actions/checkout@v2
19+
- name: Set up Python ${{ matrix.python-version }}
20+
uses: actions/setup-python@v2
21+
with:
22+
python-version: ${{ matrix.python-version }}
23+
- name: Install dependencies
24+
run: |
25+
python -m pip install --upgrade pip
26+
python -m pip install mypy
27+
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
28+
- name: MyPy check
29+
run: |
30+
mypy fsPacker
31+
- name: Install package
32+
run: |
33+
python setup.py install
34+
- name: Unittest
35+
run: |
36+
python -m fsPacker
37+

‎.github/workflows/python-publish.yml

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# This workflow will upload a Python Package using Twine when a release is created
2+
# For more information see: https://help.github.com/en/actions/language-and-framework-guides/using-python-with-github-actions#publishing-to-package-registries
3+
4+
# This workflow uses actions that are not certified by GitHub.
5+
# They are provided by a third-party and are governed by
6+
# separate terms of service, privacy policy, and support
7+
# documentation.
8+
9+
name: PyPi
10+
11+
on:
12+
release:
13+
types: [published]
14+
15+
jobs:
16+
deploy:
17+
18+
runs-on: ubuntu-latest
19+
20+
steps:
21+
- uses: actions/checkout@v2
22+
- name: Set up Python
23+
uses: actions/setup-python@v2
24+
with:
25+
python-version: '3.x'
26+
- name: Install dependencies
27+
run: |
28+
python -m pip install --upgrade pip
29+
python -m fsPacker
30+
pip install build
31+
- name: Build package
32+
run: python -m build
33+
- name: Publish package
34+
uses: pypa/gh-action-pypi-publish@27b31702a0e7fc50959f5ad993c78deac1bdfc29
35+
with:
36+
user: __token__
37+
password: ${{ secrets.PYPI_API_TOKEN }}
38+

‎.gitignore

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Byte-compiled / optimized / DLL files
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
6+
# C extensions
7+
*.so
8+
9+
# Distribution / packaging
10+
.Python
11+
build/
12+
develop-eggs/
13+
dist/
14+
downloads/
15+
eggs/
16+
.eggs/
17+
lib/
18+
lib64/
19+
parts/
20+
sdist/
21+
var/
22+
wheels/
23+
pip-wheel-metadata/
24+
share/python-wheels/
25+
*.egg-info/
26+
.installed.cfg
27+
*.egg
28+
MANIFEST
29+
30+
# PyInstaller
31+
# Usually these files are written by a python script from a template
32+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
33+
*.manifest
34+
*.spec
35+
36+
# Installer logs
37+
pip-log.txt
38+
pip-delete-this-directory.txt
39+
40+
# Unit test / coverage reports
41+
htmlcov/
42+
.tox/
43+
.nox/
44+
.coverage
45+
.coverage.*
46+
.cache
47+
nosetests.xml
48+
coverage.xml
49+
*.cover
50+
*.py,cover
51+
.hypothesis/
52+
.pytest_cache/
53+
54+
# Translations
55+
*.mo
56+
*.pot
57+
58+
# Django stuff:
59+
*.log
60+
local_settings.py
61+
db.sqlite3
62+
db.sqlite3-journal
63+
64+
# Flask stuff:
65+
instance/
66+
.webassets-cache
67+
68+
# Scrapy stuff:
69+
.scrapy
70+
71+
# Sphinx documentation
72+
docs/_build/
73+
74+
# PyBuilder
75+
target/
76+
77+
# Jupyter Notebook
78+
.ipynb_checkpoints
79+
80+
# IPython
81+
profile_default/
82+
ipython_config.py
83+
84+
# pyenv
85+
.python-version
86+
87+
# pipenv
88+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
89+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
90+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
91+
# install all needed dependencies.
92+
#Pipfile.lock
93+
94+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
95+
__pypackages__/
96+
97+
# Celery stuff
98+
celerybeat-schedule
99+
celerybeat.pid
100+
101+
# SageMath parsed files
102+
*.sage.py
103+
104+
# Environments
105+
.env
106+
.venv
107+
env/
108+
venv/
109+
ENV/
110+
env.bak/
111+
venv.bak/
112+
113+
# Spyder project settings
114+
.spyderproject
115+
.spyproject
116+
117+
# Rope project settings
118+
.ropeproject
119+
120+
# mkdocs documentation
121+
/site
122+
123+
# mypy
124+
.mypy_cache/
125+
.dmypy.json
126+
dmypy.json
127+
128+
# Pyre type checker
129+
.pyre/
130+

‎LICENSE.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
FS message packer
2+
Copyright (C) 2021 Fusion Solutions KFT <contact@fusionsolutions.io>
3+
4+
This program is free software: you can redistribute it and/or modify
5+
it under the terms of the GNU General Public License as published by
6+
the Free Software Foundation, either version 3 of the License, or
7+
(at your option) any later version.
8+
9+
This program is distributed in the hope that it will be useful,
10+
but WITHOUT ANY WARRANTY; without even the implied warranty of
11+
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12+
GNU General Public License for more details.
13+
14+
You should have received a copy of the GNU Lesser General Public License
15+
along with this program. If not, see <http://www.gnu.org/licenses/lgpl.txt>.

‎MANIFEST.in

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
recursive-include fsPacker *.py

‎README.md

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
[![Python package](https://github.com/FusionSolutions/python-fspacker/actions/workflows/python-package.yml/badge.svg)](https://github.com/FusionSolutions/python-fspacker/actions/workflows/python-package.yml)
2+
# FS Message packer
3+
4+
## Introduction
5+
6+
Message packer for socket communications.
7+
Pure-Python implementation and it is [*much slower*](#benchmark) as `pickle`, `marshal` or even `json`, but much safer for production.
8+
The following types are supported for packing and unpacking:
9+
- `None`
10+
- `bool`
11+
- `int`
12+
- `float`
13+
- `string`
14+
- `bytearray` (during unpacking it will be converted to `bytes`)
15+
- `bytes`
16+
- `list` (during unpacking it will be converted to `tuple`)
17+
- `tuple`
18+
- `dict` (dict key type can be any from this list)
19+
20+
## Installation
21+
22+
Requires python version 3.7 or later.
23+
24+
To install the latest release on [PyPI](https://pypi.org/project/python-fspacker/),
25+
simply run:
26+
27+
```shell
28+
pip3 install python-fspacker
29+
```
30+
31+
Or to install the latest version, run:
32+
33+
```shell
34+
git clone https://github.com/FusionSolutions/python-fspacker.git
35+
cd python-fspacker
36+
python3 setup.py install
37+
```
38+
39+
## Python library
40+
41+
### Usage
42+
43+
Use like `pickle` with `dump`, `dumps`, `load` and `loads` functions.
44+
45+
```python
46+
import fsPacker
47+
48+
data = fsPacker.dumps(["test"]*5)
49+
print( fsPacker.loads(data) )
50+
```
51+
52+
### Benchmark
53+
54+
Environment: Intel(R) Xeon(R) CPU E5-1650 v4 @ 3.60GHz, DIMM DDR4 Synchronous Registered (Buffered) 2133 MHz
55+
```shell
56+
$/python-fspacker: python3 -m benchmark
57+
Test data one [1 times]
58+
pickle
59+
dump size: 369436 byte
60+
dump : best: 0.00192141 <- median: 0.00303648 - average: 0.00289015 -> worst: 0.00357831
61+
loads: best: 0.00176141 <- median: 0.00193337 - average: 0.00192555 -> worst: 0.00245063
62+
marshal
63+
dump size: 474624 byte
64+
dump : best: 0.00091804 <- median: 0.00146398 - average: 0.00146232 -> worst: 0.00209858
65+
loads: best: 0.00144557 <- median: 0.00150470 - average: 0.00159074 -> worst: 0.00230714
66+
FSPacker
67+
dump size: 329289 byte
68+
dump : best: 0.02956123 <- median: 0.03191117 - average: 0.03488318 -> worst: 0.06962921
69+
loads: best: 0.02214191 <- median: 0.02311805 - average: 0.02343034 -> worst: 0.02914553
70+
Test data two [1 times]
71+
pickle
72+
dump size: 274491 byte
73+
dump : best: 0.00102806 <- median: 0.00109109 - average: 0.00112886 -> worst: 0.00150502
74+
loads: best: 0.00137711 <- median: 0.00147515 - average: 0.00148870 -> worst: 0.00195776
75+
marshal
76+
dump size: 360242 byte
77+
dump : best: 0.00073095 <- median: 0.00080529 - average: 0.00079686 -> worst: 0.00087429
78+
loads: best: 0.00122311 <- median: 0.00139272 - average: 0.00154981 -> worst: 0.00244541
79+
FSPacker
80+
dump size: 238495 byte
81+
dump : best: 0.02812932 <- median: 0.02927591 - average: 0.03252403 -> worst: 0.06384002
82+
loads: best: 0.02134671 <- median: 0.02191628 - average: 0.02265632 -> worst: 0.03327830
83+
Test data three [1000 times]
84+
pickle
85+
dump size: 97 byte
86+
dump : best: 0.00065989 <- median: 0.00066162 - average: 0.00066534 -> worst: 0.00073160
87+
loads: best: 0.00080160 <- median: 0.00080605 - average: 0.00080777 -> worst: 0.00084383
88+
marshal
89+
dump size: 79 byte
90+
dump : best: 0.00061424 <- median: 0.00061704 - average: 0.00061919 -> worst: 0.00063762
91+
loads: best: 0.00080728 <- median: 0.00081835 - average: 0.00082229 -> worst: 0.00090104
92+
FSPacker
93+
dump size: 85 byte
94+
dump : best: 0.01630956 <- median: 0.01642815 - average: 0.01642205 -> worst: 0.01668519
95+
loads: best: 0.01521912 <- median: 0.01530806 - average: 0.01553064 -> worst: 0.02047744
96+
```
97+
## Contribution
98+
99+
Bug reports, constructive criticism and suggestions are welcome. If you have some create an issue on [github](https://github.com/FusionSolutions/python-fspacker/issues).
100+
101+
## Copyright
102+
103+
All of the code in this distribution is Copyright (c) 2021 Fusion Solutions Kft.
104+
105+
The utility is made available under the GNU General Public license. The included LICENSE file describes this in detail.
106+
107+
## Warranty
108+
109+
THIS SOFTWARE IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE USE OF THIS SOFTWARE IS WITH YOU.
110+
111+
IN NO EVENT WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE LIBRARY, BE LIABLE TO YOU FOR ANY DAMAGES, EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
112+
113+
Again, see the included LICENSE file for specific legal details.

‎benchmark/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+

‎benchmark/__main__.py

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Builtin modules
2+
import timeit, zlib, pickle, marshal
3+
from os.path import dirname
4+
from statistics import median
5+
# Third party modules
6+
import fsPacker
7+
# Local modules
8+
# Program
9+
def benchmark(dumpFn, loadsFn, data, name, counter=1000, repeat=4):
10+
print(" {}".format(name))
11+
q = dumpFn(data)
12+
print(" dump size: {:>9} byte".format(len(q)))
13+
res = timeit.repeat("fn(data)", number=counter, repeat=repeat, globals={"fn":dumpFn, "data":data})
14+
print(" dump : best: {:.8F} <- median: {:.8F} - average: {:.8F} -> worst: {:.8F}".format( min(res), median(res), sum(res)/repeat, max(res)))
15+
res = timeit.repeat("fn(data)", number=counter, repeat=repeat, globals={"fn":loadsFn, "data":q})
16+
print(" loads: best: {:.8F} <- median: {:.8F} - average: {:.8F} -> worst: {:.8F}".format( min(res), median(res), sum(res)/repeat, max(res)))
17+
18+
print("Test data one [1 times]")
19+
testData1 = pickle.loads(zlib.decompress(open("{}/testData1.dat".format(dirname(__file__)), 'rb').read()))
20+
benchmark(pickle.dumps, pickle.loads, testData1, "pickle", 1, 32)
21+
benchmark(marshal.dumps, marshal.loads, testData1, "marshal", 1, 32)
22+
benchmark(fsPacker.dumps, fsPacker.loads, testData1, "FSPacker", 1, 32)
23+
24+
print("Test data two [1 times]")
25+
testData2 = pickle.loads(zlib.decompress(open("{}/testData2.dat".format(dirname(__file__)), 'rb').read()))
26+
benchmark(pickle.dumps, pickle.loads, testData2, "pickle", 1, 32)
27+
benchmark(marshal.dumps, marshal.loads, testData2, "marshal", 1, 32)
28+
benchmark(fsPacker.dumps, fsPacker.loads, testData2, "FSPacker", 1, 32)
29+
30+
print("Test data three [1000 times]")
31+
testData2 = { "jsonrpc":"2.0", "id":"00000000000", "method":"sampleMethod", "params":[True, "Joe", 88] }
32+
benchmark(pickle.dumps, pickle.loads, testData2, "pickle", 1000, 32)
33+
benchmark(marshal.dumps, marshal.loads, testData2, "marshal", 1000, 32)
34+
benchmark(fsPacker.dumps, fsPacker.loads, testData2, "FSPacker", 1000, 32)

‎benchmark/testData1.dat

84.7 KB
Binary file not shown.

‎benchmark/testData2.dat

173 KB
Binary file not shown.

0 commit comments

Comments
 (0)