Skip to content

Commit 83fbc2b

Browse files
author
Peter Demin
committed
Copy-pasting documentation
1 parent 284ef9d commit 83fbc2b

File tree

4 files changed

+364
-0
lines changed

4 files changed

+364
-0
lines changed

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,4 @@
11
*.pyc
22
build
3+
dist
4+
MANIFEST

‎README.md‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
## SUMMARY
2+
3+
Django_replicated is a Django [database router][1] designed to support more or
4+
less automatic master-slave replication. It keeps an internal state that
5+
depends on user intent to read or to write into a database. Depending on this
6+
state it automatically uses the right database (master or slave) for all
7+
SQL operations.
8+
9+
[1]: http://docs.djangoproject.com/en/dev/topics/db/multi-db/#topics-db-multi-db-routing
10+
11+
12+
## INSTALLATION
13+
14+
1. Install django_replicated distribution using "python setup.py install".
15+
16+
2. In settings.py configure your master and slave databases in a standard way:
17+
18+
DATABASES {
19+
'default': {
20+
# ENGINE, HOST, etc.
21+
},
22+
'slave1': {
23+
# ENGINE, HOST, etc.
24+
},
25+
'slave2': {
26+
# ENGINE, HOST, etc.
27+
},
28+
}
29+
30+
3. Teach django_replicated which databases are slaves:
31+
32+
DATABASE_SLAVES = ['slave1', 'slave2']
33+
34+
The 'default' database is always treated as master.
35+
36+
4. Configure a replication router:
37+
38+
DATABASE_ROUTERS = ['django_replicated.ReplicationRouter']
39+
40+
5. Configure timeout to exclude a database from the available list after an
41+
unsuccessful ping:
42+
43+
DATABASE_DOWNTIME = 20
44+
45+
The default downtime value is 60 seconds.
46+
47+
48+
## USAGE
49+
50+
Django_replicated routes SQL queries into different databases based not only on
51+
their type (insert/update/delete vs. select) but also on its own current state.
52+
This is done to avoid situation when in a single logical operations you're
53+
doing both writes and reads. If all writes would go into one database and reads
54+
would be from another one then you won't have a consistent view of the world
55+
because of two reasons:
56+
57+
- when using transactions the result of writes won't be replicated into a slave
58+
until commit
59+
- even in a non-transactional environment there's always a certain lag between
60+
updates in a master and in slaves
61+
62+
Django_replicated expects you to define what these logical operations are
63+
doing: writing/reading or only reading. Then it will try to use slave databases
64+
only for purely reading operations.
65+
66+
To define this there are several methods.
67+
68+
69+
### Middleware
70+
71+
If your project is built in accordance to principles of HTTP where GET requests
72+
don't cause changes in the system (unless by side effects) then most of the
73+
work is done by simply using a middleware :
74+
75+
MIDDLEWARE_CLASSES = [
76+
...
77+
'django_replicated.middleware.ReplicationMiddleware',
78+
...
79+
]
80+
81+
The middleware sets replication state to use slaves during handling of GET and
82+
HEAD requests and to use a master otherwise.
83+
84+
While this is usually enough there are cases when DB access is not controlled
85+
explicitly by your business logic. Good examples are implicit creation of
86+
sessions on first access, writing some bookkeeping info, implicit registration
87+
of a user account somewhere inside the system. These things can happen at
88+
arbitrary moments of time, including during GET requests.
89+
90+
Generally django_replicated handles this by always providing a master databases
91+
for write operations. If this is not enough (say you still want to read a
92+
newly created session and want to make sure that it will be read from a master)
93+
you can always instruct Django ORM to [use a certain database][2].
94+
95+
[2]: http://docs.djangoproject.com/en/dev/topics/db/multi-db/#manually-selecting-a-database
96+
97+
98+
### Decorators
99+
100+
If your system doesn't depend on the method of HTTP request to do writes and
101+
reads you can use decorators to wrap individual views into master or slave
102+
replication modes:
103+
104+
from django_replicated.decorators import use_master, use_slave
105+
106+
@use_master
107+
def my_view(request, ...):
108+
# master database used for all db operations during
109+
# execution of the view (if not explicitly overridden).
110+
111+
@use_slave
112+
def my_view(request, ...):
113+
# same with slave connection
114+
115+
116+
### GET after POST
117+
118+
There is a special case that needs addressing when working with asynchronous
119+
replication scheme. Replicas can lag behind a master database on receiving
120+
updates. In practice this mean that after submitting a POST form that redirects
121+
to a page with updated data this page may be requested from a slave replica
122+
that wasn't updated yet. And the user will have an impression that the submit
123+
didn't work.
124+
125+
To overcome this problem both ReplicationMiddleware and decorators support
126+
special technique where handling of a GET request resulting from a redirect
127+
after a POST is explicitly routed to a master database.
128+
129+
130+
### Global overrides
131+
132+
In some cases it might be necessary to override how the middleware chooses
133+
a target database based on the HTTP request method. For example you might want to
134+
route certain POST requests to a slave if you know that the request handler
135+
doesn't do any writes. The settings variable `REPLICATED_VIEWS_OVERRIDES` holds
136+
the mapping of view names (urlpatterns names) or view import paths to database
137+
names:
138+
139+
REPLICATED_VIEWS_OVERRIDES = {
140+
'api-store-event': 'slave',
141+
'app.views.do_smthg': 'master',
142+
}
143+
144+
145+
### Disabling state switching
146+
147+
There are cases when you want to disable switching of replication modes
148+
entirely. A most common example is testing your code with tests that use
149+
non-commiting transactions to preserve data between testcases. Each test is
150+
called with a default master database which it uses to load fixtures. Then if
151+
any code that the test calls will switch replication to the slave mode, it
152+
won't see any fixture data in a test slave database because the master never
153+
commits.
154+
155+
You can disable mode switching for such cases:
156+
157+
from django_replicated import utils
158+
utils.disable_state_change()
159+
160+
(There's also a similar `enalble_state_change()` function.)

‎README.rst‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
SUMMARY
2+
-------
3+
4+
Django\_replicated is a Django `database
5+
router <http://docs.djangoproject.com/en/dev/topics/db/multi-db/#topics-db-multi-db-routing>`_
6+
designed to support more or less automatic master-slave replication. It
7+
keeps an internal state that depends on user intent to read or to write
8+
into a database. Depending on this state it automatically uses the right
9+
database (master or slave) for all SQL operations.
10+
11+
INSTALLATION
12+
------------
13+
14+
1. Install django\_replicated distribution using "python setup.py
15+
install".
16+
17+
2. In settings.py configure your master and slave databases in a
18+
standard way:
19+
20+
::
21+
22+
DATABASES {
23+
'default': {
24+
# ENGINE, HOST, etc.
25+
},
26+
'slave1': {
27+
# ENGINE, HOST, etc.
28+
},
29+
'slave2': {
30+
# ENGINE, HOST, etc.
31+
},
32+
}
33+
34+
3. Teach django\_replicated which databases are slaves:
35+
36+
::
37+
38+
DATABASE_SLAVES = ['slave1', 'slave2']
39+
40+
The 'default' database is always treated as master.
41+
42+
4. Configure a replication router:
43+
44+
::
45+
46+
DATABASE_ROUTERS = ['django_replicated.ReplicationRouter']
47+
48+
5. Configure timeout to exclude a database from the available list after
49+
an unsuccessful ping:
50+
51+
::
52+
53+
DATABASE_DOWNTIME = 20
54+
55+
The default downtime value is 60 seconds.
56+
57+
USAGE
58+
-----
59+
60+
Django\_replicated routes SQL queries into different databases based not
61+
only on their type (insert/update/delete vs. select) but also on its own
62+
current state. This is done to avoid situation when in a single logical
63+
operations you're doing both writes and reads. If all writes would go
64+
into one database and reads would be from another one then you won't
65+
have a consistent view of the world because of two reasons:
66+
67+
- when using transactions the result of writes won't be replicated into
68+
a slave until commit
69+
- even in a non-transactional environment there's always a certain lag
70+
between updates in a master and in slaves
71+
72+
Django\_replicated expects you to define what these logical operations
73+
are doing: writing/reading or only reading. Then it will try to use
74+
slave databases only for purely reading operations.
75+
76+
To define this there are several methods.
77+
78+
Middleware
79+
~~~~~~~~~~
80+
81+
If your project is built in accordance to principles of HTTP where GET
82+
requests don't cause changes in the system (unless by side effects) then
83+
most of the work is done by simply using a middleware :
84+
85+
::
86+
87+
MIDDLEWARE_CLASSES = [
88+
...
89+
'django_replicated.middleware.ReplicationMiddleware',
90+
...
91+
]
92+
93+
The middleware sets replication state to use slaves during handling of
94+
GET and HEAD requests and to use a master otherwise.
95+
96+
While this is usually enough there are cases when DB access is not
97+
controlled explicitly by your business logic. Good examples are implicit
98+
creation of sessions on first access, writing some bookkeeping info,
99+
implicit registration of a user account somewhere inside the system.
100+
These things can happen at arbitrary moments of time, including during
101+
GET requests.
102+
103+
Generally django\_replicated handles this by always providing a master
104+
databases for write operations. If this is not enough (say you still
105+
want to read a newly created session and want to make sure that it will
106+
be read from a master) you can always instruct Django ORM to `use a
107+
certain
108+
database <http://docs.djangoproject.com/en/dev/topics/db/multi-db/#manually-selecting-a-database>`_.
109+
110+
Decorators
111+
~~~~~~~~~~
112+
113+
If your system doesn't depend on the method of HTTP request to do writes
114+
and reads you can use decorators to wrap individual views into master or
115+
slave replication modes:
116+
117+
::
118+
119+
from django_replicated.decorators import use_master, use_slave
120+
121+
@use_master
122+
def my_view(request, ...):
123+
# master database used for all db operations during
124+
# execution of the view (if not explicitly overridden).
125+
126+
@use_slave
127+
def my_view(request, ...):
128+
# same with slave connection
129+
130+
GET after POST
131+
~~~~~~~~~~~~~~
132+
133+
There is a special case that needs addressing when working with
134+
asynchronous replication scheme. Replicas can lag behind a master
135+
database on receiving updates. In practice this mean that after
136+
submitting a POST form that redirects to a page with updated data this
137+
page may be requested from a slave replica that wasn't updated yet. And
138+
the user will have an impression that the submit didn't work.
139+
140+
To overcome this problem both ReplicationMiddleware and decorators
141+
support special technique where handling of a GET request resulting from
142+
a redirect after a POST is explicitly routed to a master database.
143+
144+
Global overrides
145+
~~~~~~~~~~~~~~~~
146+
147+
In some cases it might be necessary to override how the middleware
148+
chooses a target database based on the HTTP request method. For example
149+
you might want to route certain POST requests to a slave if you know
150+
that the request handler doesn't do any writes. The settings variable
151+
``REPLICATED_VIEWS_OVERRIDES`` holds the mapping of view names
152+
(urlpatterns names) or view import paths to database names:
153+
154+
::
155+
156+
REPLICATED_VIEWS_OVERRIDES = {
157+
'api-store-event': 'slave',
158+
'app.views.do_smthg': 'master',
159+
}
160+
161+
Disabling state switching
162+
~~~~~~~~~~~~~~~~~~~~~~~~~
163+
164+
There are cases when you want to disable switching of replication modes
165+
entirely. A most common example is testing your code with tests that use
166+
non-commiting transactions to preserve data between testcases. Each test
167+
is called with a default master database which it uses to load fixtures.
168+
Then if any code that the test calls will switch replication to the
169+
slave mode, it won't see any fixture data in a test slave database
170+
because the master never commits.
171+
172+
You can disable mode switching for such cases:
173+
174+
::
175+
176+
from django_replicated import utils
177+
utils.disable_state_change()
178+
179+
(There's also a similar ``enalble_state_change()`` function.)

‎setup.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,31 @@
11
from distutils.core import setup
22

33

4+
with open('README.rst') as fp:
5+
readme = fp.read()
6+
7+
48
setup(
59
name='flask_replicated',
10+
description=(
11+
'Flask SqlAlchemy router for stateful master-slave replication'
12+
),
13+
long_description=readme,
14+
author='Peter Demin',
15+
author_email='poslano@gmail.com',
16+
url='https://github.com/peterdemin/python-flask-replicated',
17+
license="BSD",
18+
zip_safe=False,
19+
keywords='flask sqlalchemy replication master slave',
20+
classifiers=[
21+
'Development Status :: 5 - Production/Stable',
22+
'Intended Audience :: Developers',
23+
'License :: OSI Approved :: BSD License',
24+
'Natural Language :: English',
25+
'Programming Language :: Python :: 2',
26+
'Programming Language :: Python :: 2.6',
27+
'Programming Language :: Python :: 2.7',
28+
],
629
version='1.0',
730
py_modules=['flask_replicated'],
831
)

0 commit comments

Comments
 (0)