Hot reload #362

T4rk1n · 2018-08-31T21:01:30Z

Available with the following versions:

dash==0.29.0rc4
dash-renderer==0.15.0rc2

Reload the browser when files are changed.

Use the flask debug reloader to reset the hash on python file change.
Watch the assets files and change the hash on assets modified.
Add a route to serve the reload hash.

Dev tools params (in app.run_server or app.enable_dev_tools

dev_tools_hot_reload, bool set to true to enable hot reload (default=False).
dev_tools_hot_reload_interval, int, interval at which the renderer will request the reload hash (default=3000).
dev_tools_hot_reload_watch_interval, float, The delay between each walk of the assets folder to detect file changes.
dev_tools_silence_routes_logging, disable the route logging.

Hot reload is activated by default when debug=True !!

Need plotly/dash-renderer#73.

Resolves #66.
Fix #341.

T4rk1n · 2018-09-11T20:01:14Z

@plotly/dash Ready for reviews!

rmarren1 · 2018-09-12T00:42:03Z

dash/_watch.py

+                    if pattern and not pattern.search(f):
+                        continue
+                    path = os.path.join(current, f)
+                    info = os.stat(path)


os.path.getmtime might be cleaner here. I thought there was platform dependence issue at first (since otherwise why would they even implement os.path.getmtime if you can just os.stat(path).st_mtime, but it turns out thats exactly what os.path.getmtime does.

Yea, some methods in that module seems to be wrapper around os.stat. I'm used to os.stat so I just do that.

👍, either way is good, just seemed a bit strange, seems to break "There should be one-- and preferably only one --obvious way to do it".

I think the os module predates the PEP's.

rmarren1 · 2018-09-12T00:42:49Z

dash/_watch.py

+
+def watch(folders, on_change, pattern=None, sleep_time=0.1):
+    pattern = re.compile(pattern) if pattern else None
+    watched = collections.defaultdict(lambda: -1)


Neat, didn't know you could do this.

rmarren1 · 2018-09-12T01:07:45Z

dash/dash.py

+            self._watch_thread = threading.Thread(
+                target=lambda: _watch.watch([self._assets_folder],
+                                            self._on_assets_change,
+                                            sleep_time=0.5))


Shouldn't sleep_time be hot_reload_interval / 1000? I see that hot_reload_interval is passed to the front-end via config, but I'm not sure if it is intended to be the same on the back-end. If not (e.g. back-end should have a fixed constant refresh interval regardless of hot_reload_interval value) I think you should make this a constant set in the _watch file.

The sleep time is the poll time for the assets watch thread and hot_reload_interval is for the client requests interval. I don't think the user will need to change the sleep time so it can be a constant.

I think it makes sense to make these two times the same, or at least proportional. With this there is a .5 second lower bound on the hot reload interval, and with large intervals (e.g. 10 seconds) you would be making a lot of calls to watch that aren't really necessary, if a user has a huge app that takes a non-trivial amount of computation to traverse the asset tree, this might affect performance.

I think it makes sense to make sleep_time = (hot_reload_interval / 1000) / d where d is some constant. Then in the worst case after making a change, a user would need to wait hot_reload_interval + (hot_reload_interval / d) time for the page to change. d = 2 or d = 4 seems reasonable.

I don't think it matter much, it just the timer for when to run the check in the back end and should not be linked in any way to the request timer that can be changed. It should be a constant that work goods in the conditions. We don't want run that check too often because of the GIL and it should still be low so you can get the reload a short while after making changes. So I think a check every half a second is good with this in mind.

In my asyncio based implementation it was a really low number like yield from asyncio.sleep(0.000001) and it didn't make a difference in the performance of the app.

rmarren1 · 2018-09-12T01:16:16Z

dash/dash.py

+            self._watch_thread.start()
+
+            if not self.server.debug:
+                self.server.debug = True


I think this would get over-ridden when somebody calls app.run_server(debug=False). Should that happen? I think in run_server you can just say debug = debug or self.hot_reload

I think I will remove that from here and add it in the enable method from #369.

rmarren1 · 2018-09-12T01:35:53Z

dash/dash.py

+    def serve_reload_hash(self):
+        return flask.jsonify({
+            'reloadHash': self._reload_hash,
+            'hard': self._hard_reload


It doesn't look like the 'hard' variable is used at all in the front-end. Can we just remove it here, in _on_assets_change and in __init__?

It is used to trigger a soft reload.
https://github.com/plotly/dash-renderer/blob/71a42e2f63880ec8ce2cbc01a8e18e582857f97d/src/components/core/Reloader.react.js#L35

Oh okay, I see. So is this variable just set to true on the first asset change? I see it gets set to True in _on_assets_change but don't see it getting set back anywhere

Good catch, need to reset it after serving the response.

rmarren1 · 2018-09-12T01:39:15Z

dash/dash.py

@@ -949,6 +986,12 @@ def get_asset_url(self, path):
            self.config.routes_pathname_prefix,
            path)

+    def _on_assets_change(self, _):


Why pass the argument here? Can't we just call on_change()

The watch function I got here is from another project of mine that I adapted to run on threads instead of asyncio. When a file changed in the other project, I would log the filename thus giving the changed filename as argument in the callback. But we don't have a logging system in-place so I just put a pass param. I guess I could remove the options we're not gonna use here.

Oh okay, didn't know thats what the argument was. I think we should keep it to add logging in the future, something like File <file_name> was changed at <time>. Maybe put a comment that says the _ is the file name in case another dev makes this change.

rmarren1

Looks good! These are mostly questions, and I think there are a few things you can delete

chriddyp · 2018-09-14T00:20:20Z

This is looking really great!

A few things I ran into:

If I add a new component library in layout, the JS bundle doesn't get served. Use case: start with an empty app.layout = html.Div([]) and then incrementally build the app. When I added dcc.Graph, it didn't appear until I refreshed.
Similarly, if I created a new CSS file, I had to restart the app before I saw it.

It seems like the developer-facing API is small (just the hot_reload tag), so we should be free to change the backend implementation in the future, right? In particular, I would be interested in exploring (in the future):

A websocket based approach instead of front-end timers.
An approach where CSS files are reloaded separately without a full page reload. Use case:
- User could click through the application, getting it to a certain state where certain elements are being shown, and then style those elements with CSS without changing the state.
- CSS updates could happen a lot quicker than the Python updates, since it doesn't (in theory) necessarily require a full refresh of the underling python code

T4rk1n · 2018-09-14T14:48:48Z

If I add a new component library in layout, the JS bundle doesn't get served.

That's a hard one. Since it's a soft reload, it won't reload the index and the new resources doesn't get served.

The solution I am thinking of:

Keep a list of the dependencies in a list in the store, available in a API.requestComponentsLib.
Check the components libs scripts tags are all included in the requestComponentLib response, if not append it to the footer and wait for it to be loaded before triggering a soft reload.

if I created a new CSS file, I had to restart the app before I saw it.

I think that is related to #341. The assets folder is walked only once during initialization, my initial fix was to walk every time the index is served. But then I thought it would be better to integrate the watch of hot-reload and add it to the assets resources since that fix is not needed for deployments and incur a performance cost.

It seems like the developer-facing API is small (just the hot_reload tag),

There's also hot_reload_interval, and maybe watch_interval that can be added.

An approach where CSS files are reloaded separately without a full page reload.

Could do that by disabling the old stylesheets and append the new one with the modified timestamp in the head.

chriddyp · 2018-09-14T19:02:32Z

If I add a new component library in layout, the JS bundle doesn't get served.

That's a hard one. Since it's a soft reload, it won't reload the index and the new resources doesn't get served.

Another approach might be to solve it by solving plotly/dash-renderer#46. I think that issue will be a lot more challenging to solve though.

T4rk1n · 2018-09-14T19:15:00Z

@chriddyp

I solved it by including the keys of dash.registered_paths in the reload request and checking that the length and content is the same, if not hard reload. Was easier than I thought.

For plotly/dash-renderer#46, I think we would have to add a method to register the packages externally from the layout collection so they can be included even if not in the initial layout.

T4rk1n · 2018-09-17T18:27:32Z

An approach where CSS files are reloaded separately without a full page reload

@chriddyp I added that functionality.

updates

T4rk1n · 2018-10-03T15:44:01Z

Updated the rc versions:

dash==0.29.0rc3
dash-renderer==0.15.0rc2

chriddyp · 2018-10-03T17:17:24Z

This latest version is looking really great. The new CSS pathway is especially nice 🎉

mbkupfer · 2018-10-03T17:38:20Z

I'm having trouble with running this. When I start my app, it goes into an endless loop with the following error messages:


  File "C:\Users\mkupfer\Desktop\MYDOCU~1\01_VIS~1\PAY-RA~1\venv\lib\site-packag
es\flask\app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\mkupfer\Desktop\MYDOCU~1\01_VIS~1\PAY-RA~1\venv\lib\site-packag
es\flask\_compat.py", line 35, in reraise
    raise value
  File "C:\Users\mkupfer\Desktop\MYDOCU~1\01_VIS~1\PAY-RA~1\venv\lib\site-packag
es\flask\app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\Users\mkupfer\Desktop\MYDOCU~1\01_VIS~1\PAY-RA~1\venv\lib\site-packag
es\flask\app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "C:\Users\mkupfer\Desktop\MYDOCU~1\01_VIS~1\PAY-RA~1\venv\lib\site-packag
es\dash\dash.py", line 336, in serve_reload_hash
    'files': list(changed)
  File "C:\Users\mkupfer\Desktop\MYDOCU~1\01_VIS~1\PAY-RA~1\venv\lib\site-packag
es\flask\json\__init__.py", line 321, in jsonify
    dumps(data, indent=indent, separators=separators) + '\n',
  File "C:\Users\mkupfer\Desktop\MYDOCU~1\01_VIS~1\PAY-RA~1\venv\lib\site-packag
es\flask\json\__init__.py", line 179, in dumps
    rv = _json.dumps(obj, **kwargs)
  File "C:\Users\mkupfer\AppData\Local\Programs\Python\Python36-32\Lib\json\__in
it__.py", line 238, in dumps
    **kw).encode(obj)
  File "C:\Users\mkupfer\AppData\Local\Programs\Python\Python36-32\Lib\json\enco
der.py", line 201, in encode
    chunks = list(chunks)
  File "C:\Users\mkupfer\AppData\Local\Programs\Python\Python36-32\Lib\json\enco
der.py", line 430, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "C:\Users\mkupfer\AppData\Local\Programs\Python\Python36-32\Lib\json\enco
der.py", line 404, in _iterencode_dict
    yield from chunks
  File "C:\Users\mkupfer\AppData\Local\Programs\Python\Python36-32\Lib\json\enco
der.py", line 437, in _iterencode
    o = _default(o)
  File "C:\Users\mkupfer\Desktop\MYDOCU~1\01_VIS~1\PAY-RA~1\venv\lib\site-packag
es\flask\json\__init__.py", line 81, in default
    return _json.JSONEncoder.default(self, o)
  File "C:\Users\mkupfer\AppData\Local\Programs\Python\Python36-32\Lib\json\enco
der.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'dict_keys' is not JSON serializable

T4rk1n · 2018-10-03T17:49:00Z

@mbkupfer thanks for reporting, I think I know what's wrong, will post update soon.

T4rk1n · 2018-10-03T18:52:32Z

@mbkupfer Released dash==0.29.0rc4, should fix the json issue on py3.

T4rk1n · 2018-11-14T19:37:03Z

Had to remove the resources cache, a proper cache bust check was more expansive than remapping them.

chriddyp · 2018-11-15T00:25:31Z

Amazing work @T4rk1n ! Is there is an issue logged for the documentation items I mentioned in #362 (comment)?

T4rk1n · 2018-11-15T00:34:19Z

@chriddyp There is an issue for the dev tools https://github.com/plotly/dash-docs/issues/196

T4rk1n mentioned this pull request Aug 31, 2018

Hot reload plotly/dash-renderer#73

Merged

T4rk1n force-pushed the hot-reload branch 2 times, most recently from 9a48f6c to 8955a92 Compare September 5, 2018 17:18

T4rk1n changed the title ~~[WIP] Hot reload~~ Hot reload Sep 5, 2018

T4rk1n force-pushed the hot-reload branch from 8955a92 to 776232b Compare September 5, 2018 17:37

T4rk1n requested review from ned2 and rmarren1 September 11, 2018 20:00

rmarren1 suggested changes Sep 12, 2018

View reviewed changes

rmarren1 previously requested changes Sep 12, 2018

View reviewed changes

T4rk1n force-pushed the hot-reload branch from 3a45541 to fd15dd8 Compare September 12, 2018 14:50

T4rk1n mentioned this pull request Sep 12, 2018

remove werkzeug url print statements by default? #382

Closed

chriddyp added the Sponsored label Sep 12, 2018

T4rk1n force-pushed the hot-reload branch from 370ac65 to 227e237 Compare September 13, 2018 20:45

chriddyp self-requested a review September 13, 2018 22:50

T4rk1n force-pushed the hot-reload branch from a16cf81 to a1848f4 Compare September 14, 2018 18:13

T4rk1n force-pushed the hot-reload branch 2 times, most recently from a47c5d7 to 6736cc7 Compare September 21, 2018 18:33

T4rk1n force-pushed the hot-reload branch from a395d32 to 45521bf Compare October 3, 2018 17:51

T4rk1n added 18 commits November 14, 2018 11:56

Add package list to the reload response.

7c77bc8

Support reloading css only.

8c993fa

Add configurable watch_interval.

631da7b

Integrate hot reload with enable_dev_tools.

08bbdc1

Line length fix & add more info to the dev tools docstring.

410865e

Add hot reload dev tools arguments to run_server.

46383d1

Set to serve locally when dev tools are activated.

1e4eae6

Add silence routes logging to dev tools.

e714af3

Fix logging formatting.

d74498f

Cast registered_paths keys to list, py3 compat.

494243e

Add hot_reload_max_retry.

1ae7dca

Generate a debugger PIN and print it when logs are silenced.

f22ea7d

Fix rebase.

353d79c

Add missing dev_tools docstring.

4654e5f

Fix 🐛 with assets watch deletion.

beadd63

Proper path manipulation on assets change.

407e24c

🐐 environ -> environment.

63033ef

Add DASH_HOT_RELOAD_MAX_RETRY environment variable.

fb26306

T4rk1n force-pushed the hot-reload branch from 8ed142a to fb26306 Compare November 14, 2018 16:57

🔪 remove resources cache.

927d69b

Update version and changelog.

d410c9c

T4rk1n merged commit f590166 into master Nov 14, 2018

T4rk1n deleted the hot-reload branch November 14, 2018 21:06

rpkyle mentioned this pull request Sep 3, 2019

Need to implement hot reloading support in Dash for R plotly/dashR#125

Closed

rpkyle mentioned this pull request Sep 18, 2019

Provide support for hot reloading in Dash for R plotly/dashR#127

Merged

12 tasks

rpkyle mentioned this pull request May 18, 2020

Hot reloading support in Dash.jl plotly/Dash.jl#20

Closed

waralex mentioned this pull request May 19, 2020

Hot reload plotly/Dash.jl#25

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hot reload #362

Hot reload #362

T4rk1n commented Aug 31, 2018 •

edited

Loading

T4rk1n commented Sep 11, 2018

rmarren1 Sep 12, 2018

T4rk1n Sep 13, 2018

rmarren1 Nov 9, 2018

T4rk1n Nov 14, 2018

rmarren1 Sep 12, 2018

rmarren1 Sep 12, 2018

T4rk1n Sep 12, 2018

rmarren1 Sep 12, 2018

T4rk1n Sep 13, 2018

rmarren1 Sep 12, 2018

T4rk1n Sep 13, 2018

rmarren1 Sep 12, 2018

T4rk1n Sep 12, 2018

rmarren1 Sep 12, 2018

T4rk1n Sep 12, 2018

rmarren1 Sep 12, 2018

T4rk1n Sep 12, 2018

rmarren1 Sep 12, 2018

rmarren1 left a comment

chriddyp commented Sep 14, 2018 •

edited by T4rk1n

Loading

T4rk1n commented Sep 14, 2018

chriddyp commented Sep 14, 2018

T4rk1n commented Sep 14, 2018

T4rk1n commented Sep 17, 2018

T4rk1n commented Oct 3, 2018

chriddyp commented Oct 3, 2018

mbkupfer commented Oct 3, 2018

T4rk1n commented Oct 3, 2018

T4rk1n commented Oct 3, 2018

T4rk1n commented Nov 14, 2018

chriddyp commented Nov 15, 2018

T4rk1n commented Nov 15, 2018

Hot reload #362

Hot reload #362

Conversation

T4rk1n commented Aug 31, 2018 • edited Loading

T4rk1n commented Sep 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmarren1 left a comment

Choose a reason for hiding this comment

chriddyp commented Sep 14, 2018 • edited by T4rk1n Loading

T4rk1n commented Sep 14, 2018

chriddyp commented Sep 14, 2018

T4rk1n commented Sep 14, 2018

T4rk1n commented Sep 17, 2018

T4rk1n commented Oct 3, 2018

chriddyp commented Oct 3, 2018

mbkupfer commented Oct 3, 2018

T4rk1n commented Oct 3, 2018

T4rk1n commented Oct 3, 2018

T4rk1n commented Nov 14, 2018

chriddyp commented Nov 15, 2018

T4rk1n commented Nov 15, 2018

T4rk1n commented Aug 31, 2018 •

edited

Loading

chriddyp commented Sep 14, 2018 •

edited by T4rk1n

Loading