Updating the plugin to deal with event elements.

.github/workflows/pycodestyle.yml

@@ -0,0 +1,23 @@
+name: Pycodestyle
+
+on: [push]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.10"]
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v3
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install pycodestyle
+    - name: Analysing the code with pycodestyle
+      run: |
+        pycodestyle --ignore=E501,W503 $(git ls-files 'learning_observer/*.py' 'modules/*.py')

.gitignore

@@ -1 +1,30 @@
 *~
+\#*
+.\#*
+*__pycache__*
+webapp/logs
+webapp/static_data/teachers.yaml
+creds.yaml
+CREDS.YAML
+uncommitted
+extension.crx
+extension.pem
+extension.zip
+*egg-info*
+public_key
+*/dist
+learning_observer/learning_observer/static_data/teachers.yaml
+learning_observer/learning_observer/logs/
+learning_observer/learning_observer/static/3rd_party/
+learning_observer/learning_observer/static_data/course_lists/
+learning_observer/learning_observer/static_data/course_rosters/
+learning_observer/learning_observer/static_data/repos/
+learning_observer/learning_observer/static_data/dash_assets/
+learning_observer/learning_observer/static_data/courses.json
+learning_observer/learning_observer/static_data/students.json
+learning_observer/passwd.lo
+--*
+.venv/
+.vscode/
+build/
+dist/

CONTRIBUTORS.TXT

@@ -0,0 +1,3 @@
+Piotr Mitros
+Oren Livne
+Paul Deane

Makefile

@@ -0,0 +1,65 @@
+PYTHONFILES = $(wildcard \
+	learning_observer/learning_observer/*py \
+	learning_observer/util/*py \
+	gitserve/gitserve/*py \
+	gitserve/*py \
+	learning_observer/learning_observer/pubsub/*py \
+	learning_observer/learning_observer/stream_analytics/*py \
+)
+
+run:
+	# If you haven't done so yet, run: make install
+	# Also, run: workon learning_observer
+	cd learning_observer && python learning_observer
+
+# Build browser extension
+extension-package:
+	# I won't build this for you, but I will give you instructions, since
+	# the set of options will vary.
+	#
+	# On my last system, I needed:
+	#	google-chrome --pack-extension=extension --pack-extension-key=extension.pem --disable-setuid-sandbox --no-gpu --no-sandbox --headless
+	# On my current system
+	#	google-chrome --pack-extension=extension --pack-extension-key=extension.pem
+	# The pem file is your private key. You'll need to make one.
+	# The output will be called extension.crx
+
+codestyle:
+	# Check code style quality
+	#
+	# In pycodestyle, we ignore E501 (line too long) and W503 (which
+	# requires a choice.
+        # https://stackoverflow.com/questions/57074300/what-is-the-recommended-way-to-break-long-if-statement-w504-line-break-after-b
+	#
+	# In pylint, we ignore:
+	# 1. W0613: unused arguments (common for e.g. `request` parameter)
+	# 2. E501/C0301: line too long (obsolete with modern computers
+	#    ref: https://lkml.org/lkml/2020/5/29/1038)
+	#    We still aim for shorter lines, but it's not a showstopper if
+	#    we break the limit occasionally. We may re-examine this decision
+	#    if we have sufficient developers working on VT100 terminals.
+	# 3. W0511: TODO: We're gonna have a lot of these, and we want to
+	#    encourage leaving these around. As a coding style, we want to
+	#    get interface and overall structures right, and then clean
+	#    up e.g. performance/scaling, exception handling, test coverage,
+	#    etc. once we know what we're doing.
+	# 4. R0913: Too many arguments. That's a relic of web apps. Sorry.
+	#
+	# Generally, pycodestyle issues are showstopppers for pushing to
+	# upstream. Pylint issues are worth an occasional cleanup pass, but we
+	# can tolerate.
+
+	pycodestyle --ignore=E501,W503 $(PYTHONFILES)
+	pylint -d W0613,W0511,C0301,R0913,too-few-public-methods $(PYTHONFILES)
+
+install:
+	# Run:
+	#    mkvirtualenv learning_observer
+	#    pip install -r requirements.txt
+	#    cd learning_observer
+	#    python setup.py develop
+	#    python learning_observer
+	# And then clean up this Makefile to do it for you automatically :)
+
+sphinx:
+	cd learning_observer/docs; make html

README.md

@@ -1,6 +1,143 @@
-# Writing analysis
-This repository is for a writing analysis project. There isn't much to see here yet.
+# Writing Observer and Learning Observer
 
-Contact/maintainer: Piotr Mitros (pmitros@ets.org)
+![Writing Observer Logo](learning_observer/learning_observer/static/media/logo-clean.jpg)
 
-Licensing: Open source / free software. License TBD. 
+This repository is part of a project to provide an open source
+learning analytics dashboard to help instructors be able to manage
+student learning processes, and in particular, student writing
+processes.
+
+![linting](https://github.com/ETS-Next-Gen/writing_observer/actions/workflows/pycodestyle.yml/badge.svg)
+
+## Learning Observer
+
+_Learning Observer_ is designed as an open source, open science learning
+process data dashboarding framework. You write reducers to handle
+per-student writing data, and aggegators to make dashboards. We've
+tested this in math and writing, but our focus is on writing process
+data.
+
+It's not finished, but it's moving along quickly.
+
+## Writing Observer
+
+_Writing Observer_ is a plug-in for Google Docs which visualizes writing
+data to teachers. Our immediate goal was to provide a dashboard which
+gives rapid, actionable insights to educators supporting remote
+learning during this pandemic. We're working to expand this to support
+a broad range of write-to-learn and collaborative learning techniques.
+
+## Status
+
+There isn't much to see here for external collaborators yet. This
+repository has a series of prototypes to confirm we can:
+
+* collect the data we want;
+* extract what we need from it; and
+* route it to where we want it to go (there's *a lot* of data, with
+  complex dependencies, so this is actually a nontrivial problem)
+
+Which mitigates most of the technical risk. We also now integrate with
+Google Classroom. We also have prototype APIs for making dashboards, and
+a few prototype dashboards.
+
+For this to be useful, we'll need to provide some basic documentation
+for developers to be able to navigate this repo (in particular,
+explaining *why* this approach works).
+
+This system is designed to be *massively* scalable, but it is not
+currently implemented to be so (mostly for trivial reasons;
+e.g. scaffolding code which uses static files as a storage model). It
+will take work to flush out all of these performance issues, but we'd
+like to do that work once we better understand what we're doing and
+that the core approach and APIs are correct.
+
+Getting Started
+===============
+
+As an early prototype, getting started isn't seamless. Run:
+
+~~~~~
+make install
+~~~~~
+
+And follow the instructions. You'll probably run into bugs. Work around the bugs. Then fix up the makefile and make a PR to address those bugs :)
+
+Once that's done, run:
+
+~~~~
+make
+~~~~
+
+Again, fix up the makefile, and make a PR.
+
+You can also go into the devops directory, which has scripts in
+progress for spinning up a cloud instance and managing flocks of
+_Learning Observer_ instances.
+
+
+~~~~
+Writing Observer
+~~~~
+
+
+To setup writing_observer on top of the learning observer platform you must go into modules/writing_observer and run:
+
+   sudo python setup.py develop
+
+
+
+
+System requirements
+===================
+
+It depends on what you're planning to use the system for.
+
+The core _Learning Observer_ system works fine on an AWS nano
+instance, and that's how we do most of our testing and small-scale
+pilots. These instances have 512MB of RAM, and minimal CPU. It's
+important that this configuration remains usable.
+
+For deployment and more sophisticated uses (e.g. NLP) in larger
+numbers of classrooms, we expect to need **heavy** metal. As we're
+playing with algorithms, deep learning is turning out to work
+surprisingly well, and at the same time, requires surprisingly large
+amounts of computing power. A GPGPU with plenty of RAM is helpful if
+you want to work with more sophisticated algorithms, and is likely to
+be a requirement for many types of uses.
+
+All _Learning Observer_ development has been on Linux-based platforms
+(including Ubuntu and RHEL). There are folks outside of the core team
+who have tried to run it on Mac or on WSL, with mixed success.
+
+Running on RHEL requires the following services:
+ * REDIS
+ * nginx.
+
+
+bcrypt
+-----------------------------------------------
+A note on bcrypt.  The code uses bcrypt for some internal password management.
+We are not including it directly in the install because it acts oddly across
+platforms so you may need to install some version manually.
+
+
+Additional Notes
+===================
+At present the system also uses static content that is served from a repo.
+This allows us to actually select different sources for the static data.
+This can also point to the current copy if necessary and can be configured
+as part of the creds.yaml file to generate the repo or add it on startup.
+
+This is not ideal but it is a mechanism for embedding the alternatives.
+
+
+Contributing or learning more
+=============================
+
+We're still a small team, and the easiest way is to shoot us a quick
+email. We'll gladly walk you through anything you're interested in.
+
+Contact/core maintainer: Piotr Mitros
+
+Licensing: Open source / free software. License: AGPL.

devops/README.md

@@ -0,0 +1,13 @@
+Dev-ops scripts
+===============
+
+This contains machinery for spinning up, shutting down, and managing
+Learning Observer servers. It's usable, but very much not done yet. We
+can spin up, spin down, and list machines, but this ought to be more
+fault-tolerant, better logged, less hard-coded, etc.
+
+We would like to be cross-platform, and evenually support both
+Debian-based distros and RPM-based distros. We're not there yet
+either. We'd also like to support multiple cloud providers. We're not
+there yet either. However, we probably won't accept PRs which move us
+away from this goal.