Tooling

Notes On Tooling And Coding Standards

Wherever possible we should rely on widely-used tooling and standards for our work, to make it easier to repurpose, and to make it simpler to understand for newcomers to the project. Similarly, we should focus on keeping requirements and toolsets simple and maintenance as easy as practical. See also open data source ideas.

Likely Future Technical Direction

Short term, the goal is to create a static and no-maintenance site showcasing topically interesting data and some visualizations, to showcase our work and to attract more contributors and interest from town government. Thinking ahead, there are three obvious ways this project could head in terms of architecture:

• If the town does not have sufficient resources to help much:
  • Continue building a static site on GitHub Pages with simple layouts of data files and metadata that others can repurpose. Zero maintenance, practical to do with only volunteers, but also limited - we'd be hard pressed to keep data updated regularly or to cover all the different areas of town government that might be useful.
• If the town wants to help, but lacks resources (or funds) to drive this:
  • Build out a CKAN/DKAN site, and share administration with town employees. This can be done with some volunteer work and enough cash to pay for a simple virtual server hosting plan, using all open source software. Town employees would curate and upload data as their time permits; and our volunteers would do most of the admin and maintenance of the site. The advantage is much better security and interoperability (this would support solid API access to the data too).
• If the town wants to run a real data portal (and has some funding):
  • If this happens, it's really up to town IT to drive the technical solution using something like Socrata or ArcGIS to publish data for all town departments. Volunteers would then help as able to request or help format new datasets, and build visualizations based off of the town's official portal.

Open Data Standards And Policies

• https://sunlightfoundation.com/policy/opendatafaq/ General FAQ on open data projects.
• https://18f.gsa.gov/ Major US Government project working on open data.
• https://resources.data.gov/ Homepage for US open data.
• https://www.codeforamerica.org/ Code for America brings volunteers together to work on civic tech projects.
• https://project-open-data.cio.gov/v1.1/schema/ Specific schema used in most US open data.
• https://www.w3.org/TR/vocab-dcat/ DCAT - Data Catalog Vocabulary - is a detailed RDF standard for metadata (possibly too complex for our case).
• http://www.oecd.org/gov/digital-government/open-government-data.htm International group promoting open data for governments around the world.
• https://opendefinition.org/od/2.1/en/ Concepts and licensing around what "Open" means for data and other works.
• http://census.okfn.org/en/latest/ Open Data Census, measuring what's open.

Data Portal Software Ideas

• CKAN a comprehensive data portal with broad usage. (Affero GNU GPL v3.0)
• DKAN is a Drupal-based version of CKAN. (GPL 2.0)
• ArcGIS Hub (commercial software) is already used by the town for some things.
• Dataverse portal focused on research data. (Apache-2.0)
• Socrata (commercial software) offers an open source version of their data portal, although it requires review to see if it's sufficient. (Apache-2.0)

A view on Cambridge's data portal plans. Brookline, and Somerville also have open data portals currently.

Directory Structure

arlingtonma.info
├── src                # Scraping or data utilities - not built into production website
|  ├── townparser.rb   # Simplistic town website link scraper
├── docs               # Root of github-pages / Jekyll style website
|  ├── _data           # DO NOT USE (see data instead)
|  ├── _includes       # Jekyll include files defining layouts, etc.
|  ├── _pages          # Web pages for meta-concepts in the site: About page, data sources, etc.
|  ├── _posts          # Blog-like web pages for individual data visualizations or descriptive data overviews
|  ├── data            # Data files and corresponding .json dcat:Dataset files about them
|  ├── assets          # JS, CSS, other code files used in the production website
|    ├── css
|    |  └── main.scss  # Custom stylesheet for our site (most CSS is in minimal-mistakes theme)
|    ├── images        # Image assets for posts/pages/collections/etc.
|    ├── js            # Included and custom js files for visualizations

Current Technologies Used In This Site

The site is currently designed with: (subject to change as more contributors join!)

• GitHub Pages wherever possible (for zero-maintenance sites)
• Minimal Mistakes theme and only standard GitHub-Flavored Markdown (GFM) Jekyll tooling
• Standard JS code formatting
• D3.js data driven documents library
• C3.js charting library (built on D3)

Shane's Personal Coding Setup

Personally, I use:

• OSX Catalina
• VS Code

Considerations for Contributors' Coding Setups

The ability to contribute shouldn't require invasive changes to volunteers' existing setups. E.g. the ability to contribute should be accessible to those running different operating systems or preferring different kinds of development tools, e.g. visual IDEs (like VS Code) and command-line tools (like vim or emacs).