You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+74-7Lines changed: 74 additions & 7 deletions
Original file line number
Diff line number
Diff line change
@@ -2,13 +2,22 @@
2
2
3
3
# web-monitoring-db
4
4
5
-
This repository is the database and API underlying the EDGI [Web Monitoring Project](https://github.com/edgi-govdata-archiving/web-monitoring).
5
+
This repository is the database and API underlying the EDGI [Web Monitoring Project](https://github.com/edgi-govdata-archiving/web-monitoring). It’s a Rails app that:
6
6
7
-
It’s a Rails app that:
7
+
- Acts as a database of monitored pages and captured versions of those pages over time.
8
8
9
-
- Acts as a database of monitored pages and revisions that have been made to them
10
-
- Allows other services to add new tracked pages/versions (we are currently focused on Versionista, but this database will soon host data from other sources, such as the Internet Archive)
11
-
- Provides an API to get that version data and allow analysts or other automated tools to annotate those versions with metadata
9
+
*(The application does not record new versions itself, but relies on importing data from external services, like [the Internet Archive](https://archive.org) or [Versionista](https://versionista.com). See [“How Data Gets Loaded”](#how-data-gets-loaded) below for more.)*
10
+
11
+
- Provides an API to get that page and version data, and to allow analysts or other automated tools to annotate those versions with metadata about what has changed from version to version.
12
+
13
+
For more about how data is modeled in this project, see [“Data Model”](#data-model) below.
14
+
15
+
API documentation is available from the homepage of the application, e.g. by pointing your browser to http://localhost:3000/ or https://api.monitoring.envirodatagov.org. It’s generated from our OpenAPI docs in [`swagger.yml`](./swagger.yml).
16
+
17
+
We maintain a publicly available *staging server* at https://api-staging.monitoring.envirodatagov.org that you can test against. It runs the latest code and has non-production data — it’s safe to modify or post new versions or annotations to, but you should not rely on that data sticking around; it may get reset at any time. **For access, ask for an account on Slack or use the public user credentials:**
18
+
19
+
- Username: `public.access@envirodatagov.org`
20
+
- Password: `PUBLIC_ACCESS`
12
21
13
22
14
23
## Installation
@@ -175,7 +184,7 @@ It’s a Rails app that:
175
184
- `analysis`: Auto-analyze changes between versions and create annotations with the results.
176
185
177
186
178
-
## Manual Postgres Setup
187
+
### Manual Postgres Setup
179
188
180
189
If you don’t want to populate your DB with seed data, want to manage creation of the database yourself, or otherwise manually do database setup, run any of the following commands as desired instead of `rake db:setup`:
181
190
@@ -197,7 +206,7 @@ User.create(
197
206
```
198
207
199
208
200
-
## Docker
209
+
### Docker
201
210
202
211
The Dockerfile runs the rails server on port 3000 in the container. To build
Point your browser or ``curl`` at ``http://localhost:3000``.
213
222
214
223
224
+
## Data Model
225
+
226
+
The database models three main types of data:
227
+
228
+
- **Pages**, which represent a page on the internet. Pages are identified by a unique ID rather than their URL because pages can move or be available from multiple URLs. *(Note: we don't actually model that yet, though! See [#492](https://github.com/edgi-govdata-archiving/web-monitoring-db/issues/492) for more.)*
229
+
230
+
- **Versions**, which represent a particular page at a particular point in time. We use the term “version” instead of others more common in the archival space because we attempt to only represent *different* versions. That is, if a page changed on Wednesday and we captured copies of it on Monday, Tuesday, and Wednesday, we only make version records for Monday and Wednesday (because Tuesday was the same as Monday).
231
+
232
+
*(Note: because of technical issues around imported data, we often store more versions than we should according to the above definition [e.g. we might still have a record for Tuesday]. Versions have a `different` field that indicates whether a version is different from the previous one, and the API only returns versions that are `different` unless you explicitly request otherwise.)*
233
+
234
+
- **Annotations**, which represent an analysis about what’s changed between any two *versions* of a *page*. Annotations have a specialized `priority` and `significance`, which are numbers between 0 and 1, an `author`, indicating who made the analysis (it could be a bot account), and an `annotation` field, which is a JSON object with no specified structure (inside this field, annotations can include any data desired).
235
+
236
+
There are several other kinds of objects, but they are subservient to the ones above:
237
+
238
+
- **Changes**, which serve to connect any two *versions* of a *page*. *Annotations* are actually connected to *changes*, rather than directly to two *versions*. You can also generate diffs for a given *change*.
239
+
240
+
- **Tags**, which can be applied to pages. They help sort and categorize things. Most tags are manually applied, but the application auto-generates a few:
241
+
- `domain:<domain name>`, e.g. `domain:www.epa.gov` for a page at `https://www.epa.gov/citizen-science`
242
+
- `2l-domain:<second-level domain name>` e.g. `2l-domain:epa.gov` for a page at `https://www.epa.gov/citizen-science`
243
+
244
+
- **Maintainers**, which can be applied to pages. They represent organizations that maintain a given page. For example, the page at `https://www.epa.gov/citizen-science` is maintained by `EPA`.
245
+
246
+
- **Imports** model requests to import new data and the results of the import operation.
247
+
248
+
- **Users** model people (both human and bots) who can view, import, and annotate data. You currently have to have a user account to do anything in the application, though we hope accounts will not be needed to view public data in the future.
249
+
250
+
Actual database schemas for each of these tables is listed in [`db/schema.rb`](./db/schema.rb).
251
+
252
+
253
+
### How Data Gets Loaded
254
+
255
+
The web-monitoring-db project does not actually monitor or scrape pages on the web. Instead, we rely on importing data from other services, like [the Internet Archive](https://archive.org). Each day, a script queries other services for historical snapshots and sends the results to the `/api/v0/imports` endpoint.
256
+
257
+
Most of the data sent to `/api/v0/imports` matches up directly with the structure of the [`Version` model](./db/schema.rb). However, the `uri` field in an import is treated specially.
258
+
259
+
When new page or version data is imported, the `uri` field points to a location where the raw HTTP response body can be retrieved. If the `uri` host matches one of the values in the [`ALLOWED_ARCHIVE_HOSTS` environment variable](./.env.example), the version record that gets added to the database will simply point to that external location as a source of raw response data. Otherwise, the application downloads the data from `uri` and stores it in its `FileStorage`.
260
+
261
+
The intent is to make sure data winds up at a reliably available location, ensuring that anyone who can access the API can also access the raw response body for any version. Hosts should be listed in `ALLOWED_ARCHIVE_HOSTS` if they meet this criteria better than the application’s own file storage. The application’s storage area can be the local disk or it can be S3, depending on configuration. The component can take pluggable configurations, so we can support other storage types or locations in the future.
262
+
263
+
You can see more about this process in:
264
+
- The overview repo’s [“architecture” document](https://github.com/edgi-govdata-archiving/web-monitoring/blob/master/ARCHITECTURE.md#web-page-snapshottingcapturing-workflow)
265
+
- The [import job code](./app/jobs/import_versions_job.rb), where imports are processed.
266
+
- The [`Archiver` module code](./lib/archiver/archiver.rb), where raw HTTP response data is saved.
267
+
268
+
269
+
### File Storage
270
+
271
+
The application needs to store files for several different purposes (storing raw import data, archiving HTTP response bodies as described in the previous section, specialized logs, etc). To do this, it uses the [`FileStorage`](https://github.com/edgi-govdata-archiving/web-monitoring-db/tree/master/lib/file_storage) module, which has different implementations for different types of storage, such as [the local disk](https://github.com/edgi-govdata-archiving/web-monitoring-db/blob/master/lib/file_storage/local_file.rb) or [Amazon S3](https://github.com/edgi-govdata-archiving/web-monitoring-db/blob/master/lib/file_storage/s3.rb).
272
+
273
+
At current, the application creates two `FileStorage` instances:
274
+
275
+
1. “Archival storage” is used to store raw HTTP response bodies for each version of a page. See the [“how data gets loaded” section](#how-data-gets-loaded) for more details. Under a default configuration, this is your local disk in development and S3 in production. You can configure the S3 bucket used for it with the `AWS_ARCHIVE_BUCKET` environment variable. **Everything in this storage area is publicly available.**
276
+
277
+
2. “Working storage” is used to store internal data, such as raw import data and import logs. Under a default configuration, this is your local disk in development and S3 in production. You can configure the S3 bucket used for it with the `AWS_WORKING_BUCKET` environment variable. **Everything in this storage area should be considered private and you should not expose it to the public web.**
278
+
279
+
3. For historical reasons, EDGI’s deployment includes a third S3 bucket that is not directly accessed by the application. It’s where we store HTTP response bodies collected from [Versionista](https://versionista.com), a service we previously used for scraping government web pages. You can see it listed in [the example settings for `ALLOWED_ARCHIVE_HOSTS`](https://github.com/edgi-govdata-archiving/web-monitoring-db/blob/master/.env.example).
280
+
281
+
215
282
## Code of Conduct
216
283
217
284
This repository falls under EDGI's [Code of Conduct](https://github.com/edgi-govdata-archiving/overview/blob/master/CONDUCT.md).
0 commit comments