Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Provider Reports - Static #607

Merged
merged 10 commits into from
Dec 16, 2020
Merged

Provider Reports - Static #607

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
7b852a2
First pass without example yet
schnuerle Dec 11, 2020
0efff58
Added example
schnuerle Dec 11, 2020
131ae19
Example update
schnuerle Dec 14, 2020
531d799
Updated examples to monthly files
schnuerle Dec 14, 2020
edef4a9
Clarified Data Notes
schnuerle Dec 14, 2020
5cb97b9
Update to operations timeline
schnuerle Dec 14, 2020
a5d8036
Timezone and k-value updates
schnuerle Dec 14, 2020
55ea53f
Updates based on comments
schnuerle Dec 15, 2020
a059235
Geography typo
schnuerle Dec 15, 2020
02d56d6
Remove special group type from Metrics
schnuerle Dec 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion geography/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ Placeholder -- link to schema to be added later.
| ---------------- | --------- | --- | --------------------------------------------------------------------------------------------- |
| `name` | String | Required | Name of geography |
| `description` | String | Optional | Detailed description of geography |
| `geography_type` | Sting | Optional | Type of geography, e.g. `municipal_boundary` or `council_district` or custom text. See [Geography Types](#geography-types). |
| `geography_type` | String | Optional | Type of geography, e.g. `municipal_boundary` or `council_district` or custom text. See [Geography Types](#geography-types). |
| `geography_id` | UUID | Required | Unique ID of geography |
| `geography_json` | UUID | Required | The GeoJSON that defines the geographical coordinates. |
| `effective_date` | [timestamp][ts] | Optional | The date at which a Geography is considered "live". Must be at or after `published_date`. |
Expand Down
1 change: 0 additions & 1 deletion metrics/core_metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@ The following represent the suggested MDS core metric dimensions:
| provider_id | Transportation provider id issued by OMF and [tracked here](/providers.csv) |
| geography_id | [MDS Geography](/geography) |
| vehicle_type | [Vehicle Type](/agency#vehicle-type) defined by MDS |
| special_group_type | [Special Group Type](#special-group-type) defined by MDS |

[Top][toc]

Expand Down
117 changes: 117 additions & 0 deletions provider/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ This specification contains a data standard for *mobility as a service* provider
* [Routes](#routes)
* [Status Changes][status]
* [Status Changes - Query Parameters](#status-changes---query-parameters)
* [Reports](#reports)
* [Reports - Response](#reports---response)
* [Reports - Example](#reports---example)
* [Special Group Type](#special-group-type)
* [Data Redaction](#data-redaction)
* [Realtime Data](#realtime-data)
* [GBFS](#GBFS)
* [Data Latency Requirements][data-latency]
Expand Down Expand Up @@ -291,6 +296,118 @@ Without an `event_time` query parameter, `/status_changes` shall return a `400 B

[Top][toc]


## Reports

Reports are information that providers can send back to agencies containing aggregated data that is not contained within other MDS endpoints, like counts of special groups of riders.

The authenticated reports are monthly, historic flat files that may be pre-generated by the provider.

### Reports - Response

**Endpoint:** `/reports`
**Method:** `GET`
**[Beta feature][beta]:** Yes (as of 1.1.0)
**Schema:** TBD when out of beta
**`data` Filename:** monthly file named by year and month, e.g. `/reports/YYYY-MM.csv`
**`data` Payload:** monthly CSV files with the following structure:

| Name | Type | Comments |
| ------------------ | ----------------------------------------- | ------------------------------------------------ |
| StartDate | date | Start date of the data row, ISO 8601 format, local timezone |
| Duration | string | Value is always `P1M` for monthly. Based on [ISO 8601 duration](https://en.wikipedia.org/wiki/ISO_8601#Durations) |
| Special Group Type | [Special Group Type](#special-group-type) | Type that applies to this row |
| Geography ID | [Geography](/geography) | ID that applies to this row. Includes all IDs in /geography. When there is no /geography then return `null` for this value and counts based on the entire operating area. |
| Vehicle Type | [Vehicle Type](/agency#vehicle-type) | Type that applies to this row |
| Trip Count | integer | Count of trips taken for this row |
| Rider Count | integer | Count of unique riders for this row |

#### Data Notes

Report contents include every combination of special group types, geography IDs, and vehicle types in operation for each month since the provider began operations in the jurisdiction. New files are added monthly in addition to the previous monthly historic files.

Counts are calculated based the city's local time zone, and this time zone is returned within the `StartDate` value. For months where there is a Daylight Saving Time change, use the timezone that is in the majority of the month.

All geography IDs included in the city published [Geography](/geography) API endpoint are included in the report results. In lieu of serving an API, this can alternately be a [flat file](/geography#file-format) created by the city and sent to the provider via link. If there is no `/geography` available, then counts are for the entire agency operating area, and `null` is returned for Geography ID.

### Reports - Example

For 3 months of provider operation in a city (September 2019 through November 2019) for 3 geographies, 2 vehicle types, and 1 special group. Timezone is Eastern Time in the US which is _-4_ from UTC before November 3, 2019, and _-5_ after. Values of `-1` represent [redacted data](#data-redaction) counts.

**September 2019** `/reports/2019-09.csv`

```csv
StartDate,Duration,Special Group Type,Geography ID,Vehicle Type,Trip Count,Rider Count
2019-09-01T00:00-04,P1M,all_riders,44428624-186b-4fc3-a7fb-124f487464a1,scooter,1302,983
2019-09-01T00:00-04,P1M,low_income,44428624-186b-4fc3-a7fb-124f487464a1,scooter,201,104
2019-09-01T00:00-04,P1M,all_riders,44428624-186b-4fc3-a7fb-124f487464a1,bicycle,530,200
2019-09-01T00:00-04,P1M,low_income,44428624-186b-4fc3-a7fb-124f487464a1,bicycle,75,26
2019-09-01T00:00-04,P1M,all_riders,03db06d0-3998-406a-92c7-25a83fc2784a,scooter,687,450
2019-09-01T00:00-04,P1M,low_income,03db06d0-3998-406a-92c7-25a83fc2784a,scooter,98,45
2019-09-01T00:00-04,P1M,all_riders,03db06d0-3998-406a-92c7-25a83fc2784a,bicycle,256,104
2019-09-01T00:00-04,P1M,low_income,03db06d0-3998-406a-92c7-25a83fc2784a,bicycle,41,16
2019-09-01T00:00-04,P1M,all_riders,8ad39dc3-005b-4348-9d61-c830c54c161b,scooter,201,140
2019-09-01T00:00-04,P1M,low_income,8ad39dc3-005b-4348-9d61-c830c54c161b,scooter,35,21
2019-09-01T00:00-04,P1M,all_riders,8ad39dc3-005b-4348-9d61-c830c54c161b,bicycle,103,39
2019-09-01T00:00-04,P1M,low_income,8ad39dc3-005b-4348-9d61-c830c54c161b,bicycle,15,-1
```

**October 2019** `/reports/2019-10.csv`

```csv
StartDate,Duration,Special Group Type,Geography ID,Vehicle Type,Trip Count,Rider Count
2019-10-01T00:00-04,P1M,all_riders,44428624-186b-4fc3-a7fb-124f487464a1,scooter,1042,786
2019-10-01T00:00-04,P1M,low_income,44428624-186b-4fc3-a7fb-124f487464a1,scooter,161,83
2019-10-01T00:00-04,P1M,all_riders,44428624-186b-4fc3-a7fb-124f487464a1,bicycle,424,160
2019-10-01T00:00-04,P1M,low_income,44428624-186b-4fc3-a7fb-124f487464a1,bicycle,60,0
2019-10-01T00:00-04,P1M,all_riders,03db06d0-3998-406a-92c7-25a83fc2784a,scooter,550,360
2019-10-01T00:00-04,P1M,low_income,03db06d0-3998-406a-92c7-25a83fc2784a,scooter,78,36
2019-10-01T00:00-04,P1M,all_riders,03db06d0-3998-406a-92c7-25a83fc2784a,bicycle,205,83
2019-10-01T00:00-04,P1M,low_income,03db06d0-3998-406a-92c7-25a83fc2784a,bicycle,33,13
2019-10-01T00:00-04,P1M,all_riders,8ad39dc3-005b-4348-9d61-c830c54c161b,scooter,161,112
2019-10-01T00:00-04,P1M,low_income,8ad39dc3-005b-4348-9d61-c830c54c161b,scooter,28,-1
2019-10-01T00:00-04,P1M,all_riders,8ad39dc3-005b-4348-9d61-c830c54c161b,bicycle,82,31
2019-10-01T00:00-04,P1M,low_income,8ad39dc3-005b-4348-9d61-c830c54c161b,bicycle,-1,0
```

**November 2019** `/reports/2019-11.csv`

```csv
StartDate,Duration,Special Group Type,Geography ID,Vehicle Type,Trip Count,Rider Count
2019-11-01T00:00-05,P1M,all_riders,44428624-186b-4fc3-a7fb-124f487464a1,scooter,834,629
2019-11-01T00:00-05,P1M,low_income,44428624-186b-4fc3-a7fb-124f487464a1,scooter,129,66
2019-11-01T00:00-05,P1M,all_riders,44428624-186b-4fc3-a7fb-124f487464a1,bicycle,339,128
2019-11-01T00:00-05,P1M,low_income,44428624-186b-4fc3-a7fb-124f487464a1,bicycle,48,-1
2019-11-01T00:00-05,P1M,all_riders,03db06d0-3998-406a-92c7-25a83fc2784a,scooter,440,288
2019-11-01T00:00-05,P1M,low_income,03db06d0-3998-406a-92c7-25a83fc2784a,scooter,62,29
2019-11-01T00:00-05,P1M,all_riders,03db06d0-3998-406a-92c7-25a83fc2784a,bicycle,164,66
2019-11-01T00:00-05,P1M,low_income,03db06d0-3998-406a-92c7-25a83fc2784a,bicycle,26,0
2019-11-01T00:00-05,P1M,all_riders,8ad39dc3-005b-4348-9d61-c830c54c161b,scooter,129,90
2019-11-01T00:00-05,P1M,low_income,8ad39dc3-005b-4348-9d61-c830c54c161b,scooter,22,-1
2019-11-01T00:00-05,P1M,all_riders,8ad39dc3-005b-4348-9d61-c830c54c161b,bicycle,-1,25
2019-11-01T00:00-05,P1M,low_income,8ad39dc3-005b-4348-9d61-c830c54c161b,bicycle,0,0
```

### Special Group Type

Here are the possible values for the `special_group_type` dimension field:

| Name | Description |
| ---------- | --------------------------------------------------------------------------------------------------------------------- |
| low_income | Trips where a low income discount is applied by the provider, e.g., a discount from a qualified provider equity plan. |
| all_riders | All riders from any group |

Other special group types may be added in future MDS releases as relevant agency and provider use cases are identified. When additional special group types or metrics are proposed, a thorough review of utility and relevance in program oversight, evaluation, and policy development should be done by OMF Working Groups, as well as any privacy implications by the OMF Privacy Committee.

### Data Redaction

Some combinations of parameters may return a small count of trips, which could increase a privacy risk of re-identification. To correct for that, Reports does not return data below a certain count of results. This is called k-anonymity, and the threshold is set at a k-value of 10.

If the report returns count values from 1 through 10, then a number of value `-1` is returned to represent redacted data. Counts of `0` are still returned as usual. This requirement is applied to both counts of trips and unique riders.

[Top][toc]


## Realtime Data

### GBFS
Expand Down