Skip to content

Timeseries update #702

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 53 commits into from
Nov 22, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
d1ed16c
merge develop
maxgrossman Feb 28, 2022
4ddbd2b
fulfil api spec
maxgrossman Feb 28, 2022
48cec73
WIP timeseries page
maxgrossman Mar 4, 2022
8be7518
get csv export working
maxgrossman Mar 8, 2022
e2bf7f6
teams filter, use postgresql generate series
maxgrossman Mar 16, 2022
d20ec7a
make chart more readable
maxgrossman Mar 17, 2022
7f8acca
add default entries for bins user do not have edits in
maxgrossman Mar 18, 2022
02b026f
Merge branch 'feature/timeseries-api' into timeseries_update
maxgrossman May 19, 2022
8974889
Merge branch 'feature/timeseries-api' into timeseries_update
maxgrossman May 19, 2022
abc91f0
cleanup
maxgrossman Jun 6, 2022
1fead81
and still, i clean up.
maxgrossman Jun 6, 2022
2dd6a78
use local fetch method
maxgrossman Jun 6, 2022
58d1e79
make last binEnd the bound; coalesce when no data
maxgrossman Jun 6, 2022
99bbd1d
fix timestamp
maxgrossman Jun 6, 2022
03a94af
async, debounce, js search for faster users dropdown
maxgrossman Jun 7, 2022
54a29fa
add max width
maxgrossman Jun 7, 2022
9864974
add message if query returns nothing
maxgrossman Jun 9, 2022
7df6805
lint!
maxgrossman Jun 20, 2022
eb14fd4
only update state in updateTimeseries
maxgrossman Jun 20, 2022
01cd9c7
format; no hover labels
maxgrossman Jun 21, 2022
0d235d2
use logarithmic scale to fit many and few edits
maxgrossman Jun 21, 2022
c4a666a
show rails coasts, punt on < 1 km edits, format nums
maxgrossman Jun 22, 2022
9851d07
lint osmesa service
maxgrossman Jun 22, 2022
63f201a
found nivo version that gets us symlog
maxgrossman Jun 22, 2022
781a3c7
revert nivo upgrade
maxgrossman Jun 23, 2022
c0c8893
finally dashboard link
maxgrossman Jun 23, 2022
fddc3fb
...and on hover show user name
maxgrossman Jun 23, 2022
07c34a3
update userIdMap with userIdMap...
maxgrossman Jun 23, 2022
de4f7d6
default zero for timeseries data; fixes csv export sum
maxgrossman Jun 28, 2022
bfbffd2
undo bad merge of develop
maxgrossman Oct 14, 2022
b961249
pin react-table dependency
maxgrossman Oct 17, 2022
d59a362
start off with zero so not to double count edits/changesets
maxgrossman Oct 17, 2022
4d9b8cf
do not assume related tables will have entry for every changeset
maxgrossman Oct 18, 2022
61b2c67
update route query with full understanding of changesets_hashtags/cou…
maxgrossman Oct 19, 2022
2a63ecd
binned hashtags/countries need a left join
maxgrossman Oct 19, 2022
25cf1c7
add stats validation test and ditch rounding measurements
maxgrossman Oct 20, 2022
39e3e35
Add page header, move export button below table
LanesGood Oct 21, 2022
14e568d
remove unused hashtagPrefixFilter
maxgrossman Oct 28, 2022
62bd9ca
remove badge file
maxgrossman Oct 28, 2022
5a07406
Style page layout
LanesGood Nov 1, 2022
2dc8c93
update timeseries data using submit button
kamicut Nov 1, 2022
0cbb410
Restrict timeseries chart to top 20 users
LanesGood Nov 4, 2022
8f54915
Dynamic chart header
LanesGood Nov 4, 2022
7d5f02a
Update pages/timeseries.js
kamicut Nov 8, 2022
6242b3b
Merge pull request #708 from developmentseed/enhance/timeseries-submit
kamicut Nov 8, 2022
a8fb744
Merge pull request #709 from developmentseed/restrict-timeseries-chart
kamicut Nov 8, 2022
10fd35f
Merge branch 'timeseries_update' into enhance/timeseries-ux
LanesGood Nov 8, 2022
7ebd79c
Prevent default action on form submit button
LanesGood Nov 8, 2022
01d0d50
Align submit button width
LanesGood Nov 8, 2022
027954a
Merge pull request #707 from developmentseed/enhance/timeseries-ux
LanesGood Nov 8, 2022
b0f44c7
Make timeseries an admin area page
LanesGood Nov 8, 2022
2982453
Nest Timeseries Route under admin
LanesGood Nov 8, 2022
66480f6
Merge pull request #711 from developmentseed/admin-timeseries-nav
LanesGood Nov 9, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
**
15 changes: 4 additions & 11 deletions api/docs/api.yml
Original file line number Diff line number Diff line change
Expand Up @@ -201,16 +201,6 @@ paths:
items:
type: string
example: [ 345_task_789, 345_task_111 ]
- name: hashtagPrefixFilter
description: Filter down to these hashtag(s), using a prefix match.
in: query
explode: false
style: pipeDelimited
schema:
type: array
items:
type: string
example: [ ocelots_ ]
- name: categoriesFilter
description: 'Filter down to these osmesa categories of measurements and counts.
Each count and measurement will have [add, modified, deleted] in the response. Measurements
Expand Down Expand Up @@ -279,6 +269,10 @@ paths:
description: Result bin end date. ISO-8601 formatted datetime
string (UTC time zone).
example: 2020-12-30
binInterval:
description: 'ISO-8601 formatted time interval/duration string of binning interval.'
type: string
example: P1Y2M10DT2H30M
userStatistics:
type: array
description: Modeled after the user_statistics materialized
Expand Down Expand Up @@ -405,5 +399,4 @@ paths:
type: integer
description: Number of edits
description: Result bin of binInterval duration
x-codegen-request-body-name: hashtagPrefixFilter
components: {}
21 changes: 10 additions & 11 deletions api/src/routes/timeseries.js
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,8 @@ const osmesa = require('../services/osmesa')
* @param userIdsFilter
* @param countriesFilter
* @param hashtagsFilter
* @param hashtagPrefixFilter
* @param categoriesFilter
* @returns {{binInterval: ({isValid}|Duration), hashtagsFilter: {length}, categoriesFilter: {length}, userIdsFilter: {length}, endDate: ({isValid}|DateTime), hashtagPrefixFilter: {length}, startDate: ({isValid}|DateTime), countriesFilter: {length}}}
* @returns {{binInterval: ({isValid}|Duration), hashtagsFilter: {length}, categoriesFilter: {length}, userIdsFilter: {length}, endDate: ({isValid}|DateTime), startDate: ({isValid}|DateTime), countriesFilter: {length}}}
*/
function validateParams ({
startDate,
Expand All @@ -23,15 +22,15 @@ function validateParams ({
userIdsFilter = '',
countriesFilter = '',
hashtagsFilter = '',
hashtagPrefixFilter = '',
categoriesFilter = ''
}) {
let validStartDate, validEndDate, validBinInterval, validUserIdsFilter,
validCountriesFilter, validHashtagsFilter,
validHashtagPrefixFilter, validCategoriesFilter
validCategoriesFilter
if (!startDate) {
throw Boom.badRequest('startDate is required')
}

validStartDate = DateTime.fromISO(startDate)
if (!validStartDate.isValid) {
throw Boom.badRequest(validStartDate.invalidExplanation)
Expand All @@ -51,12 +50,10 @@ function validateParams ({
}
validCategoriesFilter = deserializeStringArray(categoriesFilter)
validCountriesFilter = deserializeStringArray(countriesFilter) // TODO: validate these are 3 letter country codes?
validHashtagPrefixFilter = deserializeStringArray(hashtagPrefixFilter)
validHashtagsFilter = deserializeStringArray(hashtagsFilter)
// require at least 1 one filter, according to api spec.
if (!validCategoriesFilter.length &&
!validCountriesFilter.length &&
!validHashtagPrefixFilter.length &&
!validHashtagsFilter.length &&
!validUserIdsFilter.length) {
throw Boom.badRequest('at least one filter is required')
Expand All @@ -68,7 +65,6 @@ function validateParams ({
userIdsFilter: validUserIdsFilter,
countriesFilter: validCountriesFilter,
hashtagsFilter: validHashtagsFilter,
hashtagPrefixFilter: validHashtagPrefixFilter,
categoriesFilter: validCategoriesFilter
}
}
Expand All @@ -80,7 +76,8 @@ function validateParams ({
* @returns {int[]}
*/
function deserializeIntArray (s) {
const result = s.split('|').map(parseInt).filter(n => !isNaN(n))
const result = s.split('|').map(p => parseInt(p)).filter(n => !isNaN(n))

if (s.length > 0 && result.length === 0) {
throw Boom.badRequest('failed to deserialize integers array')
}
Expand All @@ -103,7 +100,7 @@ function deserializeStringArray (s) {
/**
* TimeSeries route
*
* {{baseUrl}}/api/timeseries?startDate=<string>&endDate=(now)&binInterval=<string>&userIdsFilter=<integer>|<integer>&countriesFilter=<string>|<string>&hashtagsFilter=<string>|<string>&hashtagPrefixFilter=<string>|<string>&categoriesFilter=<string>|<string> *
* {{baseUrl}}/api/timeseries?startDate=<string>&endDate=(now)&binInterval=<string>&userIdsFilter=<integer>|<integer>&countriesFilter=<string>|<string>&hashtagsFilter=<string>|<string>&categoriesFilter=<string>|<string> *
*
* @param {Object} req - the request object
* @param {Object} res - the response object
Expand All @@ -112,8 +109,10 @@ function deserializeStringArray (s) {
async function get (req, res) {
try {
const params = validateParams(req.query)
const result = osmesa.getTimeSeries(params)
res.send(result)
const result = await osmesa.getTimeSeries(params)
res.send({
bins: result
})
} catch (err) {
res.boom.boomify(err)
}
Expand Down
Loading