Flow Results: row_id needs uniqueness within a Package

[markboots]

The Ona API docs for the Flow Results endpoints say:

It is expected that the row_id should be the same for records that represent one submission or interaction throw a full Flow. For example where a Flow captures the name, age and place of birth of persions in attendance, the row_id of a single persion should be the same.

whereas the Flow Results spec requires row_id to be unique for every individual data point in the package. This is because a Flow Results "row" is for a single question response, unlike an XForm/ODK "row" in a CSV, where a row is a submission of multiple question responses.

The concept of an ID for submissions could be helpful, although separate from row_id: We're discussing what to do with "session IDs" here: FLOIP/flow-results#5

Aha! Link: https://ona.aha.io/features/PROD-454

[ukanga]

The implementation tries to transform the Flow Results to a single XForm submission which is composed of multiple questions. Is there a different way to tie different rows to the same XForm submission? This could mean a revist of the spec or a total new platform integration that is not with onadata.

If we do take the approach of a unique row_id then I don't think it will be possible to visualize or utilize the information collected in a meaningful way within the existing visualization and export tools of onadata. It may call for us to implement a completely new platform to store the package responses and make them available to ona or any other platform.

[pld]

We can put the current row_id as the value for a key session_id in the options column on export. And require that any data sent to Ona include a session_id key in the options, in order to tie multiple questions together.

I do not know how we'd generate a unique row_id, we could concatenate the UUID per submission with the column number, or something like that.

[pld]

@markboots I want to point out that even if session_id is optional, it will be required to get data into Ona, since (unless I'm missing something) we'll need it to tie together flow responses. We're also evaluating the best way to generate a unique row_id, since that's not a concept we currently support. We could generate this by taking a hash of the session_id (equivalent to Submission ID for us) with the column name, but this would change if the form is edited, it would also be in a UUID format, I don't know what the format expectation was for this.

Also, if someone sends us data with row_ids, via a FLOIP endpoint, then they pull via a FLOIP endpoint, do we need to make sure they continue seeing those same row_ids or can we generate different row_ids (which will continue to be the same in the future with the above caveat)? The latter is less effort.

[markboots]

Hi team,

it will be required to get data into Ona, since (unless I'm missing something) we'll need it to tie together flow responses.

That definitely makes sense, as you're storing within a faked XForm submission. I think that's a requirement that's fine for Ona to specify: session_id is required. If session_id is gaining in importance, should we make this a first-order aspect of the spec? (ie: outright column... etc.?)

We can certainly support session_ids from our sending side. I'm sure Verboice also has a concept that maps to session IDs.

Also, if someone sends us data with row_ids, via a FLOIP endpoint, then they pull via a FLOIP endpoint, do we need to make sure they continue seeing those same row_ids or can we generate different row_ids (which will continue to be the same in the future with the above caveat)? The latter is less effort.

I think your concatenation idea with hashing the column name works fine if the sender doesn't provide row_ids on new submissions. In that case it's up to you to invent the row_id. Any kind of unique string is fine with the spec. If the form is edited, it will create a separate FLOIP package ID, or a separate version, so no issue there.

If a client does send you row_ids, it think it is actually ideal if you can serve back the provided row_ids. Any place you can hide this without too much effort in your existing storage model?

[pld]

@ukanga do we have submission level metadata maps?

…

On Wed, Feb 28, 2018 at 22:42 Mark Boots ***@***.***> wrote: Hi team, it will be required to get data into Ona, since (unless I'm missing something) we'll need it to tie together flow responses. That definitely makes sense, as you're storing within a faked XForm submission. I think that's a requirement that's fine for Ona to specify: session_id is required. If session_id is gaining in importance, should we make this a first-order aspect of the spec? (ie: outright column... etc.?) We can certainly support session_ids from our sending side. I'm sure Verboice also has a concept that maps to session IDs. Also, if someone sends us data with row_ids, via a FLOIP endpoint, then they pull via a FLOIP endpoint, do we need to make sure they continue seeing those same row_ids or can we generate different row_ids (which will continue to be the same in the future with the above caveat)? The latter is less effort. I think your concatenation idea with hashing the column name works fine if the sender doesn't provide row_ids on new submissions. In that case it's up to you to invent the row_id. Any kind of unique string is fine with the spec. If the form is edited, it will create a separate FLOIP package ID, or a separate version, so no issue there. If a client does send you row_ids, it think it is actually ideal if you can serve back the provided row_ids. Any place you can hide this without too much effort in your existing storage model? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1228 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADGEuxNNLHx-iA77A637Ise4wrVE6nSks5tZ24sgaJpZM4R2QR5> .

[mverzilli]

Hi! I'd vote against application specific requirements (such as ONA requiring session_id), because it defeats the purpose of the spec. I've been implementing FLOIP for Verboice and Surveda and one of the features that really "clicks" about the spec is that it feels truly application agnostic, and the reqs for compliance are quite minimal and domain specific.

I'd rather make session_id required in general. The collector will know what makes most sense. If it doesn't manage any concept of session internally, it can always set session_id == contact_id which would mean "In my data, there's only one session per contact per package". Contact is already mandatory in the specs, so we're really not overly complicating collector implementations by requiring a session_id.

[ukanga]

@pld nope, nothing outside of attachments and internal fields, you will see those in the JSON representation as fields starting with an underscore.

[markboots]

Hi @mverzilli, thanks for weighing in! That's a compelling argument, to keep the spec fully universal.

It sounds like we're all OK with a session_id required column. I'll update the spec!