Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(variable-hydration): fixed variable overhydration issue #18346

Merged
merged 11 commits into from
Jun 8, 2020

Conversation

asalem1
Copy link
Contributor

@asalem1 asalem1 commented Jun 2, 2020

Closes #18192

Problem

This PR aims to address the issue of overly hydrating variables when a variable selection is made.

Solution

The solution is as follows:

  • Only pass in the variable that has been selected to hydrate that specific variable
  • Create a subgraph of the variables that are being used by performing an in order tree traversal of the parent's child nodes in order to first load children, then their parents

This solution will help reduce the amount of unnecessary query requests made from the UI post order to hydrate variables. However, since it involves a lot of moving parts that currently work, I've set it behind a feature flag in order to make sure that variables are still going to work correctly

  • CHANGELOG.md updated with a link to the PR (not the Issue)
  • Rebased/mergeable

@asalem1 asalem1 requested a review from drdelambre June 2, 2020 23:33
@asalem1 asalem1 force-pushed the fix/variable-hydration-issue branch from 8d087aa to 907f974 Compare June 2, 2020 23:37
@@ -433,7 +434,10 @@ export const selectValue = (variableID: string, selected: string) => async (

await dispatch(selectValueInState(contextID, variableID, selected))
// only hydrate the changedVariable
dispatch(hydrateVariables(true))
// dispatch(hydrateChangedVariable(variableID))
if (isFlagEnabled('hydratevars')) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the most badass thing i've ever seen

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pushing the limits of feature flagging:

https://media.giphy.com/media/WsvLlmmjx9tnmeTPNc/giphy.gif

expect(actual.length).toEqual(1)
const [subgraph] = actual
// expect the subgraph to return the passed in variable
expect(subgraph.variable).toEqual(timeRangeStartVariable)
// expect the parent to be returned with the returning variable
expect(subgraph.parents[0].variable).toEqual(associatedVariable)
})
test('should filter out inputs that have already been loaded based on a previous associated variable', async () => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

omg this documentation make my week

Copy link
Contributor

@drdelambre drdelambre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for all the tests! it makes accepting changes within this area of the code easier. even though there's some feature flags in there to control the peripheral, it seems like the internals aren't entirely disconnected when toggling the flag. that keeps this PR risky. given the history around this feature, it might be good to schedule some time with russ to run through this branch locally (iirc he has a comprehensive variable dataset for OSS testing) before merging to master.

@asalem1 asalem1 changed the title fix(variable-hydration): fixed issue fix(variable-hydration): fixed variable overhydration issue Jun 4, 2020
@russorat
Copy link
Contributor

russorat commented Jun 4, 2020

We tested this locally a few hours ago and this is what we discovered:

  • Setting feature flags in OSS doesn't work (not this pr's problem)
  • The number of calls made for a single cell went from 6 to 3 when a variable value was changed: 1 call to the cell query, then a call to the variable query, then another call to the cell query. IMO, we should be able to get this down to a single call.
  • This does not improve additional calls on the explorer page

This PR is a step in the right direction and doesn't appear to have broken anything, so i'm all for merging it!

asalem1 added 11 commits June 8, 2020 08:03
… have 1 reference to a node no matter how deeply nested

Still need to update the filterUnusedVars function to load variables that might be called within variables
Also need to determine whether a child node in a graph can have multiple parents
Need to figure out whether some of the tests are still necessary.
Need to find out whether a child can have multiple parents
Need to find out whether a parent with multiple children should hydrate all the children
Need to work on the filterUnusedVars
…m unsure where to go. Need to take a break from variables
…e constraints around parent/child relationships

Added/updated tests to ensure feature stability
Need to resolve filterUnusedVariables issues
…aph implementation so I removed the comment

also add the featureflag back to the feature
… based on featureFlag

This is a temporary workaround until the featureflag can be removed so that the tests pass
@asalem1 asalem1 force-pushed the fix/variable-hydration-issue branch from 66ec9ac to 332bb51 Compare June 8, 2020 15:06
@asalem1 asalem1 merged commit 9cdeb2e into master Jun 8, 2020
@asalem1 asalem1 deleted the fix/variable-hydration-issue branch June 8, 2020 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Variable Hydration Filtering / Variable loading isn't fully functioning
3 participants