-
Notifications
You must be signed in to change notification settings - Fork 74
Implement request filter pipeline #819
Implement request filter pipeline #819
Conversation
Hello @orlandohohmeier, I will adjust some things that I learned how to do (like change The overall idea is still the same, however. Thanks, |
Hello @orlandohohmeier, the implementation is a little better now and is ready for you to start reviewing. Thanks, |
Hello @orlandohohmeier, made some changes and the implementation is getting better. If you could take a look. About the new instance of the Pipeline passed to every loaded plugin: If you think this is a problem, we can extract this instance to a variable and pass the same Pipeline object to every plugin. Also, if you think this is better we can extract all hook names to a separated file, for example: Let me know what you think. Thanks, |
export const FILTERS_PIPELINE = {}; | ||
const PRE_AJAX_REQUEST = "PRE_AJAX_REQUEST"; | ||
|
||
export const Hooks = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd move these const to a dedicated filter const file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure! Will do it.
@@ -0,0 +1,44 @@ | |||
|
|||
export const FILTERS_PIPELINE = {}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please don't export FILTERS_PIPELINE
as you otherwise risk that other components or plugins accidentally tamper with the internal state which could lead to bugs. I'd move the map into a closure make sure that only the pipeline has access.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure! The idea is to let only the PipelineStore
modify the internal state of the pipelines. Currently exporting this variable makes it possible to other components of the core UI do modify it (that's what I'm currently doing in the test cases) but this is not true for plugins, this is because plugins cannot import the module directly, all that plugins see are the objects exported to its pluginWindow.marathonPluginInterface
, so no plugin could ever reach the original exported FILTERS_PIPELINE
, right? Does this make sense?
Hi @daltonmatos, Please excuse the late response. I've been on the road for the last two weeks and wasn't able to review your code. I like the idea of a filter/pipeline mechanism to enrich the plugin system and I'm happy to help move this one forward. We might want to think about ways to ensure that the input and output types are consistent as I've seen such mechanism fail many times due to the lack of type safety and the related set of errors. Making I'd refactor the pipeline store as follows to address the open issues you've initial mentioned and ensure data integrity. export default class PipelineStore {
constructor() {
/**
* Private Context
*
* @typedef {Object} PipelineStore~Context
* @property {PipelineStore} instance - Current pipeline store instance
* @property {Map} pipelines
*/
const context = {
instance: this,
pipelines: new Map()
};
this.applyPipeline = this.applyPipeline.bind(context);
this.registerOperator = this.registerOperator.bind(context);
this.unregisterOperator = this.unregisterOperator.bind(context);
}
/**
* Apply pipeline
*
* @this PipelineStore~Context
* @param {Symbol|String} id - Pipeline identifier
* @param {*} input
* @return {*} output
*/
applyPipeline(id, input) {
if (!this.pipelines.has(id)) {
return input;
}
return this.pipelines.get(id)
.reduce((accumulator, operator) => {
try {
return operator(accumulator);
} catch (error) {
// We might want to log a proper error to help users debug
return accumulator;
}
}, input);
};
/**
* @this PipelineStore~Context
* @param {Symbol|String} id
* @param {function} operator
*/
registerOperator(id, operator) {
if (!this.pipelines.has(id)) {
this.pipelines.set(id, []);
}
const pipeline = this.pipelines.get(id);
if (!pipeline.includes(operator)) {
pipeline.push(operator);
}
};
/**
* @this PipelineStore~Context
* @param {Symbol|String} id
* @param {function} operator
*/
unregisterOperator(id, operator) {
if (!this.pipelines.has(id)) {
return;
}
const pipeline = this.pipelines.get(id);
if (pipeline.includes(operator)) {
pipeline.splice(pipeline.indexOf(operator), 1);
}
}
}; N.B.This code is illustrative and I haven't tested it at all. Creating and binding a private context helps us to ensure data integrity. Please note that it's rather hard to properly extend this class due to the usage of the private context which in this case shouldn't be a problem. |
Hello @orlandohohmeier, Thank you very much for your time to review this PR.
Indeed this is great, but if we have a new Plugin A registers a When That's why I choosed to have I don't know how is the best way to solve this. What do you think would be the best way to achieve this behavior: No plugin can mess with the internal pipelines state, but all plugins share the same pipeline internal state. I thought that having the Let me know what do you think. Thanks a lot. |
Hi @daltonmatos, you're right, with the current proposal we would run into issues as the plugins and marathon UI wouldn't share the same instance of the PipelineStore — I forgot that we don't use the plugin interface to share instance internally as well. |
Could work but I have some doubts: Are you suggesting the Because if yes, even if they share the same instance of When I created the
becasue there is no such attribute inside If your are not suggesting the About the tests on this approach, the test code should still be able to import the instance of
Do you think is it too bad to let an object importable to the core UI code? Thanks a lot!! |
Hello @orlandohohmeier, any thoughts? Thanks, |
Hi @daltonmatos, Imaging the pipeline model throws an error – where would you look for bugs? In case your local state is importable to the core UI code you would need to check not only the pipeline model but also the entire core UI to ensure that none of the components mutates that the state by accident. I was thinking about the // PipelineStore.js
export default new PipelineModel(); You could also use // PipelineStore.js
export default new class {
constructor() {
Object.defineProperty(this, "pipelines", {value: new PiplelineModel()});
}
} Both of these approaches make it impossible for any plugin to tamper with the Your test would then test that the |
Hello @orlandohohmeier, I implemented the a idea of Also extracted the constants to a separated file. Don't know if the names are ok, but let's focus on the idea and then we can change the namings, if needed. Let me know what do you think. Thanks. |
|
||
}; | ||
|
||
export default new PipelineModel(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Exporting an instance makes it unnecessary hard to test the PipelineModel
and renders the store useless. Let's export the class here and use the store to create and provide the PipelineModel
instance.
this.pipelines.clear(); | ||
}; | ||
|
||
applyFilter(operationName, filterParameters) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd rename this method to applyPipeline
as a pipeline could be used for many things other than filtering a result set otherwise I'd argue that we should restructure the model to prove a predicate that one can apply instead of applying the filter.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
this.pipelines.clear(); | ||
}; | ||
|
||
applyFilter(operationName, filterParameters) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should also document that a pipeline can't handle more than one input parameter (could be an object or list though) as the output of one operation will be the input of the next operation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where is the best place to document this? As a comment in this method?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd add a JSDoc comment for now.
applyFilter(operationName, filterParameters) { | ||
const pipeline = this.getPipeline(operationName); | ||
|
||
pipeline.forEach((filter) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's rewrite this method to properly pass the input and output parameter so that applyPipeline(FOO, "bar") === c(b(a("bar"))))
given we've registered the functions a
, b
, c
as operators for "FOO".
function a(input){
return input.toUpperCase();
}
function b(input){
return input.split("").reverse().join("");
}
function c(input){
return input.replace(/[AEUIO]/g,"");
}
pipelines = new PipelineModel();
pipelines.registerOperator("FOO", a);
pipelines.registerOperator("FOO", b);
pipelines.registerOperator("FOO", c);
pipelines.applyPipeline("FOO", "bar") === c(b(a("bar")))
#819 (comment) illustrates how to implement such a method.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
} | ||
}; | ||
|
||
getPipeline(pipelineName) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wouldn't expose this method as theoretically allows unwanted mutations. I'd also argue that as not necessary to extract the respective functionality as we only need to get the pipelines in a very few places and only in one place we want to actually create a new pipeline if there is none.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still think that we still shouldn't expose this method due to the named reasons.
@@ -0,0 +1,21 @@ | |||
import PipelineModel from "./PipelineModel"; | |||
|
|||
export default class PipelineStore { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should export and instance of PipelineStore
as the pipeline store will create and manage the PipelineModel
should be a singleton similar to all of our stores.
Hi @daltonmatos, I very much like where this is going! The current instance handling seems a little of though and doesn't solve the issues we've discussed. I've started to also comment on other parts of the implementations – feel free to ignore those comments for now. I was also wondering wether you want to implement a general purpose pipeline mechanism which one can use to apply a set of operators or provide a mechanism to filter data using the registered predicates. If you think we only need the latter I'd propose a slightly different implementation and would adjust the naming. Once again, thanks for pushing the plugin interface. |
Hello @orlandohohmeier, The idea, for now, is to just have the ability to filter some values in a very simple form. Only one parameter passed to the filters, no priority when running the registered callbacks. The generic pipeline implementation could be postponed until really needed, which is not the case for now. Let's do the minimal filtering implementation for now and think about the pipeline (if ever needed) in the future, what do you think? Let me know what are you suggestions about the naming and the different implementation you mentioned above so we can move on. Thanks a lot for your time reviewing my submissions. 🎉 |
Hello @orlandohohmeier, What do you think about making this implementation as simple as possible so we can merge it? Let me know what are the suggestions you mentioned earlier so I can implement them on this PR. Thanks, |
@daltonmatos please excuse that I didn't come back to you earlier. I've tried a few things but considering that you actually want to mutate an input object (e.g. |
Sure @orlandohohmeier, no problem. I will write this implementation. I will also use the names you suggested: Thanks, |
Hello @orlandohohmeier, please take a look at the implementation. Most of it was already done, just didn't push yet. Thanks, |
@daltonmatos the last days were super busy and I haven't had the time to review your code. Will look into it next week. Hope you don't mind. 🙂 |
No problem @orlandohohmeier. Let me know when you have some time to review this PR. |
Hello @orlandohohmeier, have you had any time to review this PR? Thanks, |
Hi @daltonmatos, unfortunately I haven't had the time to look into this. I'm super sorry. Will look into it first thing tomorrow morning. |
No problem, thanks! |
} | ||
}; | ||
|
||
getPipeline(pipelineName) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still think that we still shouldn't expose this method due to the named reasons.
PRE_AJAX_REQUEST | ||
}; | ||
|
||
export default PipelineOperations; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd use named exports, instead of exporting an Object as illustrated below.
export const PRE_AJAX_REQUEST = "PRE_AJAX_REQUEST"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
}; | ||
|
||
deregisterOperator(operationName, callback) { | ||
return this.deregisterOperator(operationName, callback); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please change this to this.deregisterOperator(operationName, callback)
as the method otherwise recurses infinitely.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Thanks. Didn't spot this. Should be return this.pipeline.deregisterOperator(operationName, operator)
}, filterParameters); | ||
}; | ||
|
||
registerOperator(operationName, callback) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎨 I'd rename callback
to operator
for all for all these methods.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
deregisterOperator(operationName, callback) { | ||
const pipeline = this.getPipeline(operationName); | ||
|
||
var callbckPosition = pipeline.indexOf(callback); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎨 I'd rename this one to operatorIndex
P.S. You could use a const
as the index shouldn't change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
src/test/units/PipelineModel.test.js
Outdated
describe("PipelineModel tests", function () { | ||
|
||
beforeEach(function (){ | ||
pipelineModel.clearPipeline(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of clearing the pipeline model I'd simple create a new instance before each run.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
src/test/units/PipelineModel.test.js
Outdated
|
||
const pipelineModel = new PipelineModel(); | ||
|
||
describe("PipelineModel tests", function () { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a test that verifies that the operators are called in the right order.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
src/test/units/PipelineModel.test.js
Outdated
expect(pipelineModel.getPipeline(PipelineOperations.PRE_AJAX_REQUEST)).to.have.lengthOf(0); | ||
}); | ||
|
||
it("should clear the pipeline when requested to", function () { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think that we should add this functionality nor a test for it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
src/test/units/PipelineModel.test.js
Outdated
pipelineModel.clearPipeline(); | ||
}); | ||
|
||
it("should return an empty list for a non-existant pipeline", function () { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove this test along with the getPipeline
getter for the reasons detailed in https://github.com/mesosphere/marathon-ui/pull/819/files#r158946727
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
src/test/units/PipelineModel.test.js
Outdated
expect(pipelineModel.getPipeline(PipelineOperations.PRE_AJAX_REQUEST)).to.have.lengthOf(0); | ||
}); | ||
|
||
it("should add a new function to the pipeline", function () { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎨 Let's always refer to the functions/callback as operators to make it easier for other to understand the feature... "should register a new operator to the pipeline"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
@daltonmatos Thanks a lot for your effort to ship this new capability. I've added a few comments here and there including stylistic ones that I flagged with the 🎨 emoji. |
All done @orlandohohmeier. Please Take a look. The suite for the Also let me know if the publushing of awaiting for your response/review. Thanks a lot! |
src/js/plugin/PluginLoader.js
Outdated
@@ -13,7 +13,7 @@ import PluginDispatcherProxy from "./PluginDispatcherProxy"; | |||
import MarathonService from "./sdk/services/MarathonService"; | |||
import MarathonActions from "./sdk/actions/MarathonActions"; | |||
import PipelineStore from "./sdk/pipeline/PipelineStore"; | |||
import PipelineOperations from "./sdk/pipeline/PipelineOperations"; | |||
import {PRE_AJAX_REQUEST} from "./sdk/pipeline/PipelineOperations"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of explicitly importing PRE_AJAX_REQUEST
I'd change the import to import * as PipelineOperations from './sdk/pipeline/PipelineOperations';
so that we don't need to manually add the operators here as well.
@@ -0,0 +1,23 @@ | |||
import PipelineModel from "./PipelineModel"; | |||
|
|||
class PipelineStore { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wonder wether we should move this one and the related files to the stores
directory as the pipeline store can be use used both by Marathon UI core as well as by plugins.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense, but I would leave it inside the plugin/sdk/stores
folder and not in js/stores
folder as I think the pipeline is feature provided by the plugin SDK (yet to be extracted from the core source-code). The UI core can use it, of couse, but importing from plugin/sdk/store/PipelineStore
.
this.pipelines = new Map(); | ||
}; | ||
|
||
applyPipeline(operationName, filterParameters) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎨 I'd change operationName
to pipelineName
as we're allowing users to register operators for different pipelines.
@@ -0,0 +1,3 @@ | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎨 I'd rename this file to PipelineNames
as these const are used to identify different pipelines not operations on a pipeline. Please also see the comment in regards to the operationName
param.
class PipelineStore { | ||
|
||
constructor() { | ||
Object.defineProperty(this, "pipeline", {value: new PipelineModel()}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎨 I'd rename the value to "pipelines" as we're allowing users to register operators for multiple different pipelines.
src/test/units/PipelineStore.test.js
Outdated
const filter = function (data) {return data;}; | ||
const filter2 = function (data) {return data;}; | ||
|
||
const pipeline_one = PipelineStore; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should be able to import the store as follows:
import {
default as TestPipelineStore
} from "../../js/plugin/sdk/pipeline/PipelineStore";
// ...
PipelineStore.pipelines === TestPipelineStore.pipelines;
I'd use sinon
to check wether the respective functions were called as illustrated in the following:
sinon.spy(PipelineStore.pipeline, "applyPipeline");
expect(PipelineStore.pipeline.applyPipeline).to.be.true;
@daltonmatos thanks for addressing the comments. 💜 Found a few more minor things but those aside the code looks good. |
All Done @orlandohohmeier, let me know what you think. I also implemented a ran locally an UI plugin that uses this new interface. Works as expected. Thanks, |
Hello @orlandohohmeier, any thoughts about this? I implemented all your suggested reviews. Thanks, |
return filterParameters; | ||
} | ||
return this.pipelines.get(pipelineName) | ||
.reduce((accumulator, pipelineCallback) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎨 I'd rename pipelineCallback
to operator
to stick with the general naming scheme.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
@daltonmatos thanks a lot for your relentless work on this one. This is a great addition which will enable new plugins. Well done! 💯 Please squash your commits before we merge this one, as not all of them work independently of any later commits, and pass the linter as well as the test suite. Thanks! |
It's now possible to plugins to change behaviors/values of the Core UI code. The main idea is to have pipelines offered by the Core and any plugins can register to these pipelines and they will be called when the pipelines run. The first pipeline implemented here makes it possible to modify any ajax request made by the UI. Here is how a plugin could register to be called when this pipeline runs: ``` const { PluginHelper, PluginMountPoints, PipelineNames, PipelineStore, } = window.marathonPluginInterface; PipelineStore.registerOperator(PipelineNames.PRE_AJAX_REQUEST, (data) => { /* Modify "data" and return a new version of it */ }); ```
Great @orlandohohmeier, Do you mind if I squash all commits into one single commit? This implementation is overall small and simple, so I think i will still be easy to review/understand even if contained in a single commit. Let me know and I will update this PR and it will be ready do be merged. Thanks, |
Hello @orlandohohmeier, I squashed all commits into one single commit. Thought this was easier than trying to pick some commits to squash. If you prefer, I still have all original commits, if you think it's necessary I can push them back here in this PR and we can choose which commits to squash. Thanks a lot for your reviews and for accepting this code change. |
@daltonmatos thanks for squashing your commits 🚀 |
Hello @orlandohohmeier,
Here is a implementation candidate for a "filters pipeline" feature. This will enrich plugin even more, as long as the core Marathon UI adds some extension points where it calls the filter pipeline.
Let me know what do you think about this implementation, so I can adjust the code to be able to eventually merge.
Things I didn't like the way it's done, but don't know how to do it differently.
PipelineStore.PRE_AJAX_REQUEST
without the need of the constructor.clearPipeline()
method is only used in the tests. I could not clean the pipeline before each tests without this method.FILTERS_PIPELINE
) should be visible only to thePipelineStore
module, and not be an attribute of thePipelineStore
class, also don't know how to achieve this.Thanks a lot!