Notion of State / Variable Expansion in Config

[PhaedrusTheGreek]

As explained in Event Dependent Configuration

Some of the configuration options in Logstash require the existence of fields in order to function. Because inputs generate events, there are no fields to evaluate within the input block—they do not exist yet!

Which is fair, but some plugins such as the http_poller run on intervals, and could theoretically access some state.

The end goal would be to say something like this:

With use of some globally maintained variable (not even sure the best way to do this)

url => "http://my-server.com/stats?time=%{now}"

Or by maintaining an environment variable outside of Logstash

url => "http://my-server.com/stats?time=${now}"

[webmstr]

Yup. Back to external scripts, I guess.

[jordansissel]

I propose the following -- using the time formatting syntax we already support with events, we could make a special handler within this plugin that allows you to say %{+<time format>} to include that time format in your URL.

Example for "now" that maybe uses unix epoch (%s in time format)

url => "http://my-server.com/stats?time=%{+%s}"

Or with today's date only:

url => "http://my-server.com/stats?time=%{+yyyy-MM-dd}"

All times would be "now" aka the HTTP request was initiated.

[jordansissel]

@PhaedrusTheGreek @hummingV Thoughts on my proposal above? I worry it may confuse users since it uses the same syntax as event formatting, but at least it's possibly a familiar format?

I'm open to other proposals.

[PhaedrusTheGreek]

@jordansissel is there any way to have global variables that could be set by a ruby filter? I know, I'm a dreamer - but, addressing only the time issue in a single plugin might result in unwanted technical debt.

[hummingV]

@PhaedrusTheGreek , @jordansissel
Having a ruby filter to set variables is a utopia but I think that will result in a major redesign.

If the ruby filter was to stay upstream, then there is no events yet to process since input plugins are the ones responsible to create events and push it down the queue. If the ruby filter was to stay downstream, then it can set global variables for the next cycle of events but that's not a very elegant solution either. All in all, without a redesign of overall logstash event flow, forcing a ruby filter for this specific scenario will also incur technical debts of its own.

My proposal is that this decision should be based on demand. Is there sufficient demand to allow filter plugins to be able to run upstream from input plugins? Or can we get away with minimal changes for this specific requirement?
It's better to be iterative and not to invest too heavily upfront. @jordansissel proposal is exactly the same as sprintf format, so the same syntax is forward-compatible even if we decide to add ruby filters later on. i.e. for now these are plugin specific; if ruby filters becomes a reality later, the same syntax can be supported without breaking the configs.

The only limitation I see is sprintf only support current time. What if we want to use other time objects instead? Can we extend the format to be the following:
%{varname+%s}, %{varname+yyyy-MM-dd}
The varname is plugin-specific declaration. Each plugin can make some set of variables available for injection.

[PhaedrusTheGreek]

@hummingV I see your point - trying to control input state by filters makes no sense.

+1 for some standard support beyond date. Has inline scripting already been ruled out?

[hummingV]

I am open to the idea of pre and post-event hooks. That is most flexible solution imo. Having them only for this plugin might be a bit awkward though. Not sure if logstash team would want an implementation at base class level.

[jordansissel]

For inputs, post-event things can be done in filters.

As for "pre-event" hooks, I don't think that solves this problem. The issue described in this ticket is for doing some computation to generate the URL used in the http request before each poll -- this is not necessarily "before the event" but is more "immediately before the next http request". It is possible that the http_poller could produce multiple individual events from a single http request.

As for solving this for more than just dates, I don't have any solutions available yet.

I believe we can solve the "now" and time formatting concern without needing a general solution. There's no mechanism to provide mutable state to an input for any plugin today. Some plugins will ask for the current time or may ask for a random number, but neither of those things are user-facing configurations.

Do you think we could solve the "now" problem without needing a scripting/general solution?

[PhaedrusTheGreek]

Formattable %{now} does solve this particular problem for me.

[webmstr]

My use case: I'm trying to hit a URI from a remote provider. The base URL is itself static. It takes some headers (supported), but requires some dynamic time indicators in the query string (start_time, end_time). I would like to send in values like "now-5min" and "now".

Based on the other comments, I would like portions of the timestamp to be constant. In my case, I'll probably run every minute, so I would want seconds to be zero'd out (12:15:00). Depending on whether the remote provide handles dates as inclusive or exclusive, you might also need ":59", e.g. 12:15:59.

Having a mechanism to track the last time successfully processed would also be useful if it could be used as the start time for the next request (similar to the timestamp tracked by the S3 input).

[jordansissel]

let's try this another way, instead of describing the solution ("I need current time" or "I need now-5min") can y'all tell a story about what you want to do? Maybe we can find a solution from the information in these stories.

[hummingV]

My story:
I am using http-poller to send queries to an Elasticsearch index (say: source). In source, I have individual timestamped http requests. At the end of each day, I query aggregated statistics from source and stash it to another ElasticSearch index (say: dest). dest is an input to our monitoring tool.

I need scheduledTime variable expansion to specify date range in the query to the source. I want this range to be precise intervals (e.g: 2016-08-20T00:00:00 to 2016-08-21T00:00:00). Btw, intervals are not necessarily 1 day each; I also want other intervals such as 15 minutes, 1 hr etc. That is why I want to expand 'scheduledTime' instead of 'current system time'. Current time could be affected by latencies in java-threads. Don't get me wrong, I can afford if the query is sent a bit late but the timestamp sent in the query should be the intended scheduledTime instead of whatever time happens to be right now.

[PhaedrusTheGreek]

Looking at my requirement again, I actually do need to be able to do date math as well.

Use case is Fetching jobstats from Yarn.

"http://host:port/path?startedTimeBegin=%{now-60}&finishedTimeEnd=%{now}&states=FINISHED