Proposals

[itdependsnetworks]

Trying to get a feel for what the appetite for a few updates into textfsm are. Using it a fair amount I have identified a few short comings (that you can work around, but are difficult.)

Missed captures
Example: I recently had a case in which on a "show interface status" I hit a condition in about %2 of interfaces. My data size what over 60K of interfaces so it was not easy to find.

Proposal: Set a flag that indicates all lines must match. Not that there must be a capture group in a given match, but that simply you have seen every line, and have profiled it already. Meaning that if it completes the rule set without a match, fail. This would be helpful when trying to match large routing tables, or large amounts of data to identify an issue early.

Repeated Data
Example: As noted in #11 when data is repeated, there is not an elegant solution to capture the multiple items

Proposal: A new key (RepeatedItem) that will reprocess a line with the assumption that there is repeated data, and to add to the list.

Trim()
Example: Often it is tough (reverse or forward lookbacks is the only way I have found) to get rid of pre/post white space.

Proposal: A new key (Trim) that will do minor post processing on the data after the match

Column Count
Example: Some output is merely based on column count (routing tables especially) and difficult to match when data may or may not be there.

Proposal: This is clearly the flimsiest of the proposals, however I would like a mechanism to identify the regex in one space, and just define space counts in the next.

[itdependsnetworks]

Ping

[buxtronix]

A few comments on this:

Missed captures: You can already do this by having a catch-all rule contain an 'Error' action.

Repeated Data: I'll discuss with harro, there are certainly use cases that make it worth trying to address properly. I'm not sure how the implementation would work yet, do you have something in mind?

Trim: We've usually preferred textfsm to do capture only and not data manipulation, but let me consider whether this simple case is feasible. Can you provide a case where this has been difficult for you?

Column count: I'm not quite sure what a specific example might be - can you provide a more concrete example here?

Thanks.

[itdependsnetworks]

Missed captures: Brilliant and much easier than my example
Repeated Data : Potentially two regex's, an outside bound and and inside bound. E.g.

Value INTERFACES RepeatedItem (\S+) "(\S+?),"

Start
  ^\${PORT}\s+${PROTO}\s+${INTERFACES}

Raw: Po101(U) LACP(a) Et3/1/1,Et3/1/2,Et3/1/3,Et3/1/4

Trim() https://github.com/networktocode/ntc-templates/blob/master/templates/cisco_ios_show_ip_bgp.template
https://github.com/networktocode/ntc-templates/tree/master/tests/cisco_ios/show_ip_bgp
Column Count Same as previous

You can see that the example it is largely positional based, and forward/reverse lookups being used. I get's complicated real quick

[mihudec]

I am just starting with textfsm, but I would also appreciate a better way to handle Repeated Data, possibly by using two regex's as @itdependsnetworks suggested. I think it would have major application for many types of outputs and would greatly reduce the need for post-processing in Python. An example of such command could be show vlan with it's interfaces, show interfaces trunk with list of allowed VLANs and so on.

Thanks.

[harro]

To second buxtronix's comments, the preference is to do post processing outside of textfsm, so it would need to be a fairly compelling case to add such functionality.

Repeated Data:

In cases like AS Path, lumping into a single field and splitting later has been our MO to date. e.g.

...
Value ASPath ((\d+ )*(i|\?))

Start
  ^\s*${Status} ${Best}${Origin}c?\s+${Prefix}\s+${NextHop}(\s+u)?\s+${Metric}\s+${LocalPref}\s+${Weight}\s+${ASPath}\s*$$ -> Next.Record

EOF

The use of string.Template for matching keeps the textfsm logic simple. I suppose for a RepeatedItem we could merge 'match' and 'separator' like the above ASPath entry, then use them separately for splitting post match.

[itdependsnetworks]

@harro the question is would you except a PR? My colleague has put in a PR for better error handling and it hasn’t been picked up yet. Willing to do the work, if going to be accepted.

[harro]

Will chat with @buxtronix about it next week. Get back to then.

[harro]

Apologies for the delay.

The appetite for functionality that can be easily provided in post processing is not high.
With trim() and ColumnCount falling under that category.

The Repeated Data function is interesting and improves parsing power, which certainly makes it worthy of consideration.

The concern being that we get a clean implementation. Presently the limited syntax and functionality allows us to keep the template and data parsing code simple. So it would be preferably if an incremental increase in functionality came with a similarly modest increase in code complexity.

[gachteme]

@harro the regex module is fully backwards compatible with re and has functionality to add the repeated data functionality.
using match.captures() instead of match.group() enables this.

a = regex.match('(ab)+(bc)', 'ababbc')
b = re.match('(ab)+(bc)', 'ababbc')
a.groups()

('ab', 'bc')

b.groups()

('ab', 'bc')

a.captures(1)

['ab', 'ab']

a.group(1)

'ab'

[sumkincpp]

Can there be an action for retrying the match but using different state?

Something like Continue <StateName> which is implicitly forbidden now. But, since it's forbidden, it's easy to implement it. This will mean "Keep current line and process it with state ".

Or may be something like KeepLine <StateName>

I.e. something like this:

SUBPART
  <all subpart matches>
  ^[^\s] -> KeepLine.Record Start

[gachteme]

@sumkincpp I've been thinking about adding an import statement that would more or less copy and paste one state into another. This would be useful for the situation you suggest and would allow the inserted matches to be at the top or middle of a state, which would be hard to do with your on failure condition. I definitely have written a few templates that had a lot of copying/pasting of parts of states that could have been reduced if there was an import feature.

To be clear I mean importing within the same file.

E.g:

Start
  \<some expressions\>
  $$IMPORT ToImportState
  \<some expressions\>

ToImportState
  \<some expressions\>
  ^\S -> Start

this solution would avoid the infinite loop problem presented by continue coupled with a state transition, while also reducing template development time and complexity.

Maybe it could also provide a feature that suppresses state transitions on imports? Thoughts?