pipeline-logparser

a library to parse and filter logs

• implementation of https://stackoverflow.com/a/57351397
• workaround for https://issues.jenkins-ci.org/browse/JENKINS-54304

Table of contents

• Library Content
• Documentation
• Installation
• Know Limitations
• Change log

Library Content

this library offer functions to retrieve logs as string or files (as run artifacts), and to parse the logs at the same time

it allows:

• to add branch prefix [branchName] in front of each line of the logs belonging to a parallel branch

  echo 'not in any branch'
  parallel (
    branch1: { echo 'in branch1' },
    branch2: { echo 'in branch2' }
  )
  

  not in any branch
  [branch1] in branch1
  [branch2] in branch2

• to filter logs by branchName

  [branch1] in branch1

• to show the name of parent branches (parent branch first) for nested branches

  parallel branch2: {
    echo 'in branch2'
    parallel branch21: { echo 'in branch2.branch21' }
  }
  

  [branch2] in branch2
  [branch2] [branch21] in branch2.branch21

• to archive logs in job artifacts (without having to allocate a node : same as ArchiveArtifacts but without node() scope)

• to hide or show VT100 markups from logs (which make raw log hard to read)

• to hide or show Pipeline technical log lines (lines starting with [Pipeline] ) from logs

  [Pipeline] Start of Pipeline
  [Pipeline] echo
  not in any branch
  [Pipeline] parallel
  [Pipeline] { (Branch: branch1)
  [Pipeline] { (Branch: branch2)
  [Pipeline] echo
  [branch1] in branch1
  [Pipeline] }
  [Pipeline] echo
  [branch2] in branch2
  [Pipeline] }
  [Pipeline] // parallel
  [Pipeline] End of Pipeline
  Finished: SUCCESS

• to access descriptors of log and branches internal ids

Documentation

import pipeline-logparser library

in Jenkinsfile import library like this

@Library('pipeline-logparser@1.1') _

identifier "pipeline-logparser" is the name of the library set by jenkins administrator in instance configuration:

• it may be different on your instance
• see below Installation

use library's functions:

• the name of the package is logparser:

  // get logs with branch prefix
  def mylog = logparser.getLogsWithBranchInfo()
  

• see complete documentation here: logparser.txt
  also available in $JOB_URL/pipeline-syntax/globals#logparser (visible only after the library has been imported once)

Installation

install the library as a "Global Pipeline Library" in "Manage jenkins > Configure System > Global Pipeline Library" (cf https://jenkins.io/doc/book/pipeline/shared-libraries/)

Note:

• it's also possible to copy the code in a Jenkinsfile and use functions from there
• but it would imply approving whatever needs to be in "Manage jenkins > In-process Script Approval" (including some unsafe API's)
• using this library as a "Global Pipeline Library" allows to avoid that (avoid getting access to unsafe API's)

Known limitations

• parsing may fail (and cause job to fail) when log is too big (millions of lines, hundreds of MB of logs) because of a lack of heap space

• when logparser functions are called the last lines of logs might not be flushed yet and shall not be in the resulting log workaround: before to call logparser add these 2 statements

  // sleep 1s and use echo to flush logs before to call logparser
  sleep 1
  echo ''
  

  example:

  @Library('pipeline-logparser@1.1') _
  
  parallel(
    branch1: { echo 'in branch1' },
    branch2: { echo 'in branch2' }
  )
  
  // sleep 1s and use echo to flush logs before to call logparser
  sleep 1
  echo ''
  
  logparser.archiveLogsWithBranchInfo('console.txt')
  

  NB: this might not always be enough

• when hidePipeline is false, the output is not fully equivalent to what we had with job-workflow plugin v2.25 and earlier : example:

  • pipeline code

    parallel(
      branch1: { echo 'in branch1' },
      branch2: { echo 'in branch2' }
    )
    

  • job-workflow plugin v2.25 output:

    [Pipeline] parallel
    [Pipeline] [branch1] { (Branch: branch1)
    [Pipeline] [branch2] { (Branch: branch2)
    [Pipeline] [branch1] echo
    [branch1] in branch1
    [Pipeline] [branch1] }
    [Pipeline] [branch2] echo
    [branch2] in branch2
    [Pipeline] [branch2] }
    [Pipeline] // parallel
    [Pipeline] End of Pipeline

  • output using logparser.getLogsWithBranchInfo():

    [Pipeline] Start of Pipeline
    [Pipeline] parallel
    [Pipeline] { (Branch: branch1)
    [Pipeline] { (Branch: branch2)
    [Pipeline] echo
    [branch1] in branch1
    [Pipeline] }
    [Pipeline] echo
    [branch2] in branch2

    So we lose a bit of information: the lines starting with [Pipeline] and which belonged to a specific branch like [Pipeline] [branch2] echo
    It might not be the most important information but sometimes it is useful to know which branch it belongs to

Change log

• 1.0 (09/2019)

  • API to get edited logs with branchName (with nested branches, without VT100 markups, etc ...)
  • API to filter logs by branch name (get logs of a specific branch)
  • API to parse logs and retrieve maps of id/offsets/branchname

• 1.0.1 (11/2019)

  • fix parsing issues (when there is only one branch and when logs of 2 branches are mixed without pipeline technical logs in between)

• 1.1 (03/2020)

  • ability to set branch name to filter using regular expression
  • ability to filter more than one branch (API change : option filter is now a list)
  • ability to filter main thread (using filter = [null])
  • new option hidePipeline to filter Pipeline technical logs (default value true to hide them)
  • fix parsing issues with old version of workflow-job plugin