Speedup builds

[t-b]

Description:

Stages are prefixed with -, Jobs with *, and tasks with ~. Bamboo only proceeds to the next stage when all jobs are
finished in one stage. Jobs are run in parallel when enough agents are free. Tasks are executed sequentially. Only
relevant tasks are shown. BA:XXX gives the bamboo agent capable of executing the job. We currently have three agents
(LM, WIN-ITC, WIN-NI).

Plan: Build release (latest)

Branch: release/2.1

• Installer and release packages 5min, BA:WIN-ITC,WIN-NI
• Unit testing 4min, BA:WIN-ITC,WIN-NI
• Uploading 10s, BA:LM

Total (best case): 10min

Plan: Build main

Branch: main

• Create assets
  • Documentation (Docker) 12min, BA:LM
  • Installer and release package 1min, BA:WIN-ITC,WIN-NI
• Compilation Stage
  • Test from repo 12min, BA:WIN-ITC,WIN-NI
• Uploading assets
  • Uploading the documentation 20s, BA:LM
  • Uploading the installer and ... 30s, BA:LM

Total (best case): 25min

Plan: Checks (Github) new

Branch: All new branches

• Create artefacts

  • CI installer 1min, BA:WIN-ITC,WIN-NI
  • Documentation (Docker) 12min, BA:LM
  • Installer and release packages 1min, BA:WIN-ITC,WIN-NI

• Testing without Hardware

  • Compilation test installer with hardware XOPs 3min, BA:WIN-ITC,WIN-NI
    ~ Compilation testing IP8-32bit
    ~ Compilation testing IP8-64it
  • Compilation test installer without hardware XOPs 3min, BA:WIN-ITC,WIN-NI
    ~ Compilation testing IP8-32bit
    ~ Compilation testing IP8-64it
  • Test from repo (IP8) 20min, BA:WIN-ITC,WIN-NI
    ~ Compilation testing IP8-32bit (with test scripts)
    ~ Compilation testing IP8-64bit (with test scripts)
    ~ Testing without hardware IP8-32bit (RunAllTests.pxp, PA_Tests.pxp)
    ~ Testing without hardware IP8-64bit (RunAllTests.pxp, PA_Tests.pxp)
  • Test from repo (IP9) 12min, BA:WIN-ITC,WIN-NI
    ~ Compilation testing IP9-32bit (with test scripts)
    ~ Compilation testing IP9-64bit (with test scripts)
    ~ Testing without hardware IP9-32bit (RunAllTests.pxp, PA_Tests.pxp)
    ~ Testing without hardware IP9-64bit (RunAllTests.pxp, PA_Tests.pxp)

• Hardware Testing

  • Testing with ITC18USB hardware 45min (HardwareTests.pxp), BA:WIN-ITC
  • Testing with NI hardware 50min (HardwareTests-NI.pxp), BA:WIN-NI

• Validation

  • Validate NWBv2 files 2min, BA:LM

Total (best case): 87min

Ideas for making the builds faster:

• Move the documentation job to the later test stage as it takes much longer than the other tasks in the first stage
• Install a second agent on the linux mint box so that we can parallelize there
• Remove 32bit support from MIES
• Remove submodules: ITCXOP2/ZeroMQ submodules as they are ~ 330MB and need to be downloaded on every job. We can grab what we need for building documentation/installer via curl. We do have the XOPs anyway in XOPs-XXX.
  • ZeroMQ
  • ITCXOP2
• Look into Hardware Tests where the time is spent
  • Make stimsets shorter?
• Test from repo (IP8) and Test from repo (IP9) should be only one job

[t-b]

Test suite execution times:

rg '<testsuite.*\bname="([^"]+)".*time="([^">]+)".*' --replace '$2 [s]: $1' --no-filename *xml | sort -n

        0.111 [s]: UTF_Testpulse.ipf
        0.132 [s]: UTF_JSONWaveNotes.ipf
        0.183 [s]: UTF_ThreadsafeDataSharing.ipf
        0.212 [s]: UTF_UpgradeWaveLocationAndGetIt.ipf
        0.223 [s]: UTF_WaveAveraging.ipf
        0.243 [s]: UTF_UpgradeDataFolderLocation.ipf
        0.566 [s]: UTF_WaveVersioning.ipf
        0.894 [s]: UTF_VeryLastTestsuite.ipf
        0.930 [s]: UTF_TraceUserData.ipf
        1.234 [s]: UTF_AnalysisFunctionHelpers.ipf
        1.804 [s]: UTF_EpochswoHardware.ipf
        1.862 [s]: UTF_Configuration.ipf
        2.114 [s]: UTF_Macros.ipf
        2.175 [s]: UTF_StimsetAPI.ipf
        2.972 [s]: UTF_PGCSetAndActivateControl.ipf
        3.943 [s]: UTF_WaveBuilderRegression.ipf
        5.786 [s]: UTF_AsynFrameworkTest.ipf
        9.281 [s]: UTF_WaveBuilder.ipf
        10.475 [s]: UTF_HardwareTestsWithBUG.ipf
        14.814 [s]: UTF_VeryBasicHardwareTests.ipf
        18.524 [s]: UTF_AnalysisFunctionParameters.ipf
        21.070 [s]: UTF_Utils.ipf
        26.745 [s]: UTF_MultiPatchSeqDaScale.ipf
        45.560 [s]: UTF_Dashboard.ipf
        45.882 [s]: UTF_Labnotebook.ipf
        55.181 [s]: UTF_SweepFormula.ipf
        59.117 [s]: UTF_ZeroMQPublishing.ipf
        65.111 [s]: UTF_ReachTargetVoltage.ipf
        66.018 [s]: UTF_AutoTestpulse.ipf
        68.411 [s]: UTF_AutoTestpulse.ipf
        111.920 [s]: UTF_SweepFormulaHardware.ipf
        121.428 [s]: UTF_DAEphys.ipf
        132.967 [s]: UTF_PA_Tests.ipf
        138.875 [s]: UTF_SweepSkipping.ipf
        161.453 [s]: UTF_SetControls.ipf
        210.859 [s]: UTF_PatchSeqPipetteInBath.ipf
        239.718 [s]: UTF_Epochs.ipf
        268.060 [s]: UTF_PatchSeqAccessResistanceSmoke.ipf
        279.252 [s]: UTF_PatchSeqSealEvaluation.ipf
        319.085 [s]: UTF_PatchSeqTrueRestingMembranePotential.ipf
        345.596 [s]: UTF_MultiPatchSeqSpikeControl.ipf
        399.543 [s]: UTF_PatchSeqRheobase.ipf
        407.057 [s]: UTF_TestPulseAndTPDuringDAQ.ipf
        434.365 [s]: UTF_PatchSeqSquarePulse.ipf
        487.917 [s]: UTF_PatchSeqRamp.ipf
        563.972 [s]: UTF_MultiPatchSeqFastRheoEstimate.ipf
        724.704 [s]: UTF_PatchSeqDAScale.ipf
        735.203 [s]: UTF_AnalysisFunctionManagement.ipf
        919.488 [s]: UTF_TrackSweepCounts.ipf
        940.674 [s]: UTF_BasicHardwareTests.ipf
        982.640 [s]: UTF_PatchSeqChirp.ipf

The total is 9456s. UTF_AutoTestpulse.ipf is duplicated, fixed in #1516.

Test case execution times:

rg '<testcase.*\bname="([^"]+)".*\btime="([^">]+)".*' --replace '$2 [s]: $1' --no-filename *xml | sort -n

test-case-execution-times.txt

The 0.000 for some test cases will be fixed with byte-physics/igortest#279.

[t-b]

Taking the results from https://github.com/AllenInstitute/MIES/actions/runs/6152479632:

Non-Hardware tests:

       0.001 [s]: @DGEN_SUITE
        0.026 [s]: @HOOK_SUITE
        0.039 [s]: @HOOK_SUITE
        0.174 [s]: UTF_JSONWaveNotes.ipf
        0.185 [s]: @DGEN_SUITE
        0.919 [s]: UTF_AnalysisFunctionHelpers.ipf
        0.975 [s]: UTF_Testpulse.ipf
        1.556 [s]: UTF_EpochswoHardware.ipf
        2.922 [s]: UTF_PGCSetAndActivateControl.ipf
        3.143 [s]: UTF_ThreadsafeDataSharing.ipf
        3.792 [s]: UTF_Configuration.ipf
        4.396 [s]: UTF_WaveBuilderRegression.ipf
        4.599 [s]: UTF_Macros.ipf
        4.748 [s]: UTF_UtilsChecks.ipf
        4.929 [s]: UTF_AsynFrameworkTest.ipf
        5.698 [s]: UTF_AnalysisBrowserTest.ipf
        6.160 [s]: UTF_WaveAveraging.ipf
        6.314 [s]: UTF_UpgradeDataFolderLocation.ipf
        6.354 [s]: UTF_StimsetAPI.ipf
        6.976 [s]: UTF_UpgradeWaveLocationAndGetIt.ipf
        10.647 [s]: UTF_WaveVersioning.ipf
        10.928 [s]: UTF_TraceUserData.ipf
        23.225 [s]: UTF_WaveBuilder.ipf
        24.331 [s]: UTF_AnalysisFunctionParameters.ipf
        42.153 [s]: UTF_DAEphyswoHardware.ipf
        46.056 [s]: UTF_Labnotebook.ipf
        60.268 [s]: UTF_ZeroMQPublishing.ipf
        86.986 [s]: @DGEN_SUITE
        97.039 [s]: UTF_SweepFormula_PSX.ipf
        98.219 [s]: UTF_HistoricEpochClipping.ipf
        103.895 [s]: UTF_HistoricDashboard.ipf
        113.181 [s]: UTF_PA_Tests.ipf
        127.352 [s]: UTF_SweepFormula.ipf
        305.043 [s]: UTF_Utils.ipf

Hardware Tests (ITC):

        0.003 [s]: @DGEN_SUITE
        0.916 [s]: UTF_VeryLastTestsuite.ipf
        0.931 [s]: @DGEN_SUITE
        3.504 [s]: @HOOK_SUITE
        3.509 [s]: @HOOK_SUITE
        9.746 [s]: UTF_Databrowser.ipf
        17.546 [s]: UTF_HardwareTestsWithBUG.ipf
        19.468 [s]: UTF_VeryBasicHardwareTests.ipf
        21.736 [s]: UTF_MultiPatchSeqDaScale.ipf
        40.663 [s]: UTF_Dashboard.ipf
        56.544 [s]: UTF_ReachTargetVoltage.ipf
        56.872 [s]: UTF_AutoTestpulse.ipf
        141.277 [s]: UTF_SweepFormulaHardware.ipf
        141.653 [s]: UTF_SweepSkipping.ipf
        148.292 [s]: UTF_SetControls.ipf
        152.901 [s]: UTF_DAEphys.ipf
        203.111 [s]: UTF_PatchSeqPipetteInBath.ipf
        260.259 [s]: UTF_PatchSeqAccessResistanceSmoke.ipf
        267.741 [s]: UTF_PatchSeqSealEvaluation.ipf
        286.858 [s]: UTF_MultiPatchSeqSpikeControl.ipf
        287.007 [s]: UTF_PatchSeqTrueRestingMembranePotential.ipf
        348.587 [s]: UTF_ConfigurationHardware.ipf
        363.039 [s]: UTF_PatchSeqRheobase.ipf
        374.512 [s]: UTF_PatchSeqSquarePulse.ipf
        395.287 [s]: UTF_Epochs.ipf
        397.471 [s]: UTF_TestPulseAndTPDuringDAQ.ipf
        450.876 [s]: UTF_MultiPatchSeqFastRheoEstimate.ipf
        452.627 [s]: UTF_PatchSeqRamp.ipf
        640.830 [s]: UTF_PatchSeqDAScale.ipf
        672.562 [s]: UTF_AnalysisFunctionManagement.ipf
        876.788 [s]: UTF_PatchSeqChirp.ipf
        955.336 [s]: UTF_TrackSweepCounts.ipf
        1043.198 [s]: UTF_BasicHardwareTests.ipf