Skip to content

Rolling upgrade failures #89915

Closed
Closed
@pgomulka

Description

@pgomulka

CI Link

https://gradle-enterprise.elastic.co/s/2v47w5344idjc

Repro line

n/a

Does it reproduce?

No

Applicable branches

main v 8.4.2

Failure history

https://gradle-enterprise.elastic.co/scans/failures?failures.failureClassification=all_failures&failures.failureMessage=Execution%20failed%20for%20task%20*%0A%3E%20process%20was%20found%20dead%20while%20waiting%20for%20ports%20files%2C%20*&search.timeZoneId=Europe/Warsaw

Failure excerpt

hard to tell what test failed, does not reproduce locally for the general ./gradlew ':qa:rolling-upgrade:v8.4.2#bwcTest'
it is possibly related to (recent change that touched TrainedModelAssignmentRebalancer) #89645

» [2022-09-08T10:44:20,221][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [v8.4.2-2] fatal error in thread [elasticsearch[v8.4.2-2][ml_utility][T#5]], exiting java.lang.AssertionError: ml.allocated_processors should parse because we set it internally: invalid value was 32.0 |  
-- | --
  | »  	at org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentRebalancer.getNodeAllocatedProcessors(TrainedModelAssignmentRebalancer.java:138) |  
  | »  	at org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentRebalancer.lambda$computeAssignmentPlan$2(TrainedModelAssignmentRebalancer.java:90) |  
  | »  	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) |  
  | »  	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) |  
  | »  	at java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1850) |  
  | »  	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) |  
  | »  	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) |  
  | »  	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) |  
  | »  	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) |  
  | »  	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) |  
  | »  	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) |  
  | »  	at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) |  
  | »  	at org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentRebalancer.computeAssignmentPlan(TrainedModelAssignmentRebalancer.java:93) |  
  | »  	at org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentRebalancer.rebalance(TrainedModelAssignmentRebalancer.java:66) |  
  | »  	at org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentClusterService.rebalanceAssignments(TrainedModelAssignmentClusterService.java:467) |  
  | »  	at org.elasticsearch.xpack.ml.inference.assignment.TrainedModelAssignmentClusterService.lambda$rebalanceAssignments$8(TrainedModelAssignmentClusterService.java:402) |  
  | »  	at org.elasticsearch.server@8.4.2-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710) |  
  | »  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) |  
  | »  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) |  
  | »  	at java.base/java.lang.Thread.run(Thread.java:833)


Metadata

Metadata

Assignees

No one assigned

    Labels

    :mlMachine learning>test-failureTriaged test failures from CITeam:MLMeta label for the ML team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions