Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]Can't trigger savepoint #3255

Open
2 of 3 tasks
westPointz opened this issue Oct 20, 2023 · 3 comments
Open
2 of 3 tasks

[Bug]Can't trigger savepoint #3255

westPointz opened this issue Oct 20, 2023 · 3 comments
Labels
bug Something isn't working

Comments

@westPointz
Copy link

Search before asking

  • I had searched in the issues and found no similar issues.

Java Version

1.8

Scala Version

2.12.x

StreamPark Version

2.1.1

Flink Version

1.16.1

deploy mode

kubernetes-application

What happened

Can't trigger savepoint

Error Exception

java.util.concurrent.CompletionException: java.lang.reflect.InvocationTargetException
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.streampark.flink.client.FlinkClient$.$anonfun$proxy$1(FlinkClient.scala:80)
	at org.apache.streampark.flink.proxy.FlinkShimsProxy$.$anonfun$proxy$1(FlinkShimsProxy.scala:60)
	at org.apache.streampark.common.util.ClassLoaderUtils$.runAsClassLoader(ClassLoaderUtils.scala:38)
	at org.apache.streampark.flink.proxy.FlinkShimsProxy$.proxy(FlinkShimsProxy.scala:60)
	at org.apache.streampark.flink.client.FlinkClient$.proxy(FlinkClient.scala:75)
	at org.apache.streampark.flink.client.FlinkClient$.triggerSavepoint(FlinkClient.scala:65)
	at org.apache.streampark.flink.client.FlinkClient.triggerSavepoint(FlinkClient.scala)
	at org.apache.streampark.console.core.service.impl.SavePointServiceImpl.lambda$trigger$0(SavePointServiceImpl.java:320)
	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
	... 3 more
Caused by: java.util.concurrent.ExecutionException: org.apache.flink.util.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted.
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
	at org.apache.streampark.flink.client.trait.FlinkClientTrait.triggerSavepoint(FlinkClientTrait.scala:569)
	at org.apache.streampark.flink.client.trait.FlinkClientTrait.triggerSavepoint$(FlinkClientTrait.scala:563)
	at org.apache.streampark.flink.client.impl.KubernetesNativeApplicationClient$.org$apache$streampark$flink$client$trait$KubernetesNativeClientTrait$$super$triggerSavepoint(KubernetesNativeApplicationClient.scala:35)
	at org.apache.streampark.flink.client.trait.KubernetesNativeClientTrait.$anonfun$doTriggerSavepoint$1(KubernetesNativeClientTrait.scala:134)
	at org.apache.streampark.flink.client.trait.KubernetesNativeClientTrait.executeClientAction(KubernetesNativeClientTrait.scala:115)
	at org.apache.streampark.flink.client.trait.KubernetesNativeClientTrait.doTriggerSavepoint(KubernetesNativeClientTrait.scala:133)
	at org.apache.streampark.flink.client.trait.KubernetesNativeClientTrait.doTriggerSavepoint$(KubernetesNativeClientTrait.scala:127)
	at org.apache.streampark.flink.client.impl.KubernetesNativeApplicationClient$.doTriggerSavepoint(KubernetesNativeApplicationClient.scala:104)
	at org.apache.streampark.flink.client.trait.FlinkClientTrait.triggerSavepoint(FlinkClientTrait.scala:169)
	at org.apache.streampark.flink.client.trait.FlinkClientTrait.triggerSavepoint$(FlinkClientTrait.scala:155)
	at org.apache.streampark.flink.client.impl.KubernetesNativeApplicationClient$.triggerSavepoint(KubernetesNativeApplicationClient.scala:35)
	at org.apache.streampark.flink.client.FlinkClientHandler$.triggerSavepoint(FlinkClientHandler.scala:56)
	at org.apache.streampark.flink.client.FlinkClientHandler.triggerSavepoint(FlinkClientHandler.scala)
	... 16 more
Caused by: org.apache.flink.util.concurrent.FutureUtils$RetryException: Could not complete the operation. Number of retries has been exhausted.
	at org.apache.flink.util.concurrent.FutureUtils.lambda$retryOperationWithDelay$6(FutureUtils.java:293)
	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:610)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1085)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)
	... 3 more
Caused by: java.util.concurrent.CompletionException: org.apache.flink.runtime.rest.util.RestClientException: Response was neither of the expected type([simple type, class org.apache.flink.runtime.rest.handler.async.AsynchronousOperationResult<org.apache.flink.runtime.rest.messages.job.savepoints.SavepointInfo>]) nor an error.
	at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367)
	at java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1074)
	... 4 more
Caused by: org.apache.flink.runtime.rest.util.RestClientException: Response was neither of the expected type([simple type, class org.apache.flink.runtime.rest.handler.async.AsynchronousOperationResult<org.apache.flink.runtime.rest.messages.job.savepoints.SavepointInfo>]) nor an error.
	at org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:550)
	at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$3(RestClient.java:514)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
	... 4 more
Caused by: org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException: class org.apache.flink.util.SerializedThrowable cannot be cast to class org.apache.flink.util.SerializedThrowable (org.apache.flink.util.SerializedThrowable is in unnamed module of loader 'app'; org.apache.flink.util.SerializedThrowable is in unnamed module of loader org.apache.streampark.flink.proxy.ChildFirstClassLoader @57564bf6) (through reference chain: org.apache.flink.runtime.rest.handler.async.AsynchronousOperationResult["operation"]->org.apache.flink.runtime.rest.messages.job.savepoints.SavepointInfo["failure-cause"])
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:390)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:349)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1822)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:567)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:449)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:542)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:565)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:449)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1405)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:362)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:195)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:322)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4569)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2867)
	at org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:523)
	... 6 more
Caused by: java.lang.ClassCastException: class org.apache.flink.util.SerializedThrowable cannot be cast to class org.apache.flink.util.SerializedThrowable (org.apache.flink.util.SerializedThrowable is in unnamed module of loader 'app'; org.apache.flink.util.SerializedThrowable is in unnamed module of loader org.apache.streampark.flink.proxy.ChildFirstClassLoader @57564bf6)
	at org.apache.flink.runtime.rest.messages.json.SerializedThrowableDeserializer.deserialize(SerializedThrowableDeserializer.java:49)
	at org.apache.flink.runtime.rest.messages.json.SerializedThrowableDeserializer.deserialize(SerializedThrowableDeserializer.java:34)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:542)
	at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:565)
	... 20 more

Screenshots

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!(您是否要贡献这个PR?)

Code of Conduct

@wsczm
Copy link

wsczm commented Nov 9, 2023

i meet the same problem can yon slove this probelm

@wolfboys wolfboys added the bug Something isn't working label Dec 7, 2023
@wolfboys
Copy link
Member

wolfboys commented Dec 7, 2023

Looks like a bug. 🐛

@Li-GL
Copy link

Li-GL commented Jan 11, 2024

i also meet the same problem

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants