Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RedisReactiveHealthIndicator makes blocking call on error #16756

Conversation

ayudovin
Copy link
Contributor

@ayudovin ayudovin commented May 8, 2019

See #16705


Caused by: java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-epoll-3
	at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:77)
	at reactor.core.publisher.Mono.block(Mono.java:1494)
	at org.springframework.data.redis.connection.ReactiveRedisConnection.close(ReactiveRedisConnection.java:60)
	at org.springframework.boot.actuate.redis.RedisReactiveHealthIndicator.lambda$doHealthCheck$1(RedisReactiveHealthIndicator.java:50)
	at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.runFinally(FluxDoFinally.java:156)
	... 203 common frames omitted

While running a Spring Cloud Gateway application with Redis as session store, I observed the stack trace above. The cause is a blocking call in: https://github.com/spring-projects/spring-boot/blob/master/spring-boot-project/spring-boot-actuator/src/main/java/org/springframework/boot/actuate/redis/RedisReactiveHealthIndicator.java#L49

There is a Mono<Void> closeLater(); https://github.com/spring-projects/spring-data-redis/blob/master/src/main/java/org/springframework/data/redis/connection/ReactiveRedisConnection.java#L68 which could be called instead.

@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label May 8, 2019
@wilkinsona
Copy link
Member

Thanks for the PR, @ayudovin. As mentioned by @mp911de, we also need to change how connections are obtained. Of the two suggestions that Mark's made, I think the use of the parallel scheduler is the one to go for. Would you like to update your PR to that effect?

@wilkinsona wilkinsona added the status: waiting-for-feedback We need additional information before we can continue label May 8, 2019
@mp911de
Copy link
Member

mp911de commented May 9, 2019

I filed DATAREDIS-977 to expose non-blocking connect methods. Right now, Mono.fromSupplier().subscribeOn(Schedulers.parallel()) or similar should be a good compromise until we can provide Mono<ReactiveRedisConnection> methods.

@ayudovin
Copy link
Contributor Author

ayudovin commented May 9, 2019

@wilkinsona, I would like to update PR

@AndreasKl
Copy link
Contributor

AndreasKl commented May 9, 2019

@ayudovin The Mono returned via closeLater is unhandled, as it is lazy the close call will never happen.

  @Test
  public void notWorking() {
    Mono.fromRunnable(() -> System.out.println("not emitted"));
  }

  @Test
  public void working() {
    Mono<Void> never = Mono.fromRunnable(() -> System.out.println("working"));
    never.block();
  }

https://github.com/spring-projects/spring-data-redis/blob/master/src/main/java/org/springframework/data/redis/connection/lettuce/LettuceReactiveRedisConnection.java#L239

@wilkinsona wilkinsona added type: bug A general bug and removed status: waiting-for-feedback We need additional information before we can continue status: waiting-for-triage An issue we've not yet triaged labels May 9, 2019
@wilkinsona wilkinsona added this to the 2.1.x milestone May 9, 2019
@philwebb philwebb changed the title RedisReactiveHealthIndicator blocks on error by calling close, which is not supported in thread reactor-http-epoll RedisReactiveHealthIndicator makes blocking call on error May 15, 2019
philwebb added a commit that referenced this pull request May 15, 2019
* pr/16756:
  Polish 'Drop blocking RedisReactiveHealthIndicator calls'
  Drop blocking RedisReactiveHealthIndicator calls
@philwebb philwebb closed this in de85737 May 15, 2019
@philwebb philwebb modified the milestones: 2.1.x, 2.1.5 May 15, 2019
@philwebb
Copy link
Member

Thanks once again @ayudovin!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug A general bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants