Health System and QBit Admin

You can use AdminBuilder to create an admin utils.

        final AdminBuilder adminBuilder = AdminBuilder.adminBuilder();

        final ServiceEndpointServer adminServer =
                adminBuilder.build();

        adminServer.startServer();

        final HealthServiceAsync healthService = adminBuilder.getHealthService();
        healthService.register("foo", 1, TimeUnit.DAYS);
        healthService.checkInOk("foo");
        healthService.register("bar", 1, TimeUnit.DAYS);
        healthService.checkInOk("bar");

You can also register for service queue health checks.

    @Bean
    public AdminBuilder qbitAdminBuilder() {

        final int port = environment.getProperty("qbit.admin.server.port", Integer.class);
        final String host = environment.getProperty("qbit.admin.server.host", String.class);


        final AdminBuilder adminBuilder = AdminBuilder.adminBuilder()
                .setPort(port).setHost(host);

        return adminBuilder;
    }


    @Bean
    public ServiceEndpointServer adminServiceEndpointServer(
            final AdminBuilder adminBuilder) {
        final ServiceEndpointServer adminServer =
                adminBuilder.build();

        adminServer.startServer();
        return adminServer;

    }

....

        final Integer healthCheckTTL = env.getProperty("qbit.app.healthCheckTTLSeconds", Integer.class);
...

        final ServiceBuilder serviceBuilder = ServiceBuilder.serviceBuilder();

        final AdminBuilder qbitAdminBuilder =
                 applicationContext.getBean("qbitAdminBuilder", AdminBuilder.class);
        final HealthServiceAsync healthServiceAsync =
               qbitAdminBuilder.getHealthServiceBuilder().buildHealthSystemReporter();

        serviceBuilder.registerHealthChecksWithTTLInSeconds(qbitAdminBuilder.getHealthService(), longName,
                healthCheckTTL == null ? 5 : healthCheckTTL);

....

You can also register for stats health checks

        final ServiceBuilder serviceBuilder = ServiceBuilder.serviceBuilder();
...
        final StatsCollector statsCollector = applicationContext.getBean("qbitStatsCollector", StatsCollector.class);

        serviceBuilder.registerStatsCollections(longName, statsCollector, flushTimeSeconds, sampleEvery );

Once you do this, then you can query for health status.

curl http://localhost:6001/services/qbit-admin/ok

Returns true if all internal service queues are running. Returns false if any are failing.

curl http://localhost:6001/services/qbit-admin/all-nodes/

Returns list of nodes healthy or not:

[
    "api.proxy.unpackerService",
    "api.proxy.healthService",
    "api.proxy.forwarderService",
    "api.proxy.bouncerService"
]

Only return healthy nodes: curl http://localhost:6001/services/qbit-admin/healthy-nodes

[
    "api.proxy.unpackerService",
    "api.proxy.healthService",
    "api.proxy.forwarderService",
    "api.proxy.bouncerService"
]

Return extended stats

curl http://localhost:6001/services/qbit-admin/load-nodes

[
    {
        "name": "api.proxy.unpackerService",
        "ttlInMS": 10000,
        "lastCheckIn": 1434003690275,
        "status": "PASS"
    },
    {
        "name": "api.proxy.healthService",
        "ttlInMS": 10000,
        "lastCheckIn": 1434003690275,
        "status": "PASS"
    },
    {
        "name": "api.proxy.forwarderService",
        "ttlInMS": 10000,
        "lastCheckIn": 1434003690275,
        "status": "PASS"
    },
    {
        "name": "api.proxy.bouncerService",
        "ttlInMS": 10000,
        "lastCheckIn": 1434003690275,
        "status": "PASS"
    }
]

Registering for health checks can be done with the service bundle builder and the service endpoint server builder as well:

        final ServiceBundleBuilder.serviceBundleBuilder =
                ServiceBundleBuilder.serviceBundleBuilder().getRequestQueueBuilder();

        final ServiceBundle serviceBundle = serviceBundleBuilder
                .setStatsCollector(statsCollector)
                .build();

        serviceBundle.start();

        serviceBundle.addService(new MyService());

Every service added to the bundle will get stats support.

A service can mark itself unhealthy:

Service marking itself unhealthy with the health system.

@RequestMapping(value = "/todo")
@Api(value = "/todo", consumes = "", description = "Todod")
public class TodoService {
...
    /**
     * Used to call the queue. This gets called when our service queue is empty or has reached its limit.
     */
    @QueueCallback({QueueCallbackType.LIMIT, QueueCallbackType.EMPTY, QueueCallbackType.IDLE})
    public void process() {
        reactor.process();
        this.jmsConnected = jmsConnectedRef.get();

        final ServiceQueue serviceQueue = ServiceContext.serviceContext().currentService();
        if (!jmsConnected) {
            serviceQueue.setFailing();
        } else {
                serviceQueue.recover();
        }
    }

The above checks to see if JMS is up. If it is not up, then this service fails. If JMS is up, then the service is marked as recovered. The two ways a service can fail is if it never checks in, or if it marks itself as failed.

Once a service is marked unhealthy. The health end points will report this status.

If you have local health end point check installed then this will show up as failed.

$ curl http://localhost:8090/__health -v 
*   Trying ::1...
* Connected to localhost (::1) port 8090 (#0)
> GET /__health HTTP/1.1
> Host: localhost:8090
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 500 SERVER ERROR
< Content-Type: application/json
< Content-Length: 6
< 
* Connection #0 to host localhost left intact
"fail"

The admin end point returns true 200 if ok and false 200 if not ok.

$ curl http://localhost:7779/__admin/ok -v
*   Trying ::1...
* Connected to localhost (::1) port 7779 (#0)
> GET /__admin/ok HTTP/1.1
> Host: localhost:7779
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 5
< 
* Connection #0 to host localhost left intact
false

Once the service recovers, in this case, it is able to connect to JMS again.

$ curl http://localhost:7779/__admin/ok -v
*   Trying ::1...
* Connected to localhost (::1) port 7779 (#0)
> GET /__admin/ok HTTP/1.1
> Host: localhost:7779
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 4
< 
* Connection #0 to host localhost left intact

true

$ curl http://localhost:8090/__health -v 
*   Trying ::1...
* Connected to localhost (::1) port 8090 (#0)
> GET /__health HTTP/1.1
> Host: localhost:8090
> User-Agent: curl/7.43.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 4
< 
* Connection #0 to host localhost left intact
"ok"

Then things go back to normal.