Update server.cpp example with correct startup sequence

[mann1x]

The HTTP listener start and the health API endpoint are moved before the model loading starts, hence the server can correctly report is loading the model

[ngxson]

Normally, all API endpoints must be registered before server start listening (svr->listen_after_bind() in this case), so I'm not convinced by this change, which registers /health before all other endpoints and the rest being registered after model loaded. During the model load, all other endpoints will throw 404 not found error which is not correct (it should be 503 Service Unavailable)

Furthermore, this change requires main thread to call svr to register new endpoints after it is spawned into new thread. This will make svr not thread-safe.

In any cases, it will be more logical to call ctx_server.load_model(params) only after all endpoints are registered. Additionally, we can add a middleware to throw 503 if the model is not yet loaded.

[mann1x]

In any cases, it will be more logical to call ctx_server.load_model(params) only after all endpoints are registered. Additionally, we can add a middleware to throw 503 if the model is not yet loaded.

Binding them before doesn't work, the model must be loaded.
They can be binded afterwards, no issues.
There's really no reason to use the other endpoints until the server reports that the model is still being loaded.
But indeed I haven't thought about 404 not being the right answer.
Made this for ollama which doesn't use any other endpoint.

I will amend it registering the other endpoints with a static 503 answer before listening and re-registering them later once the model is loaded.

[mann1x]

Furthermore, this change requires main thread to call svr to register new endpoints after it is spawned into new thread. This will make svr not thread-safe.

You are right I didn't check this.
Will try to make it work without re-registering the endpoints at all.

[ngxson]

Binding them before doesn't work, the model must be loaded.

The reference to ctx_server is never changed before/after model is loaded. If you see errors, that maybe because something accesses to ctx_server before model is loaded (for example, checking chat template). You should move them all to below code block where all endpoints are registered and HTTP is listening.

Another thing to add is inside svr->set_pre_routing_handler, there should be a middleware to check if we're accessing endpoints other than /health. If model is not loaded, the middleware must return 503 error.

[mann1x]

@ngxson

Can you please check it again and let me know?

The empty_json_model() is just a leftover of course.

What I'm wondering is if the middleware should report error 500 also for the static pages.

Thanks!

[phymbert]

What I'm wondering is if the middleware should report error 500 also for the static pages

If we can return the static pages while the model is loading, this is fine.

[mann1x]

@ngxson
Seems to work, let me know. Thanks a lot!