What is the best way to make HTTP requests during SSR?

[Zk2u]

Hey all. I forked the hacker news example to start our project and enabled rkyv support (check out #577) - everything works! (6ms API response times 🎉). However, it isn't as blazingly fast as it should be, as it's using a new reqwest client and, therefore, connection on every fetch it does while SSR'ing. This is not great.

Here's some code (api.rs):

use bytecheck::CheckBytes;
use leptos::{Scope, Serializable};
use rkyv::{
    de::deserializers::SharedDeserializeMap, ser::serializers::AllocSerializer,
    validation::validators::DefaultValidator, Archive, Deserialize, Serialize,
    from_bytes, to_bytes
};
#[cfg(not(feature = "ssr"))]
use js_sys::Uint8Array;

#[cfg(not(feature = "ssr"))]
pub async fn fetch<T, K, const N: usize>(cx: Scope, path: &str, body: K) -> Option<T>
where
    T: Serialize<AllocSerializer<1024>>,
    T: Serializable + Archive,
    T::Archived: for<'b> CheckBytes<DefaultValidator<'b>> + Deserialize<T, SharedDeserializeMap>,
    K: Serialize<AllocSerializer<N>>,
{
    let abort_controller = web_sys::AbortController::new().ok();
    let abort_signal = abort_controller.as_ref().map(|a| a.signal());

    let bytes = gloo_net::http::Request::get(path)
        .abort_signal(abort_signal.as_ref())
        .body(Uint8Array::from(to_bytes(&body).ok()?.as_slice()))
        .send()
        .await
        .map_err(|e| log::error!("{e}"))
        .ok()?
        .binary()
        .await
        .ok()?;

    // abort in-flight requests if the Scope is disposed
    // i.e., if we've navigated away from this page
    leptos::on_cleanup(cx, move || {
        if let Some(abort_controller) = abort_controller {
            abort_controller.abort()
        }
    });
    from_bytes::<T>(&bytes).ok()
}

#[cfg(feature = "ssr")]
pub async fn fetch<T, K, const N: usize>(_cx: Scope, path: &str, body: K) -> Option<T>
where
    T: Serialize<AllocSerializer<1024>>,
    T: Serializable + Archive,
    T::Archived: for<'b> CheckBytes<DefaultValidator<'b>> + Deserialize<T, SharedDeserializeMap>,
    K: Serialize<AllocSerializer<N>>,
{
    let client = reqwest::Client::new();
    let start = std::time::Instant::now();
    let bytes = client
        .get(path)
        .body(to_bytes(&body).ok()?.to_vec())
        .send()
        .await
        .map_err(|e| log::error!("{e}"))
        .ok()?
        .bytes()
        .await
        .ok()?;
    log::info!("fetch took {:?}", start.elapsed());
    from_bytes::<T>(&bytes).ok()
}

Everything is great and working quickly, apart from reqwest creating a new connection on every render. Here are some server logs:

# reqwest starts connection open
2023-02-26T15:04:05.183696Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/ 
2023-02-26T15:04:05.183893Z DEBUG hyper::client::connect::dns: resolving host="localhost" 
2023-02-26T15:04:05.184560Z DEBUG hyper::client::connect::http: connecting to [::1]:8080
2023-02-26T15:04:05.495580Z DEBUG hyper::client::connect::http: connecting to 127.0.0.1:8080
2023-02-26T15:04:05.511344Z DEBUG hyper::client::connect::http: connected to 127.0.0.1:8080
# reqwest finished connection open in 328ms
# send the actual request for the page render 
2023-02-26T15:04:05.511935Z DEBUG hyper::proto::h1::io: flushed 102 bytes
2023-02-26T15:04:05.518316Z DEBUG hyper::proto::h1::io: parsed 3 headers
2023-02-26T15:04:05.518517Z DEBUG hyper::proto::h1::conn: incoming body is content-length (80 bytes)
2023-02-26T15:04:05.518664Z DEBUG hyper::proto::h1::conn: incoming body completed
# reqwest completed in 6.9ms
2023-02-26T15:04:05.518866Z DEBUG hyper::client::pool: pooling idle connection for ("http", localhost:8080)
2023-02-26T15:04:05.519106Z  INFO valera_web::api: fetch took 335.4284ms

Ideally, I would preconnect to the API server when Actix boots up. We do a similar thing on our API server (also Actix):

fn main() -> {
    ...
    let state = web::Data::new(AppState {
        ...
        reqwest: reqwest::Client::new(),
        ...
    });

    preconnect(&state.reqwest, "rest.nexmo.com").await;
    ...
}

async fn preconnect(client: &reqwest::Client, domain: &str) {
    debug!("Preconnecting to {}", domain);
    let _ = client.head(format!("https://{}", domain)).send().await;
}

Here, the reqwest client is under an Arc at state.reqwest (i.e. one reqwest client for all threads). state,reqwest is then provided as part of the app state to each route handler who can access it. How can we do this alongside leptos (I'm guessing implementation will be server-implementation specific)? Thank you so much for your attention and participation.

[gbj]

The dead-simple solution would be to create the reqwest connection in the main function in main.rs and store it in a static OnceCell, then access it in the server API.

The slightly-more-complicated one would be to provide it with the context API by using leptos_routes_with_context instead of leptos_routes and providing the connection with provide_context()

[Zk2u]

Thanks! I managed to (sort of) get the first solution working by adding a static sync OnceCell with the client inside lib.rs behind the SSR flag. api.rs (a module of the library target) then uses crate::CLIENT.get_or_init(...). By the looks of it, this is creating a new client per thread. Running a quick bench (10k req, 10 clients) against Leptos shows it's working - P99 response times are under 4ms (mean 1.7ms), and the API requests make up most of that.

However, when running a more extended bench of 100k req over 10 clients, request latency increases to ~7-8ms and suddenly start timing out. Whether this is the API/DB slowing down or Leptos/reqwest, I will look into it.

[gbj]

Thanks! I managed to (sort of) get the first solution working by adding a static sync OnceCell with the client inside lib.rs behind the SSR flag. api.rs (a module of the library target) then uses crate::CLIENT.get_or_init(...). By the looks of it, this is creating a new client per thread.
Nice. Actually if you create it in a lazy_static instead of in main() you should be able to get it to one total client, I think.

Running a quick bench (10k req, 10 clients) against Leptos shows it's working - P99 response times are under 4ms (mean 1.7ms), and the API requests make up most of that.

Yeah, that's fast.

However, when running a more extended bench of 100k req over 10 clients, request latency increases to ~7-8ms and suddenly start timing out. Whether this is the API/DB slowing down or Leptos/reqwest, I will look into it.

Good question!

I'd welcome a PR to the example when you're done experimenting by the way -- this is super cool.

[Zk2u]

Been playing with it all day to no avail - switched to HTTP/2 and changed the number of threads available to API/leptos. Still thinking it's something to do with reqwest opening too many connections to the API or something similar. It'll run approx 16200 requests and then stop working - every time - even if you pause for a few seconds in between. Get the same error after ~16200 requests even with a lower req/s using a single client. Very odd.

Thanks! Will make another PR for the next release that adds an example + some more docs on how to use it (there are a lot of traits in there...).

[Zk2u]

I made a few changes, and it appears the issue is specific to MacOS. Successfully ran a stress test against the Leptos page for 10 minutes on Windows, and latencies are stable (though higher than MacOS, that's expected for Windows):

Concurrency Level:      10
Time taken for tests:   600.000 seconds
Complete requests:      2413825
Failed requests:        0
Total transferred:      11835005943 bytes
HTML transferred:       11571898146 bytes
Requests per second:    4023.04 [#/sec] (mean)
Time per request:       2.486 [ms] (mean)
Time per request:       0.249 [ms] (mean, across all concurrent requests)
Transfer rate:          19262.69 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       2
Processing:     1    2   0.7      2      29
Waiting:        0    1   0.5      1      26
Total:          1    2   0.7      2      29

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      3
  75%      3
  80%      3
  90%      3
  95%      3
  98%      4
  99%      4
 100%     29 (longest request)

This is using HTTP/2 to the API server using AES-256-GCM-SHA384.

Updated leptos API code:

    let client = CLIENT.get_or_init(|| {
        let dev =
            DEV_MODE.get_or_init(|| std::env::var("DEV").unwrap_or("false".to_string()) == "true");
        let mut client = Client::builder()
            .pool_max_idle_per_host(100)
            .http2_prior_knowledge();
        if dev.clone() {
            let cert = std::fs::read("../platform/cert.pem").expect("failed to read cert");
            client = client
                .danger_accept_invalid_certs(dev.clone())
                .add_root_certificate(Certificate::from_pem(&cert).expect("failed to parse cert"));
        }
        client.build().expect("failed to build client")
    });

latency in ms on the y-axis, percentile on the x-axis

Though I may have found a memory leak somewhere on the Leptos server - memory usage increased to 977MB throughout the test but didn't drop after the test stopped. The API server (also using actix) remained at 4.8MB for the whole ordeal.

[Zk2u]

Nice. Actually if you create it in a lazy_static instead of in main() you should be able to get it to one total client, I think.

The API module isn't in the binary target for some reason, so that I couldn't use a static from main.rs. It's static in lib.rs which API considers itself to be part of. That's why it's spawning per worker thread, as it's being initialised in each worker thread in actix (I think)