Using RestSharp with streaming endpoint

[JannemanDev]

Does RestSharp v106.15.0 work with a streaming endpoint? For example Twitter has an https://api.twitter.com/2/tweets/search/stream endpoint which streams tweet(s) matching certain rule(s) created with another endpoint.
Using curl https://api.twitter.com/2/tweets/search/stream -H "Authorization: Bearer ...." this works.
I also got it to work when using WebClient in C#:

webClient = new WebClient();
webClient.Headers[HttpRequestHeader.Authorization] = $"Bearer {settings.Twitter.BearerToken}";
webClient.OpenReadCompleted += (sender, openReadCompletedEventArgs) =>
{
    Stream stream = openReadCompletedEventArgs.Result;
    using (var reader = new StreamReader(stream))
    {
        while (!reader.EndOfStream)
        {
            string json = reader.ReadLine();
            Console.WriteLine(json);
        }
    }
};
webClient.OpenReadAsync(new Uri(settings.Twitter.FilteredStreamUrl.AbsoluteUri));

But I can't get it working using RestSharp:

var client = new RestClient($"{settings.Twitter.FilteredStreamUrl}");
var request = new RestRequest(RestSharp.Method.GET);
request.RequestFormat = DataFormat.Json;
request.AddHeader("Content-Type", "application/json");
request.AddHeader("Authorization", "Bearer " + settings.Twitter.BearerToken);
//request.AdvancedResponseWriter = (stream, statuscode) =>
request.ResponseWriter = (stream) =>
{
    using (var reader = new StreamReader(stream))
    {
        while (!reader.EndOfStream)
        {
            //Console.WriteLine(statuscode);
            string json = reader.ReadLine();
            Console.WriteLine(json);
        }
    }
};
client.Execute(request);

Does RestSharp support streaming endpoints? Or am I doing something wrong in my code?

[alexeyzimarev]

What do you mean by "can't get it working"?

[JannemanDev]

The code compiles and I can make a connection and ResponseWriter event is called only once right after Execute and then never again, and I don't receive any data. Twitter sends out keepalive messages (in the form of a blank line) plus it sends any tweets matching one or more rules every 10 seconds orso.

Steps To Reproduce

1. For both requests below be sure to add your Twitter bearer token:
   Authorization: Bearer ....
2. Make a Twitter filtered stream:
   POST /2/tweets/search/stream/rules
   using this in the request body:

{
    "add": [
        {
            "value": "something"
        }
    ]
}

2. Listen to the created stream:
   GET /2/tweets/search/stream

You won't see any responses when using RestSharp and when somebody tweets something. It takes approx. 10 seconds to see the tweets.

When you use curl to listen it works:
curl https://api.twitter.com/2/tweets/search/stream -H "Authorization: Bearer ...."

A typical response (below has been beautified but normally it's minified) when a match is found looks like this:

{
    "data":
    {
        "id": "1472621803210252290",
        "text": "something"
    },
    "matching_rules": [
        {
            "id": "1472621744347361287",
            "tag": "Rule: match when something is tweeted"
        }
    ]
}

[alexeyzimarev]

I'd suggest trying v107 preview. The version you use relies on heavy legacy stuff. There's an async stream reader there, you don't need to use the response writer, but call client.DownloadStreamAsync()

[JannemanDev]

Ah ok thanks. What's the easiest way? Because I can't see it in NuGet package manager in Visual Studio.
Or only by checking out the branch 107.0.0-preview ?
And then in VS2022 rebuild solution and use the RestSharp.dll in my project?

[alexeyzimarev]

What do you mean? All the preview packages are on NuGet.

[JannemanDev]

Ah thanks I missed the checkbox include prerelease. Meanwhile I already managed to compile the source and install the dll :)
For reference for other readers:

[JannemanDev]

Noticed your #1674... cool. Does that mean its impossible currently to read from an constant open streaming connection? Because I am still struggling.

This is what I tried:

            var client = new RestClient("https://api.twitter.com/2/tweets/search/stream");
            var request = new RestRequest(Method.Get)
            {
                ResponseWriter = stream =>
                {
                    using (var reader = new StreamReader(stream))
                    {
                        while (!reader.EndOfStream)
                        {
                            //Console.WriteLine(statuscode);
                            string json = reader.ReadLine();
                            Console.WriteLine(json);
                        }
                    }
                    return stream;
                }
            };

            request.RequestFormat = DataFormat.Json;
            request.AddHeader("Content-Type", "application/json");
            request.AddHeader("Authorization", "Bearer " + "...");

            await client.DownloadStreamAsync(request);

Whatever client method I call (DownloadDataAsync, DownloadStreamAsync, ExecuteAsync) I always get this error:

[alexeyzimarev]

It should be possible with DownloadStreamAsync. Which package version do you use? This issue (#1664) was fixed a few days ago.

The reason for this exception is that you set the Content-Type header for a request without content. Why are you doing that?

Also, normally you would use the JwtAuthenticator instead of adding the Authorization header.

With RestSharp, in 90% of cases you do not need to set any request headers.

Next, you need to remove the response writer. DownloadStreamAsync returns a stream. You should do what you do in the response writer loop, but use the stream returned by DownloadStreamAsync instead.

[JannemanDev]

Thanks for the tips. Also I am now using 107.0.0-preview.13 and I dont get the error but still doesn't work though.
It never makes it pass the DownloadStreamAsync call.

var client = new RestClient("https://api.twitter.com");
var request = new RestRequest("/2/tweets/search/stream", Method.Get);

JwtAuthenticator authentication = new JwtAuthenticator("...");
client.Authenticator = authentication;

Stream stream = client.DownloadStreamAsync(request).Result;

using (var reader = new StreamReader(stream))
{
    while (!reader.EndOfStream)
    {
        string json = reader.ReadLine();
        Console.WriteLine(json);
    }
}

[alexeyzimarev]

Honestly, I never tried using neither RestClient nor HttpClient for streaming endpoints. I will try this Twitter API endpoint during the week.

[JannemanDev]

No rush since I have a workaround using WebClient (see my very first code snippet in my first post in this issue). But I like RestSharp and use it in my whole project so I want to remove WebClient for consistency and because of the extra dependency.
Would be cool if RestSharp would support these kind of endpoints.
thanks for your time

[alexeyzimarev]

Basically, this code works without any changes in the current version:

    public async Task SearchStream() {
        var request = new RestRequest("tweets/search/stream") { CompletionOption = HttpCompletionOption.ResponseHeadersRead };
        var stream  = await _client.DownloadStreamAsync(request);

        var reader = new StreamReader(stream!);
        while (!reader.EndOfStream) {
            var x = await reader.ReadLineAsync();
            Console.WriteLine(x);
        }
    }