BroadcastException thrown on production server, but works on local machine

[mtanmay]

I've spent my whole day trying to solve this.

I am using redis as queue connection. I have an event TestUpdated which I am broadcasting on test.{id} private channel. The event gets fired and caught by the client properly when I am on a local machine. But on production server (forge), I get BroadcastException when TestUpdated is fired:

Illuminate\Broadcasting\BroadcastException in /home/forge/mydomain.com/vendor/laravel/framework/src/Illuminate/Broadcasting/Broadcasters/PusherBroadcaster.php:117

apps and ssl config in websockets.php:

'apps' => [
    [
        'id' => env('PUSHER_APP_ID'),
        'name' => env('APP_NAME'),
        'key' => env('PUSHER_APP_KEY'),
        'secret' => env('PUSHER_APP_SECRET'),
        'enable_client_messages' => true,
        'enable_statistics' => false,
    ],
],

'ssl' => [
        'local_cert' => env('LOCAL_CERT', null),
        'local_pk' => env('LOCAL_PK', null),
        'passphrase' => null,
        'verify_peer' => false,
    ],
  

pusher config in broadcasting.php:

'pusher' => [
            'driver' => 'pusher',
            'key' => env('PUSHER_APP_KEY'),
            'secret' => env('PUSHER_APP_SECRET'),
            'app_id' => env('PUSHER_APP_ID'),
            'options' => [
                'cluster' => env('PUSHER_APP_CLUSTER'),
                'encrypted' => true,
                'host' => '127.0.0.1',
                'port' => 6001,
                'scheme' => env('PUSHER_SCHEME'),
                'curl_options' => [
                    CURLOPT_SSL_VERIFYHOST => 0,
                    CURLOPT_SSL_VERIFYPEER => 0,
                ],
            ],
        ],

The relevant environment variables in .env:

PUSHER_APP_ID=someId
PUSHER_APP_KEY=someKey
PUSHER_APP_SECRET=someSecret
PUSHER_APP_CLUSTER=mt1
PUSHER_SCHEME=https

LOCAL_CERT=/etc/nginx/ssl/mydomain.com/123456/server.crt
LOCAL_PK=/etc/nginx/ssl/mydomain.com/123456/server.key

Client-side configuration:

window.Echo = new Echo({
        authEndpoint: 'my/endpoint',
        broadcaster: 'pusher',
        key: 'someKey',
        wsHost: process.env.NODE_ENV == 'development' ? window.location.hostname : 'mydomain.com',
        wsPort: 6001,
        wssPort: 6001,
        disableStats: true,
        encrypted: process.env.NODE_ENV == 'development' ? false : true
    });

Strangely, the presence channel works (could this be related to #149 ?). It's only the broadcasting that seems to be failing.

How do I fix this?

[francislavoie]

To clarify how it works, this lib exposes a WS server and an HTTP server. The HTTP server is used for implementing Pusher's HTTP API (see https://pusher.com/docs/channels/library_auth_reference/rest-api) which includes broadcasting.

This means that whatever service in your stack is broadcasting (in your case, your job queue workers) need to know how to send requests to your laravel-websockets HTTP server.

I don't know enough about Forge to help debug that aspect, but that's what you need to look for. This means you might need to change the 'host' in your broadcasting.php to point to whatever IP/domain your laravel-websockets is running on.

Hopefully when #140 is merged, you'll be able to use Redis for broadcasting instead and this sort of thing would go away. (This is assuming I'm reading your issue correctly)

[mtanmay]

@francislavoie Thanks a lot man! Changing the host fixes the issue :)

[TheBigK]

@francislavoie and @tanmay-das - I'm running into similar issue and have not been able to figure out a fix. Here's the error I'm getting [specifically after upgrading uBuntu to 18.10] -

[2019-05-05 14:02:43] production.ERROR:  {"exception":"[object] (Illuminate\Broadcasting\BroadcastException(code: 0):  at /var/www/site/vendor/laravel/framework/src/Illuminate/Broadcasting/Broadcasters/PusherBroadcaster.php:119)
[stacktrace]
#0 /var/www/site/vendor/laravel/framework/src/Illuminate/Broadcasting/BroadcastEvent.php(48): Illuminate\Broadcasting\Broadcasters\PusherBroadcaster->broadcast(Array, 'App\\Events\\Conv...', Array)
#1 [internal function]: Illuminate\Broadcasting\BroadcastEvent->handle(Object(Illuminate\Broadcasting\Broadcasters\PusherBroadcaster))
#2 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)
#3 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#4 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#5 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/Container.php(580): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#6 /var/www/site/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(94): Illuminate\Container\Container->call(Array)
#7 /var/www/site/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}(Object(Illuminate\Broadcasting\BroadcastEvent))
#8 /var/www/site/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(104): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Broadcasting\BroadcastEvent))
#9 /var/www/site/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(98): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#10 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(49): Illuminate\Bus\Dispatcher->dispatchNow(Object(Illuminate\Broadcasting\BroadcastEvent), false)
#11 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(86): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\SqsJob), Array)
#12 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(327): Illuminate\Queue\Jobs\Job->fire()
#13 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(277): Illuminate\Queue\Worker->process('sqs', Object(Illuminate\Queue\Jobs\SqsJob), Object(Illuminate\Queue\WorkerOptions))
#14 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(118): Illuminate\Queue\Worker->runJob(Object(Illuminate\Queue\Jobs\SqsJob), 'sqs', Object(Illuminate\Queue\WorkerOptions))
#15 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(102): Illuminate\Queue\Worker->daemon('sqs', 'default', Object(Illuminate\Queue\WorkerOptions))
#16 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(86): Illuminate\Queue\Console\WorkCommand->runWorker('sqs', 'default')
#17 [internal function]: Illuminate\Queue\Console\WorkCommand->handle()
#18 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)
#19 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#20 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#21 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/Container.php(580): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#22 /var/www/site/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\Container\Container->call(Array)
#23 /var/www/site/vendor/symfony/console/Command/Command.php(255): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#24 /var/www/site/vendor/laravel/framework/src/Illuminate/Console/Command.php(170): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#25 /var/www/site/vendor/symfony/console/Application.php(908): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#26 /var/www/site/vendor/symfony/console/Application.php(269): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#27 /var/www/site/vendor/symfony/console/Application.php(145): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#28 /var/www/site/vendor/laravel/framework/src/Illuminate/Console/Application.php(90): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#29 /var/www/site/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(122): Illuminate\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#30 /var/www/site/artisan(35): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#31 {main}

Note that I had everything working perfectly fine before the upgrade. Not sure what went wrong. Would really appreciate if you could point me in the right direction. It's almost 1.5 days I'm banging my head on this.

[francislavoie]

That error means your app can't connect to your websockets server over HTTP. The error message is truncated though so it's impossible to tell what the exception is. I don't know enough about how you set up your app to help, you didn't give nearly enough info.

[TheBigK]

Thank you for the comment, @francislavoie . I've set up the websockets server on 127.0.0.1; the same server which runs nginx.

If I turn on the Statistics dashboard, I do see connection being made to the server. However, it's only when the message is broadcasted, that the error crops up.

Could you please let me know what information could I provide to get better help?

[TheBigK]

Here's how I've setup the app on my production server -

• bootstrap.js:

window.Echo = new Echo({
    broadcaster: 'pusher',
    key: process.env.MIX_PUSHER_APP_KEY,
    cluster: process.env.MIX_PUSHER_APP_CLUSTER,
    encrypted: true,
    wsHost: window.location.hostname,
    wsPort: 6001,
    wssPort: 6001,
    disableStats: true,
    enabledTransports: ['ws', 'wss']
});

config/websockets.php

'local_cert' => '/etc/letsencrypt/live/www.mydomain.com/fullchain.pem',
'local_pk' => '/etc/letsencrypt/live/www.mydomain.com/privkey.pem',

'verify_peer' => true,  
// Have ensured that both these files are accessible by the user

Note about verify_peer => true option :
I believe this should be set to true; or commented out. When I set it to false, I get the error pasted in my last post. But if I set it to true, I get following error in Chrome's console -

webSocket connection to 'wss://www.mydomain.com:6001/app/<APP_ID>protocol=7&client=js&version=4.4.0&flash=false' failed: WebSocket opening handshake was canceled

config/broadcasting.php

        'pusher' => [
            'driver' => 'pusher',
            'key' => env('PUSHER_APP_KEY'),
            'secret' => env('PUSHER_APP_SECRET'),
            'app_id' => env('PUSHER_APP_ID'),
            'options' => [
                'cluster' => env('PUSHER_APP_CLUSTER'),
                'encrypted' => true,
                'host'  =>  '127.0.0.1',
                'port'  =>  6001,
                'scheme'    =>  'https',
            ],
        ],

Troubleshooting steps I've tried so far -

1. Tried enabling cacert option in php.ini as follows -

 curl.cainfo="/etc/php/7.3/cacert.pem"
openssl.cafile="/etc/php/7.3/cacert.pem"

It makes no difference; and the error persists.

2. Tried setting encrypted option in both bootstrap.js and broadcasting.php to false. Still, the error persists.

3. Tried reinstalling the php7.3-curl extension. No effect on the error.

4. Tried adjusting ntpdate option as suggested in some of the similar errors. No success.

Also, the error I posted above is straight from the log file generated by Laravel. It shows the truncated version and I have no idea how to get further details from it. The error itself doesn't tell me anything about where the problem is.

Note: This error appeared after I ran do-release-upgrade which upgraded my server from ubuntu 18.04 to 18.04.2. Later, I upgraded it to 18.10; just to see if it fixes the issue. I'm now on uBuntu 18.10; but the error's still there.

@francislavoie - Would really appreciate if you could help me identify the issue. I'm on it since last 72 hours and with zero progress. I thank you for your time and look forward to hearing from you.

[Meera9]

I am tired to getting this type error when start websocket server broadcast function give me error like below, please can anyone help me to sort out?

[kirilkirkov]

@Meera9 Did you find a fix?

[DeBelserArne]

I am tired to getting this type error when start websocket server broadcast function give me error like below, please can anyone help me to sort out?

Hey @Meera9 I've had these kind of errors pop up in my dev tools whenever there was something wrong when sending a Notification.

Things that sometimes helped me out:

• Clear browser cache
• Clear the chrome cache in dev tools, press F12, go to Application and press "Clear site data"
  
• If you have any middleware that protects your routes, make sure you are logged into into your system when triggering websockets.

[loginweb-dev]

I have the same problem, it works local but in a production server with ssl the events are not sent.

[shahriar-b]

I have the same problem, it works local but in a production server with ssl the events are not sent.

same here

[glebk1]

same here

[glebk1]

Please, help

[verenzuela]

any solution...? :'(