Too Many Requests of e2e.updateGroupKey and e2e.getUsersOfRoomWithoutKey

[maviterlikli]

Description:

After a client update, We have started seeing an increased number of requests on the server (v. 6.5.3). We have updated the server to latest (v6.8.0) and increased the resources on the server. However, none of them solved the issue. While we were debugging, we saw that there were lots of requests from clients to the server. The requests consist of 2 main API calls which are e2e.updateGroupKey and e2e.getUsersOfRoomWithoutKey. In under a minute, a client has sent about 2000 requests. The message IDs in the requests change and are in sequential order.

These huge amount of API requests from the clients are causing saturation on the server even tho we have allocated huge amounts of CPU and RAM to the server.

New and old clients has the issue on latest version of Rocket and in older versions

Steps to reproduce:

I do not know how to reproduce the bug. Before the bug, we had no administrative action on the server. It seems to start after a client update.

Expected behavior:

Limited number of API requests.

Actual behavior:

Server Setup Information:

• Version of Rocket.Chat Server: 6.8.0
• Operating System: Linux
• Deployment Method: Docker
• Number of Running Instances: 1
• DB Replicaset Oplog: Default
• NodeJS Version: 14.21.3
• MongoDB Version: 6.0

Client Setup Information

• Desktop App or Browser Version: 3.9.14 (older versions exist)
• Operating System: Linux (Also seen in Mac and Windows)

Additional context

Relevant logs:

We cannot provide unsanitized logs in a public channel.

[reetp]

Thanks for reporting.

Which version of Rocket are you using? CE, Starter, Pro etc?

How many users?

Can you just clarify this?

Before the bug, we had no administrative action on the server. It seems to start after a client update.

I will try and get someone to take a look.

You will have to try and sanitise some of your logs if they show errors.

[maviterlikli]

Thanks for your support.

We are currently using CE.
We have about 100 active users.
The issue mentioned did not occur after a change in the server. It occurred after our clients are updated. I am not 100% sure client update triggered the issue but it seems to be the most probable.

Logs do not show any errors. Only API call logs. and we have an unrelated error on the rocketchat_uploads indexing conflict. Apart from that, we can't see any errors.

[maviterlikli]

Is there any update?

We have also observed that in peak hours, this issue exists. For example, my client did not send huge amounts of requests before work hours but when people started to log in to RocketChat, my client started to send lots of requests as well.

Also, the mobile client does not seems to be effected by the issue.

Best regards.

[reetp]

We are currently using CE. We have about 100 active users.

OK

The issue mentioned did not occur after a change in the server. It occurred after our clients are updated. I am not 100% sure client update triggered the issue but it seems to be the most probable.

Seems so.

Logs do not show any errors. Only API call logs.

They show a call to verifyErrors.js

Has something odd happened with E2E keys somewhere in the client?

Desktop App or Browser Version: 3.9.14 (older versions exist)

Does this only happen on a client? Or does it occur on a browser too?

Are you using just one client app version eg 3.9.14, or different ones?

Is there any update?

This is open source. To access immediate support you need to be on a paid plan. Otherwise you will have to wait and see if Rocket take this one up - I think it is likely a support issue rather than a bug.

I have tried to ping the team to see if someone will look at it.

[reetp]

we have an unrelated error on the rocketchat_uploads indexing conflict

If you searched you would find this

#31876

[reetp]

Also you might want to try latest as it fixes some cache issues and that may be your problem.

https://github.com/RocketChat/Rocket.Chat.Electron/releases

[maviterlikli]

verifyErrors.js does not push any errors to server. In server docker logs shows no error. I am adding the repetitive error on the client below.

Error while fetching an original source: unsupported protocol for sourcemap request meteor://%F0%9F%92%BBapp/%3Canon%3E
Source URL: meteor://%F0%9F%92%BBapp/%3Canon%3E

This seems to go infinitely.
Which includes the error above.

I am adding console logs at the end.

Has something odd happened with E2E keys somewhere in the client?

Not that I know of. It updated automatically as expected and everything was same as the old updates.

Does this only happen on a client? Or does it occur on a browser too?

I have checked on Mozilla. It also occurs in the Mozilla web browser.

Are you using just one client app version eg 3.9.14, or different ones?

Most of us are using client 3.9.14. However, some older versions exist.

This is open source. To access immediate support you need to be on a paid plan. Otherwise you will have to wait and see if Rocket take this one up - I think it is likely a support issue rather than a bug.

Sorry :)

Also you might want to try latest as it fixes some cache issues and that may be your problem.

I will update shortly and let you know.

If you searched you would find this

#31876

I know that the error has a solution, but it is not currently a high priority for us.

In Mozilla Developer Console, the following errors exist:
[BLACK] notation is my sanitation.

Many of:

_send /client/meteorOverrides/ddpOverREST.ts:1
    bindEnvironment https://im.dt.net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:1
    (Async: promise callback)
    then https://[BLACK].net.tr/7[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:126
    _send /client/meteorOverrides/ddpOverREST.ts:1
    sendMessage https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:170
    _apply https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:170
    applyAsync https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:170
    a https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:170
    callAsync https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:170
    callAsync /client/meteorOverrides/totpOnCall.ts:1
    call https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:322
    encryptForParticipant /app/e2e/client/rocketchat.e2e.room.js:1
    encryptKeyForOtherParticipants /app/e2e/client/rocketchat.e2e.room.js:1
    encryptKeyForOtherParticipants /app/e2e/client/rocketchat.e2e.room.js:1
    provideKeyToUser /app/e2e/client/rocketchat.e2e.room.js:1
    n /client/startup/e2e.ts:1
    u https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:322
    emit https://[BLACK]net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:5
    emit https://[BLACK]net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:5
    l https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:322
    onmessage https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140
    forEachCallback https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140
    onmessage https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140
    dispatchEvent https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140
    _transportMessage https://[BLACK]net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140
    _transportMessage https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140
    emit https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140
    onmessage https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140

One of:

Object { stack: "r@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:1:4634\nn@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:170:18842\n_livedata_nosub@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:170:18914\nonMessage@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:170:21669\nbrowser.js/_launchConnection/this.socket.onmessage/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:2677\nforEachCallback@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:3731\nbrowser.js/_launchConnection/this.socket.onmessage@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:2646\nsockjs-1.6.1-min-.js/</</<[5]</r.prototype.dispatchEvent@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:8431\nsockjs-1.6.1-min-.js/</</<[14]</</</_.prototype._transportMessage/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:18864\nsockjs-1.6.1-min-.js/</</<[14]</</</_.prototype._transportMessage@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:18815\nsockjs-1.6.1-min-.js/</</<[3]</s.prototype.emit@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7315\nsockjs-1.6.1-min-.js/</</<[38]</l/this.ws.onmessage@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:44875\nEventHandlerNonNull*l@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:44817\nsockjs-1.6.1-min-.js/</</<[14]</</</_.prototype._connect@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:18052\nsockjs-1.6.1-min-.js/</</<[14]</</</_.prototype._receiveInfo@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:17314\ni@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7111\nsockjs-1.6.1-min-.js/</</<[3]</s.prototype.emit@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7315\nsockjs-1.6.1-min-.js/</</<[12]</f.prototype.doXhr/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:13272\ni@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7111\nsockjs-1.6.1-min-.js/</</<[3]</s.prototype.emit@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7315\nsockjs-1.6.1-min-.js/</</<[9]</a/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:10831\ni@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7111\nsockjs-1.6.1-min-.js/</</<[3]</s.prototype.emit@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7315\nsockjs-1.6.1-min-.js/</</<[17]</</</l.prototype._start/this.xhr.onreadystatechange@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:26226\nEventHandlerNonNull*sockjs-1.6.1-min-.js/</</<[17]</</</l.prototype._start@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:25860\nsockjs-1.6.1-min-.js/</</<[17]</</</l/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:25128\nsetTimeout handler*l@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:25104\ns@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:44294\na@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:10672\nsockjs-1.6.1-min-.js/</</<[12]</f._getReceiver@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:12897\nsockjs-1.6.1-min-.js/</</<[12]</f.prototype.doXhr@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:13088\nsockjs-1.6.1-min-.js/</</<[12]</f/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:12824\nsetTimeout handler*f@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:12800\n_@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:16284\n_launchConnection@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:2415\nindex.js/n.prototype.allowConnection@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:142:373\nmodule/<@/ee/client/ecdh.ts:1:239\nmodule@/ee/client/ecdh.ts:1:1170\nd/o.fetch/s/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:130:856\ng@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:3:1498\nr/d.prototype.require@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:3:3590\no@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:5:4659\nd/s.prototype.dynamicImport/b<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:130:579\nn.bindEnvironment/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:1:8755\npromise callback*promise_client.js/o.makeCompatible/e.prototype.then@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:126:1041\nd/s.prototype.dynamicImport@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:130:548\nmain.ts/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:322:472711\nn.bindEnvironment/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:1:8755\npromise callback*promise_client.js/o.makeCompatible/e.prototype.then@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:126:1041\nmain.ts@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:322:472700\ng@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:3:1498\nr/d.prototype.require@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:3:3590\nt@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:3:1131\n@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:322:530112\n", isClientSafe: true, error: "not-allowed", reason: undefined, details: undefined, message: "[not-allowed]", errorType: "Meteor.Error" }
​
details: undefined
​
error: "not-allowed"
​
errorType: "Meteor.Error"
​
isClientSafe: true
​
message: "[not-allowed]"
​
reason: undefined
​
stack: "r@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:1:4634\nn@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:170:18842\n_livedata_nosub@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:170:18914\nonMessage@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:170:21669\nbrowser.js/_launchConnection/this.socket.onmessage/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:2677\nforEachCallback@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:3731\nbrowser.js/_launchConnection/this.socket.onmessage@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:2646\nsockjs-1.6.1-min-.js/</</<[5]</r.prototype.dispatchEvent@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:8431\nsockjs-1.6.1-min-.js/</</<[14]</</</_.prototype._transportMessage/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:18864\nsockjs-1.6.1-min-.js/</</<[14]</</</_.prototype._transportMessage@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:18815\nsockjs-1.6.1-min-.js/</</<[3]</s.prototype.emit@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7315\nsockjs-1.6.1-min-.js/</</<[38]</l/this.ws.onmessage@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:44875\nEventHandlerNonNull*l@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:44817\nsockjs-1.6.1-min-.js/</</<[14]</</</_.prototype._connect@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:18052\nsockjs-1.6.1-min-.js/</</<[14]</</</_.prototype._receiveInfo@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:17314\ni@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7111\nsockjs-1.6.1-min-.js/</</<[3]</s.prototype.emit@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7315\nsockjs-1.6.1-min-.js/</</<[12]</f.prototype.doXhr/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:13272\ni@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7111\nsockjs-1.6.1-min-.js/</</<[3]</s.prototype.emit@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7315\nsockjs-1.6.1-min-.js/</</<[9]</a/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:10831\ni@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7111\nsockjs-1.6.1-min-.js/</</<[3]</s.prototype.emit@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:7315\nsockjs-1.6.1-min-.js/</</<[17]</</</l.prototype._start/this.xhr.onreadystatechange@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:26226\nEventHandlerNonNull*sockjs-1.6.1-min-.js/</</<[17]</</</l.prototype._start@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:25860\nsockjs-1.6.1-min-.js/</</<[17]</</</l/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:25128\nsetTimeout handler*l@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:25104\ns@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:44294\na@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:10672\nsockjs-1.6.1-min-.js/</</<[12]</f._getReceiver@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:12897\nsockjs-1.6.1-min-.js/</</<[12]</f.prototype.doXhr@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:13088\nsockjs-1.6.1-min-.js/</</<[12]</f/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:12824\nsetTimeout handler*f@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:12800\n_@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:16284\n_launchConnection@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:140:2415\nindex.js/n.prototype.allowConnection@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:142:373\nmodule/<@/ee/client/ecdh.ts:1:239\nmodule@/ee/client/ecdh.ts:1:1170\nd/o.fetch/s/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:130:856\ng@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:3:1498\nr/d.prototype.require@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:3:3590\no@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:5:4659\nd/s.prototype.dynamicImport/b<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:130:579\nn.bindEnvironment/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:1:8755\npromise callback*promise_client.js/o.makeCompatible/e.prototype.then@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:126:1041\nd/s.prototype.dynamicImport@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:130:548\nmain.ts/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:322:472711\nn.bindEnvironment/<@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:1:8755\npromise callback*promise_client.js/o.makeCompatible/e.prototype.then@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:126:1041\nmain.ts@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:322:472700\ng@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:3:1498\nr/d.prototype.require@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:3:3590\nt@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:3:1131\n@https://[BLACK].net.tr/[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js?meteor_js_resource=true:322:530112\n"
​
<prototype>: Object { constructor: r(), clone: clone(), stack: "" }
​​
clone: function clone()
​​
constructor: function r()
​​
stack: ""
​​
<prototype>: Error.prototype { stack: "", … }
[BLACK]b3a98e9af3d443d099d5c7ef5d82cd0e6.js:322:34515

[maviterlikli]

On update,

Are you using just one client app version eg 3.9.14, or different ones?

Some of us are using 3.6.15 and some older versions. I have currently updated it and the issue persists. However, it might be due to other old clients, I am not sure.

[reetp]

Some of us are using 3.6.15 and some older versions. I have currently updated it and the issue persists. However, it might be due to other old clients, I am not sure.

Not sure having a mix of clients is helpful here if you are using E2E.

I have checked on Mozilla. It also occurs in the Mozilla web browser.

OK. Chrome or other browser just to be sure?

Also are you using GridFS for storage or local storage (just dotting i's here) ?

[maviterlikli]

Not sure having a mix of clients is helpful here if you are using E2E.

I am pushing for updates. Soon we should all be using latest.

OK. Chrome or other browser just to be sure?

Chrome Browser behaves as same.

Also are you using GridFS for storage or local storage (just dotting i's here) ?

The server is on a single instance VM that uses local storage. The VM is provisioned in a Cloud environment.

[reetp]

Thanks for all that.

I have asked if someone can take a look. I can't force anyone to do so, but I do think this is worthy of more investigation.

I will be watching this bug so no need to @ anyone.

[maviterlikli]

I have several updates.

Firstly, since the app was almost unusable, we took a risk and disabled and re-enabled the E2E from the admin panel. This resolves the issue. However, we are not sure it will not happen again.

Secondly, we are a technical team and several of our leads and developers looked into the issue and they strongly think that this is a bug.

Thirdly, we observed that the system behaved as expected when most users were offline during off hours. Due to that, ALL of the users do not cause this. Some of the users have issues and they affect all clients.

Best regards.

[reetp]

Thanks for the observations.

This is in the hands of the right people, though no promises on a fix - they need to be able to see it/repeat it.