Need help with log message (LLHLS managed queue size has exceeded the threshold) #1627

mbakumenkov · 2024-05-28T17:56:49Z

mbakumenkov
May 28, 2024

Recently I have the message in server logs:
[2024-05-28 17:23:31.225] W [AW-LLHLS0:19] ManagedQueue | managed_queue.h:444 | [4051] mngq:v=#default#live:s=stream:p=pub:n=streamworker_llhls size has exceeded the threshold: queue: 937, threshold: 500, peak: 937

All outgoing traffic from server just got stopped (dropped to zero), and queue size continue rising until I restarted ome docker container.
Please help me understand what could be wrong and how this can be avoided.

bchah · 2024-05-28T20:45:28Z

bchah
May 28, 2024
Collaborator Sponsor

It's likely the threads devoted to serving HLS have become overwhelmed. In your server.xml you can set the thread count for StreamWorker and OutboundWorker. I believe LLHLS is set by OutboundWorker. Try increasing this to 8 or more depending on available CPU threads.

See: https://airensoft.gitbook.io/ovenmediaengine/performance-tuning#tuning-the-number-of-threads

7 replies

mbakumenkov May 29, 2024
Author

OME version 0.16.5 (docker)
There are 13 streams on the server
We are using AWS c5.4xlarge instances (16 vCPU, 32GB RAM) (its loaded ~30% avg without any peaks)
Bitrate of each stream is 2500 Kbit/s, it comes in 720p RTMP to the server and then transcode to 720p(bypass), 480p, 360p

Server.xml (some sensible data changed)

<?xml version="1.0" encoding="UTF-8"?>
<Server version="8">
        <Name>OvenMediaEngine</Name>
        <!-- Host type (origin/edge) -->
        <Type>origin</Type>
        <!-- Specify IP address to bind (* means all IPs) -->
        <IP>*</IP>
        <PrivacyProtection>false</PrivacyProtection>

        <!--
        To get the public IP address(mapped address of stun) of the local server.
        This is useful when OME cannot obtain a public IP from an interface, such as AWS or docker environment.
        If this is successful, you can use ${PublicIP} in your settings.
        -->
        <StunServer>stun.l.google.com:19302</StunServer>

        <Modules>
                <!--
                Currently OME only supports h2 like all browsers do. Therefore, HTTP/2 only works on TLS ports.
                -->
                <HTTP2>
                        <Enable>true</Enable>
                </HTTP2>

                <LLHLS>
                        <Enable>true</Enable>
                </LLHLS>

                <!-- P2P works only in WebRTC and is experiment feature -->
                <P2P>
                        <!-- disabled by default -->
                        <Enable>false</Enable>
                        <MaxClientPeersPerHostPeer>2</MaxClientPeersPerHostPeer>
                </P2P>
        </Modules>

        <!-- Settings for the ports to bind -->
        <Bind>
                <!-- Enable this configuration if you want to use API Server -->

                <Managers>
                        <API>
                                <Port>9091</Port>
                                <TLSPort>9092</TLSPort>
                                <WorkerCount>1</WorkerCount>
                        </API>
                </Managers>

                <Providers>
                        <RTMP>
                                <Port>19935</Port>
                                <WorkerCount>1</WorkerCount>
                        </RTMP>
                </Providers>

                <Publishers>
                        <LLHLS>
                                <!--
                                OME only supports h2, so LLHLS works over HTTP/1.1 on non-TLS port.
                                Note that LLHLS runs higher performance over HTTP/2.
                                Therefore, it is recommended to use TLS Port.
                                -->
                                <Port>${env:OME_LLHLS_STREAM_PORT:7070}</Port>
                                <TLSPort>${env:OME_LLHLS_STREAM_TLS_PORT:4443}</TLSPort>
                                <WorkerCount>1</WorkerCount>
                        </LLHLS>
                </Publishers>
        </Bind>

        <Managers>
                <Host>
                        <Names>
                                <Name>*</Name>
                        </Names>
                        <TLS>
                                <CertPath>/opt/ovenmediaengine/bin/origin_conf/certs/cert.pem</CertPath>
                                <KeyPath>/opt/ovenmediaengine/bin/origin_conf/certs/privkey.pem</KeyPath>
                                <ChainCertPath>/opt/ovenmediaengine/bin/origin_conf/certs/fullchain.pem</ChainCertPath>
                        </TLS>
                </Host>
                <API>
                        <AccessToken>xxxxxxxxxxx</AccessToken>
                        <CrossDomains>
                                <Url>*</Url>
                        </CrossDomains>
                </API>
        </Managers>

        <!-- P2P works only in WebRTC -->
        <!--
        <P2P>
                <MaxClientPeersPerHostPeer>2</MaxClientPeersPerHostPeer>
        </P2P>
        -->

        <VirtualHosts>
                <!-- You can use wildcard like this to include multiple XMLs -->
                <VirtualHost include="VHost*.xml" />
                <VirtualHost>
                        <Name>default</Name>
                        <!--Distribution is a value that can be used when grouping the same vhost distributed across multiple servers. This value is output to the events log, so you can use it to aggregate statistics. -->
                        <Distribution>ovenmediaengine.com</Distribution>

                        <!-- Settings for multi ip/domain and TLS -->
                        <Host>
                                <Names>
                                        <!-- Host names
                                                <Name>stream1.airensoft.com</Name>
                                                <Name>stream2.airensoft.com</Name>
                                                <Name>*.sub.airensoft.com</Name>
                                                <Name>192.168.0.1</Name>
                                        -->
                                        <Name>*</Name>
                                </Names>
                                <TLS>
                                        <CertPath>/opt/ovenmediaengine/bin/origin_conf/certs/cert.pem</CertPath>
                                        <KeyPath>/opt/ovenmediaengine/bin/origin_conf/certs/privkey.pem</KeyPath>
                                        <ChainCertPath>/opt/ovenmediaengine/bin/origin_conf/certs/fullchain.pem</ChainCertPath>
                                </TLS>
                        </Host>

                        <!-- Refer https://airensoft.gitbook.io/ovenmediaengine/signedpolicy
                        -->

                        <SignedPolicy>
                                <PolicyQueryKeyName>${env:SIGNED_POLICY_POLICY_QUERY_KEY:policy}</PolicyQueryKeyName>
                                <SignatureQueryKeyName>${env:SIGNED_POLICY_SIGNATURE_QUERY_KEY:signature}</SignatureQueryKeyName>
                                <SecretKey>${env:SIGNED_URL_CRYPTO_KEY:xxxxxxxxxxxxx}</SecretKey>

                                <Enables>
                                        <Providers>rtmp</Providers>
                                </Enables>
                        </SignedPolicy>

                        <!-- Settings for ProxyPass (It can specify origin for each path) -->
                        <!--
                        When Edge tries to pull a stream, it looks first in <Origins> in local, and in OriginMapStore.
                        Therefore, it is recommended to disable Origins when using OriginMapStore.
                        <OriginMapStore>
                                In order to use OriginMap, you must enable OVT Publisher in Origin and OVT Provider in Edge.
                                <RedisServer>
                                        <Host>192.168.0.160:6379</Host>
                                        <Auth>!@#ovenmediaengine</Auth>
                                </RedisServer>
                        </OriginMapStore>
                        -->

                        <!--
                                Apply the following settings for /app/<stream_name> requests.

                                This demonstrates setting <OutputProfiles>/<Providers>/<Pubishers> for a specific application.

                                (https://github.com/AirenSoft/OvenMediaEngine/issues/621#issuecomment-1026892776)
                        -->
                        <Applications>
                                <Application>
                                        <Name>live</Name>
                                        <Type>live</Type>
                                        <OutputProfiles>
                                                <OutputProfile>
                                                        <Name>stream</Name>
                                                        <OutputStreamName>${OriginStreamName}</OutputStreamName>
                                                        <Playlist>
                                                                <Name>playlist</Name>
                                                                <FileName>playlist</FileName>
                                                                <Rendition>
                                                                        <Name>720p</Name>
                                                                        <Video>720p</Video>
                                                                        <Audio>pass</Audio>
                                                                </Rendition>
                                                                <Rendition>
                                                                        <Name>480p</Name>
                                                                        <Video>480p</Video>
                                                                        <Audio>pass</Audio>
                                                                </Rendition>
                                                                <Rendition>
                                                                        <Name>360p</Name>
                                                                        <Video>360p</Video>
                                                                        <Audio>pass</Audio>
                                                                </Rendition>
                                                        </Playlist>

                                                        <Encodes>
                                                                <Video>
                                                                        <Name>720p</Name>
                                                                        <Bypass>true</Bypass>
                                                                </Video>
                                                                <Video>
                                                                        <Name>480p</Name>
                                                                        <Codec>h264</Codec>
                                                                        <Bitrate>1000000</Bitrate>
                                                                        <Framerate>30</Framerate>
                                                                        <Width>854</Width>
                                                                        <Height>480</Height>
                                                                        <Preset>medium</Preset>
                                                                        <Profile>main</Profile>
                                                                        <KeyFrameInterval>60</KeyFrameInterval>
                                                                        <ThreadCount>4</ThreadCount>
                                                                </Video>
                                                                <Video>
                                                                        <Name>360p</Name>
                                                                        <Codec>h264</Codec>
                                                                        <Bitrate>400000</Bitrate>
                                                                        <Framerate>30</Framerate>
                                                                        <Width>640</Width>
                                                                        <Height>360</Height>
                                                                        <Preset>fast</Preset>
                                                                        <Profile>baseline</Profile>
                                                                        <KeyFrameInterval>60</KeyFrameInterval>
                                                                        <ThreadCount>2</ThreadCount>
                                                                </Video>

                                                                <Audio>
                                                                        <Name>pass</Name>
                                                                        <Bypass>true</Bypass>
                                                                </Audio>
                                                        </Encodes>
                                                </OutputProfile>
                                        </OutputProfiles>
                                        <Providers>
                                                <RTMP>
                                                        <BlockDuplicateStreamName>true</BlockDuplicateStreamName>
                                                </RTMP>
                                        </Providers>
                                        <Publishers>
                                                <AppWorkerCount>1</AppWorkerCount>
                                                <StreamWorkerCount>16</StreamWorkerCount>
                                                <LLHLS>
                                                        <OriginMode>true</OriginMode>
                                                        <ChunkDuration>1.0</ChunkDuration>
                                                        <SegmentDuration>2</SegmentDuration>
                                                        <SegmentCount>10</SegmentCount>
                                                        <CrossDomains>
                                                                <Url>list</Url>
                                                                <Url>of</Url>
                                                                <Url>my</Url>
                                                                <Url>domains</Url>
                                                        </CrossDomains>
                                                </LLHLS>
                                                <FILE>
                                                        <RootPath>/var/records</RootPath>
                                                        <FilePath>/${Stream}/${Id}.mp4</FilePath>
                                                        <InfoPath>/${VirtualHost}/${Application}/${Stream}.xml</InfoPath>
                                                        <Record>
                                                                <Enable>true</Enable>
                                                                <RecordInfo>./record_info.xml</RecordInfo>
                                                        </Record>
                                                </FILE>
                                        </Publishers>
                                </Application>
                        </Applications>
                </VirtualHost>
        </VirtualHosts>
</Server>

About the log file - it`s so big, I cut it according to the playback time of the problem

[2024-05-28 17:19:38.201] I [SPLLHLS-t443:12] HTTP.Server | http_server.cpp:149  | Client(<ClientSocket: 0x7f8698e79010, #192, Connected, TCP, Nonblocking, 64.252.105.198:37806>) is connected on *:443
[2024-05-28 17:19:38.210] I [SPLLHLS-t443:12] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream6 on the LLHLS publisher. Concurrent Viewers[LLHLS(476)/Stream total(476)/App total(3261)]
[2024-05-28 17:19:38.210] I [SPLLHLS-t443:12] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(860)/Stream total(860)/App total(3260)]
[2024-05-28 17:19:38.210] I [SPLLHLS-t443:12] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream3 on the LLHLS publisher. Concurrent Viewers[LLHLS(711)/Stream total(711)/App total(3259)]
[2024-05-28 17:19:38.210] I [SPLLHLS-t443:12] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream7 on the LLHLS publisher. Concurrent Viewers[LLHLS(583)/Stream total(583)/App total(3258)]
[2024-05-28 17:19:38.210] I [SPLLHLS-t443:12] HTTP.Server | http_server.cpp:223  | Client(<ClientSocket: 0x7f8774102010, #688, Disconnected, TCP, Nonblocking, 64.252.89.202:54324>) has disconnected from *:443
[2024-05-28 17:19:38.217] I [StreamWorker:8040] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#live/stream6 on the LLHLS publisher. LLHLS(477)/Stream total(477)/App total(3259)
[2024-05-28 17:19:38.221] I [SPLLHLS-t443:12] HTTP.Server | http_connection.cpp:147  | TLS connection accepted : Server Name(bt32e10yctu1vfx.cloudfront.net) Alpn Protocol() Client (<ClientSocket: 0x7f8774082210, #297, Connected, TCP, Nonblocking, 64.252.104.199:62794>)
[2024-05-28 17:19:38.222] I [StreamWorker:7782] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#live/stream7 on the LLHLS publisher. LLHLS(584)/Stream total(584)/App total(3260)
[2024-05-28 17:19:38.227] I [StreamWorker:8044] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#live/stream6 on the LLHLS publisher. LLHLS(478)/Stream total(478)/App total(3261)
[2024-05-28 17:19:38.235] I [StreamWorker:7773] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#live/stream7 on the LLHLS publisher. LLHLS(585)/Stream total(585)/App total(3262)
[2024-05-28 17:19:38.243] I [SPLLHLS-t443:12] HTTP.Server | http_server.cpp:149  | Client(<ClientSocket: 0x7f8699410c10, #359, Connected, TCP, Nonblocking, 64.252.87.98:61084>) is connected on *:443
[2024-05-28 17:19:38.243] E [StreamWorker:7788] Socket.Pool.Worker | socket_pool_worker.cpp:803  | [#7] [0x7f8778a81510] Could not delete the socket #127 from epoll: [errno] No such file or directory (2)
#0   /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b61dfc4d ? + 0x4fdc4d
#1   /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b61d0fe9 ? + 0x4eefe9
#2   /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b61d25fa ? + 0x4f05fa
#3   /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b61d61e8 ? + 0x4f41e8
#4   /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b61d66d6 ? + 0x4f46d6
#5   /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b60d27bc ? + 0x3f07bc
#6   /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b60dfdbd ? + 0x3fddbd
#7   /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b60d2585 ? + 0x3f0585
#8   /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b5e7c8fd ? + 0x19a8fd
#9   /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b5e7d320 ? + 0x19b320
#10  /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b5e7fc19 ? + 0x19dc19
#11  /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b5e7fd3b ? + 0x19dd3b
#12  /opt/ovenmediaengine/bin/OvenMediaEngine 0x55e2b61acaf6 ? + 0x4caaf6
#13  /lib/x86_64-linux-gnu/libstdc++.so.6 0x7f8779868253 ? + 0xdc253
#14  /lib/x86_64-linux-gnu/libc.so.6     0x7f87794f0ac3 ? + 0x94ac3
#15  /lib/x86_64-linux-gnu/libc.so.6     0x7f8779581a04 clone + 0x44

[2024-05-28 17:19:38.243] W [StreamWorker:7788] Socket | socket.cpp:2109 | [#127] [0x7f86990a2a10] Socket was closed even though there was 1 commands in the socket
[2024-05-28 17:19:38.369] I [StreamWorker:8010] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#live/stream3 on the LLHLS publisher. LLHLS(712)/Stream total(712)/App total(3263)
[2024-05-28 17:19:38.377] I [StreamWorker:8025] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#live/stream2 on the LLHLS publisher. LLHLS(861)/Stream total(861)/App total(3264)
[2024-05-28 17:19:38.377] I [StreamWorker:8026] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#live/stream2 on the LLHLS publisher. LLHLS(862)/Stream total(862)/App total(3265)
[2024-05-28 17:19:38.487] I [StreamWorker:8008] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#live/stream3 on the LLHLS publisher. LLHLS(713)/Stream total(713)/App total(3266)
[2024-05-28 17:19:38.955] I [StreamWorker:8144] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#live/stream8 on the LLHLS publisher. LLHLS(103)/Stream total(103)/App total(3267)
[2024-05-28 17:19:39.048] I [StreamWorker:6637] Monitor | stream_metrics.cpp:114  | A new session has started playing #default#live/stream5 on the LLHLS publisher. LLHLS(80)/Stream total(80)/App total(3268)
[2024-05-28 17:19:39.275] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:19:39.897] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream4) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:19:42.231] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8699156210, #50, Connected, TCP, Nonblocking, 64.252.82.254:5660>) has timed out
[2024-05-28 17:19:42.231] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream7 on the LLHLS publisher. Concurrent Viewers[LLHLS(584)/Stream total(584)/App total(3267)]
[2024-05-28 17:19:42.231] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8698e9fa10, #275, Connected, TCP, Nonblocking, 64.252.105.15:34776>) has timed out
[2024-05-28 17:19:42.232] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(861)/Stream total(861)/App total(3266)]
[2024-05-28 17:19:42.232] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8698fcd810, #500, Connected, TCP, Nonblocking, 64.252.104.214:34696>) has timed out
[2024-05-28 17:19:42.232] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(860)/Stream total(860)/App total(3265)]
[2024-05-28 17:19:42.232] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f876cf80210, #373, Connected, TCP, Nonblocking, 64.252.101.103:48708>) has timed out
[2024-05-28 17:19:42.232] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream10 on the LLHLS publisher. Concurrent Viewers[LLHLS(367)/Stream total(367)/App total(3264)]
[2024-05-28 17:19:42.232] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f869905e810, #85, Connected, TCP, Nonblocking, 64.252.88.55:49876>) has timed out
[2024-05-28 17:19:42.232] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream3 on the LLHLS publisher. Concurrent Viewers[LLHLS(712)/Stream total(712)/App total(3263)]
[2024-05-28 17:19:42.232] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream7 on the LLHLS publisher. Concurrent Viewers[LLHLS(583)/Stream total(583)/App total(3262)]
[2024-05-28 17:19:42.232] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f85dddab210, #608, Connected, TCP, Nonblocking, 64.252.86.172:51288>) has timed out
[2024-05-28 17:19:42.233] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(859)/Stream total(859)/App total(3261)]
[2024-05-28 17:19:42.233] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f876cfdf810, #347, Connected, TCP, Nonblocking, 3.29.57.26:10758>) has timed out
[2024-05-28 17:19:42.233] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(858)/Stream total(858)/App total(3260)]
[2024-05-28 17:19:42.233] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f86991d7410, #212, Connected, TCP, Nonblocking, 64.252.84.19:57396>) has timed out
[2024-05-28 17:19:42.233] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(857)/Stream total(857)/App total(3259)]
[2024-05-28 17:19:42.233] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f87740f6210, #404, Connected, TCP, Nonblocking, 64.252.87.28:44832>) has timed out
[2024-05-28 17:19:42.233] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream6 on the LLHLS publisher. Concurrent Viewers[LLHLS(477)/Stream total(477)/App total(3258)]
[2024-05-28 17:19:42.233] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream7 on the LLHLS publisher. Concurrent Viewers[LLHLS(582)/Stream total(582)/App total(3257)]
[2024-05-28 17:19:42.233] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8699176610, #246, Connected, TCP, Nonblocking, 64.252.86.98:37404>) has timed out
[2024-05-28 17:19:42.234] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream3 on the LLHLS publisher. Concurrent Viewers[LLHLS(711)/Stream total(711)/App total(3256)]
[2024-05-28 17:19:42.234] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8698ff8210, #531, Connected, TCP, Nonblocking, 64.252.84.51:57688>) has timed out
[2024-05-28 17:19:42.234] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream6 on the LLHLS publisher. Concurrent Viewers[LLHLS(476)/Stream total(476)/App total(3255)]
[2024-05-28 17:19:42.234] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f86991a4410, #752, Connected, TCP, Nonblocking, 64.252.69.102:59228>) has timed out
[2024-05-28 17:19:42.234] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(856)/Stream total(856)/App total(3254)]
[2024-05-28 17:19:42.235] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream3 on the LLHLS publisher. Concurrent Viewers[LLHLS(710)/Stream total(710)/App total(3253)]
[2024-05-28 17:19:42.235] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8698e65e10, #530, Connected, TCP, Nonblocking, 64.252.89.209:23802>) has timed out
[2024-05-28 17:19:42.235] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream6 on the LLHLS publisher. Concurrent Viewers[LLHLS(475)/Stream total(475)/App total(3252)]
[2024-05-28 17:19:42.235] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8672658c10, #869, Connected, TCP, Nonblocking, 64.252.85.108:52158>) has timed out
[2024-05-28 17:19:42.235] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream7 on the LLHLS publisher. Concurrent Viewers[LLHLS(581)/Stream total(581)/App total(3251)]
[2024-05-28 17:19:43.148] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream9) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:19:43.149] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream7) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:19:43.171] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream10) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:19:43.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:19:47.235] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8698e4b410, #536, Connected, TCP, Nonblocking, 64.252.98.34:7494>) has timed out
[2024-05-28 17:19:47.236] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(855)/Stream total(855)/App total(3250)]
[2024-05-28 17:19:47.236] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8699176410, #148, Connected, TCP, Nonblocking, 64.252.66.24:30720>) has timed out
[2024-05-28 17:19:47.236] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream3 on the LLHLS publisher. Concurrent Viewers[LLHLS(709)/Stream total(709)/App total(3249)]
[2024-05-28 17:19:47.236] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f86727a8610, #620, Connected, TCP, Nonblocking, 64.252.89.160:28292>) has timed out
[2024-05-28 17:19:47.236] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream7 on the LLHLS publisher. Concurrent Viewers[LLHLS(580)/Stream total(580)/App total(3248)]
[2024-05-28 17:19:47.236] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8672665e10, #94, Connected, TCP, Nonblocking, 64.252.101.210:28122>) has timed out
[2024-05-28 17:19:47.236] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream6 on the LLHLS publisher. Concurrent Viewers[LLHLS(474)/Stream total(474)/App total(3247)]
[2024-05-28 17:19:47.236] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f87740b3010, #278, Connected, TCP, Nonblocking, 64.252.104.192:23570>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(854)/Stream total(854)/App total(3246)]
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8699438e10, #758, Connected, TCP, Nonblocking, 64.252.67.129:9788>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream3 on the LLHLS publisher. Concurrent Viewers[LLHLS(708)/Stream total(708)/App total(3245)]
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8698fcd610, #99, Connected, TCP, Nonblocking, 13.124.199.172:60054>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(853)/Stream total(853)/App total(3244)]
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f86992cfe10, #105, Connected, TCP, Nonblocking, 64.252.104.253:43030>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(852)/Stream total(852)/App total(3243)]
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8699156210, #50, Connected, TCP, Nonblocking, 64.252.82.254:5660>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8698e9fa10, #275, Connected, TCP, Nonblocking, 64.252.105.15:34776>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f86990dee10, #828, Connected, TCP, Nonblocking, 64.252.102.2:53678>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(851)/Stream total(851)/App total(3242)]
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 2 sessions has been stopped playing #default#live/stream3 on the LLHLS publisher. Concurrent Viewers[LLHLS(706)/Stream total(706)/App total(3240)]
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8698fcd810, #500, Connected, TCP, Nonblocking, 64.252.104.214:34696>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f876cf80210, #373, Connected, TCP, Nonblocking, 64.252.101.103:48708>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f869905e810, #85, Connected, TCP, Nonblocking, 64.252.88.55:49876>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f85dddab210, #608, Connected, TCP, Nonblocking, 64.252.86.172:51288>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f876cfdf810, #347, Connected, TCP, Nonblocking, 3.29.57.26:10758>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f86991d7410, #212, Connected, TCP, Nonblocking, 64.252.84.19:57396>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f87740f6210, #404, Connected, TCP, Nonblocking, 64.252.87.28:44832>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8699176610, #246, Connected, TCP, Nonblocking, 64.252.86.98:37404>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8698ff8210, #531, Connected, TCP, Nonblocking, 64.252.84.51:57688>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f86991a4410, #752, Connected, TCP, Nonblocking, 64.252.69.102:59228>) has timed out
[2024-05-28 17:19:47.237] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f869929d210, #789, Connected, TCP, Nonblocking, 64.252.98.187:39816>) has timed out
[2024-05-28 17:19:47.238] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream6 on the LLHLS publisher. Concurrent Viewers[LLHLS(473)/Stream total(473)/App total(3239)]
[2024-05-28 17:19:47.238] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(850)/Stream total(850)/App total(3238)]
[2024-05-28 17:19:47.238] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream3 on the LLHLS publisher. Concurrent Viewers[LLHLS(705)/Stream total(705)/App total(3237)]
[2024-05-28 17:19:47.238] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream7 on the LLHLS publisher. Concurrent Viewers[LLHLS(579)/Stream total(579)/App total(3236)]
[2024-05-28 17:19:47.238] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream10 on the LLHLS publisher. Concurrent Viewers[LLHLS(366)/Stream total(366)/App total(3235)]
[2024-05-28 17:19:47.238] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8698e65e10, #530, Connected, TCP, Nonblocking, 64.252.89.209:23802>) has timed out
[2024-05-28 17:19:47.238] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8672658c10, #869, Connected, TCP, Nonblocking, 64.252.85.108:52158>) has timed out
[2024-05-28 17:19:47.238] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f877404da10, #480, Connected, TCP, Nonblocking, 64.252.88.54:40494>) has timed out
[2024-05-28 17:19:47.239] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream2 on the LLHLS publisher. Concurrent Viewers[LLHLS(849)/Stream total(849)/App total(3234)]
[2024-05-28 17:19:47.239] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream7 on the LLHLS publisher. Concurrent Viewers[LLHLS(578)/Stream total(578)/App total(3233)]
[2024-05-28 17:19:47.291] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:19:48.188] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream3) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:19:48.207] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream2) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:19:48.254] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream6) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:19:51.272] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:19:52.239] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8672666610, #240, Connected, TCP, Nonblocking, 64.252.88.246:6902>) has timed out
[2024-05-28 17:19:52.240] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream3 on the LLHLS publisher. Concurrent Viewers[LLHLS(704)/Stream total(704)/App total(3232)]
[2024-05-28 17:19:52.240] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f86724bac10, #678, Connected, TCP, Nonblocking, 64.252.86.45:32976>) has timed out
[2024-05-28 17:19:52.240] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream7 on the LLHLS publisher. Concurrent Viewers[LLHLS(577)/Stream total(577)/App total(3231)]
[2024-05-28 17:19:52.240] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f87740de010, #446, Connected, TCP, Nonblocking, 64.252.88.71:16526>) has timed out
[2024-05-28 17:19:52.241] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream3 on the LLHLS publisher. Concurrent Viewers[LLHLS(703)/Stream total(703)/App total(3230)]
[2024-05-28 17:19:52.241] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f8699132a10, #390, Connected, TCP, Nonblocking, 64.252.87.148:63966>) has timed out
[2024-05-28 17:19:52.241] I [DQHTTPTimer:11] Monitor | stream_metrics.cpp:163  | 1 sessions has been stopped playing #default#live/stream3 on the LLHLS publisher. Concurrent Viewers[LLHLS(702)/Stream total(702)/App total(3229)]
[2024-05-28 17:19:52.241] I [DQHTTPTimer:11] HTTP.Server | http_connection.cpp:100  | Client(HTTP/1.1 - <ClientSocket: 0x7f867250d810, #719, Connected, TCP, Nonblocking, 64.252.86.232:34950>) has timed out
[2024-05-28 17:19:55.275] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:19:59.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:03.275] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:03.581] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream5) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:07.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:09.945] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream4) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:09.961] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream11) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:11.275] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:15.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:16.223] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:16.244] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream8) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:19.274] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:22.588] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream11) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:23.284] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:27.274] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:31.271] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:31.892] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream4) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:35.146] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream9) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:35.160] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream7) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:35.171] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream10) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:35.274] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:39.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:40.199] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream2) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:40.201] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream3) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:40.239] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream6) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:43.275] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:47.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:51.275] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:55.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:55.570] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream5) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:20:59.274] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:01.977] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream11) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:01.993] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream4) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:03.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:07.291] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:10.194] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:10.219] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream8) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:11.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:14.619] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream11) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:15.275] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:19.271] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:23.274] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:23.911] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream4) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:27.164] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream9) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:27.191] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream10) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:27.192] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream7) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:27.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:31.275] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:34.169] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream3) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:34.174] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream2) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:34.209] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream6) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:35.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:39.291] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:43.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:47.234] W [AW-LLHLS0:19] ManagedQueue | managed_queue.h:444  | [4051] mngq:v=#default#live:s=stream7:p=pub:n=streamworker_llhls size has exceeded the threshold: queue: 519, threshold: 500, peak: 519
[2024-05-28 17:21:47.275] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:49.544] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream5) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:51.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:53.232] W [AW-LLHLS0:19] ManagedQueue | managed_queue.h:444  | [4051] mngq:v=#default#live:s=stream7:p=pub:n=streamworker_llhls size has exceeded the threshold: queue: 543, threshold: 500, peak: 543
[2024-05-28 17:21:55.275] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:55.944] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream4) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:55.948] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream11) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:21:58.278] W [AW-LLHLS0:19] ManagedQueue | managed_queue.h:444  | [4051] mngq:v=#default#live:s=stream7:p=pub:n=streamworker_llhls size has exceeded the threshold: queue: 564, threshold: 500, peak: 564
[2024-05-28 17:21:59.271] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:02.187] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:02.218] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream8) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:03.275] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:04.272] W [AW-LLHLS0:19] ManagedQueue | managed_queue.h:444  | [4051] mngq:v=#default#live:s=stream7:p=pub:n=streamworker_llhls size has exceeded the threshold: queue: 588, threshold: 500, peak: 588
[2024-05-28 17:22:06.609] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream11) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:07.272] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:10.238] W [AW-LLHLS0:19] ManagedQueue | managed_queue.h:444  | [4051] mngq:v=#default#live:s=stream7:p=pub:n=streamworker_llhls size has exceeded the threshold: queue: 611, threshold: 500, peak: 611
[2024-05-28 17:22:11.291] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:15.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:16.229] W [AW-LLHLS0:19] ManagedQueue | managed_queue.h:444  | [4051] mngq:v=#default#live:s=stream7:p=pub:n=streamworker_llhls size has exceeded the threshold: queue: 635, threshold: 500, peak: 635
[2024-05-28 17:22:17.896] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream4) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:19.274] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:21.143] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream7) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:21.144] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream9) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:21.159] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream10) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:21.277] W [AW-LLHLS0:19] ManagedQueue | managed_queue.h:444  | [4051] mngq:v=#default#live:s=stream7:p=pub:n=streamworker_llhls size has exceeded the threshold: queue: 657, threshold: 500, peak: 657
[2024-05-28 17:22:23.285] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:26.167] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream3) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:26.182] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream2) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:26.208] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream6) / track (3) - a longer-than-expected (2944.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:27.291] W [AW-LLHLS0:19] FMP4 Packager | fmp4_storage.cpp:341  | LLHLS stream (#default#live/stream1) / track (0) - a longer-than-expected (4000.0 | expected : 2000) segment has created. Long or irregular intervals between keyframes might be the cause.
[2024-05-28 17:22:28.013] W [AW-LLHLS0:19] ManagedQueue | managed_queue.h:444  | [4051] mngq:v=#default#live:s=stream7:p=pub:n=streamworker_llhls size has exceeded the threshold: queue: 683, threshold: 500, peak: 683

And this managed queue keep rising until I restarted the server...

getroot May 30, 2024
Maintainer Sponsor

The number of concurrent viewers seems to be high. How many viewers are actually watching? What is the network output bandwidth?

First of all, it would be a good idea to increase the WorkerCount of the LLHLS Port to about 4.

<Publishers>
 <LLHLS>
 <!--
 OME only supports h2, so LLHLS works over HTTP/1.1 on non-TLS port.
 Note that LLHLS runs higher performance over HTTP/2.
 Therefore, it is recommended to use TLS Port.
 -->
 <Port>${env:OME_LLHLS_STREAM_PORT:7070}</Port>
 <TLSPort>${env:OME_LLHLS_STREAM_TLS_PORT:4443}</TLSPort>
 <WorkerCount>1</WorkerCount>
 </LLHLS>
 </Publishers>

If this still doesn't work, it would be a good idea to distribute traffic using a cache server such as CDN or Nginx.

mbakumenkov May 31, 2024
Author

I`m so sorry for a long reply.
So this is the output traffic of one of our servers (zero interval - is the moment of problem, described higher)

As you can see, it`s not too much.
P.S. we are using AWS CloudFront with different cache policies for manifests/chunks
P.P.S following your advice, I increased the LLHS worker count to 4 I'll watch what happens and write here if we will meet problem again

Thank you for your reply!

getroot Jun 1, 2024
Maintainer Sponsor

it’s looks like deadlock problem. Could you share your full log file that includes problem situation to [email protected]?

mbakumenkov Jun 1, 2024
Author

sure, sent email with google drive log link

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Need help with log message (LLHLS managed queue size has exceeded the threshold) #1627

{{title}}

Replies: 1 comment 7 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Need help with log message (LLHLS managed queue size has exceeded the threshold) #1627

mbakumenkov May 28, 2024

Replies: 1 comment · 7 replies

bchah May 28, 2024 Collaborator Sponsor

mbakumenkov May 29, 2024 Author

getroot May 30, 2024 Maintainer Sponsor

mbakumenkov May 31, 2024 Author

getroot Jun 1, 2024 Maintainer Sponsor

mbakumenkov Jun 1, 2024 Author

mbakumenkov
May 28, 2024

Replies: 1 comment 7 replies

bchah
May 28, 2024
Collaborator Sponsor

mbakumenkov May 29, 2024
Author

getroot May 30, 2024
Maintainer Sponsor

mbakumenkov May 31, 2024
Author

getroot Jun 1, 2024
Maintainer Sponsor

mbakumenkov Jun 1, 2024
Author