Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] sentinel error after second switch master #13309

Open
imazzeii opened this issue May 31, 2024 · 1 comment
Open

[BUG] sentinel error after second switch master #13309

imazzeii opened this issue May 31, 2024 · 1 comment

Comments

@imazzeii
Copy link

imazzeii commented May 31, 2024

Describe the bug

I have created a sentinel cluster with docker compose made of 3 sentinels (ports 5000, 5001, 5002), 1 master (port 6385) and 2 slaves (6386 and 6387); then I connect a client to sentinel node 5000 and start to execute the following steps:

time action
t0 delete key 'mykey'
t1 sleep 3 seconds
t2(+3 sec) set mykey=myvalue
t3 sleep_t3 3 seconds
t4(+6 sec) dispatch sleep command 30 seconds to redis 6385 (using a separate connection)
t5 sleep_t5 7 seconds
t6(+13 sec) set mykey=newvalue
t7 sleep_t7 50 seconds
t8(+63 sec) get new master config (using a separate connection)
dispatch sleep command 3000 seconds to new redis master (using a separate connection)
t9 sleep_t9 5 seconds
t10(+70sec) set mykey=newvalue

so at time t4 master on port 6385 enters debug mode for 30 seconds and sentinels elect a new master (everything works fine), but at time t8 the new master enters debug mode for 3000 seconds and sentinels are unable to elect a new master; if a separate client tries to connect it either hangs or connects but cannot modify data (receice error message: readonly, you can't write against a read only replica)

To reproduce

  1. (edit) download files from https://fastupload.io/f993031e9a01f2fd
  2. extract the attached file master-slave-sentinel.zip in a folder and start the sentinel cluster: docker compose -f docker-compose-sentinel.yml up -d
  3. add the entry 127.0.01 host.docker.internal in your hosts file
  4. copy the attached file test-lettuce-0.0.1-SNAPSHOT-jar-with-dependencies.jar and run with the following command: java -jar test-lettuce-0.0.1-SNAPSHOT-jar-with-dependencies.jar (the source files can be found in attachment test-lettuce-0.0.1-SNAPSHOT-sources.jar)
  5. alternatively, perform the previously described steps manually

see the attachement for the console logs

tools used: redis 7.2.5-alpine, docker desktop 4.28.0 (windows)
for the client: java jre 17, lettuce-core 6.3.1

tested on :

  • windows 10 - Intel(R) Core(TM) i7-10510U
  • windows 10 - Intel(R) Core(TM) i7-1260P

Expected behavior

the expected behaviour is that a new master is elected correctly after the second elected master stop working

Additional information

Any additional information that is relevant to the problem.
readonly_error
client_java_logs
sentinel_logs

@imazzeii
Copy link
Author

client-settings

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant