Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why is the free dragonfly io utilized so much? #3786

Closed
boomballa opened this issue Sep 25, 2024 · 1 comment
Closed

Why is the free dragonfly io utilized so much? #3786

boomballa opened this issue Sep 25, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@boomballa
Copy link

boomballa commented Sep 25, 2024

Describe the bug
Idle dragonfly with high physical machine IO footprint, but no high IO write and read operations.

Screenshots
sudo iotop -b -n 1 |head -n 50
image

pidstat -dl -p 25966
image

lsof -p 25966
image

Environment (please complete the following information):

  • OS: [Debian GNU/Linux 11]
  • Kernel: Linux 5.10.0-30-amd64 finalize blpop algorithm #1 SMP Debian 5.10.218-1 (2024-06-01) x86_64 GNU/Linux
  • Containerized: Physical machine direct deployment
  • Dragonfly Version: [dragonfly-v1.22.2]

Physical Machine Environment
CPU 40 core
Memory 128GiB
Disk 1.8T SAS(Not SSD)

Additional context
In response to this issue, where the IO footprint is high, but there are no obvious disk write and read operations, I understand that there are no more than a few reasons that can cause high IO for cached instances:

  1. save alive bgsave operation is being carried out, resulting in the RDB file write disk is more centralized.
  2. replica instance pulling bgsave files generated by the disk to cause a large number of read operations
  3. log printing is more frequent, resulting in greater pressure on the SAS disk IO.
  4. Frequent opening and closing of file handles by the dragonfly process
  5. hard disk failure

According to the phenomenon, dragonfly opened 40 threads, almost full of all the cpu io processes.
image

It's true that I didn't specify the cpu resources that dragonfly can take up, but it shouldn't produce such a large IO footprint when there are no read/write accesses, I don't know if it's a bug or if I'm just lacking understanding of dragonfly's features.

  • Attached are the contents of my conf file,I wonder if there's a misconfiguration.
--pidfile=/home/dba/dragonfly/dragonfly7379/dragonfly.pid

--log_dir=/home/dba/dragonfly/dragonfly7379
--logtostderr=false
--logtostdout=false
--max_log_size=500

--dir=/home/dba/dragonfly/dragonfly7379
--max_log_size=20
--version_check=true
--port=7379
--list_compress_depth=0
--cluster_mode=emulated

--maxclients=64000
--maxmemory=10Gib

--bind=10.46.128.11
--admin_bind=17379
--pipeline_squash=10
--tcp_keepalive=1800
--rename_command=
--hz=80
--keys_output_limit=8192

--compression_level=0
--compression_mode=0
--list_compress_depth=0

--admin_nopass=false
--multi_eval_squash_buffer=4096

--df_snapshot_format=true
--dbfilename=dragonfly7379-dump

Looking forward to the answer, thanks!

@boomballa boomballa added the bug Something isn't working label Sep 25, 2024
@boomballa boomballa changed the title Why is the idle dragonfly io hog so high? Why is the free dragonfly io utilized so much? Sep 25, 2024
@romange
Copy link
Collaborator

romange commented Sep 25, 2024

See here:
#2444 (comment)

the answer was given many times

@romange romange closed this as not planned Won't fix, can't repro, duplicate, stale Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants