Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What's the difference when starting tritonserver with mpirun --allow-run-as-root -n 1 /opt/tritonserver/bin/tritonserver vs. /opt/tritonserver/bin/tritonserver directly? #7371

Open
so2bin opened this issue Jun 25, 2024 · 0 comments

Comments

@so2bin
Copy link

so2bin commented Jun 25, 2024

Description
I am observing a difference in the behavior of TritonServer when starting it with mpirun compared to starting it directly. Specifically, when I use mpirun --allow-run-as-root -n 1 /opt/tritonserver/bin/tritonserver, the server runs and inference normally, but when I start it directly with /opt/tritonserver/bin/tritonserver, I notice higher CPU usage and slower inference speeds.

  • The following is the normally started tritonserver's CPU usage from top and process information from ps auxfww:

    • The CPU usage information:
      image

    • The processes information:
      image

  • The following is the abnormally started tritonserver's CPU and GPU informations:

    • The CPU usage information:
      image

    • The nvidia-smi information:
      image

    • The blocked start logs result from the high CPU usage:
      image

  • The following is the model files:
    image

  • Notes:

  1. Whole the configurations of these two runnings are the same, only the start command changes.
  2. All the models are running with Triton python backend.

I am wondering if there are any specific configuration that mpirun --allow-run-as-root -n 1 sets that might be influencing the behavior of TritonServer.
Thank you for your help in clarifying this matter.

Triton Information
image

To Reproduce

Expected behavior

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant