You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What's the difference when starting tritonserver with mpirun --allow-run-as-root -n 1 /opt/tritonserver/bin/tritonserver vs. /opt/tritonserver/bin/tritonserver directly?
#7371
Open
so2bin opened this issue
Jun 25, 2024
· 0 comments
Description
I am observing a difference in the behavior of TritonServer when starting it with mpirun compared to starting it directly. Specifically, when I use mpirun --allow-run-as-root -n 1 /opt/tritonserver/bin/tritonserver, the server runs and inference normally, but when I start it directly with /opt/tritonserver/bin/tritonserver, I notice higher CPU usage and slower inference speeds.
The following is the normally started tritonserver's CPU usage from top and process information from ps auxfww:
The CPU usage information:
The processes information:
The following is the abnormally started tritonserver's CPU and GPU informations:
The CPU usage information:
The nvidia-smi information:
The blocked start logs result from the high CPU usage:
The following is the model files:
Notes:
Whole the configurations of these two runnings are the same, only the start command changes.
All the models are running with Triton python backend.
I am wondering if there are any specific configuration that mpirun --allow-run-as-root -n 1 sets that might be influencing the behavior of TritonServer.
Thank you for your help in clarifying this matter.
Triton Information
To Reproduce
Expected behavior
The text was updated successfully, but these errors were encountered:
Description
I am observing a difference in the behavior of TritonServer when starting it with
mpirun
compared to starting it directly. Specifically, when I usempirun --allow-run-as-root -n 1 /opt/tritonserver/bin/tritonserver
, the server runs and inference normally, but when I start it directly with/opt/tritonserver/bin/tritonserver
, I notice higher CPU usage and slower inference speeds.The following is the normally started tritonserver's CPU usage from
top
and process information fromps auxfww
:The CPU usage information:
![image](https://private-user-images.githubusercontent.com/9431914/342646232-f472f2ca-a3e5-401d-8b1b-2a2ac468297a.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MTgxNjMsIm5iZiI6MTcxOTYxNzg2MywicGF0aCI6Ii85NDMxOTE0LzM0MjY0NjIzMi1mNDcyZjJjYS1hM2U1LTQwMWQtOGIxYi0yYTJhYzQ2ODI5N2EucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjhUMjMzNzQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YWE5OGQ5YWRhMTgzMTI3ZmZjMjdhY2I1MGM2ZGYwN2Q5OGE0MzM5MGE3MWRkZGQxZTEzNDJmYzdhYWJjOGYxYiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.GzLkxc2wU2uwxBjRXMz-NTomb69tfkfSIPJrPIh1UbE)
The processes information:
![image](https://private-user-images.githubusercontent.com/9431914/342646276-d2dba372-f936-40be-bd23-cb2a16df1d4b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MTgxNjMsIm5iZiI6MTcxOTYxNzg2MywicGF0aCI6Ii85NDMxOTE0LzM0MjY0NjI3Ni1kMmRiYTM3Mi1mOTM2LTQwYmUtYmQyMy1jYjJhMTZkZjFkNGIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjhUMjMzNzQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YTY1OGI5MDRmY2M2ODE0NDcwNTZlNTM4ODliNjIyMmVmYWVjOGY2YWRiNjJmOWU1YzIxMzUwNDNkNzg5MmFmNyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.0FLpB_zIBqEQ_bB7Dvrcda2OVIWaumSLA15ge93zV2c)
The following is the abnormally started tritonserver's CPU and GPU informations:
The CPU usage information:
![image](https://private-user-images.githubusercontent.com/9431914/342647098-4933a29c-28a8-4035-bd88-39844aa36b1d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MTgxNjMsIm5iZiI6MTcxOTYxNzg2MywicGF0aCI6Ii85NDMxOTE0LzM0MjY0NzA5OC00OTMzYTI5Yy0yOGE4LTQwMzUtYmQ4OC0zOTg0NGFhMzZiMWQucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjhUMjMzNzQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MGFlNGVhNmZiNmUxYmU4ZmZhOTc4MWRmNWNkYzkxM2UyM2MzZDlhOWY5Y2EzOGZmM2E4MDk2NDEyODBmZTU2NiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.ShBAiesPZqlSpOokNRYuXuxKHpFg9NGski56kMny_wc)
The
![image](https://private-user-images.githubusercontent.com/9431914/342647606-2cd08b8b-d3c2-44d4-b7a2-05d470fbd7fc.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MTgxNjMsIm5iZiI6MTcxOTYxNzg2MywicGF0aCI6Ii85NDMxOTE0LzM0MjY0NzYwNi0yY2QwOGI4Yi1kM2MyLTQ0ZDQtYjdhMi0wNWQ0NzBmYmQ3ZmMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjhUMjMzNzQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OWQzNDAxYmVhNGVhN2JhNzRlODMwODk0ZDQzZmFlZmM0NjMyMjMyZDM2MzNlZTRhNjU2ODc5MDAzZWI4YTczZCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.sjFhIF7-5V8UFYVg_ZpkBi2e11FNvEyiBVxxA0CgsFM)
nvidia-smi
information:The blocked start logs result from the high CPU usage:
![image](https://private-user-images.githubusercontent.com/9431914/342648242-cfee9acb-fd66-471f-9f02-7d841abf8fbf.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MTgxNjMsIm5iZiI6MTcxOTYxNzg2MywicGF0aCI6Ii85NDMxOTE0LzM0MjY0ODI0Mi1jZmVlOWFjYi1mZDY2LTQ3MWYtOWYwMi03ZDg0MWFiZjhmYmYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjhUMjMzNzQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9OGI5YjIwYzc4NmU2ZTVhMzkwYjFkYzY4YjJjZmQzYWViYzBlNThhYTdkOWIzOTg3YzI4M2ZmYmU1MmYwY2E5NyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.AFErY1_4GxF2oeMc8U2h6eQAplkl3OHbJIsZn1ahyig)
The following is the model files:
![image](https://private-user-images.githubusercontent.com/9431914/342647805-0b15b1ca-df50-4efa-a436-def5c363aeb9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MTgxNjMsIm5iZiI6MTcxOTYxNzg2MywicGF0aCI6Ii85NDMxOTE0LzM0MjY0NzgwNS0wYjE1YjFjYS1kZjUwLTRlZmEtYTQzNi1kZWY1YzM2M2FlYjkucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjhUMjMzNzQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YjIxZThkZTk5MTlmNTliNTYzNGVmMzMxNmU1Y2I0MWMyYmU2MWVmYmY2MmJiNjJlY2ViYThlZWRhOWQ0MjRlZSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.RaGlUPg_TXxD7Z4ot05pRXQczGlsFH2S4HglZx42qPk)
Notes:
python backend
.I am wondering if there are any specific configuration that
mpirun --allow-run-as-root -n 1
sets that might be influencing the behavior of TritonServer.Thank you for your help in clarifying this matter.
Triton Information
![image](https://private-user-images.githubusercontent.com/9431914/342645512-a9f8b8b3-f036-4b4b-9557-79e8fe902536.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTk2MTgxNjMsIm5iZiI6MTcxOTYxNzg2MywicGF0aCI6Ii85NDMxOTE0LzM0MjY0NTUxMi1hOWY4YjhiMy1mMDM2LTRiNGItOTU1Ny03OWU4ZmU5MDI1MzYucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDYyOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA2MjhUMjMzNzQzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ODM5OTdiMTFhNmM1YzgxZTczZTI0MDcyMDY1NWZkMzhmMjM0NDYxYjU4MWNiNjYzNjZiNDgzMWZmZWRiYWJlYiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.M11-n2wKra-i2udjWFKmhpTunhsruQWy6GGYxDKK8gI)
To Reproduce
Expected behavior
The text was updated successfully, but these errors were encountered: