[INF] Enable torch compile for inference

[oelayan7]

No description provided.

[delock]

Hi @oelayan7 do you have sample code using this feature? Or does it turn on by default? I wonder if I can try it with CPU accelerator.

[oelayan7]

Hi @delock,
No, it isn't run by default. The function .compile() needs to be called.
Torch compile was introduced in ZeRO and for training, which used CompileWrapper that was removed in #5581 and a new approach was introduced in this PR.
This change just adds this approach to inference as it was done in the other PRs.

[oelayan7]

@tjruwase the failing tests are because an HTTP failure with HF.
Can you please retrigger? (couldn't find a way to do that)

[oelayan7]

@tjruwase @tohtana @umchand Can anyone trigger the CI please?

[tohtana]

Thank you @oelayan7 @delock !

I just found that nvtx decorator breaks the graph and has an impact on performance. Can you add deepspeed.utils.nvtx.enable_nvtx = False to compile() as in #569?

Overall this looks good to me. Let's merge after you add it.

[tohtana]

Thank you @oelayan7 !

[oelayan7]

@loadams Thanks!
I fixed the formatting issue, can you please retrigger?