Add multi-gpu inference

[Kovbo]

No description provided.

[FurtherAI]

Just a note that Megatron already supports multi-gpu inference because it overrides _get_service(). Though the design could be cleaned up. So, if you're testing this, it should be tested with Unsloth.

[Kovbo]

Just a note that Megatron already supports multi-gpu inference because it overrides _get_service(). Though the design could be cleaned up. So, if you're testing this, it should be tested with Unsloth.

Yeah, you are right, probably it did not work because it required to explicitly set tensor_parallel_size?
cc @vivekkalyan