Add Llama7b FSDP test for torch.compile mode

[pankd]

Adds a test for FSDP

[regisss]

Is it a different test from the one that was added in #761 ?

[vivekgoe]

@regisss This attempts to add generic support for FSDP (similar to what you already have for DeepSpeed) such that going forward any model that runs with DeepSpeed can also be run with FSDP. Also I assume that tests in this file are "slow" tests and we can add longer tests here, we wanted to add models such as Llama-70B + FSDP which take longer time here.

Test in #761 is a short test which is intended to run as part of Gaudi2 CI. Going forward if you feel there is repetition we can consider removing it.

[vivekgoe]

@pankd please add a test for Llama-70B also.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[vivekgoe]

@regisss we are checking last few things, will let you know when it's done.

[vivekgoe]

@regisss this is ready for merge now.