There is a big model from NVIDIA that I assume is for this purpose, i.e. Megatron 530b, so it doesn't sound too unreasonable.

Edit: I assumed that the model was distillation, that is apparently not true.