[QUESTION] Why does GPTDataset not directly cache all samples document_index and sample_index, and then construct different shuffle_index for different parameters?

[eviler007]

Your question
Why does GPTDataset not directly cache all samples document_index and sample_index, and then construct different shuffle_index for different parameters?

[starkhu]

good idea！

[github-actions]

Marking as stale. No activity in 60 days.

[github-actions]

This issue was closed because it has been inactive for 7 days since being marked as stale.