Issue to gather Fixes + New features to send upstream

[stas00]

Please edit the OP to add whatever fixes we applied to the core and which need to be propagated upstream into:

1. https://github.com/microsoft/Megatron-DeepSpeed
2. https://github.com/NVIDIA/Megatron-LM

we want to do that to make it easier to sync upstream changes back to this repo.

Changes to send upstream:

Bug fixes:

• 9189c4e fix bug when restarting with no eval in round 1
• 0125aaa Fix merge functions to take in account doc_ids
• 9e75429 Fix Tensorboard logging with correct rank condition as detailed in this issue on Megatron-LM repo. Relevant part of the PR: 9e75429#diff-e2b248f8c422a601bcb0b7d93f96c1dff070f2694737e2b69f1def64ab9c1844R589
• 56c2983 - fixes issues with --rampup-batch-size help entry
• c680954 Fix document offset when merging documents, bug introduces in 0125aaa. Thanks @adammoody !
• 648ee17 - check whether python3-config is available and clearly assert if it doesn't when failing to build helpers

New functionality:

• 18201ce , e8fcbae, c680954 add tools/merge_preprocessed_data.py to support merging datasets - might be easier to just copy the new script.
• 7b99881 - new faster preprocessing script for when one has many cpu cores.
• 5069622 - new preprocessing script that uses HuggingFace Datasets as source
• Cirriculum learning: Curriculum learning support #132 + [CL] fix default placement #133

[shoeybi]

9189c4e and 9e75429 have been fixed. Will take a look at the rest later. Thank you!