Version Lock `requirements.txt`

[siddk]

Might be good to either specify versions, or ideally, switch to a conda environment for this repo. Critically, the base requirements for Megatron-DeepSpeed in requirements.txt just require "torch" whereas you actually need torch > 1.7.0, notably with CUDA 11.1 or higher for this repo to actually work!

[jaketae]

I remember discussing this with @stas00 in #46 some time ago. Linking the thread in case it becomes relevant!

[stas00]

If you know a minimal version required, by all means please set it to package>=x.y.z (but not ==) in requirements.txt

There is definitely no need for cuda-11.1, it works just fine with cuda-10.

Performance is a different story.

p.s. why is pt>1.7 required? It's not a requirement in neither Megatron-LM, nor Deepspeed. But it can be just an omission on the part of the original package maintainers.

[siddk]

My bad on CUDA-10 - I need 11.1 to build the triton dependency for DeepSpeed (I was using the current instructions in the README, which should probably be updated).

1.7 is the first PyTorch version supporting Ampere cards + TF32; for JZ, since we're on V100s probably not an issue, but for newer cards (3XXXs, A100s) this is pretty nice for performance.

Also with 1.9, better layernorm kernels, faster ops across the board, and the profiler! The profiler is amazing!

[siddk]

But yeah - I think CUDA >10.1 is fine if we're not trying to build the more complex DS kernels.

[stas00]

My bad on CUDA-10 - I need 11.1 to build the triton dependency for DeepSpeed (I was using the current instructions in the README, which should probably be updated).

1.7 is the first PyTorch version supporting Ampere cards + TF32; for JZ, since we're on V100s probably not an issue, but for newer cards (3XXXs, A100s) this is pretty nice for performance.

Both of these aren't project requirements but user's - each user knows what components they need to work on their card. We shouldn't set project requirements to the highest requirements of the user using the latest hardware.

e.g. I had to use pytorch-nightly for all of winter and some spring before Amphere was supported in pt-1.8, but that was my need and not of others.

---

But yeah - I think CUDA >10.1 is fine if we're not trying to build the more complex DS kernels.

I don't think this is so either. Deepspeed currently supports torch>=1.2. It's possible that some kernels require a higher cuda.

[siddk]

Got it; so performance is the only real issue then? Do we want to lock versions to min-runnable, or do we want to actually gate on performance?

I could imagine confusion down the line that the same models that fit on newer cards due to some DS/Megatron enhancements don’t necessarily work (or work reliably) on older cards because of an “effective” torch/ds/CUDA version.

Generally my solution for reproducibility in the future (in case there are bugs/robustness to future features) is to freeze requirements based on what we’re running on a day-to-day.

@stas00 how do you feel about keeping a “min-requirements.txt” instead and then a “locked” requirements.txt so we can have some consistency? Especially if we want to benchmark/reproduce results later?

[stas00]

Got it; so performance is the only real issue then? Do we want to lock versions to min-runnable, or do we want to actually gate on performance?

Nope. Performance is also a prerogative of a user. One should always avoid setting any hardcoded requirements because there is always a chance of conflict with other dependent package requirements.

In the case of Meg-DS, Meg has no torch requirements, DS's is 1.2, so it's safe to not set any minimal requirements.

Should we discover that the code doesn't run due to a certain problem or a bug, we will then set the minimal requirement for that particular package.

The key to understand here is that we aren't trying to make things more difficult for our users, but on the contrary - to make it easier - the less restrictions are imposed - the easier it is for more users to use the project w/o requiring a dedicated conda env.

I could imagine confusion down the line that the same models that fit on newer cards due to some DS/Megatron enhancements don’t necessarily work (or work reliably) on older cards because of an “effective” torch/ds/CUDA version.

The main back-compat issue for models is modern models trained in bf16, which don't do well under fp16.

The rest should work almost identically under older cards.

Generally my solution for reproducibility in the future (in case there are bugs/robustness to future features) is to freeze requirements based on what we’re running on a day-to-day.

For your own needs or for needs of a training, for sure. But not for different users and groups that have different requirements.

@stas00 how do you feel about keeping a “min-requirements.txt” instead and then a “locked” requirements.txt so we can have some consistency? Especially if we want to benchmark/reproduce results later?

I think this would be a waste of maintainers' time and a source of confusion to users. requirements should have only one source.

I suggest for you to create your own bash alias or a script where you set the desired requirements. This is what I do. I have little scripts for various things, since I don't like typing (and make mistakes).

When one does a benchmark one either encodes the fixed requirements in the benchmark code or one adds a documentation (or both).

To reproduce issues, package versions is often insufficient, you need SHA numbers for each project, since quite a few of the ones we use are master branches. (or main for the modern politically correct populaces).

[siddk]

Apologies @stas00 - this makes a lot of sense. A lot of my thoughts on “locking” versions are less about downstream user usability, and more about “academic reproducibility” which as you say, is complicated due to using master branch builds.

I’ll set versions locally in a bash alias like you suggest. Thanks again - closing issue for now.

[stas00]

That's said please feel free to create docs that recommend specific setups with specific versions for specific purposes - but please don't forget to maintain them so that they don't fall into disrepair.

If you'd it might be the best to be done on the sister repo: https://github.com/bigscience-workshop/bigscience where we document the processes in details.

This repo is the code part of the framework, the other repo is the information part of the project where we log experiments, share scripts, document progress, etc.

[thomasw21]

Late to the party, but torch>=1.7 might be needed to use prefix lm, essentially passing bool tensors requires that version I believe. #107 (comment)

[stas00]

Then by all means please add that minimal requirements. May be link to #107 (comment) in the PR so that there is a trace of why this was done.