Pin development dependencies & integrate dependabot

[jcugat]

Triggered by encode/httpx#1053 (comment) and #151 (comment). I set it up a while ago in my fork of httpx and seems to work quite well. We can try it here first, and expand it to httpx too if everybody agrees, or remove it if we find any issue.

After this PR is merged dependabot will be enabled to upgrade dependencies once per week: https://docs.github.com/en/github/administering-a-repository/enabling-and-disabling-version-updates

[florimondmanca]

Yes when https://github.com/encode/httpcore/pull/162/files#r470961811 is resolved via #163 :-)

[lovelydinosaur]

Great. What we might want here is a requirements.base that holds our actual requirements, and a requirements.txt that holds the frozen set of all requirements + their dependencies? (That's what I'd normally do with a deployed service, so really you need to make sure you're pinning the indirect package dependencies too)

[jcugat]

@tomchristie I've implemented your suggestion and also added ./scripts/freeze to keep both files in sync, let me know what you think.

@florimondmanca I've also integrated your changes from #163

[j178]

Hi, based on your excellent work, I added two commits:

• Install packge tools like setuptools, wheel before others, to avoid warnings like Using legacy 'setup.py install' for autoflake, since package 'wheel' is not installed.
• Remove a redundant pip install -e . in scripts/install

If there is any problem, please feel free to revert them.

[jcugat]

@j178 yup, those changes look good.

@tomchristie@florimondmanca are you happy with the overall PR now? Should we try it? 🙂

[lovelydinosaur]

Great, yup. Let's give this a whirl.

I'd probably? tend towards a ./scripts/install --upgrade instead here, but that's just being picky, and there's nothing here that's a blocker.

Nice simple tooling & workflow, w/ pinned deps and isolated upgrade PRs.

Win. 👍

[lovelydinosaur]

Grand yup, we can drop this once pip 20.3 is released, due October 2020.

https://pip.pypa.io/en/stable/news/#id26

[florimondmanca]

@jcugat I pushed an update of dependencies to resolve conflicts with master, but also to see for myself what working with this looks like…

Re @j178's comment in #162 (comment) - it looks like we're going to have some issues with multiplatform support, since we can see several transitive deps are installed on Linux (which I assume is your OS) vs macOS (mine) or even Windows, so we can't have one requirements.txt that lists everything for everyone (since pip freeze just dumps what's installed instead of adding environment markers as well).

Per pip-tools recommendations we should ideally compile and provide one requirements file per environment (basically OS and Python version), but for us this is impractical. (We don't necessarily have a 3.6, 3.7 and 3.8 installation handy to generate initial files for an OS, and that would be 9 requirements files in the repo, which is pretty huge).

So I guess for now, options are…

1. Let's not sweat it, and add a requirements.txt for any
2. Drop the "compile requirements" idea altogether, and just stick to a top-level requirements.txt with pinned dependencies.

Hold - Issues with cross-platform freezing

[florimondmanca]

Closing in favor of #184

Thanks a ton for promoting this change @jcugat!

[jcugat]

So glad to see this solved in #184, didn't realize the issues with different platforms.

Also, apologies to @florimondmanca and @tomchristie for leaving this open for so long. I've been on holidays and did a necessary break from tech, but I'm back with my batteries fully recharged! 🚀