Changes in requirements to run Studio on Python 3.8

[jredrejo]

Description

Upgrade some python packages to be able to run Studio on python 3.8

Steps to Test

• Using different versions of Python:
  pip install -r requirements.txt
make prodceleryworkers

• Do current tests pass?

Reviewers

If you are looking to assign a reviewer, here are some options:

• Jordan jayoshih (full stack)
• Aron aronasorman (back end, devops)
• Micah micahscopes (full stack)
• Kevin kollivier (back end)
• Ivan ivanistheone (Ricecooker)
• Richard rtibbles (full stack, Kolibri)
• Radina @radinamatic (documentation)

[codecov]

Codecov Report

Merging #2236 into develop will not change coverage.
The diff coverage is n/a.

@@ Coverage Diff @@## develop #2236 +/- ##
========================================
Coverage 80.80% 80.80% ========================================
Files 289 289 Lines 14022 14022 ========================================
Hits 11331 11331 Misses 2691 2691 

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3814abe...3e2fae2. Read the comment docs.

[DXCanas]

Getting some errors:

$ yarn run devsetup
yarn run v1.22.5
$ cd contentcuration && python manage.py setup --settings=contentcuration.dev_settings
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/django/core/management/__init__.py", line 206, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/django/core/management/__init__.py", line 40, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/david/Projects/studio/contentcuration/contentcuration/management/commands/setup.py", line 22, in <module>
from contentcuration.utils.db_tools import *
File "/home/david/Projects/studio/contentcuration/contentcuration/utils/db_tools.py", line 22, in <module>
from contentcuration.utils.files import duplicate_file
File "/home/david/Projects/studio/contentcuration/contentcuration/utils/files.py", line 27, in <module>
from pressurecooker.images import create_image_from_pdf_page
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/pressurecooker/images.py", line 17, in <module>
import matplotlib.pyplot as plt
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/matplotlib/pyplot.py", line 31, in <module>
import matplotlib.colorbar
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/matplotlib/colorbar.py", line 32, in <module>
import matplotlib.artist as martist
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/matplotlib/artist.py", line 16, in <module>
from .path import Path
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/matplotlib/path.py", line 21, in <module>
from . import _path, rcParams
ImportError: numpy.core.multiarray failed to import
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[DXCanas]

Also submitted a PR for some missing apt packages on Ubuntu 20, which is the reason I'm able to test this so thoroughly :)
jredrejo#1

[jredrejo]

Getting some errors:

$ yarn run devsetup
yarn run v1.22.5
$ cd contentcuration && python manage.py setup --settings=contentcuration.dev_settings
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/django/core/management/__init__.py", line 206, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/django/core/management/__init__.py", line 40, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/david/Projects/studio/contentcuration/contentcuration/management/commands/setup.py", line 22, in <module>
from contentcuration.utils.db_tools import *
File "/home/david/Projects/studio/contentcuration/contentcuration/utils/db_tools.py", line 22, in <module>
from contentcuration.utils.files import duplicate_file
File "/home/david/Projects/studio/contentcuration/contentcuration/utils/files.py", line 27, in <module>
from pressurecooker.images import create_image_from_pdf_page
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/pressurecooker/images.py", line 17, in <module>
import matplotlib.pyplot as plt
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/matplotlib/pyplot.py", line 31, in <module>
import matplotlib.colorbar
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/matplotlib/colorbar.py", line 32, in <module>
import matplotlib.artist as martist
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/matplotlib/artist.py", line 16, in <module>
from .path import Path
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/matplotlib/path.py", line 21, in <module>
from . import _path, rcParams
ImportError: numpy.core.multiarray failed to import
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

@DXCanas this is my numpy installation:

(venv) jose@jose-xps:/datos/le/studio$ pip show numpy
Name: numpy
Version: 1.15.4
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: /datos/le/studio/venv/lib/python3.8/site-packages
Requires: Required-by: wordcloud, pressurecooker, matplotlib

can you check which version do you have installed in your venv?

[DXCanas]

Getting some errors:

$ yarn run devsetup
yarn run v1.22.5
$ cd contentcuration && python manage.py setup --settings=contentcuration.dev_settings
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'
Traceback (most recent call last):
File "manage.py", line 11, in <module>
execute_from_command_line(sys.argv)
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
utility.execute()
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/django/core/management/__init__.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/django/core/management/__init__.py", line 206, in fetch_command
klass = load_command_class(app_name, subcommand)
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/django/core/management/__init__.py", line 40, in load_command_class
module = import_module('%s.management.commands.%s' % (app_name, name))
File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/david/Projects/studio/contentcuration/contentcuration/management/commands/setup.py", line 22, in <module>
from contentcuration.utils.db_tools import *
File "/home/david/Projects/studio/contentcuration/contentcuration/utils/db_tools.py", line 22, in <module>
from contentcuration.utils.files import duplicate_file
File "/home/david/Projects/studio/contentcuration/contentcuration/utils/files.py", line 27, in <module>
from pressurecooker.images import create_image_from_pdf_page
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/pressurecooker/images.py", line 17, in <module>
import matplotlib.pyplot as plt
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/matplotlib/pyplot.py", line 31, in <module>
import matplotlib.colorbar
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/matplotlib/colorbar.py", line 32, in <module>
import matplotlib.artist as martist
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/matplotlib/artist.py", line 16, in <module>
from .path import Path
File "/home/david/.virtualenvs/studio/lib/python3.8/site-packages/matplotlib/path.py", line 21, in <module>
from . import _path, rcParams
ImportError: numpy.core.multiarray failed to import
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

@DXCanas this is my numpy installation:

(venv) jose@jose-xps:/datos/le/studio$ pip show numpy
Name: numpy
Version: 1.15.4
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: /datos/le/studio/venv/lib/python3.8/site-packages
Requires: Required-by: wordcloud, pressurecooker, matplotlib

can you check which version do you have installed in your venv?

Sure! I have the same as you:

$ pip show numpy
Name: numpy
Version: 1.15.4
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: /home/david/.virtualenvs/studio/lib/python3.8/site-packages
Requires: Required-by: wordcloud, pressurecooker, matplotlib

Seems like I'm getting the same error with make prodceleryworkers now too

[rtibbles]

Note that numpy should now be removed from the Studio dependencies.

[rtibbles]

As noted from the discussion in #2533 it is probably desirable that we have a single Python version target for Studio. Perhaps we can upgrade that target to Python 3.8 when we upgrade the Django version, but for now Python 3.6 is what we are using. Closing for now.