Always install into a python venv in ci containers - #12663

Merged
leandron merged 2 commits into
apache:mainfrom
areusch:tvm-venvs
Sep 12, 2022
Merged

Always install into a python venv in ci containers#12663
leandron merged 2 commits into
apache:mainfrom
areusch:tvm-venvs

Conversation

@areusch

@areuschareusch commented Aug 31, 2022

Copy link
Copy Markdown
Contributor

This PR changes all ci_ to install TVM Python dependencies in a virtualenv separate from the system Python dependencies. This PR sets the stage for adding the poetry-based dependency generator to the CI container build process.

@areusch
areuschforce-pushed the tvm-venvs branch 2 times, most recently from 8b2922d to 89f9a53CompareAugust 31, 2022 20:05
@areusch

areusch commented Aug 31, 2022

Copy link
Copy Markdown
ContributorAuthor

Addresses #12608

@areusch
areuschforce-pushed the tvm-venvs branch 2 times, most recently from 02c0eb7 to 16bff01CompareSeptember 1, 2022 23:46
@github-actions

github-actionsBot commented Sep 2, 2022

Copy link
Copy Markdown
Contributor

Built docs for commit 16bff01 can be found here.

@areusch
areusch marked this pull request as ready for review September 2, 2022 14:18
@areusch

areusch commented Sep 2, 2022

Copy link
Copy Markdown
ContributorAuthor

@driazati@mehrdadh@leandron PTAL

areusch
areusch commented Sep 2, 2022
View reviewed changes
Comment threaddocker/python/freeze-dependencies.sh
@@ -1,4 +1,4 @@
#!/bin/bash
#!/bin/bash -eux
# Licensed to the Apache Software Foundation (ASF) under one

@areuschareuschSep 2, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, rm this file and freeze_deps.py, they are part of a follow-on

driazati
driazati reviewed Sep 2, 2022
View reviewed changes
Comment threaddocker/Dockerfile.ci_gpu
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

COPY install/ubuntu_install_cmake_source.sh /install/ubuntu_install_cmake_source.sh
# Globally disable pip cache

@driazatidriazatiSep 2, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Globally disable pip cache

Comment threaddocker/python/bootstrap/lockfiles/requirements-3.7.txt
@@ -0,0 +1,3 @@
pip

@driazatidriazatiSep 2, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are there separate requirements.txts per Python version?

Comment threaddocker/install/ubuntu_install_python.sh
lsb-core

release=$(lsb_release -sc)
if [ "${release}" == "bionic" ]; then

@driazatidriazatiSep 2, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we use miniconda instead of the distribution's Python + venv? The end result would be the same but we would be able to say "we use version X of Python in CI" instead of "it depends"

@MousiusMousiusSep 9, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I was migrating teams off of conda I used pyenv which would compose into our system with Poetry?

@areuschareusch mentioned this pull request Sep 6, 2022
Closed
@github-actions

github-actionsBot commented Sep 8, 2022

Copy link
Copy Markdown
Contributor

The list below shows some tests that ran in main 546a7da but were skipped in the CI build of 5775327:

test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr#test_rpc_large_array[mps3_an547-(16*1024)]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_aot_executor[mps3_an547]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_relay[mps3_an547]

A detailed report of ran tests is here.

@areusch

areusch commented Sep 9, 2022

Copy link
Copy Markdown
ContributorAuthor

@driazati i think this is finally ready

@leandron

leandron commented Sep 9, 2022

Copy link
Copy Markdown
Contributor

Im hopeful that #12738 will be merged once CI gets green. Happy to merge this just after that.

@areusch

areusch commented Sep 9, 2022

Copy link
Copy Markdown
ContributorAuthor

@leandron happy to hold off til then

mehrdadh
mehrdadh reviewed Sep 9, 2022
View reviewed changes

@mehrdadhmehrdadh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and thanks for working on this!
I'll wait for others to review

leandron
leandron approved these changes Sep 12, 2022
View reviewed changes

@leandronleandron left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, lets merge this improvements before CI gets stale.

@leandron
leandron merged commit a047e02 into apache:mainSep 12, 2022
@leandron

leandron commented Sep 12, 2022

Copy link
Copy Markdown
Contributor

Merged. Thanks @areusch@driazati@mehrdadh

asparkhi pushed a commit to asparkhi/tvm that referenced this pull request Sep 14, 2022
Ashutosh Parkhi
Fixed pylint issues after moving to venv in ci_lint docker
c2f7bf3
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
@asparkhiasparkhi mentioned this pull request Sep 14, 2022
Merged
leandron pushed a commit that referenced this pull request Sep 14, 2022
@asparkhi
Fixed pylint issues after moving to venv in ci_lint docker (#12775)
296565a
Following change introduced installing python dependencies inside
virtual environments: #12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
This was referenced Sep 15, 2022
Closed
Merged
driazati pushed a commit that referenced this pull request Sep 16, 2022
@asparkhi
Fix for import requests and import caffe failures (#12813)
8f8b6d8
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: #12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixes#12663
@areuschareusch mentioned this pull request Sep 16, 2022
Merged
@AndrewZhaoLuoAndrewZhaoLuo mentioned this pull request Oct 4, 2022
Closed
xinetzone pushed a commit to daobook/tvm that referenced this pull request Nov 25, 2022
@areusch@xinetzone
[CI] Always install into a python venv in ci containers (apache#12663)
12b719f
This PR changes all ci_ to install TVM Python dependencies in a
virtualenv separate from the system Python dependencies.
Sets the stage for adding the poetry-based dependency
generator to the CI container build process.
* Always install into a python venv in ci containers.
* Respect Dockerfile ENV PATH modifications in
docker/bash.sh lookups.
xinetzone pushed a commit to daobook/tvm that referenced this pull request Nov 25, 2022
@asparkhi@xinetzone
Fixed pylint issues after moving to venv in ci_lint docker (apache#12775
e8a8630
)
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
xinetzone pushed a commit to daobook/tvm that referenced this pull request Nov 25, 2022
@asparkhi@xinetzone
Fix for import requests and import caffe failures (apache#12813)
a06683e
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: apache#12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixesapache#12663
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MousiusMousiusMousius left review comments
@mehrdadhmehrdadhmehrdadh left review comments
@driazatidriazatidriazati left review comments
@leandronleandronleandron approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@areusch@leandron@Mousius@mehrdadh@driazati
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Always install into a python venv in ci containers - #12663
Merged
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Sep 12, 2022
Merged
Always install into a python venv in ci containers#12663
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Conversation
@areusch
@areuschareusch
commented
Aug 31, 2022

Copy link


Copy Markdown

Contributor
This PR changes all ci_ to install TVM Python dependencies in a virtualenv separate from the system Python dependencies. This PR sets the stage for adding the poetry-based dependency generator to the CI container build process.
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
8b2922d to
89f9a53CompareAugust 31, 2022 20:05
@areusch
areusch
commented
Aug 31, 2022

Copy link


Copy Markdown

ContributorAuthor
Addresses #12608
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
02c0eb7 to
16bff01CompareSeptember 1, 2022 23:46
@github-actions
github-actionsBot
commented
Sep 2, 2022

Copy link


Copy Markdown

Contributor
Built docs for commit 16bff01 can be found here.
@areusch
areusch
marked this pull request as ready for review
September 2, 2022 14:18
@areusch
areusch
commented
Sep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati@mehrdadh@leandron PTAL
areusch
areusch
commented
Sep 2, 2022
View reviewed changes
Comment threaddocker/python/freeze-dependencies.sh
@@ -1,4 +1,4 @@
#!/bin/bash
#!/bin/bash -eux
# Licensed to the Apache Software Foundation (ASF) under one
@areuschareuschSep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, rm this file and freeze_deps.py, they are part of a follow-on
driazati
driazati
reviewed
Sep 2, 2022
View reviewed changes
Comment threaddocker/Dockerfile.ci_gpu
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

COPY install/ubuntu_install_cmake_source.sh /install/ubuntu_install_cmake_source.sh
# Globally disable pip cache
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

# Globally disable pip cache
Comment threaddocker/python/bootstrap/lockfiles/requirements-3.7.txt
@@ -0,0 +1,3 @@
pip
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are there separate requirements.txts per Python version?
Comment threaddocker/install/ubuntu_install_python.sh
 lsb-core

release=$(lsb_release -sc)
if [ "${release}" == "bionic" ]; then
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we use miniconda instead of the distribution's Python + venv? The end result would be the same but we would be able to say "we use version X of Python in CI" instead of "it depends"
@MousiusMousiusSep 9, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I was migrating teams off of conda I used pyenv which would compose into our system with Poetry?
@areuschareusch
mentioned this pull request
Sep 6, 2022

Closed

@github-actions
github-actionsBot
commented
Sep 8, 2022

Copy link


Copy Markdown

Contributor
The list below shows some tests that ran in main 546a7da but were skipped in the CI build of 5775327:
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr#test_rpc_large_array[mps3_an547-(16*1024)]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_aot_executor[mps3_an547]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_relay[mps3_an547]

A detailed report of ran tests is here.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati i think this is finally ready
@leandron
leandron
commented
Sep 9, 2022

Copy link


Copy Markdown

Contributor
Im hopeful that #12738 will be merged once CI gets green. Happy to merge this just after that.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@leandron happy to hold off til then
mehrdadh
mehrdadh
reviewed
Sep 9, 2022
View reviewed changes
@mehrdadhmehrdadh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and thanks for working on this!

I'll wait for others to review
leandron
leandron
approved these changes
Sep 12, 2022
View reviewed changes
@leandronleandron
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, lets merge this improvements before CI gets stale.
@leandron
leandron
merged commit a047e02
into
apache:mainSep 12, 2022
@leandron
leandron
commented
Sep 12, 2022

Copy link


Copy Markdown

Contributor
Merged. Thanks @areusch@driazati@mehrdadh
asparkhi pushed a commit
to asparkhi/tvm
that referenced
this pull request
Sep 14, 2022
Ashutosh Parkhi
Fixed pylint issues after moving to venv in ci_lint docker
c2f7bf3
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
@asparkhiasparkhi
mentioned this pull request
Sep 14, 2022
 Merged

leandron pushed a commit
that referenced
this pull request
Sep 14, 2022
@asparkhi
Fixed pylint issues after moving to venv in ci_lint docker (#12775)
296565a
Following change introduced installing python dependencies inside
virtual environments: #12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661

This was referenced Sep 15, 2022

Closed

 Merged

driazati pushed a commit
that referenced
this pull request
Sep 16, 2022
@asparkhi
Fix for import requests and import caffe failures (#12813)
8f8b6d8
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: #12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixes#12663
@areuschareusch
mentioned this pull request
Sep 16, 2022
 Merged

@AndrewZhaoLuoAndrewZhaoLuo
mentioned this pull request
Oct 4, 2022

Closed

xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@areusch@xinetzone
[CI] Always install into a python venv in ci containers (apache#12663)
12b719f
This PR changes all ci_ to install TVM Python dependencies in a
virtualenv separate from the system Python dependencies.
Sets the stage for adding the poetry-based dependency
generator to the CI container build process.
* Always install into a python venv in ci containers.
* Respect Dockerfile ENV PATH modifications in
docker/bash.sh lookups.
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fixed pylint issues after moving to venv in ci_lint docker (apache#12775
e8a8630
)
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fix for import requests and import caffe failures (apache#12813)
a06683e
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: apache#12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixesapache#12663
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@MousiusMousiusMousius left review comments
@mehrdadhmehrdadhmehrdadh left review comments
@driazatidriazatidriazati left review comments
@leandronleandronleandron approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@areusch@leandron@Mousius@mehrdadh@driazati

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Always install into a python venv in ci containers - #12663
Merged
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Sep 12, 2022
Merged
Always install into a python venv in ci containers#12663
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Conversation
@areusch
@areuschareusch
commented
Aug 31, 2022

Copy link


Copy Markdown

Contributor
This PR changes all ci_ to install TVM Python dependencies in a virtualenv separate from the system Python dependencies. This PR sets the stage for adding the poetry-based dependency generator to the CI container build process.
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
8b2922d to
89f9a53CompareAugust 31, 2022 20:05
@areusch
areusch
commented
Aug 31, 2022

Copy link


Copy Markdown

ContributorAuthor
Addresses #12608
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
02c0eb7 to
16bff01CompareSeptember 1, 2022 23:46
@github-actions
github-actionsBot
commented
Sep 2, 2022

Copy link


Copy Markdown

Contributor
Built docs for commit 16bff01 can be found here.
@areusch
areusch
marked this pull request as ready for review
September 2, 2022 14:18
@areusch
areusch
commented
Sep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati@mehrdadh@leandron PTAL
areusch
areusch
commented
Sep 2, 2022
View reviewed changes
Comment threaddocker/python/freeze-dependencies.sh
@@ -1,4 +1,4 @@
#!/bin/bash
#!/bin/bash -eux
# Licensed to the Apache Software Foundation (ASF) under one
@areuschareuschSep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, rm this file and freeze_deps.py, they are part of a follow-on
driazati
driazati
reviewed
Sep 2, 2022
View reviewed changes
Comment threaddocker/Dockerfile.ci_gpu
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

COPY install/ubuntu_install_cmake_source.sh /install/ubuntu_install_cmake_source.sh
# Globally disable pip cache
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

# Globally disable pip cache
Comment threaddocker/python/bootstrap/lockfiles/requirements-3.7.txt
@@ -0,0 +1,3 @@
pip
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are there separate requirements.txts per Python version?
Comment threaddocker/install/ubuntu_install_python.sh
 lsb-core

release=$(lsb_release -sc)
if [ "${release}" == "bionic" ]; then
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we use miniconda instead of the distribution's Python + venv? The end result would be the same but we would be able to say "we use version X of Python in CI" instead of "it depends"
@MousiusMousiusSep 9, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I was migrating teams off of conda I used pyenv which would compose into our system with Poetry?
@areuschareusch
mentioned this pull request
Sep 6, 2022

Closed

@github-actions
github-actionsBot
commented
Sep 8, 2022

Copy link


Copy Markdown

Contributor
The list below shows some tests that ran in main 546a7da but were skipped in the CI build of 5775327:
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr#test_rpc_large_array[mps3_an547-(16*1024)]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_aot_executor[mps3_an547]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_relay[mps3_an547]

A detailed report of ran tests is here.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati i think this is finally ready
@leandron
leandron
commented
Sep 9, 2022

Copy link


Copy Markdown

Contributor
Im hopeful that #12738 will be merged once CI gets green. Happy to merge this just after that.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@leandron happy to hold off til then
mehrdadh
mehrdadh
reviewed
Sep 9, 2022
View reviewed changes
@mehrdadhmehrdadh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and thanks for working on this!

I'll wait for others to review
leandron
leandron
approved these changes
Sep 12, 2022
View reviewed changes
@leandronleandron
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, lets merge this improvements before CI gets stale.
@leandron
leandron
merged commit a047e02
into
apache:mainSep 12, 2022
@leandron
leandron
commented
Sep 12, 2022

Copy link


Copy Markdown

Contributor
Merged. Thanks @areusch@driazati@mehrdadh
asparkhi pushed a commit
to asparkhi/tvm
that referenced
this pull request
Sep 14, 2022
Ashutosh Parkhi
Fixed pylint issues after moving to venv in ci_lint docker
c2f7bf3
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
@asparkhiasparkhi
mentioned this pull request
Sep 14, 2022
 Merged

leandron pushed a commit
that referenced
this pull request
Sep 14, 2022
@asparkhi
Fixed pylint issues after moving to venv in ci_lint docker (#12775)
296565a
Following change introduced installing python dependencies inside
virtual environments: #12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661

This was referenced Sep 15, 2022

Closed

 Merged

driazati pushed a commit
that referenced
this pull request
Sep 16, 2022
@asparkhi
Fix for import requests and import caffe failures (#12813)
8f8b6d8
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: #12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixes#12663
@areuschareusch
mentioned this pull request
Sep 16, 2022
 Merged

@AndrewZhaoLuoAndrewZhaoLuo
mentioned this pull request
Oct 4, 2022

Closed

xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@areusch@xinetzone
[CI] Always install into a python venv in ci containers (apache#12663)
12b719f
This PR changes all ci_ to install TVM Python dependencies in a
virtualenv separate from the system Python dependencies.
Sets the stage for adding the poetry-based dependency
generator to the CI container build process.
* Always install into a python venv in ci containers.
* Respect Dockerfile ENV PATH modifications in
docker/bash.sh lookups.
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fixed pylint issues after moving to venv in ci_lint docker (apache#12775
e8a8630
)
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fix for import requests and import caffe failures (apache#12813)
a06683e
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: apache#12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixesapache#12663
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@MousiusMousiusMousius left review comments
@mehrdadhmehrdadhmehrdadh left review comments
@driazatidriazatidriazati left review comments
@leandronleandronleandron approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@areusch@leandron@Mousius@mehrdadh@driazati

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Always install into a python venv in ci containers - #12663
Merged
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Sep 12, 2022
Merged
Always install into a python venv in ci containers#12663
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Conversation
@areusch
@areuschareusch
commented
Aug 31, 2022

Copy link


Copy Markdown

Contributor
This PR changes all ci_ to install TVM Python dependencies in a virtualenv separate from the system Python dependencies. This PR sets the stage for adding the poetry-based dependency generator to the CI container build process.
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
8b2922d to
89f9a53CompareAugust 31, 2022 20:05
@areusch
areusch
commented
Aug 31, 2022

Copy link


Copy Markdown

ContributorAuthor
Addresses #12608
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
02c0eb7 to
16bff01CompareSeptember 1, 2022 23:46
@github-actions
github-actionsBot
commented
Sep 2, 2022

Copy link


Copy Markdown

Contributor
Built docs for commit 16bff01 can be found here.
@areusch
areusch
marked this pull request as ready for review
September 2, 2022 14:18
@areusch
areusch
commented
Sep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati@mehrdadh@leandron PTAL
areusch
areusch
commented
Sep 2, 2022
View reviewed changes
Comment threaddocker/python/freeze-dependencies.sh
@@ -1,4 +1,4 @@
#!/bin/bash
#!/bin/bash -eux
# Licensed to the Apache Software Foundation (ASF) under one
@areuschareuschSep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, rm this file and freeze_deps.py, they are part of a follow-on
driazati
driazati
reviewed
Sep 2, 2022
View reviewed changes
Comment threaddocker/Dockerfile.ci_gpu
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

COPY install/ubuntu_install_cmake_source.sh /install/ubuntu_install_cmake_source.sh
# Globally disable pip cache
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

# Globally disable pip cache
Comment threaddocker/python/bootstrap/lockfiles/requirements-3.7.txt
@@ -0,0 +1,3 @@
pip
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are there separate requirements.txts per Python version?
Comment threaddocker/install/ubuntu_install_python.sh
 lsb-core

release=$(lsb_release -sc)
if [ "${release}" == "bionic" ]; then
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we use miniconda instead of the distribution's Python + venv? The end result would be the same but we would be able to say "we use version X of Python in CI" instead of "it depends"
@MousiusMousiusSep 9, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I was migrating teams off of conda I used pyenv which would compose into our system with Poetry?
@areuschareusch
mentioned this pull request
Sep 6, 2022

Closed

@github-actions
github-actionsBot
commented
Sep 8, 2022

Copy link


Copy Markdown

Contributor
The list below shows some tests that ran in main 546a7da but were skipped in the CI build of 5775327:
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr#test_rpc_large_array[mps3_an547-(16*1024)]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_aot_executor[mps3_an547]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_relay[mps3_an547]

A detailed report of ran tests is here.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati i think this is finally ready
@leandron
leandron
commented
Sep 9, 2022

Copy link


Copy Markdown

Contributor
Im hopeful that #12738 will be merged once CI gets green. Happy to merge this just after that.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@leandron happy to hold off til then
mehrdadh
mehrdadh
reviewed
Sep 9, 2022
View reviewed changes
@mehrdadhmehrdadh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and thanks for working on this!

I'll wait for others to review
leandron
leandron
approved these changes
Sep 12, 2022
View reviewed changes
@leandronleandron
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, lets merge this improvements before CI gets stale.
@leandron
leandron
merged commit a047e02
into
apache:mainSep 12, 2022
@leandron
leandron
commented
Sep 12, 2022

Copy link


Copy Markdown

Contributor
Merged. Thanks @areusch@driazati@mehrdadh
asparkhi pushed a commit
to asparkhi/tvm
that referenced
this pull request
Sep 14, 2022
Ashutosh Parkhi
Fixed pylint issues after moving to venv in ci_lint docker
c2f7bf3
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
@asparkhiasparkhi
mentioned this pull request
Sep 14, 2022
 Merged

leandron pushed a commit
that referenced
this pull request
Sep 14, 2022
@asparkhi
Fixed pylint issues after moving to venv in ci_lint docker (#12775)
296565a
Following change introduced installing python dependencies inside
virtual environments: #12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661

This was referenced Sep 15, 2022

Closed

 Merged

driazati pushed a commit
that referenced
this pull request
Sep 16, 2022
@asparkhi
Fix for import requests and import caffe failures (#12813)
8f8b6d8
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: #12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixes#12663
@areuschareusch
mentioned this pull request
Sep 16, 2022
 Merged

@AndrewZhaoLuoAndrewZhaoLuo
mentioned this pull request
Oct 4, 2022

Closed

xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@areusch@xinetzone
[CI] Always install into a python venv in ci containers (apache#12663)
12b719f
This PR changes all ci_ to install TVM Python dependencies in a
virtualenv separate from the system Python dependencies.
Sets the stage for adding the poetry-based dependency
generator to the CI container build process.
* Always install into a python venv in ci containers.
* Respect Dockerfile ENV PATH modifications in
docker/bash.sh lookups.
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fixed pylint issues after moving to venv in ci_lint docker (apache#12775
e8a8630
)
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fix for import requests and import caffe failures (apache#12813)
a06683e
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: apache#12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixesapache#12663
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@MousiusMousiusMousius left review comments
@mehrdadhmehrdadhmehrdadh left review comments
@driazatidriazatidriazati left review comments
@leandronleandronleandron approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@areusch@leandron@Mousius@mehrdadh@driazati

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Always install into a python venv in ci containers - #12663
Merged
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Sep 12, 2022
Merged
Always install into a python venv in ci containers#12663
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Conversation
@areusch
@areuschareusch
commented
Aug 31, 2022

Copy link


Copy Markdown

Contributor
This PR changes all ci_ to install TVM Python dependencies in a virtualenv separate from the system Python dependencies. This PR sets the stage for adding the poetry-based dependency generator to the CI container build process.
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
8b2922d to
89f9a53CompareAugust 31, 2022 20:05
@areusch
areusch
commented
Aug 31, 2022

Copy link


Copy Markdown

ContributorAuthor
Addresses #12608
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
02c0eb7 to
16bff01CompareSeptember 1, 2022 23:46
@github-actions
github-actionsBot
commented
Sep 2, 2022

Copy link


Copy Markdown

Contributor
Built docs for commit 16bff01 can be found here.
@areusch
areusch
marked this pull request as ready for review
September 2, 2022 14:18
@areusch
areusch
commented
Sep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati@mehrdadh@leandron PTAL
areusch
areusch
commented
Sep 2, 2022
View reviewed changes
Comment threaddocker/python/freeze-dependencies.sh
@@ -1,4 +1,4 @@
#!/bin/bash
#!/bin/bash -eux
# Licensed to the Apache Software Foundation (ASF) under one
@areuschareuschSep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, rm this file and freeze_deps.py, they are part of a follow-on
driazati
driazati
reviewed
Sep 2, 2022
View reviewed changes
Comment threaddocker/Dockerfile.ci_gpu
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

COPY install/ubuntu_install_cmake_source.sh /install/ubuntu_install_cmake_source.sh
# Globally disable pip cache
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

# Globally disable pip cache
Comment threaddocker/python/bootstrap/lockfiles/requirements-3.7.txt
@@ -0,0 +1,3 @@
pip
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are there separate requirements.txts per Python version?
Comment threaddocker/install/ubuntu_install_python.sh
 lsb-core

release=$(lsb_release -sc)
if [ "${release}" == "bionic" ]; then
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we use miniconda instead of the distribution's Python + venv? The end result would be the same but we would be able to say "we use version X of Python in CI" instead of "it depends"
@MousiusMousiusSep 9, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I was migrating teams off of conda I used pyenv which would compose into our system with Poetry?
@areuschareusch
mentioned this pull request
Sep 6, 2022

Closed

@github-actions
github-actionsBot
commented
Sep 8, 2022

Copy link


Copy Markdown

Contributor
The list below shows some tests that ran in main 546a7da but were skipped in the CI build of 5775327:
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr#test_rpc_large_array[mps3_an547-(16*1024)]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_aot_executor[mps3_an547]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_relay[mps3_an547]

A detailed report of ran tests is here.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati i think this is finally ready
@leandron
leandron
commented
Sep 9, 2022

Copy link


Copy Markdown

Contributor
Im hopeful that #12738 will be merged once CI gets green. Happy to merge this just after that.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@leandron happy to hold off til then
mehrdadh
mehrdadh
reviewed
Sep 9, 2022
View reviewed changes
@mehrdadhmehrdadh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and thanks for working on this!

I'll wait for others to review
leandron
leandron
approved these changes
Sep 12, 2022
View reviewed changes
@leandronleandron
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, lets merge this improvements before CI gets stale.
@leandron
leandron
merged commit a047e02
into
apache:mainSep 12, 2022
@leandron
leandron
commented
Sep 12, 2022

Copy link


Copy Markdown

Contributor
Merged. Thanks @areusch@driazati@mehrdadh
asparkhi pushed a commit
to asparkhi/tvm
that referenced
this pull request
Sep 14, 2022
Ashutosh Parkhi
Fixed pylint issues after moving to venv in ci_lint docker
c2f7bf3
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
@asparkhiasparkhi
mentioned this pull request
Sep 14, 2022
 Merged

leandron pushed a commit
that referenced
this pull request
Sep 14, 2022
@asparkhi
Fixed pylint issues after moving to venv in ci_lint docker (#12775)
296565a
Following change introduced installing python dependencies inside
virtual environments: #12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661

This was referenced Sep 15, 2022

Closed

 Merged

driazati pushed a commit
that referenced
this pull request
Sep 16, 2022
@asparkhi
Fix for import requests and import caffe failures (#12813)
8f8b6d8
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: #12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixes#12663
@areuschareusch
mentioned this pull request
Sep 16, 2022
 Merged

@AndrewZhaoLuoAndrewZhaoLuo
mentioned this pull request
Oct 4, 2022

Closed

xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@areusch@xinetzone
[CI] Always install into a python venv in ci containers (apache#12663)
12b719f
This PR changes all ci_ to install TVM Python dependencies in a
virtualenv separate from the system Python dependencies.
Sets the stage for adding the poetry-based dependency
generator to the CI container build process.
* Always install into a python venv in ci containers.
* Respect Dockerfile ENV PATH modifications in
docker/bash.sh lookups.
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fixed pylint issues after moving to venv in ci_lint docker (apache#12775
e8a8630
)
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fix for import requests and import caffe failures (apache#12813)
a06683e
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: apache#12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixesapache#12663
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@MousiusMousiusMousius left review comments
@mehrdadhmehrdadhmehrdadh left review comments
@driazatidriazatidriazati left review comments
@leandronleandronleandron approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@areusch@leandron@Mousius@mehrdadh@driazati

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Always install into a python venv in ci containers - #12663
Merged
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Sep 12, 2022
Merged
Always install into a python venv in ci containers#12663
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Conversation
@areusch
@areuschareusch
commented
Aug 31, 2022

Copy link


Copy Markdown

Contributor
This PR changes all ci_ to install TVM Python dependencies in a virtualenv separate from the system Python dependencies. This PR sets the stage for adding the poetry-based dependency generator to the CI container build process.
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
8b2922d to
89f9a53CompareAugust 31, 2022 20:05
@areusch
areusch
commented
Aug 31, 2022

Copy link


Copy Markdown

ContributorAuthor
Addresses #12608
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
02c0eb7 to
16bff01CompareSeptember 1, 2022 23:46
@github-actions
github-actionsBot
commented
Sep 2, 2022

Copy link


Copy Markdown

Contributor
Built docs for commit 16bff01 can be found here.
@areusch
areusch
marked this pull request as ready for review
September 2, 2022 14:18
@areusch
areusch
commented
Sep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati@mehrdadh@leandron PTAL
areusch
areusch
commented
Sep 2, 2022
View reviewed changes
Comment threaddocker/python/freeze-dependencies.sh
@@ -1,4 +1,4 @@
#!/bin/bash
#!/bin/bash -eux
# Licensed to the Apache Software Foundation (ASF) under one
@areuschareuschSep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, rm this file and freeze_deps.py, they are part of a follow-on
driazati
driazati
reviewed
Sep 2, 2022
View reviewed changes
Comment threaddocker/Dockerfile.ci_gpu
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

COPY install/ubuntu_install_cmake_source.sh /install/ubuntu_install_cmake_source.sh
# Globally disable pip cache
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

# Globally disable pip cache
Comment threaddocker/python/bootstrap/lockfiles/requirements-3.7.txt
@@ -0,0 +1,3 @@
pip
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are there separate requirements.txts per Python version?
Comment threaddocker/install/ubuntu_install_python.sh
 lsb-core

release=$(lsb_release -sc)
if [ "${release}" == "bionic" ]; then
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we use miniconda instead of the distribution's Python + venv? The end result would be the same but we would be able to say "we use version X of Python in CI" instead of "it depends"
@MousiusMousiusSep 9, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I was migrating teams off of conda I used pyenv which would compose into our system with Poetry?
@areuschareusch
mentioned this pull request
Sep 6, 2022

Closed

@github-actions
github-actionsBot
commented
Sep 8, 2022

Copy link


Copy Markdown

Contributor
The list below shows some tests that ran in main 546a7da but were skipped in the CI build of 5775327:
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr#test_rpc_large_array[mps3_an547-(16*1024)]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_aot_executor[mps3_an547]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_relay[mps3_an547]

A detailed report of ran tests is here.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati i think this is finally ready
@leandron
leandron
commented
Sep 9, 2022

Copy link


Copy Markdown

Contributor
Im hopeful that #12738 will be merged once CI gets green. Happy to merge this just after that.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@leandron happy to hold off til then
mehrdadh
mehrdadh
reviewed
Sep 9, 2022
View reviewed changes
@mehrdadhmehrdadh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and thanks for working on this!

I'll wait for others to review
leandron
leandron
approved these changes
Sep 12, 2022
View reviewed changes
@leandronleandron
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, lets merge this improvements before CI gets stale.
@leandron
leandron
merged commit a047e02
into
apache:mainSep 12, 2022
@leandron
leandron
commented
Sep 12, 2022

Copy link


Copy Markdown

Contributor
Merged. Thanks @areusch@driazati@mehrdadh
asparkhi pushed a commit
to asparkhi/tvm
that referenced
this pull request
Sep 14, 2022
Ashutosh Parkhi
Fixed pylint issues after moving to venv in ci_lint docker
c2f7bf3
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
@asparkhiasparkhi
mentioned this pull request
Sep 14, 2022
 Merged

leandron pushed a commit
that referenced
this pull request
Sep 14, 2022
@asparkhi
Fixed pylint issues after moving to venv in ci_lint docker (#12775)
296565a
Following change introduced installing python dependencies inside
virtual environments: #12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661

This was referenced Sep 15, 2022

Closed

 Merged

driazati pushed a commit
that referenced
this pull request
Sep 16, 2022
@asparkhi
Fix for import requests and import caffe failures (#12813)
8f8b6d8
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: #12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixes#12663
@areuschareusch
mentioned this pull request
Sep 16, 2022
 Merged

@AndrewZhaoLuoAndrewZhaoLuo
mentioned this pull request
Oct 4, 2022

Closed

xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@areusch@xinetzone
[CI] Always install into a python venv in ci containers (apache#12663)
12b719f
This PR changes all ci_ to install TVM Python dependencies in a
virtualenv separate from the system Python dependencies.
Sets the stage for adding the poetry-based dependency
generator to the CI container build process.
* Always install into a python venv in ci containers.
* Respect Dockerfile ENV PATH modifications in
docker/bash.sh lookups.
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fixed pylint issues after moving to venv in ci_lint docker (apache#12775
e8a8630
)
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fix for import requests and import caffe failures (apache#12813)
a06683e
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: apache#12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixesapache#12663
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@MousiusMousiusMousius left review comments
@mehrdadhmehrdadhmehrdadh left review comments
@driazatidriazatidriazati left review comments
@leandronleandronleandron approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@areusch@leandron@Mousius@mehrdadh@driazati

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Always install into a python venv in ci containers - #12663
Merged
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Sep 12, 2022
Merged
Always install into a python venv in ci containers#12663
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Conversation
@areusch
@areuschareusch
commented
Aug 31, 2022

Copy link


Copy Markdown

Contributor
This PR changes all ci_ to install TVM Python dependencies in a virtualenv separate from the system Python dependencies. This PR sets the stage for adding the poetry-based dependency generator to the CI container build process.
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
8b2922d to
89f9a53CompareAugust 31, 2022 20:05
@areusch
areusch
commented
Aug 31, 2022

Copy link


Copy Markdown

ContributorAuthor
Addresses #12608
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
02c0eb7 to
16bff01CompareSeptember 1, 2022 23:46
@github-actions
github-actionsBot
commented
Sep 2, 2022

Copy link


Copy Markdown

Contributor
Built docs for commit 16bff01 can be found here.
@areusch
areusch
marked this pull request as ready for review
September 2, 2022 14:18
@areusch
areusch
commented
Sep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati@mehrdadh@leandron PTAL
areusch
areusch
commented
Sep 2, 2022
View reviewed changes
Comment threaddocker/python/freeze-dependencies.sh
@@ -1,4 +1,4 @@
#!/bin/bash
#!/bin/bash -eux
# Licensed to the Apache Software Foundation (ASF) under one
@areuschareuschSep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, rm this file and freeze_deps.py, they are part of a follow-on
driazati
driazati
reviewed
Sep 2, 2022
View reviewed changes
Comment threaddocker/Dockerfile.ci_gpu
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

COPY install/ubuntu_install_cmake_source.sh /install/ubuntu_install_cmake_source.sh
# Globally disable pip cache
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

# Globally disable pip cache
Comment threaddocker/python/bootstrap/lockfiles/requirements-3.7.txt
@@ -0,0 +1,3 @@
pip
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are there separate requirements.txts per Python version?
Comment threaddocker/install/ubuntu_install_python.sh
 lsb-core

release=$(lsb_release -sc)
if [ "${release}" == "bionic" ]; then
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we use miniconda instead of the distribution's Python + venv? The end result would be the same but we would be able to say "we use version X of Python in CI" instead of "it depends"
@MousiusMousiusSep 9, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I was migrating teams off of conda I used pyenv which would compose into our system with Poetry?
@areuschareusch
mentioned this pull request
Sep 6, 2022

Closed

@github-actions
github-actionsBot
commented
Sep 8, 2022

Copy link


Copy Markdown

Contributor
The list below shows some tests that ran in main 546a7da but were skipped in the CI build of 5775327:
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr#test_rpc_large_array[mps3_an547-(16*1024)]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_aot_executor[mps3_an547]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_relay[mps3_an547]

A detailed report of ran tests is here.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati i think this is finally ready
@leandron
leandron
commented
Sep 9, 2022

Copy link


Copy Markdown

Contributor
Im hopeful that #12738 will be merged once CI gets green. Happy to merge this just after that.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@leandron happy to hold off til then
mehrdadh
mehrdadh
reviewed
Sep 9, 2022
View reviewed changes
@mehrdadhmehrdadh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and thanks for working on this!

I'll wait for others to review
leandron
leandron
approved these changes
Sep 12, 2022
View reviewed changes
@leandronleandron
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, lets merge this improvements before CI gets stale.
@leandron
leandron
merged commit a047e02
into
apache:mainSep 12, 2022
@leandron
leandron
commented
Sep 12, 2022

Copy link


Copy Markdown

Contributor
Merged. Thanks @areusch@driazati@mehrdadh
asparkhi pushed a commit
to asparkhi/tvm
that referenced
this pull request
Sep 14, 2022
Ashutosh Parkhi
Fixed pylint issues after moving to venv in ci_lint docker
c2f7bf3
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
@asparkhiasparkhi
mentioned this pull request
Sep 14, 2022
 Merged

leandron pushed a commit
that referenced
this pull request
Sep 14, 2022
@asparkhi
Fixed pylint issues after moving to venv in ci_lint docker (#12775)
296565a
Following change introduced installing python dependencies inside
virtual environments: #12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661

This was referenced Sep 15, 2022

Closed

 Merged

driazati pushed a commit
that referenced
this pull request
Sep 16, 2022
@asparkhi
Fix for import requests and import caffe failures (#12813)
8f8b6d8
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: #12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixes#12663
@areuschareusch
mentioned this pull request
Sep 16, 2022
 Merged

@AndrewZhaoLuoAndrewZhaoLuo
mentioned this pull request
Oct 4, 2022

Closed

xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@areusch@xinetzone
[CI] Always install into a python venv in ci containers (apache#12663)
12b719f
This PR changes all ci_ to install TVM Python dependencies in a
virtualenv separate from the system Python dependencies.
Sets the stage for adding the poetry-based dependency
generator to the CI container build process.
* Always install into a python venv in ci containers.
* Respect Dockerfile ENV PATH modifications in
docker/bash.sh lookups.
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fixed pylint issues after moving to venv in ci_lint docker (apache#12775
e8a8630
)
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fix for import requests and import caffe failures (apache#12813)
a06683e
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: apache#12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixesapache#12663
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@MousiusMousiusMousius left review comments
@mehrdadhmehrdadhmehrdadh left review comments
@driazatidriazatidriazati left review comments
@leandronleandronleandron approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@areusch@leandron@Mousius@mehrdadh@driazati

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Always install into a python venv in ci containers - #12663
Merged
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Sep 12, 2022
Merged
Always install into a python venv in ci containers#12663
leandron merged 2 commits into
apache:mainfrom 
areusch:tvm-venvs
Conversation
@areusch
@areuschareusch
commented
Aug 31, 2022

Copy link


Copy Markdown

Contributor
This PR changes all ci_ to install TVM Python dependencies in a virtualenv separate from the system Python dependencies. This PR sets the stage for adding the poetry-based dependency generator to the CI container build process.
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
8b2922d to
89f9a53CompareAugust 31, 2022 20:05
@areusch
areusch
commented
Aug 31, 2022

Copy link


Copy Markdown

ContributorAuthor
Addresses #12608
@areusch
areuschforce-pushed
the
tvm-venvs
branch
2 times, most recently
from
02c0eb7 to
16bff01CompareSeptember 1, 2022 23:46
@github-actions
github-actionsBot
commented
Sep 2, 2022

Copy link


Copy Markdown

Contributor
Built docs for commit 16bff01 can be found here.
@areusch
areusch
marked this pull request as ready for review
September 2, 2022 14:18
@areusch
areusch
commented
Sep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati@mehrdadh@leandron PTAL
areusch
areusch
commented
Sep 2, 2022
View reviewed changes
Comment threaddocker/python/freeze-dependencies.sh
@@ -1,4 +1,4 @@
#!/bin/bash
#!/bin/bash -eux
# Licensed to the Apache Software Foundation (ASF) under one
@areuschareuschSep 2, 2022

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops, rm this file and freeze_deps.py, they are part of a follow-on
driazati
driazati
reviewed
Sep 2, 2022
View reviewed changes
Comment threaddocker/Dockerfile.ci_gpu
ENV PYTHONNOUSERSITE 1 # Disable .local directory from affecting CI.

COPY install/ubuntu_install_cmake_source.sh /install/ubuntu_install_cmake_source.sh
# Globally disable pip cache
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


Suggested change

# Globally disable pip cache
Comment threaddocker/python/bootstrap/lockfiles/requirements-3.7.txt
@@ -0,0 +1,3 @@
pip
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are there separate requirements.txts per Python version?
Comment threaddocker/install/ubuntu_install_python.sh
 lsb-core

release=$(lsb_release -sc)
if [ "${release}" == "bionic" ]; then
@driazatidriazatiSep 2, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we use miniconda instead of the distribution's Python + venv? The end result would be the same but we would be able to say "we use version X of Python in CI" instead of "it depends"
@MousiusMousiusSep 9, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I was migrating teams off of conda I used pyenv which would compose into our system with Poetry?
@areuschareusch
mentioned this pull request
Sep 6, 2022

Closed

@github-actions
github-actionsBot
commented
Sep 8, 2022

Copy link


Copy Markdown

Contributor
The list below shows some tests that ran in main 546a7da but were skipped in the CI build of 5775327:
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr#test_rpc_large_array[mps3_an547-(16*1024)]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_aot_executor[mps3_an547]
test: Cortex_M -> ctypes.tests.micro.zephyr.test_zephyr_aot_exec#test_relay[mps3_an547]

A detailed report of ran tests is here.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@driazati i think this is finally ready
@leandron
leandron
commented
Sep 9, 2022

Copy link


Copy Markdown

Contributor
Im hopeful that #12738 will be merged once CI gets green. Happy to merge this just after that.
@areusch
areusch
commented
Sep 9, 2022

Copy link


Copy Markdown

ContributorAuthor
@leandron happy to hold off til then
mehrdadh
mehrdadh
reviewed
Sep 9, 2022
View reviewed changes
@mehrdadhmehrdadh
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and thanks for working on this!

I'll wait for others to review
leandron
leandron
approved these changes
Sep 12, 2022
View reviewed changes
@leandronleandron
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, lets merge this improvements before CI gets stale.
@leandron
leandron
merged commit a047e02
into
apache:mainSep 12, 2022
@leandron
leandron
commented
Sep 12, 2022

Copy link


Copy Markdown

Contributor
Merged. Thanks @areusch@driazati@mehrdadh
asparkhi pushed a commit
to asparkhi/tvm
that referenced
this pull request
Sep 14, 2022
Ashutosh Parkhi
Fixed pylint issues after moving to venv in ci_lint docker
c2f7bf3
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
@asparkhiasparkhi
mentioned this pull request
Sep 14, 2022
 Merged

leandron pushed a commit
that referenced
this pull request
Sep 14, 2022
@asparkhi
Fixed pylint issues after moving to venv in ci_lint docker (#12775)
296565a
Following change introduced installing python dependencies inside
virtual environments: #12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661

This was referenced Sep 15, 2022

Closed

 Merged

driazati pushed a commit
that referenced
this pull request
Sep 16, 2022
@asparkhi
Fix for import requests and import caffe failures (#12813)
8f8b6d8
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: #12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixes#12663
@areuschareusch
mentioned this pull request
Sep 16, 2022
 Merged

@AndrewZhaoLuoAndrewZhaoLuo
mentioned this pull request
Oct 4, 2022

Closed

xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@areusch@xinetzone
[CI] Always install into a python venv in ci containers (apache#12663)
12b719f
This PR changes all ci_ to install TVM Python dependencies in a
virtualenv separate from the system Python dependencies.
Sets the stage for adding the poetry-based dependency
generator to the CI container build process.
* Always install into a python venv in ci containers.
* Respect Dockerfile ENV PATH modifications in
docker/bash.sh lookups.
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fixed pylint issues after moving to venv in ci_lint docker (apache#12775
e8a8630
)
Following change introduced installing python dependencies inside
virtual environments: apache#12663
Previous to this fix, a different version of python was being
picked up that didn't catch the issues fixed in this commit.
Change-Id: Ie290d9474a799311e07d293fa1b8299326b11661
xinetzone pushed a commit
to daobook/tvm
that referenced
this pull request
Nov 25, 2022
@asparkhi@xinetzone
Fix for import requests and import caffe failures (apache#12813)
a06683e
Recently virtual environments were introduced in the
docker images which was a great contribution to
localize errors: apache#12663. In this fix, link to the caffe is
created inside this virtual env instead of adding it
to the system path of python. This fix also removes
importing request package where not needed.
Fixesapache#12663
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@MousiusMousiusMousius left review comments
@mehrdadhmehrdadhmehrdadh left review comments
@driazatidriazatidriazati left review comments
@leandronleandronleandron approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@areusch@leandron@Mousius@mehrdadh@driazati