ARROW-5725: [Crossbow] Port conda recipes to azure pipelines - #4649

Closed
kszucs wants to merge 39 commits into
apache:masterfrom
kszucs:crossbow-azure
Closed

ARROW-5725: [Crossbow] Port conda recipes to azure pipelines #4649
kszucs wants to merge 39 commits into
apache:masterfrom
kszucs:crossbow-azure

Conversation

@kszucs

@kszucskszucs commented Jun 21, 2019•
edited
Loading

Copy link
Copy Markdown
Member

@kszucskszucs added the WIP PR is work in progress label Jun 21, 2019
@kszucskszucs mentioned this pull request Jun 24, 2019
Closed
@kszucs

kszucs commented Jun 24, 2019

Copy link
Copy Markdown
MemberAuthor

@ursabot crossbow package -g conda

@ursabot

ursabot commented Jun 24, 2019

Copy link
Copy Markdown

AMD64 Conda Crossbow (#24978) builder has been succeeded.

Revision: 14a15b61fcf1ccb00010c70f9937959319e17fd4

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-60

TaskStatus
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
conda-win-vs2015-py36Appveyor Status
conda-linux-gcc-py37unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-osx-clang-py36unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure

@kszucs

kszucs commented Jun 24, 2019

Copy link
Copy Markdown
MemberAuthor

@ursabot crossbow package -g conda

@ursabot

ursabot commented Jun 24, 2019

Copy link
Copy Markdown

AMD64 Conda Crossbow (#24997) builder has been succeeded.

Revision: 5bfdf1a047e9da6b9638d4ece2c6e46778ab5a6c

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-61

TaskStatus
conda-win-vs2015-py36Appveyor Status
conda-osx-clang-py36unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-linux-gcc-py37unsupported CI service azure
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status

@kszucs

kszucs commented Jun 25, 2019

Copy link
Copy Markdown
MemberAuthor

@ursabot crossbow package -g conda

@ursabot

ursabot commented Jun 25, 2019

Copy link
Copy Markdown

AMD64 Conda Crossbow (#25036) builder has been succeeded.

Revision: f9a2f5ca12232e66e931b16ce733695789dc3e91

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-62

TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor

@kszucs

kszucs commented Jun 25, 2019

Copy link
Copy Markdown
MemberAuthor

@ursabot crossbow package -g conda

@ursabot

ursabot commented Jun 25, 2019

Copy link
Copy Markdown

AMD64 Conda Crossbow (#25299) builder has been succeeded.

Revision: ec85d4c5c096cf47b3e416eef45f203f80f9a41b

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-63

TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor

@kszucskszucs removed the WIP PR is work in progress label Jun 25, 2019
@kszucs

kszucs commented Jun 25, 2019

Copy link
Copy Markdown
MemberAuthor

One of the azure build has lost connection, but I cannot restart it on the UI.

@kszucs

kszucs commented Jun 25, 2019

Copy link
Copy Markdown
MemberAuthor

@ursabot crossbow package -g conda

@ursabot

ursabot commented Jun 25, 2019

Copy link
Copy Markdown

AMD64 Conda Crossbow (#25411) builder has been succeeded.

Revision: ea32b2e5cea273a9713b491bfc09ceab9a9def48

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-65

TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor

@kszucs

kszucs commented Jun 25, 2019

Copy link
Copy Markdown
MemberAuthor

@ursabot crossbow package -g conda

@ursabot

ursabot commented Jun 25, 2019

Copy link
Copy Markdown

AMD64 Conda Crossbow (#25445) builder has been succeeded.

Revision: 597fc8a839f46b99eae8ea06176a1a4d64162441

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-67

TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor

@kszucskszucs changed the title [Crossbow] Port conda recipes to azure pipelinesARROW-5725: [Crossbow] Port conda recipes to azure pipelines Jun 25, 2019
@kszucs

kszucs commented Jun 25, 2019

Copy link
Copy Markdown
MemberAuthor

@ursabot crossbow package -g conda

@ursabot

ursabot commented Jun 25, 2019

Copy link
Copy Markdown

AMD64 Conda Crossbow (#25501) builder has been succeeded.

Revision: 2605fbaf06ebf419b72b22839dd182483ebf171d

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-69

TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor

@kszucs

kszucs commented Jun 25, 2019

Copy link
Copy Markdown
MemberAuthor

The builds are passing, but the artifacts are not uploaded correctly.

@kszucs

kszucs commented Jun 25, 2019

Copy link
Copy Markdown
MemberAuthor

@ursabot crossbow package -g conda

@ursabot

ursabot commented Jun 25, 2019

Copy link
Copy Markdown

AMD64 Conda Crossbow (#25538) builder has been succeeded.

Revision: be7037aeb464a2e3456e4ddef7122b6c79e3055f

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-70

TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor

pitrou
pitrou reviewed Jun 25, 2019
View reviewed changes

@pitroupitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I have anything constructive to say about this, sorry. Some questions though...

Comment threaddev/tasks/conda-recipes/.ci_support/linux_python2.7.yaml
Comment threaddev/tasks/conda-recipes/README.md Outdated
Comment threaddev/tasks/conda-recipes/README.md Outdated
Comment threaddev/tasks/conda-recipes/README.md Outdated
Comment threaddev/tasks/conda-recipes/README.md Outdated
Comment threaddev/tasks/conda-recipes/appveyor.win.yml
@@ -1,20 +1,3 @@
# Licensed to the Apache Software Foundation (ASF) under one

@pitroupitrouJun 25, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really mind, but is there a reason why you removed the license headers?

@kszucskszucsJun 25, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because these files are mostly ported from the [py]arrow[-cpp] feedstocks. There is a note in the LICENSE.txt that files under dev/tasks/conda-recipes have another license. Also added to the rat exclude list.

Comment threaddev/tasks/conda-recipes/arrow-cpp/build.sh

# Include g++'s system headers
if [ "$(uname)" == "Linux" ]; then
SYSTEM_INCLUDES=$(echo | ${CXX} -E -Wp,-v -xc++ - 2>&1 | grep '^ ' | awk '{print "-isystem;" substr($1, 1)}' | tr '\n' ';')

@pitroupitrouJun 25, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally there would be clean way to write this...
btw, here I get:

$ CXX=g++
$ echo$(echo |${CXX} -E -Wp,-v -xc++ - 2>&1| grep '^ '| awk '{print "-isystem;" substr($1, 1)}'| tr '\n'';')
-isystem;6:;-isystem;7:;-isystem;8:;-isystem;9:;-isystem;10:;-isystem;11:;-isystem;12:;-isystem;13:;

@kszucskszucsJun 25, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to any suggestions.

@pitroupitrouJun 26, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I have none, sorry.

Comment threaddev/tasks/conda-recipes/azure.linux.yml
Comment threaddev/tasks/docker-tests/azure.linux.yml
@kszucs

kszucs commented Jun 25, 2019

Copy link
Copy Markdown
MemberAuthor

@ursabot crossbow package conda-win-vs2015-py36 conda-win-vs2015-py37

@ursabot

ursabot commented Jun 25, 2019

Copy link
Copy Markdown

AMD64 Conda Crossbow (#25665) builder has been succeeded.

Revision: 20f6cec

Submitted crossbow builds: ursa-labs/crossbow @ ursabot-73

TaskStatus
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor

@kszucs

kszucs commented Jun 26, 2019

Copy link
Copy Markdown
MemberAuthor

@pitrou the artifacts are properly uploaded to github releases now

@pitrou

pitrou commented Jun 26, 2019

Copy link
Copy Markdown
Member

Side note, would it be possible for ursabot to post links to artifacts in its report above? Finding files in the github releases is a bit cumbersome.

@kszucs

kszucs commented Jun 26, 2019

Copy link
Copy Markdown
MemberAuthor

It is more complicated then the report above.

  1. python crossbow.py submit ... creates the branches and triggers the builds on various CIs
  2. python crossbow.py status ... polls the github api about these statuses, checks and release assets

The ursabot crossbow command is not a service like buildbot, thus it is not notified about changes with hooks, and the ursabot crossbow command is currently only a proxy to python crossbow.py submit ..., executed via buildbot.

On the long term I could port the functionality of crossbow into ursabot, but it requires the development of a series of meta services, a meta-scheduler, a meta-builder, a meta-worker. The scheduler would manipulate the crossbow repository directly (preferably via github api instead of native git bindings), the triggered meta builders would follow the logs of other services via their api and we'd be notified about crossbow's status and check updates by github hooks, then we could properly report back all of the results (including the artifacts).

TLDR it is possible, but requires more time.

@pitrou

pitrou commented Jun 26, 2019

Copy link
Copy Markdown
Member

I don't mean post the links when they are available, but upfront (so they would give a 404 until the build is finished). It depends whether the URLs are deterministic or not...

@kszucs

kszucs commented Jun 26, 2019

Copy link
Copy Markdown
MemberAuthor

This is how the badges are working. For the conda packages the assets are not deterministic: https://github.com/apache/arrow/pull/4649/files#diff-0abe5ae3589de012350559065a4194a6R155

pitrou
pitrou approved these changes Jun 26, 2019
View reviewed changes

@pitroupitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. No objection from me.

@jorisvandenbosschejorisvandenbossche mentioned this pull request Apr 8, 2021
Closed
@asfimportasfimport mentioned this pull request Jun 26, 2019
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pitroupitroupitrou approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kszucs@ursabot@pitrou
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines  - #4649
Closed
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Closed
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines #4649
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Conversation
@kszucs
@kszucskszucs
commented
Jun 21, 2019•

edited

Loading

Copy link


Copy Markdown

Member
@kszucskszucs
added
the 
WIP
PR is work in progress
label
Jun 21, 2019
@kszucskszucs
mentioned this pull request
Jun 24, 2019
 Closed

@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24978) builder has been succeeded.
Revision: 14a15b61fcf1ccb00010c70f9937959319e17fd4
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-60
TaskStatus
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
conda-win-vs2015-py36Appveyor Status
conda-linux-gcc-py37unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-osx-clang-py36unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24997) builder has been succeeded.
Revision: 5bfdf1a047e9da6b9638d4ece2c6e46778ab5a6c
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-61
TaskStatus
conda-win-vs2015-py36Appveyor Status
conda-osx-clang-py36unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-linux-gcc-py37unsupported CI service azure
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25036) builder has been succeeded.
Revision: f9a2f5ca12232e66e931b16ce733695789dc3e91
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-62
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25299) builder has been succeeded.
Revision: ec85d4c5c096cf47b3e416eef45f203f80f9a41b
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-63
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
removed
the 
WIP
PR is work in progress
label
Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
One of the azure build has lost connection, but I cannot restart it on the UI.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25411) builder has been succeeded.
Revision: ea32b2e5cea273a9713b491bfc09ceab9a9def48
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-65
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25445) builder has been succeeded.
Revision: 597fc8a839f46b99eae8ea06176a1a4d64162441
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-67
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
changed the title
[Crossbow] Port conda recipes to azure pipelinesARROW-5725: [Crossbow] Port conda recipes to azure pipelines Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25501) builder has been succeeded.
Revision: 2605fbaf06ebf419b72b22839dd182483ebf171d
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-69
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
The builds are passing, but the artifacts are not uploaded correctly.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25538) builder has been succeeded.
Revision: be7037aeb464a2e3456e4ddef7122b6c79e3055f
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-70
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
pitrou
pitrou
reviewed
Jun 25, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I have anything constructive to say about this, sorry. Some questions though...
Comment threaddev/tasks/conda-recipes/.ci_support/linux_python2.7.yaml
Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/appveyor.win.yml
@@ -1,20 +1,3 @@
# Licensed to the Apache Software Foundation (ASF) under one
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really mind, but is there a reason why you removed the license headers?
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because these files are mostly ported from the [py]arrow[-cpp] feedstocks. There is a note in the LICENSE.txt that files under dev/tasks/conda-recipes have another license. Also added to the rat exclude list.
Comment threaddev/tasks/conda-recipes/arrow-cpp/build.sh

# Include g++'s system headers
if [ "$(uname)" == "Linux" ]; then
 SYSTEM_INCLUDES=$(echo | ${CXX} -E -Wp,-v -xc++ - 2>&1 | grep '^ ' | awk '{print "-isystem;" substr($1, 1)}' | tr '\n' ';')
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally there would be clean way to write this...

btw, here I get:
$ CXX=g++
$ echo$(echo |${CXX} -E -Wp,-v -xc++ - 2>&1| grep '^ '| awk '{print "-isystem;" substr($1, 1)}'| tr '\n'';')
-isystem;6:;-isystem;7:;-isystem;8:;-isystem;9:;-isystem;10:;-isystem;11:;-isystem;12:;-isystem;13:;
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to any suggestions.
@pitroupitrouJun 26, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I have none, sorry.
Comment threaddev/tasks/conda-recipes/azure.linux.yml
Comment threaddev/tasks/docker-tests/azure.linux.yml
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package conda-win-vs2015-py36 conda-win-vs2015-py37
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25665) builder has been succeeded.
Revision: 20f6cec
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-73
TaskStatus
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
@pitrou the artifacts are properly uploaded to github releases now
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
Side note, would it be possible for ursabot to post links to artifacts in its report above? Finding files in the github releases is a bit cumbersome.
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
It is more complicated then the report above.
  1. python crossbow.py submit ... creates the branches and triggers the builds on various CIs
  2. python crossbow.py status ... polls the github api about these statuses, checks and release assets
The ursabot crossbow command is not a service like buildbot, thus it is not notified about changes with hooks, and the ursabot crossbow command is currently only a proxy to python crossbow.py submit ..., executed via buildbot.
On the long term I could port the functionality of crossbow into ursabot, but it requires the development of a series of meta services, a meta-scheduler, a meta-builder, a meta-worker. The scheduler would manipulate the crossbow repository directly (preferably via github api instead of native git bindings), the triggered meta builders would follow the logs of other services via their api and we'd be notified about crossbow's status and check updates by github hooks, then we could properly report back all of the results (including the artifacts).
TLDR it is possible, but requires more time.
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
I don't mean post the links when they are available, but upfront (so they would give a 404 until the build is finished). It depends whether the URLs are deterministic or not...
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
This is how the badges are working. For the conda packages the assets are not deterministic: https://github.com/apache/arrow/pull/4649/files#diff-0abe5ae3589de012350559065a4194a6R155
pitrou
pitrou
approved these changes
Jun 26, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. No objection from me.
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Apr 8, 2021
 Closed

@asfimportasfimport
mentioned this pull request
Jun 26, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@kszucs@ursabot@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines  - #4649
Closed
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Closed
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines #4649
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Conversation
@kszucs
@kszucskszucs
commented
Jun 21, 2019•

edited

Loading

Copy link


Copy Markdown

Member
@kszucskszucs
added
the 
WIP
PR is work in progress
label
Jun 21, 2019
@kszucskszucs
mentioned this pull request
Jun 24, 2019
 Closed

@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24978) builder has been succeeded.
Revision: 14a15b61fcf1ccb00010c70f9937959319e17fd4
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-60
TaskStatus
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
conda-win-vs2015-py36Appveyor Status
conda-linux-gcc-py37unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-osx-clang-py36unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24997) builder has been succeeded.
Revision: 5bfdf1a047e9da6b9638d4ece2c6e46778ab5a6c
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-61
TaskStatus
conda-win-vs2015-py36Appveyor Status
conda-osx-clang-py36unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-linux-gcc-py37unsupported CI service azure
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25036) builder has been succeeded.
Revision: f9a2f5ca12232e66e931b16ce733695789dc3e91
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-62
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25299) builder has been succeeded.
Revision: ec85d4c5c096cf47b3e416eef45f203f80f9a41b
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-63
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
removed
the 
WIP
PR is work in progress
label
Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
One of the azure build has lost connection, but I cannot restart it on the UI.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25411) builder has been succeeded.
Revision: ea32b2e5cea273a9713b491bfc09ceab9a9def48
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-65
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25445) builder has been succeeded.
Revision: 597fc8a839f46b99eae8ea06176a1a4d64162441
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-67
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
changed the title
[Crossbow] Port conda recipes to azure pipelinesARROW-5725: [Crossbow] Port conda recipes to azure pipelines Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25501) builder has been succeeded.
Revision: 2605fbaf06ebf419b72b22839dd182483ebf171d
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-69
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
The builds are passing, but the artifacts are not uploaded correctly.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25538) builder has been succeeded.
Revision: be7037aeb464a2e3456e4ddef7122b6c79e3055f
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-70
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
pitrou
pitrou
reviewed
Jun 25, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I have anything constructive to say about this, sorry. Some questions though...
Comment threaddev/tasks/conda-recipes/.ci_support/linux_python2.7.yaml
Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/appveyor.win.yml
@@ -1,20 +1,3 @@
# Licensed to the Apache Software Foundation (ASF) under one
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really mind, but is there a reason why you removed the license headers?
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because these files are mostly ported from the [py]arrow[-cpp] feedstocks. There is a note in the LICENSE.txt that files under dev/tasks/conda-recipes have another license. Also added to the rat exclude list.
Comment threaddev/tasks/conda-recipes/arrow-cpp/build.sh

# Include g++'s system headers
if [ "$(uname)" == "Linux" ]; then
 SYSTEM_INCLUDES=$(echo | ${CXX} -E -Wp,-v -xc++ - 2>&1 | grep '^ ' | awk '{print "-isystem;" substr($1, 1)}' | tr '\n' ';')
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally there would be clean way to write this...

btw, here I get:
$ CXX=g++
$ echo$(echo |${CXX} -E -Wp,-v -xc++ - 2>&1| grep '^ '| awk '{print "-isystem;" substr($1, 1)}'| tr '\n'';')
-isystem;6:;-isystem;7:;-isystem;8:;-isystem;9:;-isystem;10:;-isystem;11:;-isystem;12:;-isystem;13:;
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to any suggestions.
@pitroupitrouJun 26, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I have none, sorry.
Comment threaddev/tasks/conda-recipes/azure.linux.yml
Comment threaddev/tasks/docker-tests/azure.linux.yml
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package conda-win-vs2015-py36 conda-win-vs2015-py37
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25665) builder has been succeeded.
Revision: 20f6cec
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-73
TaskStatus
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
@pitrou the artifacts are properly uploaded to github releases now
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
Side note, would it be possible for ursabot to post links to artifacts in its report above? Finding files in the github releases is a bit cumbersome.
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
It is more complicated then the report above.
  1. python crossbow.py submit ... creates the branches and triggers the builds on various CIs
  2. python crossbow.py status ... polls the github api about these statuses, checks and release assets
The ursabot crossbow command is not a service like buildbot, thus it is not notified about changes with hooks, and the ursabot crossbow command is currently only a proxy to python crossbow.py submit ..., executed via buildbot.
On the long term I could port the functionality of crossbow into ursabot, but it requires the development of a series of meta services, a meta-scheduler, a meta-builder, a meta-worker. The scheduler would manipulate the crossbow repository directly (preferably via github api instead of native git bindings), the triggered meta builders would follow the logs of other services via their api and we'd be notified about crossbow's status and check updates by github hooks, then we could properly report back all of the results (including the artifacts).
TLDR it is possible, but requires more time.
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
I don't mean post the links when they are available, but upfront (so they would give a 404 until the build is finished). It depends whether the URLs are deterministic or not...
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
This is how the badges are working. For the conda packages the assets are not deterministic: https://github.com/apache/arrow/pull/4649/files#diff-0abe5ae3589de012350559065a4194a6R155
pitrou
pitrou
approved these changes
Jun 26, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. No objection from me.
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Apr 8, 2021
 Closed

@asfimportasfimport
mentioned this pull request
Jun 26, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@kszucs@ursabot@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines  - #4649
Closed
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Closed
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines #4649
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Conversation
@kszucs
@kszucskszucs
commented
Jun 21, 2019•

edited

Loading

Copy link


Copy Markdown

Member
@kszucskszucs
added
the 
WIP
PR is work in progress
label
Jun 21, 2019
@kszucskszucs
mentioned this pull request
Jun 24, 2019
 Closed

@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24978) builder has been succeeded.
Revision: 14a15b61fcf1ccb00010c70f9937959319e17fd4
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-60
TaskStatus
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
conda-win-vs2015-py36Appveyor Status
conda-linux-gcc-py37unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-osx-clang-py36unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24997) builder has been succeeded.
Revision: 5bfdf1a047e9da6b9638d4ece2c6e46778ab5a6c
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-61
TaskStatus
conda-win-vs2015-py36Appveyor Status
conda-osx-clang-py36unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-linux-gcc-py37unsupported CI service azure
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25036) builder has been succeeded.
Revision: f9a2f5ca12232e66e931b16ce733695789dc3e91
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-62
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25299) builder has been succeeded.
Revision: ec85d4c5c096cf47b3e416eef45f203f80f9a41b
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-63
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
removed
the 
WIP
PR is work in progress
label
Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
One of the azure build has lost connection, but I cannot restart it on the UI.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25411) builder has been succeeded.
Revision: ea32b2e5cea273a9713b491bfc09ceab9a9def48
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-65
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25445) builder has been succeeded.
Revision: 597fc8a839f46b99eae8ea06176a1a4d64162441
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-67
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
changed the title
[Crossbow] Port conda recipes to azure pipelinesARROW-5725: [Crossbow] Port conda recipes to azure pipelines Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25501) builder has been succeeded.
Revision: 2605fbaf06ebf419b72b22839dd182483ebf171d
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-69
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
The builds are passing, but the artifacts are not uploaded correctly.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25538) builder has been succeeded.
Revision: be7037aeb464a2e3456e4ddef7122b6c79e3055f
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-70
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
pitrou
pitrou
reviewed
Jun 25, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I have anything constructive to say about this, sorry. Some questions though...
Comment threaddev/tasks/conda-recipes/.ci_support/linux_python2.7.yaml
Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/appveyor.win.yml
@@ -1,20 +1,3 @@
# Licensed to the Apache Software Foundation (ASF) under one
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really mind, but is there a reason why you removed the license headers?
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because these files are mostly ported from the [py]arrow[-cpp] feedstocks. There is a note in the LICENSE.txt that files under dev/tasks/conda-recipes have another license. Also added to the rat exclude list.
Comment threaddev/tasks/conda-recipes/arrow-cpp/build.sh

# Include g++'s system headers
if [ "$(uname)" == "Linux" ]; then
 SYSTEM_INCLUDES=$(echo | ${CXX} -E -Wp,-v -xc++ - 2>&1 | grep '^ ' | awk '{print "-isystem;" substr($1, 1)}' | tr '\n' ';')
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally there would be clean way to write this...

btw, here I get:
$ CXX=g++
$ echo$(echo |${CXX} -E -Wp,-v -xc++ - 2>&1| grep '^ '| awk '{print "-isystem;" substr($1, 1)}'| tr '\n'';')
-isystem;6:;-isystem;7:;-isystem;8:;-isystem;9:;-isystem;10:;-isystem;11:;-isystem;12:;-isystem;13:;
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to any suggestions.
@pitroupitrouJun 26, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I have none, sorry.
Comment threaddev/tasks/conda-recipes/azure.linux.yml
Comment threaddev/tasks/docker-tests/azure.linux.yml
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package conda-win-vs2015-py36 conda-win-vs2015-py37
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25665) builder has been succeeded.
Revision: 20f6cec
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-73
TaskStatus
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
@pitrou the artifacts are properly uploaded to github releases now
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
Side note, would it be possible for ursabot to post links to artifacts in its report above? Finding files in the github releases is a bit cumbersome.
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
It is more complicated then the report above.
  1. python crossbow.py submit ... creates the branches and triggers the builds on various CIs
  2. python crossbow.py status ... polls the github api about these statuses, checks and release assets
The ursabot crossbow command is not a service like buildbot, thus it is not notified about changes with hooks, and the ursabot crossbow command is currently only a proxy to python crossbow.py submit ..., executed via buildbot.
On the long term I could port the functionality of crossbow into ursabot, but it requires the development of a series of meta services, a meta-scheduler, a meta-builder, a meta-worker. The scheduler would manipulate the crossbow repository directly (preferably via github api instead of native git bindings), the triggered meta builders would follow the logs of other services via their api and we'd be notified about crossbow's status and check updates by github hooks, then we could properly report back all of the results (including the artifacts).
TLDR it is possible, but requires more time.
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
I don't mean post the links when they are available, but upfront (so they would give a 404 until the build is finished). It depends whether the URLs are deterministic or not...
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
This is how the badges are working. For the conda packages the assets are not deterministic: https://github.com/apache/arrow/pull/4649/files#diff-0abe5ae3589de012350559065a4194a6R155
pitrou
pitrou
approved these changes
Jun 26, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. No objection from me.
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Apr 8, 2021
 Closed

@asfimportasfimport
mentioned this pull request
Jun 26, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@kszucs@ursabot@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines  - #4649
Closed
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Closed
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines #4649
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Conversation
@kszucs
@kszucskszucs
commented
Jun 21, 2019•

edited

Loading

Copy link


Copy Markdown

Member
@kszucskszucs
added
the 
WIP
PR is work in progress
label
Jun 21, 2019
@kszucskszucs
mentioned this pull request
Jun 24, 2019
 Closed

@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24978) builder has been succeeded.
Revision: 14a15b61fcf1ccb00010c70f9937959319e17fd4
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-60
TaskStatus
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
conda-win-vs2015-py36Appveyor Status
conda-linux-gcc-py37unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-osx-clang-py36unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24997) builder has been succeeded.
Revision: 5bfdf1a047e9da6b9638d4ece2c6e46778ab5a6c
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-61
TaskStatus
conda-win-vs2015-py36Appveyor Status
conda-osx-clang-py36unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-linux-gcc-py37unsupported CI service azure
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25036) builder has been succeeded.
Revision: f9a2f5ca12232e66e931b16ce733695789dc3e91
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-62
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25299) builder has been succeeded.
Revision: ec85d4c5c096cf47b3e416eef45f203f80f9a41b
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-63
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
removed
the 
WIP
PR is work in progress
label
Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
One of the azure build has lost connection, but I cannot restart it on the UI.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25411) builder has been succeeded.
Revision: ea32b2e5cea273a9713b491bfc09ceab9a9def48
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-65
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25445) builder has been succeeded.
Revision: 597fc8a839f46b99eae8ea06176a1a4d64162441
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-67
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
changed the title
[Crossbow] Port conda recipes to azure pipelinesARROW-5725: [Crossbow] Port conda recipes to azure pipelines Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25501) builder has been succeeded.
Revision: 2605fbaf06ebf419b72b22839dd182483ebf171d
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-69
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
The builds are passing, but the artifacts are not uploaded correctly.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25538) builder has been succeeded.
Revision: be7037aeb464a2e3456e4ddef7122b6c79e3055f
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-70
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
pitrou
pitrou
reviewed
Jun 25, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I have anything constructive to say about this, sorry. Some questions though...
Comment threaddev/tasks/conda-recipes/.ci_support/linux_python2.7.yaml
Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/appveyor.win.yml
@@ -1,20 +1,3 @@
# Licensed to the Apache Software Foundation (ASF) under one
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really mind, but is there a reason why you removed the license headers?
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because these files are mostly ported from the [py]arrow[-cpp] feedstocks. There is a note in the LICENSE.txt that files under dev/tasks/conda-recipes have another license. Also added to the rat exclude list.
Comment threaddev/tasks/conda-recipes/arrow-cpp/build.sh

# Include g++'s system headers
if [ "$(uname)" == "Linux" ]; then
 SYSTEM_INCLUDES=$(echo | ${CXX} -E -Wp,-v -xc++ - 2>&1 | grep '^ ' | awk '{print "-isystem;" substr($1, 1)}' | tr '\n' ';')
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally there would be clean way to write this...

btw, here I get:
$ CXX=g++
$ echo$(echo |${CXX} -E -Wp,-v -xc++ - 2>&1| grep '^ '| awk '{print "-isystem;" substr($1, 1)}'| tr '\n'';')
-isystem;6:;-isystem;7:;-isystem;8:;-isystem;9:;-isystem;10:;-isystem;11:;-isystem;12:;-isystem;13:;
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to any suggestions.
@pitroupitrouJun 26, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I have none, sorry.
Comment threaddev/tasks/conda-recipes/azure.linux.yml
Comment threaddev/tasks/docker-tests/azure.linux.yml
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package conda-win-vs2015-py36 conda-win-vs2015-py37
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25665) builder has been succeeded.
Revision: 20f6cec
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-73
TaskStatus
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
@pitrou the artifacts are properly uploaded to github releases now
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
Side note, would it be possible for ursabot to post links to artifacts in its report above? Finding files in the github releases is a bit cumbersome.
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
It is more complicated then the report above.
  1. python crossbow.py submit ... creates the branches and triggers the builds on various CIs
  2. python crossbow.py status ... polls the github api about these statuses, checks and release assets
The ursabot crossbow command is not a service like buildbot, thus it is not notified about changes with hooks, and the ursabot crossbow command is currently only a proxy to python crossbow.py submit ..., executed via buildbot.
On the long term I could port the functionality of crossbow into ursabot, but it requires the development of a series of meta services, a meta-scheduler, a meta-builder, a meta-worker. The scheduler would manipulate the crossbow repository directly (preferably via github api instead of native git bindings), the triggered meta builders would follow the logs of other services via their api and we'd be notified about crossbow's status and check updates by github hooks, then we could properly report back all of the results (including the artifacts).
TLDR it is possible, but requires more time.
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
I don't mean post the links when they are available, but upfront (so they would give a 404 until the build is finished). It depends whether the URLs are deterministic or not...
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
This is how the badges are working. For the conda packages the assets are not deterministic: https://github.com/apache/arrow/pull/4649/files#diff-0abe5ae3589de012350559065a4194a6R155
pitrou
pitrou
approved these changes
Jun 26, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. No objection from me.
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Apr 8, 2021
 Closed

@asfimportasfimport
mentioned this pull request
Jun 26, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@kszucs@ursabot@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines  - #4649
Closed
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Closed
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines #4649
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Conversation
@kszucs
@kszucskszucs
commented
Jun 21, 2019•

edited

Loading

Copy link


Copy Markdown

Member
@kszucskszucs
added
the 
WIP
PR is work in progress
label
Jun 21, 2019
@kszucskszucs
mentioned this pull request
Jun 24, 2019
 Closed

@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24978) builder has been succeeded.
Revision: 14a15b61fcf1ccb00010c70f9937959319e17fd4
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-60
TaskStatus
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
conda-win-vs2015-py36Appveyor Status
conda-linux-gcc-py37unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-osx-clang-py36unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24997) builder has been succeeded.
Revision: 5bfdf1a047e9da6b9638d4ece2c6e46778ab5a6c
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-61
TaskStatus
conda-win-vs2015-py36Appveyor Status
conda-osx-clang-py36unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-linux-gcc-py37unsupported CI service azure
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25036) builder has been succeeded.
Revision: f9a2f5ca12232e66e931b16ce733695789dc3e91
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-62
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25299) builder has been succeeded.
Revision: ec85d4c5c096cf47b3e416eef45f203f80f9a41b
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-63
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
removed
the 
WIP
PR is work in progress
label
Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
One of the azure build has lost connection, but I cannot restart it on the UI.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25411) builder has been succeeded.
Revision: ea32b2e5cea273a9713b491bfc09ceab9a9def48
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-65
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25445) builder has been succeeded.
Revision: 597fc8a839f46b99eae8ea06176a1a4d64162441
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-67
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
changed the title
[Crossbow] Port conda recipes to azure pipelinesARROW-5725: [Crossbow] Port conda recipes to azure pipelines Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25501) builder has been succeeded.
Revision: 2605fbaf06ebf419b72b22839dd182483ebf171d
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-69
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
The builds are passing, but the artifacts are not uploaded correctly.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25538) builder has been succeeded.
Revision: be7037aeb464a2e3456e4ddef7122b6c79e3055f
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-70
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
pitrou
pitrou
reviewed
Jun 25, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I have anything constructive to say about this, sorry. Some questions though...
Comment threaddev/tasks/conda-recipes/.ci_support/linux_python2.7.yaml
Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/appveyor.win.yml
@@ -1,20 +1,3 @@
# Licensed to the Apache Software Foundation (ASF) under one
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really mind, but is there a reason why you removed the license headers?
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because these files are mostly ported from the [py]arrow[-cpp] feedstocks. There is a note in the LICENSE.txt that files under dev/tasks/conda-recipes have another license. Also added to the rat exclude list.
Comment threaddev/tasks/conda-recipes/arrow-cpp/build.sh

# Include g++'s system headers
if [ "$(uname)" == "Linux" ]; then
 SYSTEM_INCLUDES=$(echo | ${CXX} -E -Wp,-v -xc++ - 2>&1 | grep '^ ' | awk '{print "-isystem;" substr($1, 1)}' | tr '\n' ';')
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally there would be clean way to write this...

btw, here I get:
$ CXX=g++
$ echo$(echo |${CXX} -E -Wp,-v -xc++ - 2>&1| grep '^ '| awk '{print "-isystem;" substr($1, 1)}'| tr '\n'';')
-isystem;6:;-isystem;7:;-isystem;8:;-isystem;9:;-isystem;10:;-isystem;11:;-isystem;12:;-isystem;13:;
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to any suggestions.
@pitroupitrouJun 26, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I have none, sorry.
Comment threaddev/tasks/conda-recipes/azure.linux.yml
Comment threaddev/tasks/docker-tests/azure.linux.yml
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package conda-win-vs2015-py36 conda-win-vs2015-py37
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25665) builder has been succeeded.
Revision: 20f6cec
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-73
TaskStatus
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
@pitrou the artifacts are properly uploaded to github releases now
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
Side note, would it be possible for ursabot to post links to artifacts in its report above? Finding files in the github releases is a bit cumbersome.
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
It is more complicated then the report above.
  1. python crossbow.py submit ... creates the branches and triggers the builds on various CIs
  2. python crossbow.py status ... polls the github api about these statuses, checks and release assets
The ursabot crossbow command is not a service like buildbot, thus it is not notified about changes with hooks, and the ursabot crossbow command is currently only a proxy to python crossbow.py submit ..., executed via buildbot.
On the long term I could port the functionality of crossbow into ursabot, but it requires the development of a series of meta services, a meta-scheduler, a meta-builder, a meta-worker. The scheduler would manipulate the crossbow repository directly (preferably via github api instead of native git bindings), the triggered meta builders would follow the logs of other services via their api and we'd be notified about crossbow's status and check updates by github hooks, then we could properly report back all of the results (including the artifacts).
TLDR it is possible, but requires more time.
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
I don't mean post the links when they are available, but upfront (so they would give a 404 until the build is finished). It depends whether the URLs are deterministic or not...
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
This is how the badges are working. For the conda packages the assets are not deterministic: https://github.com/apache/arrow/pull/4649/files#diff-0abe5ae3589de012350559065a4194a6R155
pitrou
pitrou
approved these changes
Jun 26, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. No objection from me.
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Apr 8, 2021
 Closed

@asfimportasfimport
mentioned this pull request
Jun 26, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@kszucs@ursabot@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines  - #4649
Closed
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Closed
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines #4649
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Conversation
@kszucs
@kszucskszucs
commented
Jun 21, 2019•

edited

Loading

Copy link


Copy Markdown

Member
@kszucskszucs
added
the 
WIP
PR is work in progress
label
Jun 21, 2019
@kszucskszucs
mentioned this pull request
Jun 24, 2019
 Closed

@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24978) builder has been succeeded.
Revision: 14a15b61fcf1ccb00010c70f9937959319e17fd4
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-60
TaskStatus
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
conda-win-vs2015-py36Appveyor Status
conda-linux-gcc-py37unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-osx-clang-py36unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24997) builder has been succeeded.
Revision: 5bfdf1a047e9da6b9638d4ece2c6e46778ab5a6c
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-61
TaskStatus
conda-win-vs2015-py36Appveyor Status
conda-osx-clang-py36unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-linux-gcc-py37unsupported CI service azure
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25036) builder has been succeeded.
Revision: f9a2f5ca12232e66e931b16ce733695789dc3e91
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-62
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25299) builder has been succeeded.
Revision: ec85d4c5c096cf47b3e416eef45f203f80f9a41b
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-63
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
removed
the 
WIP
PR is work in progress
label
Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
One of the azure build has lost connection, but I cannot restart it on the UI.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25411) builder has been succeeded.
Revision: ea32b2e5cea273a9713b491bfc09ceab9a9def48
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-65
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25445) builder has been succeeded.
Revision: 597fc8a839f46b99eae8ea06176a1a4d64162441
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-67
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
changed the title
[Crossbow] Port conda recipes to azure pipelinesARROW-5725: [Crossbow] Port conda recipes to azure pipelines Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25501) builder has been succeeded.
Revision: 2605fbaf06ebf419b72b22839dd182483ebf171d
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-69
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
The builds are passing, but the artifacts are not uploaded correctly.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25538) builder has been succeeded.
Revision: be7037aeb464a2e3456e4ddef7122b6c79e3055f
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-70
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
pitrou
pitrou
reviewed
Jun 25, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I have anything constructive to say about this, sorry. Some questions though...
Comment threaddev/tasks/conda-recipes/.ci_support/linux_python2.7.yaml
Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/appveyor.win.yml
@@ -1,20 +1,3 @@
# Licensed to the Apache Software Foundation (ASF) under one
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really mind, but is there a reason why you removed the license headers?
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because these files are mostly ported from the [py]arrow[-cpp] feedstocks. There is a note in the LICENSE.txt that files under dev/tasks/conda-recipes have another license. Also added to the rat exclude list.
Comment threaddev/tasks/conda-recipes/arrow-cpp/build.sh

# Include g++'s system headers
if [ "$(uname)" == "Linux" ]; then
 SYSTEM_INCLUDES=$(echo | ${CXX} -E -Wp,-v -xc++ - 2>&1 | grep '^ ' | awk '{print "-isystem;" substr($1, 1)}' | tr '\n' ';')
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally there would be clean way to write this...

btw, here I get:
$ CXX=g++
$ echo$(echo |${CXX} -E -Wp,-v -xc++ - 2>&1| grep '^ '| awk '{print "-isystem;" substr($1, 1)}'| tr '\n'';')
-isystem;6:;-isystem;7:;-isystem;8:;-isystem;9:;-isystem;10:;-isystem;11:;-isystem;12:;-isystem;13:;
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to any suggestions.
@pitroupitrouJun 26, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I have none, sorry.
Comment threaddev/tasks/conda-recipes/azure.linux.yml
Comment threaddev/tasks/docker-tests/azure.linux.yml
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package conda-win-vs2015-py36 conda-win-vs2015-py37
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25665) builder has been succeeded.
Revision: 20f6cec
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-73
TaskStatus
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
@pitrou the artifacts are properly uploaded to github releases now
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
Side note, would it be possible for ursabot to post links to artifacts in its report above? Finding files in the github releases is a bit cumbersome.
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
It is more complicated then the report above.
  1. python crossbow.py submit ... creates the branches and triggers the builds on various CIs
  2. python crossbow.py status ... polls the github api about these statuses, checks and release assets
The ursabot crossbow command is not a service like buildbot, thus it is not notified about changes with hooks, and the ursabot crossbow command is currently only a proxy to python crossbow.py submit ..., executed via buildbot.
On the long term I could port the functionality of crossbow into ursabot, but it requires the development of a series of meta services, a meta-scheduler, a meta-builder, a meta-worker. The scheduler would manipulate the crossbow repository directly (preferably via github api instead of native git bindings), the triggered meta builders would follow the logs of other services via their api and we'd be notified about crossbow's status and check updates by github hooks, then we could properly report back all of the results (including the artifacts).
TLDR it is possible, but requires more time.
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
I don't mean post the links when they are available, but upfront (so they would give a 404 until the build is finished). It depends whether the URLs are deterministic or not...
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
This is how the badges are working. For the conda packages the assets are not deterministic: https://github.com/apache/arrow/pull/4649/files#diff-0abe5ae3589de012350559065a4194a6R155
pitrou
pitrou
approved these changes
Jun 26, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. No objection from me.
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Apr 8, 2021
 Closed

@asfimportasfimport
mentioned this pull request
Jun 26, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@kszucs@ursabot@pitrou

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines  - #4649
Closed
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Closed
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines #4649
kszucs wants to merge 39 commits into
apache:masterfrom 
kszucs:crossbow-azure
Conversation
@kszucs
@kszucskszucs
commented
Jun 21, 2019•

edited

Loading

Copy link


Copy Markdown

Member
@kszucskszucs
added
the 
WIP
PR is work in progress
label
Jun 21, 2019
@kszucskszucs
mentioned this pull request
Jun 24, 2019
 Closed

@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24978) builder has been succeeded.
Revision: 14a15b61fcf1ccb00010c70f9937959319e17fd4
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-60
TaskStatus
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
conda-win-vs2015-py36Appveyor Status
conda-linux-gcc-py37unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-osx-clang-py36unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
@kszucs
kszucs
commented
Jun 24, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 24, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#24997) builder has been succeeded.
Revision: 5bfdf1a047e9da6b9638d4ece2c6e46778ab5a6c
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-61
TaskStatus
conda-win-vs2015-py36Appveyor Status
conda-osx-clang-py36unsupported CI service azure
conda-osx-clang-py27unsupported CI service azure
conda-linux-gcc-py36unsupported CI service azure
conda-osx-clang-py37unsupported CI service azure
conda-linux-gcc-py37unsupported CI service azure
conda-linux-gcc-py27unsupported CI service azure
conda-win-vs2015-py37Appveyor Status
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25036) builder has been succeeded.
Revision: f9a2f5ca12232e66e931b16ce733695789dc3e91
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-62
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25299) builder has been succeeded.
Revision: ec85d4c5c096cf47b3e416eef45f203f80f9a41b
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-63
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
removed
the 
WIP
PR is work in progress
label
Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
One of the azure build has lost connection, but I cannot restart it on the UI.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25411) builder has been succeeded.
Revision: ea32b2e5cea273a9713b491bfc09ceab9a9def48
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-65
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25445) builder has been succeeded.
Revision: 597fc8a839f46b99eae8ea06176a1a4d64162441
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-67
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucskszucs
changed the title
[Crossbow] Port conda recipes to azure pipelinesARROW-5725: [Crossbow] Port conda recipes to azure pipelines Jun 25, 2019
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25501) builder has been succeeded.
Revision: 2605fbaf06ebf419b72b22839dd182483ebf171d
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-69
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
The builds are passing, but the artifacts are not uploaded correctly.
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package -g conda
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25538) builder has been succeeded.
Revision: be7037aeb464a2e3456e4ddef7122b6c79e3055f
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-70
TaskStatus
conda-linux-gcc-py27Azure
conda-linux-gcc-py36Azure
conda-linux-gcc-py37Azure
conda-osx-clang-py27Azure
conda-osx-clang-py36Azure
conda-osx-clang-py37Azure
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
pitrou
pitrou
reviewed
Jun 25, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I have anything constructive to say about this, sorry. Some questions though...
Comment threaddev/tasks/conda-recipes/.ci_support/linux_python2.7.yaml
Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/README.md
Outdated

Comment threaddev/tasks/conda-recipes/appveyor.win.yml
@@ -1,20 +1,3 @@
# Licensed to the Apache Software Foundation (ASF) under one
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really mind, but is there a reason why you removed the license headers?
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because these files are mostly ported from the [py]arrow[-cpp] feedstocks. There is a note in the LICENSE.txt that files under dev/tasks/conda-recipes have another license. Also added to the rat exclude list.
Comment threaddev/tasks/conda-recipes/arrow-cpp/build.sh

# Include g++'s system headers
if [ "$(uname)" == "Linux" ]; then
 SYSTEM_INCLUDES=$(echo | ${CXX} -E -Wp,-v -xc++ - 2>&1 | grep '^ ' | awk '{print "-isystem;" substr($1, 1)}' | tr '\n' ';')
@pitroupitrouJun 25, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally there would be clean way to write this...

btw, here I get:
$ CXX=g++
$ echo$(echo |${CXX} -E -Wp,-v -xc++ - 2>&1| grep '^ '| awk '{print "-isystem;" substr($1, 1)}'| tr '\n'';')
-isystem;6:;-isystem;7:;-isystem;8:;-isystem;9:;-isystem;10:;-isystem;11:;-isystem;12:;-isystem;13:;
@kszucskszucsJun 25, 2019

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm open to any suggestions.
@pitroupitrouJun 26, 2019

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately I have none, sorry.
Comment threaddev/tasks/conda-recipes/azure.linux.yml
Comment threaddev/tasks/docker-tests/azure.linux.yml
@kszucs
kszucs
commented
Jun 25, 2019

Copy link


Copy Markdown

MemberAuthor
@ursabot crossbow package conda-win-vs2015-py36 conda-win-vs2015-py37
@ursabot
ursabot
commented
Jun 25, 2019

Copy link


Copy Markdown

AMD64 Conda Crossbow (#25665) builder has been succeeded.
Revision: 20f6cec
Submitted crossbow builds: ursa-labs/crossbow @ ursabot-73
TaskStatus
conda-win-vs2015-py36Appveyor
conda-win-vs2015-py37Appveyor
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
@pitrou the artifacts are properly uploaded to github releases now
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
Side note, would it be possible for ursabot to post links to artifacts in its report above? Finding files in the github releases is a bit cumbersome.
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
It is more complicated then the report above.
  1. python crossbow.py submit ... creates the branches and triggers the builds on various CIs
  2. python crossbow.py status ... polls the github api about these statuses, checks and release assets
The ursabot crossbow command is not a service like buildbot, thus it is not notified about changes with hooks, and the ursabot crossbow command is currently only a proxy to python crossbow.py submit ..., executed via buildbot.
On the long term I could port the functionality of crossbow into ursabot, but it requires the development of a series of meta services, a meta-scheduler, a meta-builder, a meta-worker. The scheduler would manipulate the crossbow repository directly (preferably via github api instead of native git bindings), the triggered meta builders would follow the logs of other services via their api and we'd be notified about crossbow's status and check updates by github hooks, then we could properly report back all of the results (including the artifacts).
TLDR it is possible, but requires more time.
@pitrou
pitrou
commented
Jun 26, 2019

Copy link


Copy Markdown

Member
I don't mean post the links when they are available, but upfront (so they would give a 404 until the build is finished). It depends whether the URLs are deterministic or not...
@kszucs
kszucs
commented
Jun 26, 2019

Copy link


Copy Markdown

MemberAuthor
This is how the badges are working. For the conda packages the assets are not deterministic: https://github.com/apache/arrow/pull/4649/files#diff-0abe5ae3589de012350559065a4194a6R155
pitrou
pitrou
approved these changes
Jun 26, 2019
View reviewed changes
@pitroupitrou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. No objection from me.
@jorisvandenbosschejorisvandenbossche
mentioned this pull request
Apr 8, 2021
 Closed

@asfimportasfimport
mentioned this pull request
Jun 26, 2019

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@pitroupitroupitrou approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@kszucs@ursabot@pitrou