Uh oh!
There was an error while loading. Please reload this page.
- Notifications
You must be signed in to change notification settings - Fork 4
add nix expressions for building and running CountESS and github workflow for CI#1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Uh oh!
There was an error while loading. Please reload this page.
Changes from all commits
File filter
Filter by extension
Conversations
Uh oh!
There was an error while loading. Please reload this page.
Jump to
Uh oh!
There was an error while loading. Please reload this page.
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| name: "Test" | ||
| on: | ||
| pull_request: | ||
| push: | ||
| jobs: | ||
| tests: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v3 | ||
| - uses: cachix/install-nix-action@v19 | ||
| - run: nix build | ||
| - run: nix flake check |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -5,3 +5,4 @@ build/ | ||
| .*.sw? | ||
| *.csv | ||
| .direnv/ | ||
| result* | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| { | ||
| python3Packages, | ||
| cmake, | ||
| }: let | ||
| fqfa = with python3Packages; | ||
| buildPythonPackage rec { | ||
| pname = "fqfa"; | ||
| version = "1.2.3"; | ||
| src = fetchPypi { | ||
| inherit pname version; | ||
| sha256 = "sha256-ex0IJfbFWTdFQhXWxtUeH5OQPvaU7LoRnKL9QWyKBos="; | ||
| }; | ||
| format = "pyproject"; | ||
| nativeBuildInputs = [hatchling]; | ||
| pythonImportsCheck = ["fqfa"]; | ||
| }; | ||
| ttkthemes = with python3Packages; | ||
| buildPythonPackage rec { | ||
| pname = "ttkthemes"; | ||
| version = "3.2.2"; | ||
| src = fetchPypi { | ||
| inherit pname version; | ||
| sha256 = "sha256-AdrtAB8v8OTzKDKg2epIF2wMUFIDsDB1a93jvRvLIdI="; | ||
| }; | ||
| propagatedBuildInputs = [ | ||
| tkinter | ||
| pillow | ||
| ]; | ||
| }; | ||
| blosc2 = with python3Packages; | ||
| buildPythonPackage rec { | ||
| pname = "blosc2"; | ||
| version = "2.0.0"; | ||
| src = fetchPypi { | ||
| inherit pname version; | ||
| sha256 = "sha256-8ZsLNnT2yCW0kPANgmSwxUDCzcEex+gReNOLg8V3kKE="; | ||
| }; | ||
| nativeBuildInputs = [scikit-build cmake cython]; | ||
| propagatedBuildInputs = [ | ||
| numpy | ||
| msgpack | ||
| ]; | ||
| dontUseCmakeConfigure = true; | ||
| format = "pyproject"; | ||
| }; | ||
| in | ||
| with python3Packages; | ||
| buildPythonApplication { | ||
| pname = "CountESS"; | ||
| version = "git"; | ||
| src = ./.; | ||
| propagatedBuildInputs = [ | ||
| dask | ||
| distributed | ||
| fqfa | ||
| more-itertools | ||
| levenshtein | ||
| numpy | ||
| pandas | ||
| (tables.overrideAttrs (attrs: rec { | ||
| version = "3.8.0"; | ||
| src = fetchPypi { | ||
| pname = "tables"; | ||
| inherit version; | ||
| sha256 = "sha256-NPP6I2bOILGPHfVzp3wdJzBs4fKkHZ+e/2IbUZLqh4g="; | ||
| }; | ||
| nativeBuildInputs = attrs.nativeBuildInputs ++ [sphinx]; | ||
| propagatedBuildInputs = | ||
| attrs.propagatedBuildInputs | ||
| ++ [ | ||
| blosc2 | ||
| py-cpuinfo | ||
| ]; | ||
| doInstallCheck = false; | ||
| })) | ||
| pyarrow | ||
| ttkthemes | ||
| ]; | ||
| nativeBuildInputs = [cython]; | ||
| preConfigure = '' | ||
jbedo marked this conversation as resolved.
Uh oh!There was an error while loading. Please reload this page. | ||
| substituteInPlace setup.py \ | ||
| --replace "pyarrow~=10.0.1" "pyarrow"\ | ||
| --replace "pandas~=1.5.2" "pandas" # this is 1.5.2 but not detecting version correctly | ||
| ''; | ||
| meta = { | ||
| mainProgram = "countess_gui"; | ||
| }; | ||
| } | ||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Uh oh!
There was an error while loading. Please reload this page.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| { | ||
| inputs = { | ||
| nixpkgs.url = "github:nixos/nixpkgs/release-22.11"; | ||
| flake-utils.url = "github:numtide/flake-utils"; | ||
| }; | ||
| outputs = { | ||
| self, | ||
| nixpkgs, | ||
| flake-utils, | ||
| }: | ||
| flake-utils.lib.eachDefaultSystem (system: let | ||
| pkgs = import nixpkgs {inherit system;}; | ||
| in { | ||
| packages = rec { | ||
| countess = pkgs.callPackage ./. {}; | ||
| default = countess; | ||
| }; | ||
| }); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know much about Nix, and I suspect this is Just How It Is Done In Nix, but is there any way to prevent/detect these versions getting out of step with the versions in setup.py?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sadly no, with the base nixpkgs only there's no mechanism available to read python requirements and solve the constraint problem (nixpkgs does not allow dynamic resolution of dependencies, i.e., import from derivation in nix lingo, for some good reasons). This is what's done in more complex solutions (like https://github.com/DavHau/mach-nix) but the dependencies here are not numerous so I think it's worth sticking to the far simpler nixpkgs solution and updating manually as needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, probably. I guess my real question is: will the Nix package build process break if they get out of step?
If so we can add a CI rule for it or something, thus reminding us to fix it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes it can break, CI is a good idea. Not sure what CI you're currently using for this project, but there's a github workflow that will build the flake that's easy to setup https://github.com/marketplace/actions/install-nix#usage-with-flakes. Would you like me to add that to this PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sounds great, thanks, there's no CI at the moment but it's definitely something we need ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done, and working over here: https://github.com/jbedo/CountESS/actions/runs/4251500407/jobs/7393927424
There is a caveat though, since tables 3.8.0 is specified in this expression (nixpkgs currently only has 3.7.0) it is not cached and so gets built as part of the CI. The tests take a while to run, so the total time used in the github workflow is >20m. It's tempting to just disable the test suite to speed this up. Thoughts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've disabled the install checks for tables, it not a good tradeoff to do this repeatedly in the CI and also isn't a great user experience for somebody running CountESS to have to wait ~20m for it to start. When tables 3.8.0 is in nixpkgs we can bump the flake and the problem will go away. Build down to ~5m: https://github.com/jbedo/CountESS/actions/runs/4257342035/jobs/7407356911