Uh oh!
There was an error while loading. Please reload this page.
FIX: refactor Network and DAG SOLVER to fix bad pruning - #26
Open
ankostis wants to merge 30 commits into
Open
Conversation
were writing in text-mode in PY3. and failing as encoding error.
receiving partial inputs, needed for other operations.
+ The x2 TCs added just before are now passing.
NOTE dict are not deterministic in <PY3.6. So this commit would not improve determinism in those pythons. + build: add `boltons` dependency for ndexedSet. + doc: mark all set usage if affect determinism. + e.g. see reproducibility problem in yahoo#14.
the later described in yahoo#21.
not after compose(). + All TCs pass ok. + NOTE this is not yet what is described in yahoo#21.
ankostisforce-pushed
the
refact-new_dag_solver
branch
from
October 3, 2019 13:55
7e851b1 to
3ee344bCompareto pass +TC checking DeleteInst vary when inputs change. - x4 TCs still failing, and need revamp of dag-solution.
+ Read the next doc-only commit to understand changes. + Renamed: + net.steps --> net.execution_plan. + (old)compile() --> _build_execution_plan() + _find_necessary_steps() --> (new)compile() + _solve_dag() compile() became the master function invoking _solve_dag & _build-execution_plan(), and do the caching. + refact(compute()): extract common tasks from sequential/parallel. + refact show_layers() to allow full-print, geting also string (not just printing), and using custom classes for representation. + Raise AssertionError when invalid class in plan. it's a logic error, not a language type-error.
Probaly unreported bug in v1.2.4 for '_neccessary_steps_cache`.
ankostisforce-pushed
the
refact-new_dag_solver
branch
4 times, most recently
from
October 3, 2019 15:12
e67e57c to
31aae2fCompare+ Pruning behaves correctly also when outputs given; this happens by breaking incoming provide-links to any given intermedediate inputs. + Unsatisfied detection now includes those without outputs due to broken links (above). + Remove some uneeded "glue" from unsatisfied-detection code, leftover from previous compile() refactoring. + Renamed satisfiable --> satisfied. + Improved unknown output requested raise-message. + x3 TCs PASS, x1 in yahoo#24 and the first x2 in yahoo#25. - 1x TCs in yahoo#25 still FAIL, and need "Pinning" of given-inputs (the operation MUST and MUST NOT run in these cases).
ankostisforce-pushed
the
refact-new_dag_solver
branch
from
October 3, 2019 17:13
31aae2f to
0830b7cCompareankostis added a commit
to ankostis/graphtik
that referenced
this pull request
Oct 4, 2019
…must run + Insert "PinInstructions" in the execution-plan to avoid overwritting. + Add `_overwrite_collector` in `compose()` to collect re-calculated values. + FIX the last TC in yahoo#25.
ankostisforce-pushed
the
refact-new_dag_solver
branch
from
October 4, 2019 02:37
008d501 to
b870451Compare…must run - WIP: PARALLEL execution not adding PINS! + Insert "PinInstructions" in the execution-plan to avoid overwritting. + Add `_overwrite_collector` in `compose()` to collect re-calculated values. + FIX the last TC in yahoo#25.
ankostisforce-pushed
the
refact-new_dag_solver
branch
from
October 4, 2019 02:41
b870451 to
0dc1293CompareCodecov Report
@@ Coverage Diff @@## master #26 +/- ##
==========================================
+ Coverage 77.87% 78.31% +0.44%
==========================================
Files 5 5 Lines 348 392 +44 ==========================================
+ Hits 271 307 +36 - Misses 77 85 +8
Continue to review full report at Codecov.
|
ankostisforce-pushed
the
refact-new_dag_solver
branch
from
October 4, 2019 10:53
7c94891 to
91c7b11Compare- STILL buggy PIN on PARALLEL, 2 DISABLED TCs FAIL: - test_pruning_with_given_intermediate_and_asked_out() - test_unsatisfied_operations_same_out() + move check if value in asked outputs before cache-evicting it in build-execution-plan time - compute methods don't need outputs anymore. + test: speed up parallel/multihtread TCs by reducing delays & repetitions. + refact: network rightfully adopted stray functions for parallel processing - they all worke on the net.graph, + upd: networkx api by indexing on `dag.nodes` views. + enh: add log message when deleting in parallel (in par with sequential code). + refact: var-renames, if-then-else simplifications, pythonisms. + doc: A lot!
ankostisforce-pushed
the
refact-new_dag_solver
branch
from
October 4, 2019 21:25
91c7b11 to
06f6554Compareankostisforce-pushed
the
refact-new_dag_solver
branch
from
October 4, 2019 21:35
4603bbb to
1cc733eCompareankostis added a commit
to ankostis/graphtik
that referenced
this pull request
Oct 4, 2019
- WIP: x4 TCs FAIL and still not discovered th bug :-( + BUT ALL+AUGMENTED PARALLEL TCs pass (yahoo#26 were failing some) + refact: net stores also `pruned_dag` (not only `steps`). + refact: _solve_dag() --> _prune_dag(). + doc: +a lot. + TODO: store pruned_dag in own ExePlan class.
ankostis
commented
Oct 4, 2019
Author
Can merge now:
|
This was referenced Oct 5, 2019
ankostis referenced
this pull request
in syamajala/graphkit
Oct 8, 2019
ankostis added a commit
to ankostis/graphtik
that referenced
this pull request
Oct 8, 2019
many commits ago. Never got it bc TC were not checking merges!
ankostis added a commit
to ankostis/graphtik
that referenced
this pull request
Oct 8, 2019
many commits ago. Never got it bc TC were not checking merges!
ankostis
commented
Oct 8, 2019
Author
|
3 tasks
ankostis
commented
Oct 15, 2019
Author
- WIP: x4 TCs FAIL and still not discovered th bug :-( + BUT ALL+AUGMENTED PARALLEL TCs pass (yahoo#26 were failing some) + refact: net stores also `pruned_dag` (not only `steps`). + refact: _solve_dag() --> _prune_dag(). + doc: +a lot. + TODO: store pruned_dag in own ExePlan class.
... bugged in the opening commit d403783 of this PR, and discovered 68(!) commits later, and all that time had to live with x4 broken TCs with asked-outputs.
+ Partial fix deterministic results (yahoo#22-2.4.3i).
bc subgraph was taken on plain string outputs. + minor upd err-msg terminology.
due to bad node check, evicting parallels it nevered kicked in.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR refactors the DAG-solver to prune correctly when intermediate data are given as input.
test_pruning_with_given_intermediate_and_asked_out()test_unsatisfied_operations_same_out()ENH: NEW DAG SOLVER
to any given intermedediate inputs and retrofitting satisfaction detector
to include those operations without outputs due to these broken links.
REFACT: Network class COMPILE+COMPUTE:
_build-execution_plan(), and do the caching.
(not just printing).
net.graphattribute; pave the way for a separate class.networkxAPI when indexingdag.nodesviews.it's a logic error, not a language type-error.
I confirm that this contribution is made under the terms of the license found in the root directory of this repository's source tree and that I have the authority necessary to make this contribution on behalf of its copyright owner.