shell script: users affected by a relocation

[yakra]

https://forum.travelmapping.net/index.php?topic=4804.msg26915#msg26915

idea: a shall script that, given a route section (input in standard 4-field .list line style format) that's been relocated in the middle of a route, finds all .list files affected. Even those using two "good" unchanged points on either side of the relocation.

• For both point labels in .list lines referencing this route, look up line numbers,

  file=HighwayData/hwy_data/ME/usame/me.me077.wpt #for example
  LabelTable=`cat -n $file | sed -r -e 's~^ *~~' -e 's~http.*~~' | tr -d '+*' | tr '\t' ' '`
  echo "$LabelTable" | grep -i " $ALabelFromSomeonesList " | cut -f1 -d' '

  find out whether before, within, or after relocated section
  • 1st point within -> flag
  • 2nd point within -> flag
  • before & before -> don't
  • before & after -> flag
  • after & before -> flag
  • after & after -> don't
• Invalid labels will yield an empty string for a line number. Skip over stuff.
• 6-field lines on one chopped route will be easy. Convert to 4-field using sed.
• Multi-region 6-field lines will be more complicated.
• smash case on args & .list files for xform & 4->6 field conversions to work properly
• + & * are stripped from LabelTable. Strip from args & .list files too.
• search .list files: consolidate multiple sed commands
• Rewrite compile list phase, using git log --pretty. Incrementally populate both GitHubURLs and GitHub here, showing progress as we go. Then implement those changes in whouses.sh.
• account for chopped Routes flipped around for their ConnectedRoutes

[michihdeu]

We might use this magic for generating the Route updated 2022-02-13: PA US13 log file entries down the road....

[yakra]

Nope. This is a self-contained shell script, entirely different from the siteupdate program.
(I probably should have opened this issue in yakra/tmtools, but anyway.)

[michihdeu]

I think that the algorithm might be used for it - if we want. Am I wrong?

[yakra]

This operates on the old, about-to-be-replaced .wpt data; siteupdate operates on the new .wpt data.

Even if we somehow automated the creation of some kind of data structure storing the old<->new diffs and track when a user's travels go into the areas of a route covered by those diffs, we'd get lots of false positives from NMP fixes or minor recenterings, adding/deleting waypoints along the same existing route, etc. Probably too many to be useful.

[yakra]

Wait. You're not wrong.
At least, in the oversimplified way I was conceptualizing it.
It's still not something I want to pursue though.

What I didn't consider:

• Sure, this would produce false positives, but what we already have produces even more FPs.
• This would get rid of some (but not all) of the FPs we already have.

However...

• There'd need to be some way of distinguishing the most recent newsworthy update from other more recent non-newsworthy updates. This can be done, but it'd be a pain.
• Even if implemented successfully, it would be painfully slow. The info would have to be gotten via git, before launching siteupdate, the same way listupdates.txt (used for the Route updated 2022-02-13: PA US13 log file entries) is created now.
  Extrapolating from that, this could take about 6 minutes. Almost as much time as a full Py site update, even more time than a full C++ one.
  And that figure is unrealistically optimistic because this process is more complicated:
  • git log -n 1 $root.wpt -> git log --until=$ChangeDate --since=$ChangeDateMinus1WhichIsAPitA
  • then we'd have to git diff $OldCommit $NewCommit and process & save the data into a format usable by siteupdate.
• WRT getting the data into a format usable by siteupdate, maintaining compatibility with both mid-route relocations and truncations at either end could be complicated. Ow, my head.

All in all...
False positives are rare enough and transient enough in most logs for regularly-updated .list files that I don't think it's too big deal.
IMO, the gain is not worth the pain here.
This isn't something that translates very well out of a self-contained shell script to be invoked as needed, as people are interested.

[yakra]

painfully slow blah blah via git yadda yadda 6 minutes yadda yadda unrealistically optimistic

date
systems=$(grep -v '^#' systems.csv | egrep 'active$|preview$' | cut -f1 -d';' | sed 's~.*~hwy_data/_systems/&.csv~')
for root in `tail -n +2 updates.csv | cut -f4 -d';' | egrep -v ' |^$' | sort | uniq`; do
 rteinfo=$(grep -m 1 -hF $root $systems)
 sys=$(echo "$rteinfo" | cut -f1 -d';')
 rg=$(echo "$rteinfo" | cut -f2 -d';')
 if [ -e "hwy_data/$rg/$sys/$root.wpt" ]; then
 git log -n 1 --pretty='%P %H' hwy_data/$rg/$sys/$root.wpt > /dev/null
 fi
done; date

Unrealistically optimistic time is more like:

• 10m 54s on BiggaTomato
• 8m 56s on noreaster. Its SSD doesn't help very much.
• 8m 42 on lab2

This is still finding only the most recent commit, no diff, rather than finding 2 relevant to a given change & performing a diff.