Datacheck: FP matching

[yakra]

DataProcessing/siteupdate/python-teresco/siteupdate.py

Lines 1158 to 1187 in 18f3a08

	def match(self,fpentry):
	"""Check if the fpentry from the csv file matches in all fields"""
	# quick and easy checks first
	if self.route.root != fpentry[0] or self.code != fpentry[4]:
	return False
	# now label matches
	if len(self.labels) > 0 and self.labels[0] != fpentry[1]:
	return False
	if len(self.labels) > 1 and self.labels[1] != fpentry[2]:
	return False
	if len(self.labels) > 2 and self.labels[2] != fpentry[3]:
	return False
	# finally make sure info matches if we got this far
	return self.info == fpentry[5]
	def match_except_info(self,fpentry):
	"""Check if the fpentry from the csv file matches in all fields
	except the info field"""
	# quick and easy checks first
	if self.route.root != fpentry[0] or self.code != fpentry[4]:
	return False
	# now label matches
	if len(self.labels) > 0 and self.labels[0] != fpentry[1]:
	return False
	if len(self.labels) > 1 and self.labels[1] != fpentry[2]:
	return False
	if len(self.labels) > 2 and self.labels[2] != fpentry[3]:
	return False
	# call it a match except info if the info does not match
	return self.info != fpentry[5]

Proposed:

 def match_except_info(self,fpentry):
 """Check if the fpentry from the csv file matches in all fields
 except the info field"""
 # quick and easy checks first
 if self.route.root != fpentry[0] or self.code != fpentry[4]:
 return False
 # now label matches
 if len(self.labels) > 0 and self.labels[0] != fpentry[1]:
 return False
 if len(self.labels) > 1 and self.labels[1] != fpentry[2]:
 return False
 if len(self.labels) > 2 and self.labels[2] != fpentry[3]:
 return False
 return True

 def match_info(self,fpentry):
 return self.info == fpentry[5]

---

DataProcessing/siteupdate/python-teresco/siteupdate.py

Lines 3281 to 3290 in 18f3a08

	for fp in datacheckfps:
	#print("Comparing: " + str(d) + " to " + str(fp))
	if d.match(fp):
	#print("Match!")
	d.fp = True
	toremove.append(fp)
	break
	if d.match_except_info(fp):
	fpfile.write("FP_ENTRY: " + fp[0] + ';' + fp[1] + ';' + fp[2] + ';' + fp[3] + ';' + fp[4] + ';' + fp[5] + '\n')
	fpfile.write("CHANGETO: " + fp[0] + ';' + fp[1] + ';' + fp[2] + ';' + fp[3] + ';' + fp[4] + ';' + d.info + '\n')

Proposed:

 for fp in datacheckfps:
 #print("Comparing: " + str(d) + " to " + str(fp))
 if d.match_except_info(fp):
 if d.match_info(fp):
 #print("Match!")
 d.fp = True
 toremove.append(fp)
 break
 else:
 fpfile.write("FP_ENTRY: " + fp[0] + ';' + fp[1] + ';' + fp[2] + ';' + fp[3] + ';' + fp[4] + ';' + fp[5] + '\n')
 fpfile.write("CHANGETO: " + fp[0] + ';' + fp[1] + ';' + fp[2] + ';' + fp[3] + ';' + fp[4] + ';' + d.info + '\n')

[yakra]

While in there making changes,

DataProcessing/siteupdate/python-teresco/siteupdate.py

Line 3327 in 18f3a08

logfile.write(";;;;")

should only have three semicolons.

[jteresco]

No objections from me. Thanks!

[yakra]

Removing matched FPs straight away instead of adding them to the toremove list and removing later could save time on successive passes thru the inner for loop

[yakra]

C++, not Python:

• I estimate that a theoretical version of my code most comparable to how siteupdate.py is set up now would have taken 28.8 s Marking datacheck false positives.
• Matching the FP fields other than Info once instead of twice brings it down to 19.5 s.
• Removing matched FP entries from the datacheckfps list right away instead of adding to a toremove list and re-matching & removing later brings it down to 4.1 s.
• DIFFs looked good, though I haven't implemented all datachecks yet.

If similar improvements can be realized in siteupdate.py that'd be pretty good.

[yakra]

Python on BiggaTomato:

• Marking datacheck false positives. starts out at 39.9 s.
• Matching the FP fields other than Info once instead of twice brings it down to 20.5 s.
• Removing matched FP entries from the datacheckfps list right away instead of adding to a toremove list and removing later brings it down to 4.3 s.
• DIFFs looked good.

Indeed pretty good. Percentage-wise, that's an even better improvement than making the equivalent changes to the C++ version.