Verify the etree_parse and etree_iterparse benchmarks are working appropriately

[brettcannon]

BPO	25638
Nosy	@brettcannon, @pitrou, @scoder, @serhiy-storchaka
PRs	[WIP] bpo-35502: Fix memory leak in xml.etree iterparse() #11169
Dependencies	bpo-25814: Propagate all errors from ElementTree.iterparse
Files	etree_iterparse.patch etree_iterparse_2.patch etree_start_handler_no_attrib.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-03-12.17:21:48.895>
created_at = <Date 2015-11-16.19:41:23.215>
labels = ['extension-modules', 'expert-XML', 'library', 'invalid', 'performance']
title = 'Verify the etree_parse and etree_iterparse benchmarks are working appropriately'
updated_at = <Date 2018-12-14.22:10:01.246>
user = 'https://github.com/brettcannon'

bugs.python.org fields:

activity = <Date 2018-12-14.22:10:01.246>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2016-03-12.17:21:48.895>
closer = 'brett.cannon'
components = ['Extension Modules', 'Library (Lib)', 'XML', 'Benchmarks']
creation = <Date 2015-11-16.19:41:23.215>
creator = 'brett.cannon'
dependencies = ['25814']
files = ['41110', '41167', '41276']
hgrepos = []
issue_num = 25638
keywords = ['patch']
message_count = 20.0
messages = ['254746', '254748', '254751', '254752', '254753', '254763', '254764', '255021', '255024', '255027', '255050', '255394', '255936', '256013', '256039', '256059', '256158', '256167', '256172', '261651']
nosy_count = 6.0
nosy_names = ['brett.cannon', 'pitrou', 'scoder', 'eli.bendersky', 'python-dev', 'serhiy.storchaka']
pr_nums = ['11169']
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue25638'
versions = ['Python 3.6']

[brettcannon]

If you look at bit.ly/pycon-ca-keynote you will notice that the etree_parse and etree_iterparse benchmarks were horrible for everyone. Because of how badly everyone seemed to do, I think the benchmarks should be verified to be doing reasonable things on implementations other than CPython 2.7.

[pitrou]

Would you have a quick summary for those not willing to watch a whole keynote?

[brettcannon]

That link is to a Jupyter notebook so you don't have to watch anything. Plus the video is not even up yet so you can't skip the keynote even if you wanted to since you can't watch it yet. :)

[pitrou]

Ok, so when you say "horrible for everyone", this is really IronPython and Jython, right? :-) Other runtimes seem to do ok (perhaps not stellar, but ok).

[brettcannon]

Well, Jython and IronPython obviously did the worst, but even Python 3 didn't do as well as I would have expected, so I still want to double-check the benchmarks to see if it's obvious why CPython 2.7 beats out everyone.

[serhiy-storchaka]

I think these histograms would look better with logarithmic scale.

[brettcannon]

Let's not pollute the issue with a critique of my notebook. You can feel free to email me personally to discuss it if you want, including why I purposefully didn't use a logarithmic scale.

[serhiy-storchaka]

Sorry Brett.

How tests were ran? There are two implementations of ElementTree, accelerated and non-accelerated. xml.etree.ElementTree by default is accelerated in Python 3, but non-accelerated in Python 2.

$ python2.7 bm_elementtree.py -n 7 --take_geo_mean 
0.463665158795
$ python2.7 bm_elementtree.py -n 7 --take_geo_mean --etree-module=xml.etree.ElementTree
5.46309932568
$ python3.4 bm_elementtree.py -n 7 --take_geo_mean --etree-module=xml.etree.ElementTree
0.813397633467649
$ python3.4 bm_elementtree.py -n 7 --take_geo_mean --etree-module=xml.etree.ElementTree --no-accelerator
5.31174765817514

If run the test with the same options --etree-module=xml.etree.ElementTree, it will use accelerated implementation in Python 3 and non-accelerated in Python 2.

[brettcannon]

The commands I used are in the notebook for each implementation and you can get the same result with python3 perf.py -b etree python2 python3.

[serhiy-storchaka]

The slowing down Python 3 can be related to adding XMLPullParser (bpo-17741).

[serhiy-storchaka]

Proposed patch optimizes iterparse(). Now it is only 33% slower than in 2.7 (was 2.6 times slower).