Json.dump() bug when using generator

[biloup]

BPO	33850
Nosy	@doerwalter, @ericvsmith, @serhiy-storchaka
Superseder	bpo-27613: Empty iterator with fake len is rendered as a single bracket ] when using json's iterencode
Files	json-dump-generators-bug.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-06-13.13:12:52.881>
created_at = <Date 2018-06-13.08:44:21.503>
labels = ['type-bug', 'library']
title = 'Json.dump() bug when using generator'
updated_at = <Date 2018-06-13.13:12:52.879>
user = 'https://bugs.python.org/biloup'

bugs.python.org fields:

activity = <Date 2018-06-13.13:12:52.879>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2018-06-13.13:12:52.881>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2018-06-13.08:44:21.503>
creator = 'biloup'
dependencies = []
files = ['47640']
hgrepos = []
issue_num = 33850
keywords = ['patch']
message_count = 4.0
messages = ['319436', '319442', '319459', '319461']
nosy_count = 4.0
nosy_names = ['doerwalter', 'eric.smith', 'serhiy.storchaka', 'biloup']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '27613'
type = 'behavior'
url = 'https://bugs.python.org/issue33850'
versions = ['Python 3.6']

[biloup]

I use a class to write easily json when having generator.

class StreamArray(list):
 def __init__(self, generator):
 super().__init__()
 self.generator = generator
 def __iter__(self):
 return self.generator
 def __len__(self):
 return 1

Below a test comparing json.dump and json.dumps.

>>> import json
>>> class StreamArray(list):
... def __init__(self, generator):
... super().__init__()
... self.generator = generator
... def __iter__(self):
... return self.generator
... def __len__(self):
... return 1
... 
>>> g = (i for i in range(0))
>>> json.dumps({"a": StreamArray(g)})
'{"a": []}'
>>> f = open("/tmp/test.json", "w+")
>>> g = (i for i in range(0))
>>> json.dump({"a": StreamArray(g)}, f)
>>> f.close()
>>> print(open("/tmp/test.json").read())
{"a": ]}

I don't know if it's me or if there is actually a problem, could you help me ?

[ericvsmith]

You should ask your question on this mailing list: https://mail.python.org/mailman/listinfo/python-list

The bug tracker is not a place for asking how to use Python. If you actually find a bug in Python, you can re-open this issue. I do not believe that what you show here is a Python bug.

Good luck!

[doerwalter]

The problem here is that StreamArray lies about the length of the iterator. This confuses json.encoder._make_iterencode._iterencode_list(), (which is called by json.dump()), because it first does a check for "if not lst" and then assumes in the loop that it will be entered at least once.

(Note that json.dumps() doesn't have that problem, because it calls JSONEncoder.encode() with _one_shot=True which leads to a totally different code path).

We could declare that bug as "don't do that then", but the problem is easily solvable, because we can check whether the loop was entered. The attached patch should do the trick.

An even better approach would IMHO be, that the encoder supports a special flag that enables JSON serialization of generators directly, so it's no longer required to masquerade generators as list

[serhiy-storchaka]

This is a duplicate of bpo-27613.