json.dumps to check for obj.__json__ before raising TypeError

[DanielWard]

BPO	27362
Nosy	@rhettinger, @etrepum, @stevendaprano, @bitdancer, @berkerpeksag, @serhiy-storchaka, @Vgr255
Files	json-customize.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/etrepum'
closed_at = None
created_at = <Date 2016-06-21.13:09:18.565>
labels = ['3.7', 'type-feature', 'library']
title = 'json.dumps to check for obj.__json__ before raising TypeError'
updated_at = <Date 2018-02-01.16:33:45.801>
user = 'https://bugs.python.org/DanielWard'

bugs.python.org fields:

activity = <Date 2018-02-01.16:33:45.801>
actor = 'ulope'
assignee = 'bob.ippolito'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2016-06-21.13:09:18.565>
creator = 'Daniel Ward'
dependencies = []
files = ['46701']
hgrepos = []
issue_num = 27362
keywords = ['patch']
message_count = 11.0
messages = ['268991', '268993', '268995', '268999', '269002', '269003', '269021', '269091', '289001', '289006', '289007']
nosy_count = 10.0
nosy_names = ['rhettinger', 'bob.ippolito', 'steven.daprano', 'r.david.murray', 'berker.peksag', 'serhiy.storchaka', 'abarry', 'ulope', 'Daniel Ward', 'Ollie Ford']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue27362'
versions = ['Python 3.7']

Linked PRs

• gh-71549: Add support for JSON serialization of custom types #153607

[DanielWard]

To help prevent retrospective JSONEncoder overrides when failing to serialize a given object, the intention of this issue is to propose that the JSON encoder checks if a given object has a __json__ attribute, using that rather than raising a TypeError.

This will help in maintaining easier-to-follow code and keeps the responsibility of determining how an object should be represented in JSON objects firmly within the object itself.

The obj.__json__ callable/attribute should behave in the same way as __repr__ or __str__, for example.

I'm happy to look in to contributing this enhancement myself if that's preferred. Any pointers as to how I go about contributing are greatly appreciated.

[Vgr255]

I'm not too familiar with the json package, but what should __json__ return when called?

[DanielWard]

Sure, so for example:

=========

import json


class ObjectCounter:

 def __init__(self, name, count):
 self.name = name
 self.count = count

 def __json__(self):
 return '[{name}] {count}'.format(name=self.name, count=self.count)


object_counter = ObjectCounter('DC1', 3789)
my_json_string = json.dumps({'success': True, 'counter': object_counter})

============

In the above example, the value stored in my_json_string would be:

'{"success": true, "counter": "[DC1] 3789"}'

This is an untested and quick example, but I hope it explains what I'm aiming to achieve. Without the __json__ method, the json.dumps call would raise an exception along the lines of the below message, unless we create a new JSONEncoder object and call json.dumps(..., cls=MyJSONEncoder), which becomes difficult to manage and follow on larger projects.

TypeError: <ObjectCounter instance at XXX> is not JSON serializable

[Vgr255]

So __json__ returns a string meant to be serializable. I'm not too keen on using a dunder name (although my word doesn't weigh anything ;) and I'd personally prefer something like as_json_string(). I think the idea in general is good, though. Mind submitting a patch?

[stevendaprano]

For starters, dunder names like __json__ are reserved for Python's own use, so you would have to get the core developers to officially bless this use.

But... I'm not really sure that "the responsibility of determining how an object should be represented in JSON objects firmly within the object itself" is a good idea. For a general purpose protocol, I don't think you can trust any object to return valid JSON. What if my object.__json__ returned "}key='c" or some other invalid string? Whose responsibility is it to check that __json__ returns valid JSON?

I don't think there is any need to make this an official protocol. You know your own objects, you know if you can trust them, and you can call any method you like. So your example becomes:

 my_json_string = json.dumps(
 {'success': True, 'counter': object_counter.to_json()})

which is okay because that's clearly *your* responsibility to make sure that your object's to_json method returns a valid string. If you make it an official language wide protocol, it's unclear whose responsibility it is: the object (dangerous!), the caller (difficult), the Python interpreter (unlikely), json.dumps (unlikely).

[DanielWard]

I don't think I explained the response very well, effectively the __json__ call would return an object which is JSON-serializable. This would include dict objects containing JSON-serializable objects albeit natively-supporting JSON serialisation or by means of subsequent obj.__json__ calls.

The reason I gave it __json__ is purely for easily-remembered implementation, separating it out from calls which may potentially clash with existing codebases, because let's face it, people don't often get to start again ;)

I'm not adverse to changing the method name at all, but I do believe this is a progressive way to go regarding JSON-serialization.

[berkerpeksag]

This was discussed on python-ideas before:

• https://groups.google.com/forum/#!msg/python-ideas/MbvpK21nUNA/DwYpOFOVuYMJ
• https://groups.google.com/forum/#!msg/python-ideas/JVLuo-L0reU/zLNnBa9THqgJ

I don't think there was an agreement on the idea so I suggest to send your proposal to python-ideas first.

[bitdancer]

Pretty much any project that makes non-trivial use of json ends up implementing a jsonification protocol, usually by creating either a __json__ method or (more commonly, I think) a to_json method.

But, yeah, this is python-ideas material and would get into the stdlib only as an officially blessed protocol, in which case using __json__ would make sense. So I'm going to close the issue pending a consensus on python-ideas. If it gets accepted the issue can be reopened.

[serhiy-storchaka]

This could fix other issues:

• bpo-16535 -- for Decimal.
• bpo-20774 -- for deque.
• bpo-24313 -- for NumPy numeric types.
• bpo-26263 -- for array.

Currently the blessed way of JSON encoder customization is to implement the default method in JSONEncoder subclass or pass the default argument to dump(). But that requires changing every JSON serialization call and handling all non-standard types in one function. I think it would be handly to pick the type-specific serialization function from: 1) per-encoder dispatch table, 2) global dispatch type (registry), 3) __json__ method. This can be done after the default function fails or be included in the default default method.

This will add JSON support of standard library types (e.g. collections other than list, tuple and dict, numbers other than int and float) and will help to implement task specific serialization of user classes.

[rhettinger]

I concur with David Murray that this should be kicked around on python-dev or python-ideas first. Also, we should ask Bob Ippolito for his thoughts.