Support more date-time formats for JSON deserialization

[gregsdennis]

Description

Originally reported to JsonSchema.Net here.

The report says that System.Text.Json deserialization only supports ISO 8601 date-time formats. However JSON Schema specifies that RFC 3339 formats are to be used.

Thus a problem exists when a payload that has been validated by the schema:

{
 "type": "string",
 "format": "date-time"
}

(with format validation enabled) fails to deserialize because there is a mismatch between the format required by JSON Schema and the formats supported by System.Text.Json's deserialization.

This restriction is especially odd since DateTime.Parse() supports these formats.

Reproduction Steps

Please see attached solution.

[Test]
public void Test1()
{
	var serializerOptions = new JsonSerializerOptions
	{
		WriteIndented = true,
		Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping
	};

	var jsonDate = JsonNode.Parse("\"2023-03-20 05:41:23.3881075\"");

	Console.WriteLine(jsonDate.AsJsonString());

	JsonSchema schema = new JsonSchemaBuilder()
		.Type(SchemaValueType.String)
		.Format(Formats.DateTime);

	var results = schema.Evaluate(jsonDate, new EvaluationOptions
	{
		OutputFormat = OutputFormat.List,
		RequireFormatValidation = true
	});

	Console.WriteLine(JsonSerializer.Serialize(results, serializerOptions));
	Assert.IsTrue(results.IsValid);

	var date = jsonDate.Deserialize<DateTime>(); // fails

	Console.WriteLine(date);
}

[Test]
public void DateTimeParsing()
{
	var date = DateTime.Parse("2023-03-20 05:41:23.3881075"); // works

	Console.WriteLine(date);
}

Expected behavior

Date/time deserialization is more tolerant and accepts any format that can be handled by DateTime.Parse().

Actual behavior

Date/time deserialization MUST be ISO 8601-1:2019.

Regression?

Unsure.

Known Workarounds

A custom converter should be able to handle this, but it shouldn't really be necessary.

Configuration

These tests run in .Net 6. JsonSchema.Net is .Net Standard 2.0.

Other information

Maybe serialization could be configurable.

Date-time format comparison: https://ijmacd.github.io/rfc3339-iso8601/

Tagging subscribers to this area: @dotnet/area-system-text-json, @gregsdennis
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

Originally reported to JsonSchema.Net here.

The report says that System.Text.Json deserialization only supports ISO 8601 date-time formats. However JSON Schema specifies that RFC 3339 formats are to be used.

Thus a problem exists when a payload that has been validated by the schema:

{
 "type": "string",
 "format": "date-time"
}

(with format validation enabled) fails to deserialize because there is a mismatch between the format required by JSON Schema and the formats supported by System.Text.Json's deserialization.

This restriction is especially odd since DateTime.Parse() supports these formats.

Reproduction Steps

Please see attached solution.

[Test]
public void Test1()
{
	var serializerOptions = new JsonSerializerOptions
	{
		WriteIndented = true,
		Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping
	};

	var jsonDate = JsonNode.Parse("\"2023-03-20 05:41:23.3881075\"");

	Console.WriteLine(jsonDate.AsJsonString());

	JsonSchema schema = new JsonSchemaBuilder()
		.Type(SchemaValueType.String)
		.Format(Formats.DateTime);

	var results = schema.Evaluate(jsonDate, new EvaluationOptions
	{
		OutputFormat = OutputFormat.List,
		RequireFormatValidation = true
	});

	Console.WriteLine(JsonSerializer.Serialize(results, serializerOptions));
	Assert.IsTrue(results.IsValid);

	var date = jsonDate.Deserialize<DateTime>(); // fails

	Console.WriteLine(date);
}

[Test]
public void DateTimeParsing()
{
	var date = DateTime.Parse("2023-03-20 05:41:23.3881075"); // works

	Console.WriteLine(date);
}

Expected behavior

Date/time deserialization is more tolerant and accepts any format that can be handled by DateTime.Parse().

Actual behavior

Date/time deserialization MUST be ISO 8601-1:2019.

Regression?

Unsure.

Known Workarounds

A custom converter should be able to handle this, but it shouldn't really be necessary.

Configuration

These tests run in .Net 6. JsonSchema.Net is .Net Standard 2.0.

Other information

Maybe serialization could be configurable.

Date-time format comparison: https://ijmacd.github.io/rfc3339-iso8601/

Author:	gregsdennis
Assignees:	-
Labels:	area-System.Text.Json
Milestone:	-

[layomia]

The primitive types supported by the serializer each support only one format for consistent implementation simplicity and to encourage the design of type/data models that would yield better performance during (de)serialization routines.

Here are a few similar issues that we've closed following this principle - https://github.com/dotnet/runtime/issues?q=is%3Aissue+label%3Aarea-System.Text.Json+formats+is%3Aclosed. As you note, a custom converter can handle this.

This issue has been marked needs-author-action and may be missing some important information.

[gregsdennis]

The primitive types supported by the serializer each support only one format for consistent implementation simplicity and to encourage the design of type/data models that would yield better performance during (de)serialization routines.

I completely agree that this needs to be a goal, and serializing to specific formats can help accomplish this.

However, we need to be tolerant in what we accept. (Ref: Postel's Law / Robustness Principle).

I suggest that the deserializer expect the strict format, and, if that fails, allow other formats.

Maybe this can be hidden behind a serialization option, but it should definitely be supported.

[eiriktsarpalis]

However, we need to be tolerant in what we accept. (Ref: Postel's Law / Robustness Principle).

I suggest that the deserializer expect the strict format, and, if that fails, allow other formats.

I think that's reasonable provided that we don't compromise the fast path.

[layomia]

I suggest that any such feature be opt-in to avoid breaking people who expect/depend on the strict ISO representation. Perhaps a public converter like what we have in JsonMetadataServices, but not something like JsonNumberHandling which somewhat taints the generic nature of the JsonConverter system.

[jeffhandley]

I like that idea, @layomia. Something that can be optionally composed in is a good idea, with the converter's behavior being standalone.