JSON Fields for Nested Pydantic Models?

[scuervo91]

First Check

• I added a very descriptive title to this issue.
• I used the GitHub search to find a similar issue and didn't find it.
• I searched the SQLModel documentation, with the integrated search.
• I already searched in Google "How to X in SQLModel" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to SQLModel but to Pydantic.
• I already checked if it is not related to SQLModel but to SQLAlchemy.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from tortoise.models import Model 
from tortoise.fields import UUIDField, DatetimeField,CharField, BooleanField, JSONField, ForeignKeyField, CharEnumField, IntField
from tortoise.contrib.pydantic import pydantic_model_creator

class Schedule(Model):
 id = UUIDField(pk=True)
 created_at = DatetimeField(auto_now_add=True)
 modified_at = DatetimeField(auto_now=True)
 case = JSONField()
 type = CharEnumField(SchemasEnum,description='Schedule Types')
 username = ForeignKeyField('models.Username')
 description = CharField(100)
 
schedule_pydantic = pydantic_model_creator(Schedule,name='Schedule')

Description

I have already implemented an API using FastAPI to store Pydantic Models. These models are themselves nested Pydantic models so the way they interact with a Postgres DataBase is throught JsonField. I've been using Tortoise ORM as the example shows.

Is there an equivalent model in SQLModel?

Operating System

Linux

Operating System Details

WSL 2 Ubuntu 20.04

SQLModel Version

0.0.4

Python Version

3.8

Additional Context

No response

[OXERY]

I also wondered how to store JSON objects without converting to string. SQL Alchemy supports storing these directly

[TheJedinator]

@OXERY && @scuervo91 - I was able to get something that works Using this:

regions: dict = Field(sa_column=Column(JSON), default={'all': 'true'})

That said: this is a postgresql JSONB column in my database. But it works.

For a nested Object you could use a pydantic model as the Type and do it the same way. Hope this helps as I was having a difficult time figuring out a solution as well :)

[OXERY]

I also got it working, on SQLite and Postgresql:
mygreatfield: Dict[Any, Any] = Field(index=False, sa_column=Column(JSON))
needs from sqlmodel import Field, SQLModel, Column, JSON as well as from typing import Dict, Any

[psarka]

@TheJedinator Could you help a bit more with the nested object? I tried to "use the pydantic model as the Type" but I can't get it to work :( Here is my snippet:

from sqlalchemy import Column
from sqlalchemy.dialects.postgresql import JSONB
from sqlmodel import Field
from sqlmodel import Session
from sqlmodel import SQLModel

from engine import get_sqlalchemy_engine


class J(SQLModel):
 j: int


class A(SQLModel, table=True):
 a: int = Field(primary_key=True)
 b: J = Field(sa_column=Column(JSONB))


engine = get_sqlalchemy_engine()
SQLModel.metadata.create_all(engine)

with Session(engine) as session:
 a = A(a=1, b=J(j=1))
 session.add(a)
 session.commit()
 session.refresh(a)

Throws an error

sqlalchemy.exc.StatementError: (builtins.TypeError) Object of type J is not JSON serializable
[SQL: INSERT INTO a (b, a) VALUES (%(b)s, %(a)s)]
[parameters: [{'a': 1, 'b': J(j=1)}]]

[TheJedinator]

@psarka

j = J(j=1)
db_j = J.from_orm(j)
a = A(a=1, b=db_j)

This should resolve your issue in preparing the object for the database. What I'm seeing in the error is that the Raw Object is being included in the statement rather than the instance...

If this doesn't help I can definitely put some more time in to looking at what's going on.

[psarka]

Thank you! Unfortunately I get the same error :(

I found one workaround - registering a custom_serializer for the sqlalchemy engine, like so:

def custom_serializer(d):
 return json.dumps(d, default=lambda v: v.json())

def get_sqlalchemy_engine():
 return create_engine("postgresql+psycopg2://", creator=get_conn, json_serializer=custom_serializer)

But if there is a cleaner way, I would gladly use that instead.

[TheJedinator]

Hey @psarka

I just actually tried what I told and sorry have mislead... I did get a working solution though 😄

It was actually the opposite function that you need to use, here's the example you supplied with the amendments to make it work:

with Session(engine) as session:
 j = J(j=1)
 j_dumped = J.json(j)
 a = A(a=1, b=j_dumped)
 session.add(a)
 session.commit()
 session.refresh(a)

[psarka]

Hmm, this doesn't (or at least shouldn't) typecheck :)

But I see what you did there, essentially it's the same as registring a custom serializer, but manually.

[TheJedinator]

It does type check when you create the J Object (which it should) So if you tried to supply a string it would fail J(j="foo")

This allows for the type checking of the object, the A class requires a serialized version of J in order for it to be entered in to the database.

It is essentially the same as registering a custom serializer but allows you to be explicit about using it.

[HenningScheufler]

A hacky method with type checking that work with sqlite is

from sqlalchemy import Column
from typing import List
# from sqlalchemy.dialects.postgresql import JSONB
from sqlmodel import Field
from sqlmodel import Session
from pydantic import validator
from sqlmodel import SQLModel, JSON,create_engine

# from engine import get_sqlalchemy_engine
sqlite_file_name = "test.db"
sqlite_url = f"sqlite:///{sqlite_file_name}"

engine = create_engine(sqlite_url)


class J2(SQLModel):
 test: List[int]

class J(SQLModel):
 j: int
 nested: J2


class A(SQLModel, table=True):
 a: int = Field(primary_key=True)
 b: J = Field(sa_column=Column(JSON))

 @validator('b')
 def val_b(cls, val):
 return val.dict()

SQLModel.metadata.create_all(engine)

with Session(engine) as session:
 a = A(a=1, b=J(j=1,nested=J2(test=[100,100,100])))
 session.add(a)
 session.commit()
 session.refresh(a)

[hakanoktay]

hi,
I created a "JSON Field" based on what is written here. I am using SQLite.

from sqlmodel import SQLModel,Relationship,Field,JSON
from typing import Optional,List, Dict
from sqlalchemy import Column
from pydantic import validator


#
class J2(SQLModel):
 id: int
 title:str

#
class Companies(SQLModel, table=True):
 id:Optional[int]=Field(default=None,primary_key=True)
 name:str
 adddresses: List['J2'] = Field(sa_column=Column(JSON))


 @validator('adddresses')
 def val_b(cls, val):
 print(val)
 return val.dict()

Given error.

TypeError: Type is not JSON serializable: J2

when i print it, it returns

[J2(id=1, title='address1'), J2(id=2, title='address2')]

how can i handle that? Why is this J2 added, how can I get rid of it, i can't turn it to .dict(), i cannot serialise it... can you give an idea?

[HenningScheufler]

Does this work?

 @validator('adddresses')
 def val_b(cls, value):
 print(value)
 return [v.dict() for v in value]

[hakanoktay]

Does this work?

 @validator('adddresses')
 def val_b(cls, value):
 print(value)
 return [v.dict() for v in value]

@HenningScheufler thank you for your help, it worked perfect.