Skip to content

Reevaluate dtypes on importing from SBDF file #75

Description

@bbassett-tibco

Given the following script:

from spotfire import sbdf

x = sbdf.import_data("spotfire/test/files/sbdf/10001.sbdf")
print("column Boolean")
print(x['Boolean'].loc[[0,1,2]])
print("column String")
print(x['String'].loc[[0,1,2]])

Output:

column Boolean
0    False
1     True
2     None
Name: Boolean, dtype: object
column String
0      The
1    quick
2     None
Name: String, dtype: object

Note how both of these columns report as being dtype object. However, it's probably better to use the nullable boolean (https://pandas.pydata.org/docs/user_guide/boolean.html) and string (https://pandas.pydata.org/docs/user_guide/text.html#text) dtypes for these columns. (We're actually using the nullable integer and float dtypes for other column types, so our current requirements on Pandas should be untouched.)

Current workarounds are to manually cast the columns using Pandas' astype method:

>>> x['Boolean'].loc[[0,1,2]].astype('boolean')
0    False
1     True
2     <NA>
Name: Boolean, dtype: boolean
>>> x['String'].loc[[0,1,2]].astype('string')
0      The
1    quick
2     <NA>
Name: String, dtype: string

Make this the native result of the import_data function, and make sure that these dtypes correctly export to SBDF in the export_data function.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions