Skip to content

prepare_iter_for_array PR - #43

Closed
chaburkland wants to merge 23 commits into
static-frame:masterfrom
chaburkland:9/prepare_iter_for_array
Closed

prepare_iter_for_array PR#43
chaburkland wants to merge 23 commits into
static-frame:masterfrom
chaburkland:9/prepare_iter_for_array

Conversation

@chaburkland

Copy link
Copy Markdown
Collaborator

Closes#9

@chaburklandchaburkland self-assigned this Apr 26, 2021
@chaburkland
chaburkland marked this pull request as ready for review April 28, 2021 22:29
@chaburkland

Copy link
Copy Markdown
CollaboratorAuthor

Current performance

cls func ak ref ref/ak
IsGenCopyValues main 0.2276659 0.39814382 1.74880746
PrepareIterForArray iter_small 1.07673633 1.28456536 1.19301757
PrepareIterForArray iter_large 1.46419539 2.07376239 1.41631534 

@flexatone

Copy link
Copy Markdown
Contributor

I have optimized the Python implementation of this function, which reduces and clarifies the requirements of this function. The new implementation is as follows:

defprepare_iter_for_array(
values: tp.Iterable[tp.Any],
restrict_copy: bool=False
) ->tp.Tuple[DtypeSpecifier, bool, tp.Sequence[tp.Any]]:
is_gen, copy_values=is_gen_copy_values(values)
ifnotis_genandlen(values) ==0: #type: ignorereturnNone, False, values#type: ignoreifrestrict_copy:
copy_values=Falsev_iter=valuesifis_genelseiter(values)
ifcopy_values:
values_post= []
resolved=None# None is valid specifier if the type is not ambiguoushas_tuple=Falsehas_str=Falsehas_non_str=Falsehas_inexact=Falsehas_big_int=Falseforvinv_iter:
ifcopy_values:
# if a generator, have to make a copy while iteratingvalues_post.append(v)
value_type=type(v)
if (value_typeisstrorvalue_typeisnp.str_orvalue_typeisbytesorvalue_typeisnp.bytes_):
# must compare to both string typeshas_str=Trueelifhasattr(v, '__len__'):
# identify SF types by if they have STATIC attr they also must be assigned after array creation, so we treat them like tupleshas_tuple=Trueresolved=objectbreakelifisinstance(v, Enum):
# must check isinstance, as Enum types are always derived from Enumresolved=objectbreakelse:
has_non_str=Trueifvalue_typeinINEXACT_TYPES:
has_inexact=Trueelifvalue_typeisintandabs(v) >INT_MAX_COERCIBLE_TO_FLOAT:
has_big_int=Trueif (has_strandhas_non_str) or (has_big_intandhas_inexact):
resolved=objectbreakifcopy_values:
# v_iter is an iter, we need to finish itvalues_post.extend(v_iter)
returnresolved, has_tuple, values_postreturnresolved, has_tuple, values#type: ignore

@brandtbucher
brandtbucher removed their request for review November 17, 2022 21:30
@flexatone

Copy link
Copy Markdown
Contributor

A new PR has a fresh implementation of this with shared resources for 1D elementwise function application.

#226

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement prepare_iter_for_array

2 participants

@chaburkland@flexatone