hiro/ray
1
0
Fork 0
mirror of https://github.com/vale981/ray synced 2025-03-05 18:11:42 -05:00
Code Issues Projects Releases Packages Wiki Activity Actions

[Ray Dataset] fix the type infer of pd.dataframe (when dtype is object.) (#25563)

Browse source 
this is a temp fix of #25556. When the dtype from the pandas dataframe gives object, we set the dtype to be None and make use of the auto-inferring of the type in the conversion.
This commit is contained in:
Jimmy Yao 2022-06-14 12:49:04 -07:00 • committed by GitHub
parent 0d8cbb1cae
commit 57d02eec2e
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 7 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
python/ray/air/_internal/tensorflow_utils.py
View file

@ -47,6 +47,13 @@ def convert_pandas_to_tf_tensor(
# them. If the columns contain different types (for example, `float32`s
# and `int32`s), then `tf.concat` raises an error.
dtype: np.dtype = np.find_common_type(df.dtypes, [])
# if the columns are `ray.data.extensions.tensor_extension.TensorArray`,
# the dtype will be `object`. In this case, we need to set the dtype to
# none, and use the automatic type casting of `tf.convert_to_tensor`.
if isinstance(dtype, object):
dtype = None
except TypeError:
# `find_common_type` fails if a series has `TensorDtype`. In this case,
# don't cast any of the series and continue.