make_array returns FixedSizeList which has proven to be difficult to work with.

[dadepo]

I admit this might be due to my "noobness", but dealing with FixedSizeList in DataFusion has proven to be difficult and I was wondering maybe I could get some insight that would make it more productive.

Basically I am having to deal with FixedSizeList because I am making use of DataFusion's make_array, which returns a FixedSizeList but there are a couple of issues i have ran into that makes it cumbersome.

First, I can't use functions like array_to_json_array or record_batches_to_json_rows because FixedSizeList is not supported by arrow_json. I created an issue for this here apache/arrow-rs#4248

Second, I cannot use arrow_cast to cast it as arrow_cast also currently does not support FixedSizeList

Third I am not sure how to define a varadic UDF that takes a FixedSizeList. ScalarUDF::new expects Signature and for variadic UDF, I can create a value using Signature::variadic(vec![DataType::Utf8, DataType::UInt8], Volatility::Immutable) but if I want to include FixedSizeList, I am not sure how to do this.

The type definition of DataType::FixedSizeList is

FixedSizeList(FieldRef, i32)

where FieldRef is pub type FieldRef = Arc<Field>; and Field is

pub struct Field {
    name: String,
    data_type: DataType,
    nullable: bool,
    dict_id: i64,
    dict_is_ordered: bool,
    /// A map of key-value pairs containing additional custom meta data.
    metadata: HashMap<String, String>,
}

Now at the point of creating DataType::FixedSizeList as needed by Signature::variadic I am not sure how to set the argument it requires.

In general I have been stumped by FixedSizeList in so many places where it is either not supported by the greater eco-system or where how to work with it feel cumbersome or not that obvious.

Any advice that can help here? Is there a way to wield FixedSizeList so it does not feel this difficult to work with?

[alamb]

I admit this might be due to my "noobness", but dealing with FixedSizeList in DataFusion has proven to be difficult and I was wondering maybe I could get some insight that would make it more productive.

I do not think this is anything to do with you. In general I think DataFusion's support for nested / structured types could be much better -- we have started collecting the ideas here: #2326

Second, I cannot use arrow_cast to cast it as arrow_cast also currently does not support FixedSizeList

I think we should can/should add support for this type in arrow_cast for sure

[alamb]

I think make_array now returns ListArray which is easier to work with:

DataFusion CLI v33.0.0
❯ select arrow_typeof(make_array(1, 2, 3));
+------------------------------------------------------------------------------------------------------------------+
| arrow_typeof(make_array(Int64(1),Int64(2),Int64(3)))                                                             |
+------------------------------------------------------------------------------------------------------------------+
| List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) |
+------------------------------------------------------------------------------------------------------------------+
1 row in set. Query took 0.003 seconds.

❯