[HUDI-6804] Fix hive read schema evolution MOR table

[Zouxxyy]

Change Logs

Current hive read schema evolution MOR table will error, e.g. :

-- spark-sqlsethoodie.schema.on.read.enable=true;
createtableif not exists hudi_mor_test_tbl (
id bigint,
name string,
num int,
ts bigint,
ds string
) using hudi tblproperties (
type ='mor',
primaryKey ='id',
preCombineField ='ts'
)
partitioned by (ds);
insert into hudi_mor_test_tbl partition(ds ='20211211') select1, 'a1', 1000,100;
update hudi_mor_test_tbl set name ='a2'where id =1;
altertable hudi_mor_test_tbl rename column name to name_new; -- hiveselect id,name_new from hudi_mor_test_tbl_rt;
Failed with exception java.io.IOException:java.lang.ArrayIndexOutOfBoundsException: 25

Impact

Fix above

Risk level (write none, low medium or high below)

low

Documentation Update

None

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[danny0405]

+1

[xiarixiaoyao]

@Zouxxyy
thanks for your fix.
could you pls point out the specific reason for the error? Thank you
i cannot reproduce this problem on hive 3.1.1 (hudi 0.11) with my cluster

[Zouxxyy]

thanks for your fix. could you pls point out the specific reason for the error? Thank you i cannot reproduce this problem on hive 3.1.1 (hudi 0.11) with my cluster

The core change is internalSchemaOption = Option.of(prunedInternalSchema); You can remove it and then run the ut add in this patch

hudi 0.11 may not have this patch #6989 and #6358

/** * Get final Read Schema for support evolution. * step1: find the fileSchema for current dataBlock. * step2: determine whether fileSchema is compatible with the final read internalSchema. * step3: merge fileSchema and read internalSchema to produce final read schema. * * @param dataBlock current processed block * @return final read schema. */privateOption<Pair<Function<HoodieRecord, HoodieRecord>, Schema>> composeEvolvedSchemaTransformer(
HoodieDataBlockdataBlock) {
if (internalSchema.isEmptySchema()) {
returnOption.empty();
}
longcurrentInstantTime = Long.parseLong(dataBlock.getLogBlockHeader().get(INSTANT_TIME));
InternalSchemafileSchema = InternalSchemaCache.searchSchemaAndCache(currentInstantTime,
hoodieTableMetaClient, false);
InternalSchemamergedInternalSchema = newInternalSchemaMerger(fileSchema, internalSchema,
true, false).mergeSchema();
SchemamergedAvroSchema = AvroInternalSchemaConverter.convert(mergedInternalSchema, readerSchema.getFullName());
returnOption.of(Pair.of((record) -> {
returnrecord.rewriteRecordWithNewSchema(
dataBlock.getSchema(),
this.hoodieTableMetaClient.getTableConfig().getProps(),
mergedAvroSchema,
Collections.emptyMap());
}, mergedAvroSchema));
}

[hudi-bot]

CI report:

• ceaaa3e Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build