trinodb/trino

Support ORC decimal type evolution in ORC reader

Open

#5,128 opened on Sep 10, 2020

View on GitHub
 (5 comments) (0 reactions) (0 assignees)Java (2,678 forks)batch import
good first issue

Repository metrics

Stars
 (9,113 stars)
PR merge metrics
 (Avg merge 5d 9h) (306 merged PRs in 30d)

Description

I have an external hive partitioned table, when I change the table's structure (mostly modify column precision), presto query will trigger the following errors:

io.prestosql.spi.PrestoException: Failed to read ORC file: /tmp/test_part_tbl/logdate=20200911/000000_0
   at io.prestosql.plugin.hive.orc.OrcPageSource.handleException(OrcPageSource.java:138)
   at io.prestosql.plugin.hive.orc.OrcPageSourceFactory.lambda$createOrcPageSource$10
                          (OrcPageSourceFactory.java:344)
	at io.prestosql.orc.OrcBlockFactory$OrcBlockLoader.load(OrcBlockFactory.java:83)
	at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
	at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
	at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
   at io.prestosql.plugin.hive.orc.OrcPageSource$SourceColumn$MaskingBlockLoader.load(OrcPageSource.java:275)
	at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
	at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
	at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
	at io.prestosql.plugin.hive.HivePageSource$CoercionLazyBlockLoader.load(HivePageSource.java:549)
	at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
	at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
	at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
	at io.prestosql.spi.Page.getLoadedPage(Page.java:273)
	at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:304)
	at io.prestosql.operator.Driver.processInternal(Driver.java:379)
	at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
	at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
	at io.prestosql.operator.Driver.processFor(Driver.java:276)
	at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
	at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
	at io.prestosql.$gen.Presto_340____20200902_012116_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.IllegalArgumentException: target scale must be larger than source scale
	at io.prestosql.spi.type.Decimals.rescale(Decimals.java:298)
	at io.prestosql.orc.reader.DecimalColumnReader.readShortNotNullBlock(DecimalColumnReader.java:184)
	at io.prestosql.orc.reader.DecimalColumnReader.readNonNullBlock(DecimalColumnReader.java:164)
	at io.prestosql.orc.reader.DecimalColumnReader.readBlock(DecimalColumnReader.java:125)
	at io.prestosql.orc.OrcBlockFactory$OrcBlockLoader.load(OrcBlockFactory.java:76)
	... 24 more

The above error can be repeated by doing the following instructions:

  1. create table and add partitions via beeline
create external table test_part_tbl(id  int, asset decimal(10,2))
partitioned by (logdate string)
stored as orc
location '/tmp/test_part_tbl'
tblproperties('external.table.purge'='true');

alter table test_part_tbl add partition(logdate='20200910');

insert into test_part_tbl partition (logdate='20200910') values(1,10.22);
  1. modify column and add another partition and insert a record
alter table test_part_tbl add partition(logdate='20200911');

alter table test_part_tbl change column asset asset decimal(10,3);
insert into test_part_tbl partition (logdate='20200911') values(1,10.23);
insert into test_part_tbl partition (logdate='20200911') values(1,10.233);
  1. query test_part_tbl via presto
presto> select * from  default.test_part_tbl;

Query 20200910_053546_00098_cczqj, FAILED, 1 node
Splits: 19 total, 1 done (5.26%)
2.16 [1 rows, 310B] [0 rows/s, 144B/s]

Query 20200910_053546_00098_cczqj failed: Failed to read ORC file: 
hdfs://cfzq/tmp/test_part_tbl/logdate=20200911/000000_0
  1. query test_part_tbl via hive
select * from test_part_tbl;
+-------------------+----------------------+------------------------+
| test_part_tbl.id  | test_part_tbl.asset  | test_part_tbl.logdate  |
+-------------------+----------------------+------------------------+
| 1                 | 10.220               | 20200910               |
| 1                 | 10.230               | 20200911               |
| 1                 | 10.233               | 20200911               |
+-------------------+----------------------+------------------------+
3 rows selected (0.487 seconds)

If I drop partition 20200911 and add it again , query will result all data.

I use Hortonworks HDP 3.1.4.0-315 version

  • Hadoop HDFS: 3.1.1.3.1
  • Hive: 3.1.0
  • OS: CentOS 7.7.1908
  • PrestoSQL: 340

Contributor guide