good first issue
Repository metrics
- Stars
- (9,113 stars)
- PR merge metrics
- (Avg merge 5d 9h) (306 merged PRs in 30d)
Description
I have an external hive partitioned table, when I change the table's structure (mostly modify column precision), presto query will trigger the following errors:
io.prestosql.spi.PrestoException: Failed to read ORC file: /tmp/test_part_tbl/logdate=20200911/000000_0
at io.prestosql.plugin.hive.orc.OrcPageSource.handleException(OrcPageSource.java:138)
at io.prestosql.plugin.hive.orc.OrcPageSourceFactory.lambda$createOrcPageSource$10
(OrcPageSourceFactory.java:344)
at io.prestosql.orc.OrcBlockFactory$OrcBlockLoader.load(OrcBlockFactory.java:83)
at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
at io.prestosql.plugin.hive.orc.OrcPageSource$SourceColumn$MaskingBlockLoader.load(OrcPageSource.java:275)
at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
at io.prestosql.plugin.hive.HivePageSource$CoercionLazyBlockLoader.load(HivePageSource.java:549)
at io.prestosql.spi.block.LazyBlock$LazyData.load(LazyBlock.java:381)
at io.prestosql.spi.block.LazyBlock$LazyData.getFullyLoadedBlock(LazyBlock.java:360)
at io.prestosql.spi.block.LazyBlock.getLoadedBlock(LazyBlock.java:276)
at io.prestosql.spi.Page.getLoadedPage(Page.java:273)
at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:304)
at io.prestosql.operator.Driver.processInternal(Driver.java:379)
at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
at io.prestosql.operator.Driver.processFor(Driver.java:276)
at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
at io.prestosql.$gen.Presto_340____20200902_012116_2.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.IllegalArgumentException: target scale must be larger than source scale
at io.prestosql.spi.type.Decimals.rescale(Decimals.java:298)
at io.prestosql.orc.reader.DecimalColumnReader.readShortNotNullBlock(DecimalColumnReader.java:184)
at io.prestosql.orc.reader.DecimalColumnReader.readNonNullBlock(DecimalColumnReader.java:164)
at io.prestosql.orc.reader.DecimalColumnReader.readBlock(DecimalColumnReader.java:125)
at io.prestosql.orc.OrcBlockFactory$OrcBlockLoader.load(OrcBlockFactory.java:76)
... 24 more
The above error can be repeated by doing the following instructions:
- create table and add partitions via beeline
create external table test_part_tbl(id int, asset decimal(10,2))
partitioned by (logdate string)
stored as orc
location '/tmp/test_part_tbl'
tblproperties('external.table.purge'='true');
alter table test_part_tbl add partition(logdate='20200910');
insert into test_part_tbl partition (logdate='20200910') values(1,10.22);
- modify column and add another partition and insert a record
alter table test_part_tbl add partition(logdate='20200911');
alter table test_part_tbl change column asset asset decimal(10,3);
insert into test_part_tbl partition (logdate='20200911') values(1,10.23);
insert into test_part_tbl partition (logdate='20200911') values(1,10.233);
- query
test_part_tblvia presto
presto> select * from default.test_part_tbl;
Query 20200910_053546_00098_cczqj, FAILED, 1 node
Splits: 19 total, 1 done (5.26%)
2.16 [1 rows, 310B] [0 rows/s, 144B/s]
Query 20200910_053546_00098_cczqj failed: Failed to read ORC file:
hdfs://cfzq/tmp/test_part_tbl/logdate=20200911/000000_0
- query
test_part_tblvia hive
select * from test_part_tbl;
+-------------------+----------------------+------------------------+
| test_part_tbl.id | test_part_tbl.asset | test_part_tbl.logdate |
+-------------------+----------------------+------------------------+
| 1 | 10.220 | 20200910 |
| 1 | 10.230 | 20200911 |
| 1 | 10.233 | 20200911 |
+-------------------+----------------------+------------------------+
3 rows selected (0.487 seconds)
If I drop partition 20200911 and add it again , query will result all data.
I use Hortonworks HDP 3.1.4.0-315 version
- Hadoop HDFS: 3.1.1.3.1
- Hive: 3.1.0
- OS: CentOS 7.7.1908
- PrestoSQL: 340