本文共 3749 字，大约阅读时间需要 12 分钟。

今天使用spak从postgresql 往mongodb 导入数据时出现以下错误：

9/02/25 16:47:21 INFO DAGScheduler: Job 0 failed: foreachPartition at MongoSpark.scala:117, took 16.897605 sorg.apache.spark.SparkException: Job aborted due to stage failure: Task 16 in stage 0.0 failed 4 times, most recent failure: Lost task 16.3 in stage 0.0 (TID 77, tod4, executor 5): java.lang.IllegalArgumentException: requirement failed: Decimal precision 39 exceeds max precision 38	at scala.Predef$.require(Predef.scala:224)	at org.apache.spark.sql.types.Decimal.set(Decimal.scala:114)	at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:453)	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$org$apache$spark$sql$execution$datasources$jdbc$JdbcUtils$$makeGetter$3$$anonfun$12.apply(JdbcUtils.scala:398)	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$org$apache$spark$sql$execution$datasources$jdbc$JdbcUtils$$makeGetter$3$$anonfun$12.apply(JdbcUtils.scala:398)	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.org$apache$spark$sql$execution$datasources$jdbc$JdbcUtils$$nullSafeConvert(JdbcUtils.scala:500)	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$org$apache$spark$sql$execution$datasources$jdbc$JdbcUtils$$makeGetter$3.apply(JdbcUtils.scala:398)	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$org$apache$spark$sql$execution$datasources$jdbc$JdbcUtils$$makeGetter$3.apply(JdbcUtils.scala:396)	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anon$1.getNext(JdbcUtils.scala:347)	at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anon$1.getNext(JdbcUtils.scala:329)	at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$1.hasNext(WholeStageCodegenExec.scala:614)	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)	at scala.collection.Iterator$class.isEmpty(Iterator.scala:330)	at scala.collection.AbstractIterator.isEmpty(Iterator.scala:1336)	at scala.collection.TraversableOnce$class.nonEmpty(TraversableOnce.scala:111)	at scala.collection.AbstractIterator.nonEmpty(Iterator.scala:1336)	at com.mongodb.spark.MongoSpark$$anonfun$save$1.apply(MongoSpark.scala:117)	at com.mongodb.spark.MongoSpark$$anonfun$save$1.apply(MongoSpark.scala:117)	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2067)	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2067)	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)	at org.apache.spark.scheduler.Task.run(Task.scala:109)	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)	at java.lang.Thread.run(Thread.java:748)

报错显示是Decimal精度问题，spark api文档中对DecimalType的描述如下，它的最大精度是38。

查看postgresql中原表的数据类型。

转载地址：http://ezjmb.baihongyu.com/