Spark Sql依赖报错：ClassNotFoundException: org.codehaus.janino.InternalCompilerException

kuikui1452

738人浏览 · 2025-05-14 22:09:43

kuikui1452 · 2025-05-14 22:09:43 发布

Exception in thread “main” java.lang.NoClassDefFoundError: org/codehaus/janino/InternalCompilerException
at org.apache.spark.sql.catalyst.expressions.objects.GetExternalRowField.(objects.scala:1835)
at org.apache.spark.sql.catalyst.encoders.RowEncoder $.$ anonfun$serializerFor $3 (R o wE n co d er . sc a l a : 186) a t sc a l a . co ll ec t i o n . T r a v ers ab l e L ik e .$ anonfun$flatMap $1 (T r a v ers ab l e L ik e . sc a l a : 293) a t sc a l a . co ll ec t i o n . I n d e x e d S e qOpt imi ze d . f ore a c h (I n d e x e d S e qOpt imi ze d . sc a l a : 36) a t sc a l a . co ll ec t i o n . I n d e x e d S e qOpt imi ze d . f ore a c h$ (IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps $o f R e f . f ore a c h (A rr a y Op s . sc a l a : 198) a t sc a l a . co ll ec t i o n . T r a v ers ab l e L ik e . f l a tM a p (T r a v ers ab l e L ik e . sc a l a : 293) a t sc a l a . co ll ec t i o n . T r a v ers ab l e L ik e . f l a tM a p$ (TraversableLike.scala:290)
at scala.collection.mutable.ArrayOps $o f R e f . f l a tM a p (A rr a y Op s . sc a l a : 198) a t or g . a p a c h e . s p a r k . s ql . c a t a l ys t . e n co d ers . R o wE n co d er$ .serializerFor(RowEncoder.scala:183)
at org.apache.spark.sql.catalyst.encoders.RowEncoder $. a ppl y (R o wE n co d er . sc a l a : 72) a t or g . a p a c h e . s p a r k . s ql . D a t a se t$ . $an o n f u n$ ofRows $1 (D a t a se t . sc a l a : 91) a t or g . a p a c h e . s p a r k . s ql . Sp a r k S ess i o n . w i t h A c t i v e (Sp a r k S ess i o n . sc a l a : 775) a t or g . a p a c h e . s p a r k . s ql . D a t a se t$ .ofRows(Dataset.scala:88)
at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:440)
at org.apache.spark.sql.execution.datasources.json.TextInputJsonDataSource $. cre a t e B a seD a t a se t (J so n D a t a S o u rce . sc a l a : 124) a t or g . a p a c h e . s p a r k . s ql . e x ec u t i o n . d a t a so u rces . j so n . T e x t I n p u t J so n D a t a S o u rce$ .infer(JsonDataSource.scala:97)
at org.apache.spark.sql.execution.datasources.json.JsonDataSource.inferSchema(JsonDataSource.scala:65)
at org.apache.spark.sql.execution.datasources.json.JsonFileFormat.inferSchema(JsonFileFormat.scala:59)
at org.apache.spark.sql.execution.datasources.DataSource. $an o n f u n$ getOrInferFileFormatSchema $11 (D a t a S o u rce . sc a l a : 210) a t sc a l a . Opt i o n . or El se (Opt i o n . sc a l a : 447) a t or g . a p a c h e . s p a r k . s ql . e x ec u t i o n . d a t a so u rces . D a t a S o u rce . g e tO r I n f er F i l e F or ma tS c h e ma (D a t a S o u rce . sc a l a : 207) a t or g . a p a c h e . s p a r k . s ql . e x ec u t i o n . d a t a so u rces . D a t a S o u rce . reso l v e R e l a t i o n (D a t a S o u rce . sc a l a : 411) a t or g . a p a c h e . s p a r k . s ql . D a t a F r am e R e a d er . l o a d V 1 S o u rce (D a t a F r am e R e a d er . sc a l a : 274) a t or g . a p a c h e . s p a r k . s ql . D a t a F r am e R e a d er .$ anonfun$load $3 (D a t a F r am e R e a d er . sc a l a : 245) a t sc a l a . Opt i o n . g e tO r El se (Opt i o n . sc a l a : 189) a t or g . a p a c h e . s p a r k . s ql . D a t a F r am e R e a d er . l o a d (D a t a F r am e R e a d er . sc a l a : 245) a t or g . a p a c h e . s p a r k . s ql . D a t a F r am e R e a d er . j so n (D a t a F r am e R e a d er . sc a l a : 405) a t or g . a p a c h e . s p a r k . s ql . D a t a F r am e R e a d er . j so n (D a t a F r am e R e a d er . sc a l a : 386) a t co m . a t gu i gu . s p a r k s ql . C re a t eD a t a F r am e$ .main(CreateDataFrame.scala:11)
at com.atguigu.sparksql.CreateDataFrame.main(CreateDataFrame.scala)
Caused by: java.lang.ClassNotFoundException: org.codehaus.janino.InternalCompilerException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
… 31 more

Caused by: java.lang.ClassNotFoundException: org.codehaus.janino.InternalCompilerException：使用的spark 版本冲突，原因是使用的spark的版本为3.2.1 的，Spark 3.2.1 需要 Janino 3.0.16（官方兼容版本），而非 3.1.6

<dependency>
    <groupId>org.codehaus.janino</groupId>
    <artifactId>janino</artifactId>
    <version>3.0.16</version>
</dependency>
<dependency>
    <groupId>org.codehaus.janino</groupId>
    <artifactId>commons-compiler</artifactId>
    <version>3.0.16</version>
</dependency>