Spark3 读 hive 1.1.0 遇到的问题

Exception in thread “main” org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table test1. Invalid method name: ‘get_table_req’;

Exception in thread "main" org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table test1. Invalid method name: 'get_table_req';
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:113)
	at org.apache.spark.sql.hive.HiveExternalCatalog.tableExists(HiveExternalCatalog.scala:855)
	at org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener.tableExists(ExternalCatalogWithListener.scala:146)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.tableExists(SessionCatalog.scala:432)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.requireTableExists(SessionCatalog.scala:185)
	at org.apache.spark.sql.catalyst.catalog.SessionCatalog.getTableMetadata(SessionCatalog.scala:445)
	at org.apache.spark.sql.execution.datasources.v2.V2SessionCatalog.loadTable(V2SessionCatalog.scala:66)
	at org.apache.spark.sql.connector.catalog.CatalogV2Util$.loadTable(CatalogV2Util.scala:283)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.loaded$lzycompute$1(Analyzer.scala:1010)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.loaded$1(Analyzer.scala:1010)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.$anonfun$lookupRelation$3(Analyzer.scala:1022)
	at scala.Option.orElse(Option.scala:447)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupRelation(Analyzer.scala:1021)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$9.applyOrElse(Analyzer.scala:977)

原因：版本不兼容，spark 3.x默认支持hive 2.x，这篇文章讲解的比较清楚
https://blog.csdn.net/OldDirverHelpMe/article/details/105325439

解决方式一：配置spark.sql.hive.metastore.version

pom.xml

	<properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <spark.version>3.0.1</spark.version>
        <scala.version>2.12</scala.version>
    </properties>

    <dependencies>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.12</artifactId>
            <version>3.0.1</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive-thriftserver_2.12</artifactId>
            <version>3.0.1</version>
            <scope>provided</scope>
        </dependency>

    </dependencies>

resources

resources:
	core-site.xml
	hdfs-site.xml
	hive-site.xml
	yarn-site.xml

spark 读 hive代码

package com.persist
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.sql.SparkSession

/**
 * Created by xxx on 2021/10/13.
 */
object Spark3ReadHiveTest {
  def main(args: Array[String]): Unit = {
  
    // 本地读取服务器安全配置，根据各自情况可删除
    val conf = new Configuration
    System.setProperty("java.security.krb5.conf", "/xxx/etc/hive/krb5.conf")
    conf.addResource(new Path("/xxx/etc/hive/conf/hdfs-site.xml"))
    conf.set("hadoop.security.authentication", "Kerberos")
    UserGroupInformation.setConfiguration(conf)
    UserGroupInformation.loginUserFromKeytab("xxx", "/xxx/etc/xxx.keytab")
    println("login user: " + UserGroupInformation.getLoginUser())

	// spark
    val spark = SparkSession
      .builder()
      .appName("zjj-spark")
      .master("local[*]")
      .config("spark.sql.hive.metastore.version", "1.2.1") 
      .config("spark.sql.hive.metastore.jars", "maven") // 生产环境不建议配置maven
      //.config("spark.sql.hive.metastore.jars", "/Users/xxx/etc/hive/hive1_2_1jars/*") // 配置jar包路径，包下载https://download.csdn.net/download/z1941563559/33083988 
      .enableHiveSupport()
      .getOrCreate()

    spark.sql("show tables").show()
    spark.sql("select * from public.test1").show()

    spark.stop()

  }
}

官网配置 spark.sql.hive.metastore.version 介绍：
https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html#interacting-with-different-versions-of-hive-metastore

// 生产环境不建议配置maven，jar包下载路径 https://download.csdn.net/download/z1941563559/33083988 
.config("spark.sql.hive.metastore.jars", "/Users/xxx/etc/hive/hive1_2_1jars/*") 

解决方式二：修改pom依赖低版本的hive jar

pom.xml

	<properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <spark.version>3.0.1</spark.version>
        <scala.version>2.12</scala.version>
    </properties>

    <dependencies>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_${scala.version}</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_${scala.version}</artifactId>
            <version>${spark.version}</version>
            <scope>provided</scope>
        </dependency>

        <!--        去掉依赖两个jar的依赖-->
        <!--        <dependency>-->
        <!--            <groupId>org.apache.spark</groupId>-->
        <!--            <artifactId>spark-hive_2.12</artifactId>-->
        <!--            <version>3.0.1</version>-->
        <!--            <scope>provided</scope>-->
        <!--        </dependency>-->
        <!--        <dependency>-->
        <!--            <groupId>org.apache.spark</groupId>-->
        <!--            <artifactId>spark-hive-thriftserver_2.12</artifactId>-->
        <!--            <version>3.0.1</version>-->
        <!--            <scope>provided</scope>-->
        <!--        </dependency>-->

        <!--        spark-hive_2.12，及spark-hive-thriftserver_2.12重新依赖，并排除对hive的传递依赖-->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive_2.12</artifactId>
            <version>3.0.1</version>
            <scope>provided</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.hive</groupId>
                    <artifactId>hive-common</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.hive</groupId>
                    <artifactId>hive-exec</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.hive</groupId>
                    <artifactId>hive-metastore</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.hive</groupId>
                    <artifactId>hive-serde</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.hive</groupId>
                    <artifactId>hive-shims</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-hive-thriftserver_2.12</artifactId>
            <version>3.0.1</version>
            <scope>provided</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.hive</groupId>
                    <artifactId>hive-cli</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.hive</groupId>
                    <artifactId>hive-jdbc</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.apache.hive</groupId>
                    <artifactId>hive-beeline</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <!--        hive版本降级为1.2.1，spark3原本依赖2.3.7-->
        <!--        spark-hive-thriftserver_2.12的传递依赖-->
        <dependency>
            <groupId>org.spark-project.hive</groupId>
            <artifactId>hive-cli</artifactId>
            <version>1.2.1.spark2</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.spark-project.hive</groupId>
            <artifactId>hive-jdbc</artifactId>
            <version>1.2.1.spark2</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.spark-project.hive</groupId>
            <artifactId>hive-beeline</artifactId>
            <version>1.2.1.spark2</version>
            <scope>provided</scope>
        </dependency>

        <!--        spark-hive_2.12的传递依赖-->
        <dependency>
            <groupId>org.spark-project.hive</groupId>
            <artifactId>hive-common</artifactId>
            <version>1.2.1.spark2</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.spark-project.hive</groupId>
            <artifactId>hive-metastore</artifactId>
            <version>1.2.1.spark2</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.spark-project.hive</groupId>
            <artifactId>hive-exec</artifactId>
            <version>1.2.1.spark2</version>
            <scope>provided</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.apache.commons</groupId>
                    <artifactId>commons-lang3</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <!--        hive-exec中传递依赖的commons-lang3需要升级回3.9版本-->
        <dependency>
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.9</version>
            <scope>provided</scope>
        </dependency>

    </dependencies>

resources

resources:
	core-site.xml
	hdfs-site.xml
	hive-site.xml
	yarn-site.xml

spark读hive代码

package com.persist
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.security.UserGroupInformation
import org.apache.spark.sql.SparkSession

/**
 * Created by xxx on 2021/10/13.
 */
object Spark3ReadHiveTest {
  def main(args: Array[String]): Unit = {
  
    // 本地读取服务器安全配置，根据各自情况可删除
    val conf = new Configuration
    System.setProperty("java.security.krb5.conf", "/xxx/etc/hive/krb5.conf")
    conf.addResource(new Path("/xxx/etc/hive/conf/hdfs-site.xml"))
    conf.set("hadoop.security.authentication", "Kerberos")
    UserGroupInformation.setConfiguration(conf)
    UserGroupInformation.loginUserFromKeytab("xxx", "/xxx/etc/xxx.keytab")
    println("login user: " + UserGroupInformation.getLoginUser())

	// spark
    val spark = SparkSession
      .builder()
      .appName("zjj-spark")
      .master("local[*]")
      .enableHiveSupport()
      .getOrCreate()

    spark.sql("show tables").show()
    spark.sql("select * from public.test1").show()

    spark.stop()

  }
}

解决方式三：重编译spark3指定hive版本（未尝试）

～～我不会

解决过程中遇到的其他问题

问题1：Exception in thread “main” org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

解决：集群core-site.xml放到项目resources目录下

Exception in thread "main" org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

问题2：Can‘t get Master Kerberos principal for use as renewer
解决：集群yarn-site.xml放到项目resources目录下