Windows下IDEA调试spark程序问题详解

Failed to locate the winutils binary in the hadoop binary path 问题

Windows环境下会出现这样的提示。Linux则不会——我只知道有这样的现象，目前还不知道为何会这样

使用的sbt构建的项目，sbt配置如下

name := "sbt"

version := "0.1"

scalaVersion := "2.12.9"

libraryDependencies ++= Seq(
  "com.fasterxml.jackson.core" % "jackson-annotations" % "2.11.0",
  "org.apache.spark" %% "spark-core" % "3.0.0",
  "org.apache.spark" %% "spark-sql" % "3.0.0",
  "org.apache.spark" %% "spark-streaming" % "3.0.0",
  "org.apache.spark" %% "spark-streaming-kafka-0-10-assembly" % "2.4.3",
  "mysql" % "mysql-connector-java" % "8.0.20",
  "org.apache.kafka" % "kafka-clients" % "2.5.0",
  "com.alibaba" % "fastjson" % "1.2.71"
)

spark样例代码

package _2020_07_25

import org.apache.spark.{SparkConf, SparkContext}

object sparkDemo {
  def main(args: Array[String]): Unit = {
    val sparkconf = new SparkConf().setMaster("local").setAppName("wordCount")
    val sc = new SparkContext(sparkconf)

    printf("init ok ")
  }
}

运行后报出
Failed to locate the winutils binary in the hadoop binary path
的警告提示，解决的办法就是配置一下Windows的HADOOP_HOME环境变量，步骤如下

在 https://github.com/srccodes/hadoop-common-2.2.0-bin 下载该项目的zip包，将其解压到某个指定的目录，比如
C:\Users\Administrator\IdeaProjects\hadoop-common-2.2.0-bin-master (C:\Users\Administrator\IdeaProjects是我的IDEA的工程目录)

新建一个系统环境变量为HADOOP_HOME，将其设置为 C:\Users\Administrator\IdeaProjects\hadoop-common-2.2.0-bin-master 注意不要包含bin目录（它会自动加上bin目录，加上反而会出错）

重启IDEA，该警告提示消失。

参考：https://blog.csdn.net/lepton126/article/details/85987197
https://blog.csdn.net/MASILEJFOAISEGJIAE/article/details/89281982