spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

时间：2023-01-06 18:54:17

　　不多说，直接上干货！

　　问题详情

　 电脑8G，目前搭建3节点的spark集群，采用YARN模式。

master分配2G，slave1分配1G,slave2分配1G。（在安装虚拟机时）

export SPARK_WORKER_MERMORY=1g  （在spark-env.sh）

export JAVA_HOME=/usr/local/jdk/jdk1.8.0_60         （必须写）

export SCALA_HOME=/usr/local/scala/scala-2.10.5    （必须写）

export HADOOP_HOME=/usr/local/hadoop/hadoop-2.6.0    （必须写）

export HADOOP_CONF_DIR=/usr/local/hadoop/hadoop-2.6.0/etc/hadoop   （必须写）

export SPARK_MASTER_IP=192.168.80.10　　

export SPARK_WORKER_MERMORY=1G     （官网上说，至少1g）

　其实这个问题解决办法很简单，就是3个节点的是，你的内存最好还是大点。如master分配4G，slave1分配2G,slave2分配2G！！！！（尽可能的大）

当然，很多博友们，跟我的情况是一样的，在学校阶段，电脑只有8g已经是最大电脑内存限制了。

一般是由于有多个用户同时向集群提交任务或一个用户向集群同时提交了多个任务导致Yarn资源的分配错误。解决这个问题，只需要更改Hadoop的配置文件：/etc/hadoop/conf/capacity-scheduler.xml，把选项：yarn.scheduler.capacity.maximum-am-resource-percent从0.1改成0.5。顾名思义，这个选项是增加Yarn可调度的资源量，当然也可以视具体情况增加更多。也可见，默认情况下，Yarn没有将很多资源分配给任务的能力。

具体，见

Spark on YARN模式的安装（spark-1.6.1-bin-hadoop2.6.tgz + hadoop-2.6.0.tar.gz）（master、slave1和slave2）（博主推荐）

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)
[spark@master logs]$  $SPARK_HOME/bin/spark-submit  \

> --class org.apache.spark.examples.JavaSparkPi \

> --master yarn-cluster \

> --num-executors 1 \

> --driver-memory 512m \

> --executor-memory 512m \

> --executor-cores 1 \

>  /usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-examples-1.6.1-hadoop2.6.0.jar

注意：
driver-memory不指定也可以，默认使用512M
executor-memory不指定的化， 默认是1G
 
 
17/04/09 17:03:55 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

17/04/09 17:03:55 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.80.10:8032

17/04/09 17:03:56 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers

17/04/09 17:03:56 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)

17/04/09 17:03:56 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead

17/04/09 17:03:56 INFO yarn.Client: Setting up container launch context for our AM

17/04/09 17:03:56 INFO yarn.Client: Setting up the launch environment for our AM container

17/04/09 17:03:56 INFO yarn.Client: Preparing resources for our AM container

17/04/09 17:03:59 INFO yarn.Client: Uploading resource file:/usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1491728358337_0001/spark-assembly-1.6.1-hadoop2.6.0.jar

17/04/09 17:04:19 INFO yarn.Client: Uploading resource file:/usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-examples-1.6.1-hadoop2.6.0.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1491728358337_0001/spark-examples-1.6.1-hadoop2.6.0.jar

17/04/09 17:04:49 INFO yarn.Client: Uploading resource file:/tmp/spark-d152ed1b-09ca-47c8-8457-58f7e52ff419/__spark_conf__6499474209714260387.zip -> hdfs://master:9000/user/spark/.sparkStaging/application_1491728358337_0001/__spark_conf__6499474209714260387.zip

17/04/09 17:04:50 INFO spark.SecurityManager: Changing view acls to: spark

17/04/09 17:04:50 INFO spark.SecurityManager: Changing modify acls to: spark

17/04/09 17:04:50 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)

17/04/09 17:04:50 INFO yarn.Client: Submitting application 1 to ResourceManager

17/04/09 17:04:51 INFO impl.YarnClientImpl: Submitted application application_1491728358337_0001

17/04/09 17:04:52 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:04:52 INFO yarn.Client:

     client token: N/A

     diagnostics: N/A

     ApplicationMaster host: N/A

     ApplicationMaster RPC port: -1

     queue: default

     start time: 1491728691236

     final status: UNDEFINED

     tracking URL: http://master:8088/proxy/application_1491728358337_0001/

     user: spark

17/04/09 17:04:53 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:04:54 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:04:55 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:04:56 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:04:57 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:04:58 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:04:59 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:05:00 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:05:01 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:05:02 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:05:03 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:05:04 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:05:05 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:05:06 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

17/04/09 17:05:07 INFO yarn.Client: Application report for application_1491728358337_0001 (state: ACCEPTED)

^C17/04/09 17:05:08 INFO util.ShutdownHookManager: Shutdown hook called

17/04/09 17:05:08 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-d152ed1b-09ca-47c8-8457-58f7e52ff419

.........
然后最后失败
spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

　问题分析

　　　　其实，这一情况，是我刚开始启动了太多的Application导致累积，使得内存超出了导致！

　解决方法

　　　　按照办法1，办法2，办法3...的顺序来

 办法1:（一般能解决）

　　先，

[spark@master ~]$ cd $SPARK_HOME
[spark@master spark-1.6.1-bin-hadoop2.6]$ sbin/stop-all.sh

[spark@master spark-1.6.1-bin-hadoop2.6]$ cd $HADOOP_HOME
[spark@master hadoop-2.6.0]$ sbin/stop-all.sh

　　再，

[spark@master spark-1.6.1-bin-hadoop2.6]$ cd $HADOOP_HOME
[spark@master hadoop-2.6.0]$ sbin/start-all.sh

[spark@master ~]$ cd $SPARK_HOME
[spark@master spark-1.6.1-bin-hadoop2.6]$ sbin/start-all.sh

　　在spark-env.sh，至少得

export SPARK_WORKER_MERMORY=1G

　　改成

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)
[spark@master spark-1.6.1-bin-hadoop2.6]$ $SPARK_HOME/bin/spark-submit \
--class org.apache.spark.examples.JavaSparkPi \
--master yarn-cluster \
--num-executors 1 \
--driver-memory 1g \
--executor-memory 1g \
--executor-cores 1 \
$SPARK_HOME/lib/spark-examples-1.6.1-hadoop2.6.0.jar

[spark@master spark-1.6.1-bin-hadoop2.6]$ $SPARK_HOME/bin/spark-submit --class org.apache.spark.examples.JavaSparkPi --master yarn-cluster --num-executors 1 --driver-memory 1g --executor-memory 1g --executor-cores 1 $SPARK_HOME/lib/spark-examples-1.6.1-hadoop2.6.0.jar
17/04/09 20:01:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/09 20:01:27 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.80.10:8032
17/04/09 20:01:31 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers
17/04/09 20:01:32 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/04/09 20:01:32 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
17/04/09 20:01:32 INFO yarn.Client: Setting up container launch context for our AM
17/04/09 20:01:32 INFO yarn.Client: Setting up the launch environment for our AM container
17/04/09 20:01:32 INFO yarn.Client: Preparing resources for our AM container
17/04/09 20:01:37 INFO yarn.Client: Uploading resource file:/usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-assembly-1.6.1-hadoop2.6.0.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1491735225663_0006/spark-assembly-1.6.1-hadoop2.6.0.jar
17/04/09 20:01:57 INFO yarn.Client: Uploading resource file:/usr/local/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-examples-1.6.1-hadoop2.6.0.jar -> hdfs://master:9000/user/spark/.sparkStaging/application_1491735225663_0006/spark-examples-1.6.1-hadoop2.6.0.jar
17/04/09 20:02:18 INFO yarn.Client: Uploading resource file:/tmp/spark-8d8662b1-6410-4b72-9751-57bb0e2785d0/__spark_conf__5191259430263541003.zip -> hdfs://master:9000/user/spark/.sparkStaging/application_1491735225663_0006/__spark_conf__5191259430263541003.zip
17/04/09 20:02:18 INFO spark.SecurityManager: Changing view acls to: spark
17/04/09 20:02:19 INFO spark.SecurityManager: Changing modify acls to: spark
17/04/09 20:02:19 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)
17/04/09 20:02:22 INFO yarn.Client: Submitting application 6 to ResourceManager
17/04/09 20:02:23 INFO impl.YarnClientImpl: Submitted application application_1491735225663_0006
17/04/09 20:02:25 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:25 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1491739342315
     final status: UNDEFINED
     tracking URL: http://master:8088/proxy/application_1491735225663_0006/
     user: spark
17/04/09 20:02:26 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:32 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:35 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:36 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:37 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:38 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:39 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:42 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:43 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:44 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:45 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:46 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:47 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:48 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:49 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:50 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:51 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:53 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:54 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:55 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:56 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:57 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:58 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:02:59 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:00 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:01 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:02 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:04 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)

17/04/09 20:03:05 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:06 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:07 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:08 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:09 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:10 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:11 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:12 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:13 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:14 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:15 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:16 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:17 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:18 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:19 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:20 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:21 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:23 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:24 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:25 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:26 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:27 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:28 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:29 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:30 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:31 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:32 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:33 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:34 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:36 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:38 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:39 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:40 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:41 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:42 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:43 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:44 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:45 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:46 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:47 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:48 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:50 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:51 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:52 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:53 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:54 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:55 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:56 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:57 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:58 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:03:59 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:00 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:01 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:02 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:03 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:04 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:05 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:06 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:07 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:08 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:09 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:10 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:11 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:12 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:13 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:15 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:16 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:17 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:18 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:19 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:20 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:21 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:22 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:23 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:24 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:25 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:26 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:27 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:28 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:29 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:30 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:31 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:32 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:33 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:34 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:35 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:36 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:37 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:38 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:39 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:40 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:41 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:43 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:44 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:45 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:46 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:47 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:48 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:49 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:50 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:51 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:52 INFO yarn.Client: Application report for application_1491735225663_0006 (state: ACCEPTED)
17/04/09 20:04:53 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)   只要最后出现这个了，就说明成功了！
17/04/09 20:04:53 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 192.168.80.11
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1491739342315
     final status: UNDEFINED
     tracking URL: http://master:8088/proxy/application_1491735225663_0006/
     user: spark
17/04/09 20:04:54 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:55 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:56 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:57 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:58 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:04:59 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:00 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:01 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:02 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:04 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:05 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:06 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:07 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:08 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:09 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:10 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:11 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:12 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:13 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:14 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:15 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:16 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:17 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:19 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:20 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:21 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:22 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:23 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:25 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:26 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:27 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:28 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:29 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:30 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:31 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:32 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:33 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:34 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:35 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:36 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:37 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:38 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:39 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:40 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:41 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:42 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:43 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:44 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:45 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:46 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:05:47 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
....
17/04/09 20:06:57 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:06:58 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:06:59 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:00 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:01 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:02 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:03 INFO yarn.Client: Application report for application_1491735225663_0006 (state: RUNNING)
17/04/09 20:07:04 INFO yarn.Client: Application report for application_1491735225663_0006 (state: FINISHED)
17/04/09 20:07:04 INFO yarn.Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 192.168.80.11
     ApplicationMaster RPC port: 0
     queue: default
     start time: 1491739342315
     final status: SUCCEEDED
     tracking URL: http://master:8088/proxy/application_1491735225663_0006/A
     user: spark
17/04/09 20:07:04 INFO yarn.Client: Deleting staging directory .sparkStaging/application_1491735225663_0006
17/04/09 20:07:07 INFO util.ShutdownHookManager: Shutdown hook called
17/04/09 20:07:07 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-8d8662b1-6410-4b72-9751-57bb0e2785d0
[spark@master spark-1.6.1-bin-hadoop2.6]$
spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

　　如果之后，再次去执行别的jar包，出现又是无法运行了，则

spark on yarn模式里需要有时手工释放linux内存。

 解决办法

方法2： 首先想到是集群中内存资源不足，可以检查下每台机器是否有足够剩余内存( free -g)；也可能是其他已经提交的Spark应用占了大部分资源；

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

[spark@master spark-1.6.1-bin-hadoop2.6]$ free -g

             total       used       free     shared    buffers     cached

Mem:             1          1          0          0          0          0

-/+ buffers/cache:          0          0

Swap:            1          0          1

[spark@master spark-1.6.1-bin-hadoop2.6]$

　　那么，就要去杀死之前那些还正在运行（多余没用）的Application。

　　怎么做？

　　进入你的$HADOOP_HOME/bin下

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

　　执行命令

[spark@master bin]$ yarn application -list

　　然后可以看到应用的信息

　　如需杀死当前某个作业，使用kill application-id的命令如下

yarn application -kill application_1437456051228_1725

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

方法3： 如果1>正常，我们可以看看YARN集群是否启动成功。注意“坑”可能就在这里： 即使Slave上的nodemanager进程存在，要注意检查resource manager日志，看看各个node manager是否启动成功，有些人的问题就出现在这里：进程在，但是日志显示node manager状态为UNHEALTHY，所以YARN集群能识别到的总内存资源为0。检查了UNHEALTHY的原因，是因为/tmp下一个目录被识别为bad, 因为是临时目录，把每个node manager的对应目录删掉，然后重启YARN集群，最终问题解决。

ResourceManager日志存放位置是${HADOOP_HOME}/logs/yarn-*-resourcemanager-*.log

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

NodeManager日志存放位置是各个NodeManager节点上${HADOOP_HOME}/logs/yarn-*-nodemanager-*.log

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

　　先，

[spark@master ~]$ cd $SPARK_HOME
[spark@master spark-1.6.1-bin-hadoop2.6]$ sbin/stop-all.sh

[spark@master spark-1.6.1-bin-hadoop2.6]$ cd $HADOOP_HOME
[spark@master hadoop-2.6.0]$ sbin/stop-all.sh

　　再，

[spark@master spark-1.6.1-bin-hadoop2.6]$ cd $HADOOP_HOME
[spark@master hadoop-2.6.0]$ sbin/start-all.sh

[spark@master ~]$ cd $SPARK_HOME
[spark@master spark-1.6.1-bin-hadoop2.6]$ sbin/start-all.sh

方法4：如果还是有问题的话，只能去日志文件中查看出错的异常，查看更多一些的报错信息。

　　　　　　但是，方法1和方法2是足够可以解决该问题的!

方法5：

　　　　除非你的内存足够，可以扩大内存。我的只是自己电脑最大才8G

方法6：

　　在yarn-site.xml下

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)
<property>

          <name>yarn.nodemanager.pmem-check-enabled</name>

          <value>false</value>

</property>

<property>

          <name>yarn.nodemanager.vmem-check-enabled</name>

          <value>false</value>

</property>
spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

　　具体见

http://dongxicheng.org/mapreduce-nextgen/hadoop-yarn-memory-cpu-scheduling/

【YARN中内存资源的调度和隔离】

【YARN中CPU资源的调度和隔离】

方法7：

运行hadoop的时候提示物理内存或虚拟内存溢出的解决方案running beyond physical memory或者beyond vitual memory limits

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

方法8

在yarn-site.xml中增加：

   <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>spark_shuffle,mapreduce_shuffle</value>

    </property>

   <property>

	<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>

	<value>org.apache.spark.network.yarn.YarnShuffleService</value>

   </property>

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)

cp /usr/local/spark-2.3.0-bin-hadoop2.7/yarn/spark-2.3.0-yarn-shuffle.jar /usr/local/hadoop-2.7.3/share/hadoop/yarn/lib/

方法9

　　　　　　最坏的打算，就是之前有快照，回到初始位置吧！重新安装，哎

欢迎大家，加入我的微信公众号：大数据躺过的坑        人工智能躺过的坑
 
 
 

同时，大家可以关注我的个人博客：

   http://www.cnblogs.com/zlslch/   和     http://www.cnblogs.com/lchzls/      http://www.cnblogs.com/sunnyDream/   

   详情请见：http://www.cnblogs.com/zlslch/p/7473861.html

　　人生苦短，我愿分享。本公众号将秉持活到老学到老学习无休止的交流分享开源精神，汇聚于互联网和个人学习工作的精华干货知识，一切来于互联网，反馈回互联网。
　　目前研究领域：大数据、机器学习、深度学习、人工智能、数据挖掘、数据分析。
语言涉及：Java、Scala、Python、Shell、Linux等
。同时还涉及平常所使用的手机、电脑和互联网上的使用技巧、问题和实用软件。 只要你一直关注和呆在群里，每天必须有收获

spark跑YARN模式或Client模式提交任务不成功（application state: ACCEPTED）(转)的更多相关文章

  1. Spark On Yarn的两种模式yarn-cluster和yarn-client深度剖析

    Spark On Yarn的优势 每个Spark executor作为一个YARN容器(container)运行.Spark可以使得多个Tasks在同一个容器(container)里面运行 1. Sp ...

  2. Spark on YARN两种运行模式介绍

    本文出自:Spark on YARN两种运行模式介绍http://www.aboutyun.com/thread-12294-1-1.html(出处: about云开发)   问题导读 1.Spark ...

  3. spark跑YARN模式或Client模式提交任务不成功（application state&colon; ACCEPTED）

    不多说,直接上干货! 问题详情 电脑8G,目前搭建3节点的spark集群,采用YARN模式. master分配2G,slave1分配1G,slave2分配1G.(在安装虚拟机时) export SPA ...

  4. Spark on yarn的两种模式 yarn-cluster 和 yarn-client

    从深层次的含义讲,yarn-cluster和yarn-client模式的区别其实就是Application Master进程的区别,yarn-cluster模式下,driver运行在AM(Applic ...

  5. Spark on YARN模式的安装（spark-1&period;6&period;1-bin-hadoop2&period;6&period;tgz &plus; hadoop-2&period;6&period;0&period;tar&period;gz）（master、slave1和slave2）（博主推荐）

    说白了 Spark on YARN模式的安装,它是非常的简单,只需要下载编译好Spark安装包,在一台带有Hadoop YARN客户端的的机器上运行即可.  Spark on YARN简介与运行wor ...

  6. spark on yarn模式里需要有时手工释放linux内存

    为什么要提出这个问题? spark跑YARN模式或Client模式提交任务不成功(application state: ACCEPTED) 然后执行 [spark@master spark--bin- ...

  7. spark on yarn模式下内存资源管理（笔记2）

    1.spark 2.2内存占用计算公式 https://blog.csdn.net/lingbo229/article/details/80914283 2.spark on yarn内存分配** 本 ...

  8. Java虚拟机6：内存溢出和内存泄露、并行和并发、Minor GC和Full GC、Client模式和Server模式的区别

    前言 之前的文章尤其是讲解GC的时候提到了很多的概念,比如内存溢出和内存泄露.并行与并发.Client模式和Server模式.Minor GC和Full GC,本文详细讲解下这些概念的区别. 内存溢出 ...

  9. java虚拟机--jvm client模式与server模式的区别

    JVM Server模式与client模式启动,最主要的差别在于:-Server模式启动时,速度较慢,但是一旦运行起来后,性能将会有很大的提升.JVM如果不显式指定是-Server模式还是-clien ...

随机推荐

  1. iOS之02-第一个OC的类

    OC是一门面向对象的语言,因此它也有类.对象.静态\动态方法.成员变量的概念.这讲就来创建第一个OC的类. 第一个类的源码: /* 人 类名:Person 属性(成员变量\实例变量):体重.年龄 行为 ...

  2. 概率dp入门

    概率DP主要用于求解期望.概率等题目. 转移方程有时候比较灵活. 一般求概率是正推,求期望是逆推.通过题目可以体会到这点. poj2096:Collecting Bugs #include <i ...

  3. CENTOS 下编译HTK

    在centenos下编译HTK碰到缺少libX11库,需要安装 libX11-dev libXext-dev libXtst-dev

  4. excel设置下拉菜单，并且不同值会显示不同颜色

    工作中常常要用的excel,每次都会有新的需求,然后不会,然后百度,然后过段时间可能就又忘了,于是就想说,自己记录下来~~~因为自己用的都是2010,其实哪个版本都差不多,都是应该可以找到相应的按钮滴 ...

  5. 设置windows窗口ICON 【windows 编程】【API】【原创】

    1. ICON介绍 最近开始接触windows 编程,因此将自己所接触的一些零散的知识进行整理并记录.本文主要介绍了如何更改windows对话框窗口的ICON图标.这里首先介绍一下windows IC ...

  6. 关于Mysql索引的笔记

    MySQL索引原理 索引目的 索引的目的在于提高查询效率,可以类比字典,如果要查“mysql”这个单词,我们肯定需要定位到m字母,然后从下往下找到y字母,再找到剩下的sql.如果没有索引,那么你可能需 ...

  7. spring &commat;Scheduled 执行2次

    今天遇到定时任务Scheduled 执行2次的情况,做一个简单的记录. 网上有好多办法,我几乎都试了一遍,我的情况下面的办法可用. 1. autodeploy属性值设置为false,如果此项设为tru ...

  8. node c&plus;&plus;多线程插件构想

    最近想写一个node的c++插件实现线程.提供的api使用回调并进行二次包装使其返回一个promise,并且要求需要在工作线程里执行的函数为async函数.如果是node7.0以下的版本,函数必须返回 ...

  9. minimun path sum&lpar;最小路径和&rpar;

    Given a m x n grid filled with non-negative numbers, find a path from top left to bottom right which ...

  10. Notes ： &lt&semi;Hands-on ML with Sklearn &amp&semi; TF&gt&semi; Chapter 1

    <Hands-on ML with Sklearn & TF> Chapter 1 what is ml from experience E with respect to som ...

相关文章