Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十三）kafka+spark streaming打包好的程序提交时提示虚拟内存不足（Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 G）

时间：2022-05-19 08:22:57

异常问题：Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

spark-submit提交脚本：

[spark@master work]$ more submit.sh

#! /bin/bash

jars=""

for file in `ls /home/spark/work/jars/*.jar`

do

        jars=$file,$jars

        #echo $jars

done

echo "------------------------------------"

echo $jars

echo "------------------------------------"

/opt/spark-2.2.1-bin-hadoop2.7/bin/spark-submit \

--jars $jars \

--master yarn \

--verbose \

--driver-java-options "-XX:+TraceClassPaths" \

--num-executors 2 \

--executor-memory 1G \

--executor-cores 1 \

--driver-memory 1G \

--class com.dx.streaming.producer.TestProducer \

/home/spark/work/kafka-streaming-test.jar

在master节点上提交后，抛出异常：

18/07/09 05:00:16 INFO yarn.Client: Application report for application_1531069155353_0001 (state: ACCEPTED)

18/07/09 05:00:17 INFO yarn.Client: Application report for application_1531069155353_0001 (state: FAILED)

18/07/09 05:00:17 INFO yarn.Client:

         client token: N/A

         diagnostics: Application application_1531069155353_0001 failed 2 times due to AM Container for appattempt_1531069155353_0001_000002 exited with  exitCode: -103

Failing this attempt.Diagnostics: [2018-07-09 05:00:16.132]Container [pid=5674,containerID=container_e22_1531069155353_0001_02_000001] is running beyond virtual memory limits. Current usage: 195.8 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.

Dump of the process-tree for container_e22_1531069155353_0001_02_000001 :

        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE

        |- 5674 5673 5674 5674 (bash) 0 0 115896320 350 /bin/bash -c /opt/jdk1.8.0_171/bin/java -server -Xmx512m -Djava.io.tmpdir=/opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/tmp -Dspark.yarn.app.container.log.dir=/opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg '192.168.0.120:40486' --properties-file /opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/__spark_conf__/__spark_conf__.properties 1> /opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/stdout 2> /opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/stderr

        |- 5687 5674 5674 5674 (java) 208 35 2323283968 49770 /opt/jdk1.8.0_171/bin/java -server -Xmx512m -Djava.io.tmpdir=/opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/tmp -Dspark.yarn.app.container.log.dir=/opt/hadoop-2.9.0/logs/userlogs/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg 192.168.0.120:40486 --properties-file /opt/hadoop-2.9.0/tmp/nm-local-dir/usercache/spark/appcache/application_1531069155353_0001/container_e22_1531069155353_0001_02_000001/__spark_conf__/__spark_conf__.properties

[2018-07-09 05:00:16.133]

[2018-07-09 05:00:16.152]Container killed on request. Exit code is 143[2018-07-09 05:00:16.152]

[2018-07-09 05:00:16.156]Container exited with a non-zero exit code 143. [2018-07-09 05:00:16.156]

For more detailed output, check the application tracking page: http://master:8088/cluster/app/application_1531069155353_0001 Then click on links to logs of each attempt.

. Failing the application.

         ApplicationMaster host: N/A

         ApplicationMaster RPC port: -1

         queue: default

         start time: 1531083596749

         final status: FAILED

         tracking URL: http://master:8088/cluster/app/application_1531069155353_0001

         user: spark

18/07/09 05:00:17 INFO yarn.Client: Deleted staging directory hdfs://HA/user/spark/.sparkStaging/application_1531069155353_0001

18/07/09 05:00:17 ERROR spark.SparkContext: Error initializing SparkContext.

org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)

        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)

▽       at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)

        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)

        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)

        at scala.Option.getOrElse(Option.scala:121)

        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)

        at com.dx.streaming.producer.TestProducer.main(TestProducer.java:39)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

18/07/09 05:00:17 INFO server.AbstractConnector: Stopped Spark@60d84f61{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}

18/07/09 05:00:17 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.0.120:4040

18/07/09 05:00:17 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!

18/07/09 05:00:17 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors

18/07/09 05:00:17 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down

18/07/09 05:00:17 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices

(serviceOption=None,

 services=List(),

 started=false)

18/07/09 05:00:17 INFO cluster.YarnClientSchedulerBackend: Stopped

18/07/09 05:00:17 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

18/07/09 05:00:17 INFO memory.MemoryStore: MemoryStore cleared

18/07/09 05:00:17 INFO storage.BlockManager: BlockManager stopped

18/07/09 05:00:17 INFO storage.BlockManagerMaster: BlockManagerMaster stopped

18/07/09 05:00:17 WARN metrics.MetricsSystem: Stopping a MetricsSystem that is not running

18/07/09 05:00:17 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

18/07/09 05:00:17 INFO spark.SparkContext: Successfully stopped SparkContext

Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:85)

        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:62)

        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:173)

        at org.apache.spark.SparkContext.<init>(SparkContext.scala:509)

        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)

        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)

        at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)

        at scala.Option.getOrElse(Option.scala:121)

        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)

        at com.dx.streaming.producer.TestProducer.main(TestProducer.java:39)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)

        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)

        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)

        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)

        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

18/07/09 05:00:17 INFO util.ShutdownHookManager: Shutdown hook called

18/07/09 05:00:17 INFO util.ShutdownHookManager: Deleting directory /opt/spark-2.2.1-bin-hadoop2.7/spark-255c65cf-19d8-481a-94bc-843e058eb669

错误原因：

Container is running beyond virtual memory limits. Current usage: 119.5 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.

YARN的虚拟内存计算方式导致，上例中用户程序申请的内存为1Gb，YARN根据此值乘以一个比例（默认为2.1）得出申请的虚拟内存的值，当YARN计算的用户程序所需虚拟内存值大于计算出来的值时，就会报出以上错误。

解决方案：

上网一查，发现有两种方法能解决这个问题:

1.将yarn.nodemanager.vmem-check-enabled的值改为false，即不检查VM的值；（修改yarn-site.xml）参考《https://discuss.elastic.co/t/unable-to-start-elasticsearch-yarn-container-is-running-beyond-virtual-memory-limits/26102/6》

<property>

    <name>yarn.nodemanager.vmem-check-enabled</name>

    <value>false</value>

</property>

2.将yarn.scheduler.minimum-allocation-mb的值调高一些，默认是1024mb,或者修改yarn.nodemanager.vmem-pmem-ratio的值，默认为2.1，将该值改得更大。

于是，这里我采用了第一种方法，关闭vn检查，需要重新格式化namenode，参考《Kafka：ZK+Kafka+Spark Streaming集群环境搭建（十）安装hadoop2.9.0搭建HA》，再次提交，果然好了。终于解决了这个困扰一天的问题啦！

相关文章