spark application提交应用的两种方式

时间:2023-03-08 17:43:32

bin/spark-submit --help

... ...

--deploy-mode DEPLOY_MODE   Whether to launch the driver program locally ("client") or
                               on one of the worker machines inside the cluster ("cluster")
                               (Default: client).

... ...

--deploy-mode定义了spark运行时的两种方式:

1)client模式:
     spark应用driver运行在本地,应用输出日志可以在本地查看。

2)cluster模式:
     spark应用driver运行在cluster中一台节点上,应用输出日志只能通过ui界面查看。

例1:spark运行在client模式下:

bin/hdfs dfs -mkdir -p /user/beifeng/mapreduce/wordcount/

bin/spark-submit \

--master spark://chavin.king:7077 \

--deploy-mode client \

jars/sparkApp.jar

例2:spark运行在cluster模式下:

bin/hdfs dfs -mkdir -p /user/beifeng/mapreduce/wordcount/

bin/spark-submit \

--master spark://chavin.king:7077 \

--deploy-mode cluster \

jars/sparkApp.jar