Spark学习(4)----ScalaTest

时间:2022-01-26 02:25:02

一、例子:

1、一个简单例子:https://www.jianshu.com/p/ceabf3437dd7

2、Funsuite例子:https://www.programcreek.com/scala/org.scalatest.FunSuite

3、SparkFunsuite例子:https://www.programcreek.com/scala/org.apache.spark.SparkFunSuite

4、一个spark学习博客:https://liuxiaofei.com.cn/blog/category/spark/spark-core/

  一个scalatest博客:https://blog.csdn.net/zhouyan8603/article/details/85560861

5、一个GBTRegressorSuite例子:https://fossies.org/linux/spark/mllib/src/test/scala/org/apache/spark/ml/regression/GBTRegressorSuite.scala

6、scalatest在线文档:http://tool.oschina.net/apidocs/apidoc?api=scalatest-1.7.2

7、scalatest官方文档:http://www.scalatest.org/getting_started_with_fun_suite

        http://doc.scalatest.org/3.0.1/#org.scalatest.FunSuite

7、spark官方文档:http://spark.apache.org/docs/2.3.0/

二、 记录一个折腾了两天的错误

在本地写了ScalaTest的测试,并且可以通过,但是一放上服务器,一直报错,错误有:

【只要有dataframe.show()出现就报错,有collect() 就报错】
java.lang.UnsupportedOperationException: empty.max
java.lang.IllegalArgumentException: Can not set final [B field org.codehaus.janino.util.ClassFile$CodeAttribute.code to org.codehaus.janino.util.ClassFile$CodeAttribute
以上这些错误可能还是版本问题,因为我在maven的pom.xml中加了test的相关环境:【红字原来没加,就会报错】
    <dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<classifier>tests</classifier>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<classifier>tests</classifier>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<classifier>tests</classifier>
<scope>test</scope>
</dependency>

所以要加上红色字体,表明这是test,就不会变异打包到服务器上跑,因为服务器上原本也有spark环境,这样两个spark环境就会有冲突。