Spark1.6.2 java实现读取json数据文件插入MySql数据库

时间：2021-01-15 15:02:46
public class Main implements Serializable {

/**

*

*/

private static final long serialVersionUID = -8513279306224995844L;

private static final String MYSQL_USERNAME = "demo";

private static final String MYSQL_PWD = "demo";

private static final String MYSQL_CONNECTION_URL = "jdbc:mysql://192.168.1.91:3306/demo";

private static final JavaSparkContext sc = new JavaSparkContext(new SparkConf().setAppName("SparkSaveToDb").setMaster("local[*]"));

private static final SQLContext sqlContext = new SQLContext(sc);

public static void main(String[] args) {

// Sample data-frame loaded from a JSON file

DataFrame usersDf = sqlContext.read().json("users.json");

// Save data-frame to MySQL (or any other JDBC supported databases)

Properties connectionProperties = new Properties();

connectionProperties.put("user", MYSQL_USERNAME);

connectionProperties.put("password", MYSQL_PWD);

// write dataframe to jdbc mysql

usersDf.write().mode(SaveMode.Append).jdbc(MYSQL_CONNECTION_URL, "users", connectionProperties);

}

}

我们为了写入数据方便测试，需要一个json文件，类似下方：

{"id":994,"name":"Betty","email":"bsmithrl@simplemachines.org","city":"Eláteia","country":"Greece","ip":"9.19.204.44"},

{"id":995,"name":"Anna","email":"alewisrm@canalblog.com","city":"Shangjing","country":"China","ip":"14.207.119.126"},

{"id":996,"name":"David","email":"dgarrettrn@japanpost.jp","city":"Tsarychanka","country":"Ukraine","ip":"111.252.63.159"},

{"id":997,"name":"Heather","email":"hgilbertro@skype.com","city":"Koilás","country":"Greece","ip":"29.57.181.250"},

{"id":998,"name":"Diane","email":"ddanielsrp@statcounter.com","city":"Mapiripán","country":"Colombia","ip":"19.205.181.99"},

{"id":999,"name":"Philip","email":"pfullerrq@reuters.com","city":"El Cairo","country":"Colombia","ip":"210.248.121.194"},

{"id":1000,"name":"Maria","email":"mfordrr@shop-pro.jp","city":"Karabash","country":"Russia","ip":"224.21.41.52"}

读取文件时，users.json需要与jar包在同一目录下，测试采用本地运行方式：

DataFrame usersDf = sqlContext.read().json("users.json");

其中，代码中的这行mode(SaveMode.Append)要特别注意，这个使得每次写入的数据是增加到数据表中。否则会一直提 示：Exception in thread “main” java.lang.RuntimeException: Table users already exists.

usersDf.write().mode(SaveMode.Append).jdbc(MYSQL_CONNECTION_URL, "users", connectionProperties）

相关文章