Apache Spark, Hive, and Spring Boot — Testing Guide

• apache-spark-integration-testing-example

  1 3 0.0 Java

  An example of integration tests for Apache Spark Spring Boot application that transfers data from Apache Hive table to Aerospike Database.

  In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• shadow

  15 3,783 9.3 Groovy

  Gradle plugin to create fat/uber JARs, apply file transforms, and relocate packages for applications and libraries. Gradle version of Maven's Shade plugin.

  

  The result .jar is going to submitted to Apache Spark cluster (e.g. spark-submit command). So, it should contain all runtime artefacts. Unfortunately, the standard Spring Boot packaging does not put the dependencies in the way Apache Spark expects it. So, we'll use shadow-jar Gradle plugin. Take a look at the example below.

• initializr

  302 3,463 8.9 Java

  A quickstart generator for Spring projects

  

  The project is bootstrapped with Spring Initializr. Nothing special here. But the dependencies list should be clarified.

• Apache Spark

  109 40,073 10.0 Scala

  Apache Spark - A unified analytics engine for large-scale data processing

  

  In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.

• Apache Hive

  15 5,564 9.6 Java

  Apache Hive

  

  In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.

• Aerospike

  16 1,094 9.0 C

  Aerospike Database Server – flash-optimized, in-memory, nosql database

  

  In this article, I'm showing you how to create a Spring Boot app that loads data from Apache Hive via Apache Spark to the Aerospike Database. More than that, I'm giving you a recipe for writing integration tests for such scenarios that can be run either locally or during the CI pipeline execution. The code examples are taken from this repository.

Suggest a related project