How to be Test Driven with Spark: Chapter 5: Leverage spark in a container

1. docker

   1 88 204 10.0 Go

   FreeBSD port of docker, take a look at PORTING-FREEBSD.md in freebsd-compat branch (by kvasdopil)

   

   If you are unfamiliar with the concept of containers and docker images, I suggest you have a look at docker. It will be leveraged here to start the Spark server for the tests; it's important to mention there are other opensource alternatives like podman or nerdctl to allow containerization.

2. Stream

   getstream.io featured

   Stream - Scalable APIs for Chat, Feeds, Moderation, & Video. Stream helps developers build engaging apps that scale to millions with performant and flexible Chat, Feeds, Moderation, and Video APIs and SDKs powered by a global edge network and enterprise-grade infrastructure.

   

3. nerdctl

   2 38 9,039 9.9 Go

   contaiNERD CTL - Docker-compatible CLI for containerd, with support for Compose, Rootless, eStargz, OCIcrypt, IPFS, ...

   

   If you are unfamiliar with the concept of containers and docker images, I suggest you have a look at docker. It will be leveraged here to start the Spark server for the tests; it's important to mention there are other opensource alternatives like podman or nerdctl to allow containerization.

4. podman

   3 402 27,505 10.0 Go

   Podman: A tool for managing OCI containers and pods.

   

   If you are unfamiliar with the concept of containers and docker images, I suggest you have a look at docker. It will be leveraged here to start the Spark server for the tests; it's important to mention there are other opensource alternatives like podman or nerdctl to allow containerization.

5. spark_tdd

   4 3 0 3.8

   

   The same error should appear, indeed the spark_session needs to be adapted to connect to the server you have just created. In test/conftest.py:

6. testcontainers-python

   5 13 1,877 8.4 Python

   Testcontainers is a Python library that providing a friendly API to run Docker container. It is designed to create runtime environment to use during your automatic tests.

   

   To improve this, you will leverage testcontainers which empower you with easy docker creation and removal at the test level.

7. ffizer

   6 2 116 8.8 Rust

   ffizer is a files and folders initializer / generator. Create any kind (or part) of project from template.

   

   Templatize the repository for easier reusage with the help of ffizer

8. spec

   7 63 4,415 3.5

   Development Containers: Use a container as a full-featured development environment. (by devcontainers)

   

   Leverage devcontainer to improve ci and local development

9. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

Suggest a related project