Apache Spark Unit Testing Part 1 — Core Components

Dependencies

Core components

SparkFunSuite

SharedSparkContext

LocalSparkContext

JsonTestUtils

Smuggle

1.2 Benchmark

Benchmark, BenchmarkBase

1.3 Security

EncryptionFunSuite

1.3 Util

SparkConfWithEnv

--

--

--

I believe that science makes the World better. Big Data, Machine Learning, Quantum Computing.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Generate CSR for Digital Certificates in Azure in 5 mins

CSVs on Rails!

Object-Oriented Programming

CS373 Spring 2020: Laith Alsukhni — Week 14

Shiden Growth NFT Contest #1 in collaboration with ShMonsters

How I got started with OpenMRS?

Introducing MYOB Future Maker Academy Test Labs

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Eugene Lopatkin

Eugene Lopatkin

I believe that science makes the World better. Big Data, Machine Learning, Quantum Computing.

More from Medium

Understanding Apache Hive Transforms

from the above table we can easily understand that 1st offset will process the 24 rows(4+12+08)…

Integrating Confluent Schema Registry with Apache Spark applications

Apache Spark: aggregateByKey vs combineByKey