Half a year ago, someone guy hijacked a Yandex Drive car that I rented. Hijacked because the application did not close the car. This has happened many times before, but car-sharing users know that Yandex is not in a hurry, that you are not using a car. When the car…

Testing with HDFS Best way ever I think is to start testing of your Spark application with preparing of integration tests. There are a lot of internal and external tools for building this kind of tests. I prefer to use internal testing tools of used frameworks in highest priority with relation to external…

Overview In our company in the production we have a hundreds of scheduling machine learning (ML) models and calculation scripts on a constant basis. And our Data Engineering division had taken a task to make some CD/CD component that was most suitable for our goals: Wide amount of usable libraries from…

Third part of article series is about Spark Streaming unit testing. This series pretend to fill the gap between code and documentation inside Spark Unit Testing domain. Spark has a huge Framework that allow to developers to test their code in any various cases. There is one package spark streaming…

Second part of article series about how to use Spark repository classes for Unit Testing. This series pretend to fill the gap between code and documentation inside Spark Unit Testing domain. Spark has a huge Framework that allow to developers to test their code in any various cases. …

This article is about how to use own Spark repository classes for Unit Testing and pretend to fill the gap between code and documentation inside Spark Unit Testing domain. Spark has a huge Framework that allow to developers to test their code in any various cases. …