Target audience: Intermediate
Estimated reading time: 5'
This post illustrates the histogram-based function approximation for very large data sets using Apache Spark and Scala.
Note: Implementation relies on Scala 2.11.8, Spark 2.0
Overview
The previous post introduces and describes a model to approximate or fit a function to a given data set, using an histogram implemented in Scala. (see Histogram-based approximation using Scala).
This post re-implements the dynamic resizable histogram using Apache Spark with limited changes in the original code base.
This post re-implements the dynamic resizable histogram using Apache Spark with limited changes in the original code base.
Histogram
The Apache Spark version of histogram-based approximation reuses the classes Bin and Histogram described in the previous post. We made some minor changes to the constructors of the Histogram. As a reminder the Histogram class was defined as follows:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
| final class Histogram(var step: Double,
private var values: mutable.ArrayBuffer[Bin]) {
var min = values.head._x - 0.5*step
var max = values.last._x + 0.5*step
private val maxNumBins = (values.length+1)<<RADIX
def this(first: Double, step: Double, initNumBins: Int) =
this(step, Range(0, initNumBins-1)
.scanLeft(first)((s, n) => s + step)
./:(new mutable.ArrayBuffer[Bin])(
(vals, x) => vals += (new Bin(x)) )
)
def this(params: HistogramParams) =
this(params.first, params.step, params.initNumBins)
def train(x: Double, y: Double): Unit
final def predict(x: Double): Double
// ….
}
|
The class Histogram has three constructors, for convenience's sake:
- Instantiation of an histogram using an estimate of the _x value of the first bin, the width of the bins, step and the initial number of bins, initNumBins. This constructor is invoked for the first batch of data .
- Instantiation of an histogram using an initial given step and an existing array of bins/buckets of width step. This constructor is invoked for subsequent batch of data (line 8-12).
- Instantiation of an histogram using its parameters params (line 15).
The parameters for the histogram are encapsulated in the following HistogramParams class described in a previous post.
I added an implementation of the concatenation of array bins which is required by the reducer implemented by a fold action on Spark data nodes (line 6)
1
2
3
4
5
6
7
8
9
10
11
12
| object Histogram {
final val EPS = 1e-12
final val RADIX = 8
final def sqr(x: Double): Double = x*x
def +(itBins1: Array[Bin], itBins2: Array[Bin]): Array[Bin]={
val histBins = new mutable.ArrayBuffer[Bin]
itBins1.scanLeft(histBins)((newBins, bin) => newBins +=bin)
itBins2.scanLeft(histBins)((newBins, bin) => newBins +=bin)
histBins.toArray
}
}
|
Test data generation
For testing purpose, we need to generate a data set of very large (or even infinite) size. For this purpose we load an existing data set from a file which is duplicated then modified with a random uniform noise to create a large number of batch to be recursively processed by Spark data nodes.
The amplitude of the randomization (synthetic noise) is significant enough to generate different data point while preserving the original data pattern so the algorithm can be easily validated.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
| final class DataGenerator(sourceName: String, nTasks: Int) {
private final val DELIM = " "
private final val RATIO = 0.05
var datasetSize: Int = _
// Generate the RDD from an existing data set template
// and a noise function
def apply(sc: SparkContext): RDD[(Float, Float)] = {
val r = new Random(System.currentTimeMillis +
Random.nextLong)
val src = Source.fromFile(sourceName)
val input = src.getLines.map(_.split(DELIM))
./:(new mutable.ArrayBuffer[(Float, Float)])
((buf, xy) => {
val x = addNoise(xy(0).trim.toFloat, r)
val y = addNoise(xy(1).trim.toFloat, r)
buf += ((x, y))
})
datasetSize = input.size
val data_rdd = sc.makeRDD(input, nTasks)
src.close
data_rdd
}
private def addNoise(value: Float, r: Random): Float =
value*(1.0 + RATIO*(r.nextDouble - 0.5)).toFloat
}
|
The data is generated for a specific number of tasks nTasks and loading the data from a local file sourceName (line 1). The generation of the histogram is implemented by the method apply.
Once loaded from file (line 12), the input is generated locally as an array of (Float, Float) (line 12 -13).
Next, the algorithm adds randomization addNoise (line 15, 16) before converting into a RDD (line 21).
Apache Spark client
The first step is to create a Spark configuration and context for a given number of concurrent tasks.
1
2
3
4
5
6
| val numTasks: Int = 12
val conf = new SparkConf().setAppName("Simple Application")
.setMaster(s"local[$numTasks]")
val sc = new SparkContext(conf)
sc.setLogLevel("ERROR")
|
The main goal of the Spark driver is to apply a tail recursive function across all the batches of data. The histogram is computed/updated on the Spark data node, then reduced to a single histogram which is then broadcasted to all the nodes to be updated with the next batch of data.
The RDDs are generated by an instance of the DataGenerator class as input (line 1). The input is loaded recursively into multiple partitions using the Spark method mapPartitionsWithIndex (line 11). Each partition generates a new array of bins (line30) once the class Histogram is instantiated (line 17) then trained (line 24). The data is then grouped by partition index (line 28). finally the fold method is invoked to generate the array of bins (line 30).
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
| val input = new DataGenerator("small.data", numTasks)
@scala.annotation.tailrec
def execute(iter: Int,
initParam_brdcast: Broadcast[HistogramParams]): ArrayBuffer[Bin]={
val initParam = initParam_brdcast.value
if( iter <= 0)
initParam.values
else {
val hist_rdd = input(sc).mapPartitionsWithIndex(
(idx: Int, it: Iterator[(Float, Float)]) => {
// The first invocation used the broadcasted parameters for
// the histogram and the subsequent invocation update
// the distribution of histogram bins
val histogram =
if (initParam.values.length == 0)
new Histogram(initParam)
else
new Histogram(step, initParam.values)
// Train the histogram with a new batch of data
it.foreach { case (x, y) => histogram.train(x, y) }
histogram.getValues.map((idx, _)).iterator
})
.groupBy(_._1)
.map { case (n, itBins) => itBins.map(_._2).toArray }
.fold(Array[Bin]())(Histogram +)
execute(iter-1, sc.broadcast(
HistogramParams(0.0, initParam.step, 0, new ArrayBuffer[Bin]++hist_rdd)
)
}
}
|
Note:
The invocation of the recursive method execute on the Apache Spark driver (line 4) broadcasts the current (recently updated) parameters of the histogram initParms_brdcast as one of the arguments of the recursive call (line 5). Upon receiving the broadcast values, the partitions load the next batch of data, generates and processes the RDD input.apply(sc) and update the histogram parameters (line 11).
Evaluation
Let's write a test driver to generate the Histogram using Spark RDD. After their initialization (line 1-4), the parameters of the histogram are simply broadcasted to the remote data/worker nodes (line 7,8).
1
2
3
4
5
6
7
8
9
| val initNumBins = 2048
val range = (-1.0, 2.0)
var step = (range._2 - range._1)/initNumBins
val maxRecursions = 4
val hist = execute(maxRecursions,
sc.broadcast(
HistogramParams(range._1 + 0.5 * step, step, initNumBins))
)
|
References
- Histogram-based approximation using Scala
- Statistics methods - Chapter 2, Descriptive statistics - Pfaffenberger, Patternson - Irwin publications 1981.
- github.com/patnicolas
