scala> import org.apache.spark.storage.StorageLevel
import org.apache.spark.storage.StorageLevel
scala> val lines = sc.textFile("hdfs:///user/raj/data.txt", 3)
lines: org.apache.spark.rdd.RDD[String] = hdfs:///user/raj/data.txt MapPartitionsRDD[1] at textFile at <console>:28
scala> // No of partitions
scala> lines.partitions.size
res0: Int = 3
scala> // flatMap() : One of many transformation
scala> val words = lines.flatMap(x => x.split(" "))
words: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[2] at flatMap at <console>:30
scala> // Persist the data
scala> val units = words.map ( word => (word, 1) ).
| persist(StorageLevel.MEMORY_ONLY)
units: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[3] at map at <console>:32
scala>
scala> val counts = units.reduceByKey ( (x, y) => x + y )
counts: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[4] at reduceByKey at <console>:34
// Text file is read to compute the 'counts' RDD
scala> counts.toDebugString
res1: String =
(3) ShuffledRDD[4] at reduceByKey at <console>:34 []
+-(3) MapPartitionsRDD[3] at map at <console>:32 []
| MapPartitionsRDD[2] at flatMap at <console>:30 []
| hdfs:///user/raj/data.txt MapPartitionsRDD[1] at textFile at <console>:28 []
| hdfs:///user/raj/data.txt HadoopRDD[0] at textFile at <console>:28 []
scala> // First Action
scala> counts.collect()
res2: Array[(String, Int)] = Array((another,1), (This,2), (is,2), (a,1), (test,2))
scala> val counts2 = units.reduceByKey((x, y) => x * y)
counts2: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[5] at reduceByKey at <console>:34
// Cache value is read to compute the 'counts2' RDD
scala> counts2.toDebugString
res3: String =
(3) ShuffledRDD[5] at reduceByKey at <console>:34 []
+-(3) MapPartitionsRDD[3] at map at <console>:32 []
| CachedPartitions: 3; MemorySize: 696.0 B; ExternalBlockStoreSize: 0.0 B; DiskSize: 0.0 B
| MapPartitionsRDD[2] at flatMap at <console>:30 []
| hdfs:///user/raj/data.txt MapPartitionsRDD[1] at textFile at <console>:28 []
| hdfs:///user/raj/data.txt HadoopRDD[0] at textFile at <console>:28 []
scala> // Second Action
scala> counts2.collect()
res4: Array[(String, Int)] = Array((another,1), (This,1), (is,1), (a,1), (test,1))
18 June 2016
A Word Count Example with Cached Partition
Broadcast Variable Example
scala> // Sending a value from Driver to Worker Nodes without scala> // using Broadcast variable scala> val input = sc.parallelize(List(1, 2, 3)) input: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[17] at parallelize at <console>:27 scala> val localVal = 2 localVal: Int = 2 scala> val added = input.map( x => x + localVal) added: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[18] at map at <console>:31 scala> added.foreach(println) 4 3 5 scala> //** Local variable is once again transferred to worked nodes scala> // for the next operation scala> val multiplied = input.map( x => x * 2) multiplied: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[19] at map at <console>:29 scala> multiplied.foreach(println) 4 6 2
scala> // Sending a read-only value using Broadcast variable scala> // Can be used to send large read-only values to all worker scala> // nodes efficiently scala> val broadcastVar = sc.broadcast(2) broadcastVar: org.apache.spark.broadcast.Broadcast[Int] = Broadcast(14) scala> val added = input.map(x => broadcastVar.value + x) added: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[20] at map at <console>:31 scala> added.foreach(println) 5 3 4 scala> val multiplied = input.map(x => broadcastVar.value * x) multiplied: org.apache.spark.rdd.RDD[Int] = MapPartitionsRDD[21] at map at <console>:31 scala> multiplied.foreach(println) 6 4 2 scala>
28 May 2016
A Word Count example using 'spark-shell'
[raj@Rajkumars-MacBook-Pro ~]$spark-shell --master local[*]
2016-05-28 15:37:24.325 java[3907:6309927] Unable to load realm info from SCDynamicStore
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.1
/_/
Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_45)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
SQL context available as sqlContext.
scala> val lines = sc.parallelize(List("This is a word", "This is another word"), 7)
lines: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[0] at parallelize at :27
scala> // No of partitions
scala> lines.partitions.size
res0: Int = 7
scala> // flatMap() : One of many transformation
scala> val words = lines.flatMap(line => line.split(" "))
words: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at flatMap at :29
scala> val units = words.map ( word => (word, 1) )
units: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[2] at map at :31
scala> val counts = units.reduceByKey ( (x, y) => x + y )
counts: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[3] at reduceByKey at :33
scala> counts.toDebugString
res1: String =
(7) ShuffledRDD[3] at reduceByKey at :33 []
+-(7) MapPartitionsRDD[2] at map at :31 []
| MapPartitionsRDD[1] at flatMap at :29 []
| ParallelCollectionRDD[0] at parallelize at :27 []
scala> // collect() : One of many actions
scala> counts.collect()
res2: Array[(String, Int)] = Array((This,2), (is,2), (another,1), (a,1), (word,2))
04 May 2016
Accumulator : Example
Note : Use Accumulator only in action to get correct values. Do not use Accumulator in Transformation ; Use it only for debugging purpose in Transformation
scala> val input = sc.parallelize(List(1, 2, 3, 4, 5,
| 6, 7, 8, 9, 10,
| 11, 12, 13, 14, 15
| ))
input: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:27
scala> println("No of partitions -> " + input.partitions.size)
No of partitions -> 8
scala> val myAccum = sc.accumulator(0, "My Accumulator")
myAccum: org.apache.spark.Accumulator[Int] = 0
scala> // Used inside an action
scala> input.foreach{ x =>
| //Thread.sleep(50000)
| myAccum += 1
| }
scala> println("myAccum -> " + myAccum.value)
myAccum -> 15
c15 > a1506 December 2015
Setting Partitioner for RDD
When doing Join on Pair RDDs, if one of the dataset we are using is a Master data, it makes a lot of sense to persist the data, as we do not want the RDD being created every time an action associated with the dataset is executed.
Apart from enabling persistence for the master dataset, we can also avoid the shuffling of the master dataset by providing a Partitioner
Example
Following are the operations, in which shuffling can be avoided/minimized by using a Partitioner
- cogroup()
- groupWith()
- join()
- leftOuterJoin()
- rightOuterJoin()
- groupByKey()
- reduceByKey()
- combineByKey()
- lookup()
Operations that preserves the Partitioner of the Parent RDD (or) result in Partitioner being set
- cogroup()
- groupWith()
- join()
- leftOuterJoin()
- rightOuterJoin()
- groupByKey()
- reduceByKey()
- combineByKey()
- partitionBy()
- sort()
- mapValues()
Operations that will not set a Partitioner
- map()
Learning Spark : Partitioning : 63, 66
Apart from enabling persistence for the master dataset, we can also avoid the shuffling of the master dataset by providing a Partitioner
Example
scala> val masterdata =
| sc.
| parallelize(Seq(
| ("math", 55),
| ("math", 56),
| ("english", 57),
| ("english", 58),
| ("science", 59),
| ("science", 54))).
| partitionBy(new HashPartitioner(100)).
| persist()
masterdata: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[42] at partitionBy at :31
scala> masterdata.partitioner
res16: Option[org.apache.spark.Partitioner] = Some(org.apache.spark.HashPartitioner@64)
scala> masterdata.partitions.length
res17: Int = 100
Following are the operations, in which shuffling can be avoided/minimized by using a Partitioner
- cogroup()
- groupWith()
- join()
- leftOuterJoin()
- rightOuterJoin()
- groupByKey()
- reduceByKey()
- combineByKey()
- lookup()
Operations that preserves the Partitioner of the Parent RDD (or) result in Partitioner being set
- cogroup()
- groupWith()
- join()
- leftOuterJoin()
- rightOuterJoin()
- groupByKey()
- reduceByKey()
- combineByKey()
- partitionBy()
- sort()
- mapValues()
Operations that will not set a Partitioner
- map()
Reference
Learning Spark : Partitioning : 63, 66
parallelize() Examples
scala> //for loop Example
scala> val rdd = sc.parallelize(for {
| x <- 1 to 3
| y <- 1 to 2
| } yield (x, None))
rdd: org.apache.spark.rdd.RDD[(Int, None.type)] = ParallelCollectionRDD[38] at parallelize at :21
scala>
scala> rdd.collect()
res15: Array[(Int, None.type)] = Array((1,None), (1,None), (2,None), (2,None), (3,None), (3,None))
Reference
for loop with yield : http://alvinalexander.com/scala/scala-for-loop-yield-examples-yield-tutorialActions on Pair RDDs : countByKey(), collectAsMap() & lookup()
Example
scala> val rdd = sc.parallelize(Seq(
| ("math", 55),
| ("math", 56),
| ("english", 57),
| ("english", 58),
| ("science", 59),
| ("science", 54)))
rdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[31] at parallelize at :21
scala> //Example : countByKey()
scala> val result1 = rdd.countByKey()
result1: scala.collection.Map[String,Long] = Map(math -> 2, english -> 2, science -> 2)
scala> //Example : collectAsMap()
scala> val reslt2 = rdd.collectAsMap()
reslt2: scala.collection.Map[String,Int] = Map(math -> 56, science -> 54, english -> 58)
scala> //Example : lookup()
scala> val result3 = rdd.lookup("math")
result3: Seq[Int] = WrappedArray(55, 56)
sortByKey() Example
sortByKey() is part of OrderedRDDFunctions that works on Key/Value pairs. The official documentation for OrderedRDDFunctions states that,
Note : sortByKey() results in range-partitioned RDDs
Learning Spark : Range Partition : 64
class OrderedRDDFunctions[K, V, P <: Product2[K, V]] extends Logging with Serializable Extra functions available on RDDs of (key, value) pairs where the key is sortable through an implicit conversion. They will work with any key type K that has an implicit Ordering[K] in scope. Ordering objects already exist for all of the standard primitive types. Users can also define their own orderings for custom types, or to override the default ordering. The implicit ordering that is in the closest scope will be used.Syntax : sortByKey()
def sortByKey(ascending: Boolean = true, numPartitions: Int = self.partitions.length): RDD[(K, V)] Sort the RDD by key, so that each partition contains a sorted range of the elements. Calling collect or save on the resulting RDD will return or output an ordered list of records (in the save case, they will be written to multiple part-X files in the filesystem, in order of the keys).Example
scala> //Input
scala> val rdd = sc.parallelize(Seq(
| ("math", 55),
| ("math", 56),
| ("english", 57),
| ("english", 58),
| ("science", 59),
| ("science", 54)))
rdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[17] at parallelize at :21
scala> rdd.collect()
res8: Array[(String, Int)] = Array((math,55), (math,56), (english,57), (english,58), (science,59), (science,54))
scala> //Default Sorting : Ascending order
scala> val sorted1 = rdd.sortByKey()
sorted1: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[20] at sortByKey at :23
scala> //Result
scala> sorted1.collect()
res9: Array[(String, Int)] = Array((english,57), (english,58), (math,55), (math,56), (science,59), (science,54))
scala> //Custom Sorting : Descending order (using implicit 'Ordering')
scala> {
| //Let us define an implicit sorting for the method sortByKey()
| //We have used '{' above to limit the scope of the implicit ordering
| implicit val sortIntegersByString = new Ordering[String] {
| override def compare(a: String, b: String) = {
| val result = a.compare(b)
| //We use -ve to sort the key in descending order
| -result
| }
| }
| val sorted2 = rdd.sortByKey()
|
| //Result
| sorted2.collect()
| }
res10: Array[(String, Int)] = Array((science,59), (science,54), (math,55), (math,56), (english,57), (english,58))
scala> //Default Sorting : Descending order (done using the 'ascending' flag argument)
scala> val sorted3 = rdd.sortByKey(false)
sorted3: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[26] at sortByKey at :23
scala> //Result
scala> sorted3.collect()
res11: Array[(String, Int)] = Array((science,59), (science,54), (math,55), (math,56), (english,57), (english,58))
Note : sortByKey() results in range-partitioned RDDs
Reference
Learning Spark : Range Partition : 64
join(), leftOuterJoin() & rightOuterJoin() Example
Example
scala> //Input Data
scala> val rdd1 = sc.parallelize(Seq(
| ("math", 55),
| ("math", 56),
| ("english", 57),
| ("english", 58),
| ("science", 59),
| ("science", 54)))
rdd1: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[45] at parallelize at :21
scala> val rdd2 = sc.parallelize(Seq(
| ("math", 60),
| ("math", 65),
| ("science", 61),
| ("science", 62),
| ("history", 63),
| ("history", 64)))
rdd2: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[46] at parallelize at :21
scala> //join() Example
scala> val joined = rdd1.join(rdd2)
joined: org.apache.spark.rdd.RDD[(String, (Int, Int))] = MapPartitionsRDD[49] at join at :25
scala> //Result
scala> joined.collect()
res15: Array[(String, (Int, Int))] = Array((math,(55,60)), (math,(55,65)), (math,(56,60)), (math,(56,65)), (science,(59,61)), (science,(59,62)), (science,(54,61)), (science,(54,62)))
scala> //leftOuterJoin() Example
scala> val leftJoined = rdd1.leftOuterJoin(rdd2)
leftJoined: org.apache.spark.rdd.RDD[(String, (Int, Option[Int]))] = MapPartitionsRDD[52] at leftOuterJoin at :25
scala> //Result
scala> leftJoined.collect()
res16: Array[(String, (Int, Option[Int]))] = Array((math,(55,Some(60))), (math,(55,Some(65))), (math,(56,Some(60))), (math,(56,Some(65))), (english,(57,None)), (english,(58,None)), (science,(59,Some(61))), (science,(59,Some(62))), (science,(54,Some(61))), (science,(54,Some(62))))
scala> //rightOuterJoin() Example
scala> val rightJoined = rdd1.rightOuterJoin(rdd2)
rightJoined: org.apache.spark.rdd.RDD[(String, (Option[Int], Int))] = MapPartitionsRDD[55] at rightOuterJoin at :25
scala> //Result
scala> rightJoined.collect()
res17: Array[(String, (Option[Int], Int))] = Array((math,(Some(55),60)), (math,(Some(55),65)), (math,(Some(56),60)), (math,(Some(56),65)), (history,(None,63)), (history,(None,64)), (science,(Some(59),61)), (science,(Some(59),62)), (science,(Some(54),61)), (science,(Some(54),62)))
cogroup() Example
Example
Multiple Pair RDDs can be combined using cogroup()
Multiple Pair RDDs can be combined using cogroup()
scala> val rdd1 = sc.parallelize(Seq(
| ("key1", 1),
| ("key2", 2),
| ("key1", 3)))
rdd1: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[25] at parallelize at :21
scala> val rdd2 = sc.parallelize(Seq(
| ("key1", 5),
| ("key2", 4)))
rdd2: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[26] at parallelize at :21
//cogroup() Example
scala> val grouped = rdd1.cogroup(rdd2)
grouped: org.apache.spark.rdd.RDD[(String, (Iterable[Int], Iterable[Int]))] = MapPartitionsRDD[28] at cogroup at :25
//Result
scala> grouped.collect()
res10: Array[(String, (Iterable[Int], Iterable[Int]))] = Array((key1,(CompactBuffer(1, 3),CompactBuffer(5))), (key2,(CompactBuffer(2),CompactBuffer(4))))
// Iterate through each value in key
// and increment the value by '1'
scala> val updated = grouped.map{x =>
| {
| val key = x._1
| //println("Key -> " + key)
| val value = x._2
| val itl1 = value._1
| val itl2 = value._2
| val res1 = itl1.map{ x =>
| {
| //println("It1 : Key -> " + key + ", Val -> " + (x + 1))
| x + 1
| }
| }
| val res2 = itl2.map{ x =>
| {
| //println("It2 : Key -> " + key + ", Val -> " + (x + 1))
| x + 1
| }
| }
| //println("End")
| (key, (res1, res2))
| }
| }
updated: org.apache.spark.rdd.RDD[(String, (Iterable[Int], Iterable[Int]))] = MapPartitionsRDD[18] at map at :33
scala> updated.collect()
res17: Array[(String, (Iterable[Int], Iterable[Int]))] = Array((key1,(List(2, 4),List(6))), (key2,(List(3),List(5))))
groupBy() & groupByKey() Example
groupByKey() operates on Pair RDDs and is used to group all the values related to a given key. groupBy() can be used in both unpaired & paired RDDs. When used with unpaired data, the key for groupBy() is decided by the function literal passed to the method
Example
Note
groupByKey() always results in Hash-Partitioned RDDs
Learning Spark : Hash-Partition : 64
Example
scala> val inputrdd = sc.parallelize(Seq(
| ("key1", 1),
| ("key2", 2),
| ("key1", 3)))
inputrdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[12] at parallelize at :21
//groupByKey() Example
scala> val grouped1 = inputrdd.groupByKey()
grouped1: org.apache.spark.rdd.RDD[(String, Iterable[Int])] = ShuffledRDD[13] at groupByKey at :23
scala> grouped1.collect()
res6: Array[(String, Iterable[Int])] = Array((key1,CompactBuffer(1, 3)), (key2,CompactBuffer(2)))
//groupBy() Example : Find Odd & Even numbers
scala> val grouped2 = inputrdd.groupBy{ x =>
| if((x._2 % 2) == 0) {
| "evennumbers"
| }else {
| "oddnumbers"
| }
| }
grouped2: org.apache.spark.rdd.RDD[(String, Iterable[(String, Int)])] = ShuffledRDD[15] at groupBy at :23
scala> grouped2.collect()
res7: Array[(String, Iterable[(String, Int)])] = Array((evennumbers,CompactBuffer((key2,2))), (oddnumbers,CompactBuffer((key1,1), (key1,3))))
Note
groupByKey() always results in Hash-Partitioned RDDs
Reference
Learning Spark : Hash-Partition : 64
05 December 2015
Setting up Partition size
Example
Learning Spark : Page 57
scala> val inputrdd = sc.parallelize(Seq(
| ("key1", 1),
| ("key2", 2),
| ("key1", 3)))
inputrdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[80] at parallelize at :21
scala> val noPartition = inputrdd.reduceByKey((x, y) => x + y)
noPartition: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[81] at reduceByKey at :23
scala> noPartition.partitions.length
res50: Int = 8
scala> //Here Partition size is given as a second argument
scala> val withPartition = inputrdd.reduceByKey((x, y) => x + y, 11)
withPartition: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[82] at reduceByKey at :23
scala> withPartition.partitions.length
res51: Int = 11
scala> val repartitioned = withPartition.repartition(16)
repartitioned: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[86] at repartition at :25
scala> repartitioned.partitions.length
res52: Int = 16
scala> val coalesced = if(4 < repartitioned.partitions.length) {
| //Note : Use coalesce() only when the new partition size is
| // less than the current partition size of the RDD
| repartitioned.coalesce(4)
| }else {
| repartitioned
| }
coalesced: org.apache.spark.rdd.RDD[(String, Int)] = CoalescedRDD[87] at coalesce at :30
scala> coalesced.partitions.length
res53: Int = 4
Reference
Learning Spark : Page 57
Combiner in Pair RDDs : combineByKey()
Similar to combiner in MapReduce, when working with key/value pairs, combineByKey() interface can be used to customize the combiner functionality. Methods like reduceByKey() by default use their own combiner to combine the data locally in each Partition, for a given key
Syntax
Here,
1st Argument : createCombiner is called when a key(in the RDD element) is found for the first time in a given Partition. This method creates an initial value for the accumulator for that key
2nd Argument : mergeValue is called when the key already has an accumulator
3rd Argument : mergeCombiners is called when more that one partition has accumulator for the same key
Example
Let us calculate the average in each subject using combineByKey()
Note
The map side aggregation done using combineByKey() can also be disabled(which is the case with methods like groupByKey() where the functionality of the combiner is not needed)
Learning Spark : Page 54
Similar to aggregate()(which is used with single element RDD), combineByKey() allows user to return different RDD element type compared to the element type of Input RDD
Syntax
def combineByKey[C](createCombiner: (V) ⇒ C, mergeValue: (C, V) ⇒ C, mergeCombiners: (C, C) ⇒ C, partitioner: Partitioner, mapSideCombine: Boolean = true, serializer: Serializer = null): RDD[(K, C)] Generic function to combine the elements for each key using a custom set of aggregation functions. Turns an RDD[(K, V)] into a result of type RDD[(K, C)], for a "combined type" C Note that V and C can be different -- for example, one might group an RDD of type (Int, Int) into an RDD of type (Int, Seq[Int]). Users provide three functions: - createCombiner, which turns a V into a C (e.g., creates a one-element list) - mergeValue, to merge a V into a C (e.g., adds it to the end of a list) - mergeCombiners, to combine two C's into a single one. In addition, users can control the partitioning of the output RDD, and whether to perform map-side aggregation (if a mapper can produce multiple items with the same key).
Here,
1st Argument : createCombiner is called when a key(in the RDD element) is found for the first time in a given Partition. This method creates an initial value for the accumulator for that key
2nd Argument : mergeValue is called when the key already has an accumulator
3rd Argument : mergeCombiners is called when more that one partition has accumulator for the same key
Example
Let us calculate the average in each subject using combineByKey()
scala> val inputrdd = sc.parallelize(Seq(
| ("maths", 50), ("maths", 60),
| ("english", 65),
| ("physics", 66), ("physics", 61), ("physics", 87)),
| 1)
inputrdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[41] at parallelize at <console>:27
scala> inputrdd.getNumPartitions
res55: Int = 1
scala> val reduced = inputrdd.combineByKey(
| (mark) => {
| println(s"Create combiner -> ${mark}")
| (mark, 1)
| },
| (acc: (Int, Int), v) => {
| println(s"""Merge value : (${acc._1} + ${v}, ${acc._2} + 1)""")
| (acc._1 + v, acc._2 + 1)
| },
| (acc1: (Int, Int), acc2: (Int, Int)) => {
| println(s"""Merge Combiner : (${acc1._1} + ${acc2._1}, ${acc1._2} + ${acc2._2})""")
| (acc1._1 + acc2._1, acc1._2 + acc2._2)
| }
| )
reduced: org.apache.spark.rdd.RDD[(String, (Int, Int))] = ShuffledRDD[42] at combineByKey at <console>:29
scala> reduced.collect()
Create combiner -> 50
Merge value : (50 + 60, 1 + 1)
Create combiner -> 65
Create combiner -> 66
Merge value : (66 + 61, 1 + 1)
Merge value : (127 + 87, 2 + 1)
res56: Array[(String, (Int, Int))] = Array((maths,(110,2)), (physics,(214,3)), (english,(65,1)))
scala> val result = reduced.mapValues(x => x._1 / x._2.toFloat)
result: org.apache.spark.rdd.RDD[(String, Float)] = MapPartitionsRDD[43] at mapValues at <console>:31
scala> result.collect()
res57: Array[(String, Float)] = Array((maths,55.0), (physics,71.333336), (english,65.0))
Note
The map side aggregation done using combineByKey() can also be disabled(which is the case with methods like groupByKey() where the functionality of the combiner is not needed)
Reference
Learning Spark : Page 54
mapValues() Example
When we use map() with a Pair RDD, we get access to both Key & value. There are times we might only be interested in accessing the value(& not key). In those case, we can use mapValues() instead of map().
Note
Operations like map() always cause the new RDD to no retain the parent partitioning information
In this example we use mapValues() along with reduceByKey() to calculate average for each subject
scala> val inputrdd = sc.parallelize(Seq(("maths", 50), ("maths", 60), ("english", 65)))
inputrdd: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[29] at parallelize at :21
scala> val mapped = inputrdd.mapValues(mark => (mark, 1));
mapped: org.apache.spark.rdd.RDD[(String, (Int, Int))] = MapPartitionsRDD[30] at mapValues at :23
scala> val reduced = mapped.reduceByKey((x, y) => (x._1 + y._1, x._2 + y._2))
reduced: org.apache.spark.rdd.RDD[(String, (Int, Int))] = ShuffledRDD[31] at reduceByKey at :25
scala> val average = reduced.map { x =>
| val temp = x._2
| val total = temp._1
| val count = temp._2
| (x._1, total / count)
| }
average: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[32] at map at :27
scala>
| average.collect()
res30: Array[(String, Int)] = Array((english,65), (maths,55))
Note
Operations like map() always cause the new RDD to no retain the parent partitioning information
Reference
Learning Spark : Partitioning : 64Pair RDD generation
Using parallelize() method
By creating Tuples using the parallelize() method, Pair RDD can be created
scala> val inputrdd = sc.parallelize(Seq(("key1", "val1"), ("key1", "val1")))
inputrdd: org.apache.spark.rdd.RDD[(String, String)] = ParallelCollectionRDD[3] at parallelize at :21
scala> inputrdd.reduceByKey((x, y) => x.concat(y)).collect()
res0: Array[(String, String)] = Array((key1,val1val1))
Note :
reduceByKey() is available in class PairRDDFunctions
reduceByKey() is available in class PairRDDFunctions
Using map() method
By using map() method to create key/value pairs, Pair RDD can be created
scala> val inputrdd = sc.parallelize(List("key1 1", "key1 3"))
inputrdd: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[15] at parallelize at :21
scala> val mapped = inputrdd.map { x =>
| val splitted = x.split(" ")
| (splitted(0), splitted(1).toInt)
| }
mapped: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[16] at map at :23
scala> val reduced = mapped.reduceByKey((x, y) => x + y)
reduced: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[17] at reduceByKey at :25
scala> reduced.collect()
res24: Array[(String, Int)] = Array((key1,4))
24 November 2015
countByValue() Example
countByValue() is an action that returns the Map of each unique value with its count
Syntax
Example
Learning Spark : 41
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD
Syntax
def countByValue()(implicit ord: Ordering[T] = null): Map[T, Long] Return the count of each unique value in this RDD as a local map of (value, count) pairs. Note that this method should only be used if the resulting map is expected to be small, as the whole thing is loaded into the driver's memory. To handle very large results, consider using rdd.map(x => (x, 1L)).reduceByKey(_ + _), which returns an RDD[T, Long] instead of a map.
Example
scala> val inputrdd = sc.parallelize{ Seq(10, 4, 3, 3) }
inputrdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[28] at parallelize at :47
scala> inputrdd.countByValue()
res34: scala.collection.Map[Int,Long] = Map(10 -> 1, 3 -> 2, 4 -> 1)
Reference
Learning Spark : 41
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD
takeSample() Example
takeSample() is an action that is used to return a fixed-size sample subset of an RDD
Syntax
Example
Learning Spark : 41
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD
Syntax
def takeSample(withReplacement: Boolean, num: Int, seed: Long = Utils.random.nextLong): Array[T] Return a fixed-size sampled subset of this RDD in an array withReplacement whether sampling is done with replacement num size of the returned sample seed seed for the random number generator returns sample of specified size in an array
Example
scala> val inputrdd = sc.parallelize{ Seq(10, 4, 5, 3, 11, 2, 6) }
inputrdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[22] at parallelize at :47
scala> inputrdd.takeSample(false, 3, System.nanoTime.toInt)
res29: Array[Int] = Array(6, 11, 10)
scala> inputrdd.takeSample(false, 3, System.nanoTime.toInt)
res30: Array[Int] = Array(5, 11, 4)
scala> inputrdd.takeSample(true, 3, System.nanoTime.toInt)
res31: Array[Int] = Array(10, 11, 5)
Reference
Learning Spark : 41
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD
top() & takeOrdered() Example
top() & takeOrdered() are actions that return the N elements based on the default ordering or the Customer ordering provided by us
Syntax
Example
In this example, let us return the top 5 elements based on ascending order
takeOrdered() does the opposite of top()
Learning Spark : 60
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD
Syntax
def top(num: Int)(implicit ord: Ordering[T]): Array[T] Returns the top k (largest) elements from this RDD as defined by the specified implicit Ordering[T]. This does the opposite of takeOrdered. For example: sc.parallelize(Seq(10, 4, 2, 12, 3)).top(1) // returns Array(12) sc.parallelize(Seq(2, 3, 4, 5, 6)).top(2) // returns Array(6, 5) num k, the number of top elements to return ord the implicit ordering for T returns an array of top elements
Example
In this example, let us return the top 5 elements based on ascending order
scala> val inputrdd = sc.parallelize{ Seq(10, 4, 5, 3, 11, 2, 6) }
inputrdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[20] at parallelize at :44
scala>
scala> implicit val sortIntegersByString = new Ordering[Int] {
| override def compare(a: Int, b: Int) = {
| //a.toString.compare(b.toString)
| if(a > b) {
| -1
| }else{
| +1
| }
| }
| }
sortIntegersByString: Ordering[Int] = $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$anon$1@4be552af
scala> inputrdd.top(5)
res28: Array[Int] = Array(2, 3, 4, 5, 6)
takeOrdered() does the opposite of top()
Reference
Learning Spark : 60
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.RDD
foreach() Example
foreach() is an action. Unlike other actions, foreach do not return any value. It simply operates on all the elements in the RDD. foreach() can be used in situations, where we do not want to return any result, but want to initiate a computation. A good example is ; inserting elements in RDD into database. Let us look at an example for foreach()
scala> val testData=Array(1,2,3) testData: Array[Int] = Array(1, 2, 3) scala> val inputrdd = sc.parallelize(testData) inputrdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[2] at parallelize at:23 scala> scala> inputrdd.foreach{ x => { | println(x) | }} 3 1 2 scala> scala> //Checkout this ; out addition has no scala> //impact on the actual elements in RDD scala> inputrdd.collect() res6: Array[Int] = Array(1, 2, 3)
Reference
Learning spark : Page 41
Subscribe to:
Posts (Atom)