Description
Problem 1
Data: /home/public/google
Description: The data for this problem comes from Google’s project for counting word frequencies in its
entire Google Books collection. You are given two files: one file reports 1 grams (single words) and
another file reports 2 grams (pairs of words; one following each other in text). The data represents the
number of occurrences of a particular word (or pairs of words) in a given year across all books available
by Google Books. The number of volumes/books containing a word (or pairs of words) is also reported.
Write a MapReduce program that reports the average number of volumes per year for words containing
the following three substrings: ‘nu,’ ‘chi,’ ‘haw’.
Example:
The 1-gram file format –the regex “\\s+” will match any kind of whitespace (space, tab etc):
word \\s+ year \\s+ number of occurrences \\s+ number of volumes \\s+ …
The 2-gram (bigram) file format:
word \\s+ word \\s+ year \\s+ number of occurrences \\s+ number of volumes \\s+…
The final output should show the year, substring, and average number of volumes across both bigram
and unigram formats where the substring appears in that year. For example:
2000,nu,345
2010,nu,200
1998,chi,31
The ‘year’ column may include erroneous values which can be a string. If the year field is a string, the
record should be discarded.
If each word in the bigram includes the string, it should be counted twice in the average.
For example, for the bigram “nugi hinunu” with volume count of 10, when calculating the average, its
contribution to the numerator should be 2 times 10 and in the denominator, it should be 2. A unigram
counts only once regardless of the number of occurrences of “nu” in the word.
Do this in the most efficient way with a single MapReduce job. Beyond the file formats described above,
you are not allowed to make any structural assumptions on the data; e.g., that the 2 gram file contains
more fields compared to the 1 gram file.
Problem 2
Data: /home/public/google
Description: The data set is the same as in the previous exercise. You need to compute the standard
deviation of all volume values (across all records and both files) in MapReduce. The output should be a
single value. You must not use more than 2 MapReduce jobs.
Problem 3
Data: /home/public/music
Description: The data for this problem is a subset of the million-song database
(https://www.kaggle.com/c/msdchallenge#description), and its size is 42GB. The file is in csv format.
Your task is to extract the song title (column 1), artist’s name (column 2) and duration (column 3) for all
songs published between the years 2000 and 2010. The year is in column 166 – note that some year
entries can be erroneous and should be discarded. Additionally, you can assume that all songs are
unique, so there is no need to remove any duplicates. You should do this as efficiently as possible in
MapReduce.
Problem 4
Data: /home/public/music
Description: The data set is the same as in the previous exercise. For each artist in the data set, compute
the maximum duration across all of their songs. The output should be: artist, max duration.
In addition, the management of your firm wants the artists’ names to be sorted across all files based on
the first character. This means that each output file of your MapReduce job must be sorted by the first
character of an artist’s name. You cannot take the output files and then sort them in a spreadsheet
software. You are only allowed to concatenate the output files in the end.
In order to save computing resources for your firm, you have to use 5 reducers.
