01 / Objective & overall result
Trace how data moves when a distributed processing job is launched.
The lab combines Hadoop and Spark with Docker to follow a file end to end: HDFS storage, YARN orchestration, MapReduce execution, then the equivalent transformation with PySpark.
The NameNode sees both storage nodes and no under-replicated blocks during validation.
WordCount and Average Word Length both appear as FINISHED / SUCCEEDED in YARN.
PySpark returns exactly the same counts as Java: hadoop 1, hello 2, world 1.
What the lab proves: not a performance benchmark, but an understanding and validation of two different distributed execution paths in a reproducible environment.
02 / Architecture
Two subsystems in the same Docker environment.
Store and execute through HDFS / YARN
Files are stored in HDFS. Java MapReduce jobs are tracked by YARN.
Submit a job to the master, then the worker
The PySpark WordCount is submitted to the Spark master and reads the file from the shared volume.
With the services running, the first check is whether HDFS sees both data nodes.
03 / HDFS storage
The NameNode sees two DataNodes in service.
HDFS separates metadata from physical storage. The NameNode tracks where blocks live, while DataNodes store the data. The lab adds a second DataNode so this separation is visible rather than using a single storage container.
Once storage is available, the first workload can follow the classic Hadoop path: HDFS, MapReduce, then HDFS output.
04 / Java MapReduce
A simple WordCount to follow every stage of the Hadoop engine.
The control file contains hello hadoop hello world. The mapper turns each word into a (word, 1) pair. Hadoop then groups identical keys during the shuffle, and the reducer sums the values for each word.
Hadoop path
hello appears twice, while hadoop and world appear once.That result becomes a reference point: the same computation is rewritten in PySpark to verify that the logic stays the same with another execution API.
05 / PySpark
Same transformation, different execution engine.
The PySpark job reads the same file from the shared volume. flatMap splits lines into words, map pairs each word with 1 and reduceByKey sums values by key. The Spark Master delegates execution to the registered worker.
The business logic is identical, but the technical path changes. The Hadoop job uses HDFS and YARN with Java MapReduce. The Spark job is submitted to the standalone master and uses its worker. This is not a speed benchmark between the two engines.
After this functional check, a second MapReduce job tests a numeric aggregation and exposes a more concrete type-contract issue.
06 / Real debugging
The second job had a type contract mismatch between the declared output and the value actually produced.
Expected result
An average word length must be decimal.
Problem
The reducer writes a DoubleWritable, while the driver still declared an IntWritable.
Fix
Align the job output contract with the actual reducer type, then rebuild the JAR.
Validation
The job succeeds in YARN and returns Average Word Length = 5.333333333333333.
07 / What the evidence confirms
HDFS, MapReduce and Spark validated in a containerized lab.
The important point is consistency across the evidence: the UIs show cluster state, terminal outputs show results, and the two WordCount implementations return the same content. The lab therefore validates a complete path from HDFS storage to YARN orchestration and MapReduce/Spark execution, without claiming that one engine is faster.
08 / Limits
A containerized learning lab on a single machine.
- All containers run on one physical machine: if the host stops, the whole lab stops.
- Only one NodeManager and one Spark Worker are used; the project demonstrates orchestration and processing paths, not scale-out performance.
- No controlled benchmark supports a claim that Spark is faster than MapReduce in this project.
- The lab uses Hadoop 3.2.1 and Spark 2.4.0 images, without high availability, Kerberos or production monitoring.