Apache Spark - Try it using docker!


Apache Spark is an opensource tool for data analysis and transformation. It uses RDD (resilient distributed dataset) a read only dataset distributed over a cluster of machine that is mantained is a fault tolerant way.

It obtain high performance for batch and streaming data. It uses:

  • DAG scheduler
  • Query Optimizer
  • A Physical Execution Engine.


Docker is a tool to create an isolated system into a host computer

I use docker every day to create and simulate more scenario. Today i will explain how to create a master and worker apache spark node.

docker-compose is a tool to create an environment consist of a set of container

docker-spark is big-data-europe's github repository. We can create an environment using docker-compose.yaml file:

version: "3.3"

    image: bde2020/spark-master:2.4.0-hadoop2.7
    container_name: spark-master
      - "8080:8080"
      - "7077:7077"
      - INIT_DAEMON_STEP=setup_spark
    image: bde2020/spark-worker:2.4.0-hadoop2.7
    container_name: spark-worker-1
      - spark-master
      - "8081:8080"
      - "SPARK_MASTER=spark://spark-master:7077"
    image: bde2020/spark-worker:2.4.0-hadoop2.7
    container_name: spark-worker-2
      - spark-master
      - "8082:8080"
      - "SPARK_MASTER=spark://spark-master:7077"

Now, we can create a cluster into our machine using the command:

Now my spark cluster is running!


Apache Spark runs on Master-Slave architecture, into our cluster there are:
  1. one master node
    • this node exposes WebUI (see below) and manage the jobs into workers
  2. two workers


Now we are able to send a new job to the spark cluster or open a spark-shell linked to our cluster (see below):

It is so easy use spark with docker but.....


java 8 version is not able to retrieve right resource informations, read this post to fix it (external resource)


This guide talks about a local cluster uses for test or develop purpose. But, i am sure, you are asking: "Can i use this configurations into real production cluster?"

Docker has a cluster manager called swarm to link different machine. It is stable and easy to use!!!

YES: docker swarm and apache spark can be used into a real production cluster!

Good bye and have fun with Apache Spark!


  1. Nice and good article. It is very useful for me to learn and understand easily. Thanks for sharing your valuable information and time. Please keep updating big data online training

  2. I like your post very much. It is very much useful for my research. I hope you to share more info about this. Keep posting Spark Online Training India

  3. Thank you so much for this nice information. Hope so many people will get aware of this and useful as well. And please keep update like this.

    Big Data Services

    Data Lake Services

    Advanced Analytics Solutions

    Full Stack Development Services

  4. Thanks for sharing your innovative ideas to our vision. I have read your blog and I gathered some new information through your blog. Your blog is really very informative and unique. Keep posting like this. Awaiting for your further update.If you are looking for any apache spark scala related information, please visit our website Apache spark training institute in Bangalore

  5. Thanks for sharing useful information. I learned something new from your bog. Its very interesting and informative. keep updating. If you are looking for any apache spark scala related information, please visit our website Apache spark training institute in Bangalore

  6. The information which you have provided in this blog is really useful to everyone. Thanks for sharing.
    Docker and Kubernetes Training
    Docker and Kubernetes Online Training
    Kubernetes Online Training
    Docker Online Training

  7. Wow Very Nice Post I really like This Post. Please share more post.
    Apache Spark Training Institutes in Pune

  8. Very nice post.thank you so much for sharing your knowledge with us.

    keep updating more Concepts.

    big data hadoop course


Posta un commento

Post popolari in questo blog

Hadoop, how to create a single node cluster using docker

How to install IOTA node with docker