Running Apache Superset in a Docker
As a response to questions from a previous article, this author explores how to set up Apache Superset as a Docker image.
Join the DZone community and get the full member experience.Join For Free
A couple of days back, I wrote the post about how to run Apache Superset in the production environment for serving hundreds or thousands of users. Superset community members and users appreciated the post for which I am thankful; however, over the Superset Slack and Gitter channels, many users asked various questions on setting Superset as a Docker container and how to use/run it. In this post, I am trying to explore more about the Docker image of a Superset, and I am hoping that after reading the post you will acquire a conceptual understanding of setting Superset as a Docker container and benefits of it.
First, let’s quickly understand what exactly terms "container" and "image" mean and how they are related to Docker.
As per Wikipedia, any structure which holds product for storage, packaging, and shipping is a container. Same applies for the container in a software world.
A container is a standard unit of software that packages up the code and all its dependencies, so the application runs quickly and reliably from one computing environment to another.
Now, let’s look what a term image means.
A container image is a lightweight, standalone, executable package of software that includes everything needed to run an application: code, run-time, system tools, system libraries, and settings.
Finally, relationship with Docker.
Container images become containers at runtime and in the case of Docker containers — images become containers when they run on Docker Engine. Available for both Linux and Windows-based applications, containerized software will always run the same, regardless of the infrastructure.
There are many other container runtime environments, but Docker among them is the most popular one.
Back to Superset Docker Image
There are multiple active repositories and images of Superset available over GitHub and DockerHub. Below is a list of some of them.
Why so many repositories? Are they different? Aren’t they suppose to be the same and provide the same functionality, i.e., packaging the Superset and its dependencies? Yes, they should be identical, but there are multiple different ways and modes to start the Superset. An image should be generic for handling all method and commands which is not the case, and that’s why there are multiple repositories.
I started working on Superset with the perspective of running it in a completely distributed manner so that hundreds or thousands of users can access the Superset concurrently. In the beginning, I was exploring the Apache Superset code but realized that several changes are required to run Superset multiple containers for a distributed architecture and that’s why I decided to have a separate repository.
Features of the Docker image of Superset
- There are multiple ways to start the container, either by using the command
docker-composeor by using the
- Superset all components, i.e., web application, celery worker, celery flower UI can run in the same container or different containers.
- All database plugins and packages are installed by default.
- Container first runs sets required Superset metadata database along with sample data and the Fab-manager user account with credentials
username: admin & password: admin
- Apart from the packaged Superset config file in a container image, a custom config file like superset_config.py can be mounted to the container. There's need to rebuild the image for changing configurations.
- The default configuration uses MySQL as a Superset metadata database and Redis as a cache and celery broker which can be easily replaced.
Starting the container using the command
docker-compose will start three containers.
mysql5.7 as the metadata database,
redis3.4 as a cache and celery broker, and Superset container.
- Expect multiple environment variables defined in the docker-compose.yml file. Default environment variables are present in the file
- Default environment variables can be overridden either by editing a
.envfile or passing through commands using environments
- Permissible value of
SUPERSET_ENVcan be either
- In mode
localone celery worker and Superset flask-based superset web application run.
- In mode
prodtwo celery workers and Gunicorn-based Superset web application run.
Starting the container by using command
docker run can be a used for a complete distributed setup, requires metadata database & Redis URL for starting the container.
- Single or multiple server(using load balancer) containers can be spawned. In the server, Gunicorn based superset web application runs.
- Multiple celery workers container running on same or different machines. In worker, celery worker & flower UI runs.
How to Run
First, copy superset_config.py, docker-compose.yml, and .env files in your execution environment. Please follow the directory structure like below.
docker-superset |__config | |__superset_config.py | |__docker-files | |__docker-compose.yml | |__.env
- Running a container using the command
Starting a Superset image as a
superset container in a local mode:
cd docker-superset/docker-files/ && docker-compose up -d
Starting a Superset image as a
superset container in a prod mode:
cd docker-superset/docker-files/ && SUPERSET_ENV=prod SUPERSET_VERSION=<version-tag> docker-compose up -d
Running a container using the command
Starting a superset image as a
cd docker-superset && docker run -p 8088:8088 -v config:/home/superset/config/ abhioncbr/docker-superset:<version-tag> cluster server <superset_metadata_db_url> <redis_url>
Starting a superset image as a
cd docker-superset && docker run -p 5555:5555 -v config:/home/superset/config/ abhioncbr/docker-superset:<version-tag> cluster worker <superset_metadata_db_url> <redis_url>
Note: There is no need to build an image if you are not making changes in the image. You can pull an image from DockerHub using the below command which can be any superset-version or one with tag "latest".
docker pull abhioncbr/docker-superset:<version-tag>
Extending Superset Docker Image
- No changes are required for adding new environment variables. For example, adding a BigQuery connection with Superset which
GOOGLE_APPLICATION_CREDENTIALScan be easily provided through a
docker-compose.ymlfile or passing through command.
- Also, changes done in the file
superset_config.pyare easily reflectable into the container by mounting the config file into the container.
- For any further changes or bug, please contact me or contribute to the repository.
Happy Superset Exploration!!!
Opinions expressed by DZone contributors are their own.
Boosting Application Performance With MicroStream and Redis Integration
Integration Architecture Guiding Principles, A Reference
Incident Response Guide
Database Integration Tests With Spring Boot and Testcontainers