It’s much easier to iterate your platform on containers before deciding to use more “traditional” computing systems in Staging or Production. It’s not a requirement to use Docker or Kubernetes, but even if the system is using containers all the way up to Production, many of the DevOps cycles can be done more quickly because of how quickly environments can be refreshed.
Docker / Kubernetes
These distributions are also distributed as Docker containers which make them easy to use and build prototypes and then scale in production using Kubernetes. What’s Kubernetes? It’s now become the platform of choice to distribute stateless apps (API, UI, Queue Processors) and recently stateful apps (Databases, indices, Queues) as containers. Since containers are running on the host operating system, it’s a very efficient use of hardware whether it’s physical or virtual.
Proven Open source technology
Used by everyone supported
Option to run in physical, virtual, or inside other containers.
Allows for local testing to global deployment using the same code.
Docker, like other containerization engines, is now generally compliant with the OCI (Open Container Initiative. As long as they are OCI compliant, images can be run on Docker Swarm, Kubernetes, or DCOS/Mesos. Kubernetes just happens to be the most widely used and has the largest community. A Kubernetes overview by The New stack gives an excellent introduction if you are curious.
Proven Open source technology
Support for Docker Containers
Option to use physical, virtual, containers on hybrid or multi cloud.
Allows uniform platform across clouds.
These are technologies that make STACK easier.
Setup – Containers are easy to orchestrate with compose and more manageable on large platforms with Kubernetes
Training – Containers can be used to bring up the whole platform in one computer such as a person’s laptop.
Administration – Containers are immutable , even if the data is stored elsewhere. It makes updates and upgrades seamless through clean deployments.
Configuration – Since all infrastructure is managed as code that orchestrates containers , it allows developers to try out configurations very easily.
Knowledge Management – Since you can inherit certain features from parent containers, you can maintain a lot of knowledge of the whole infrastructure in code. Someone who can read a Compose YAML, Kubernetes service YAML, or read a HELM Chart knows exactly how the system is being built and brought up in all environments.
Deployment can be done on anything that can run an operating system. Using these components that are both open source and commercially supported, we have options to deploy it on any system that we need.
Here’s an architecture that may be used to build a really rock solid deployment that provides a general resiliency in the event of a global spike in traffic as well as more specific seasonal traffic.
We’ve talked about the components that can power a global data & analytics platform, in the next part of this series we will design an example that could very well be your next platform design. We’ll even account for a hybrid / multi-cloud design for some of the components.
If you want me or our company, to come and talk to your company about your Global Data & Analytics Platform, feel free to email me or my team at Anant.