One of the reasons I fell in love with Git several years ago was because of how simple and intuitive the command line tool was. Using command-line tools is almost always a productivity boost, and using Git on the command line is no exception. However, I found myself firing up SourceTree (a Git GUI application) on a daily basis. Why?
Atomic commits.
What are Atomic Commits?
One of the best practices in version control, no matter what tool you use, is to keep your commits atomic.
What does that mean?
There are a few definitions for atomic in the dictionary, but this one is the most relevant:
atomic: of or forming a single irreducible unit or component in a larger system
When making code changes, you want to make commits that are generally smaller and that encompass only one irreducible feature, fix, or improvement. I recently created a simple contact form for a small site, and here is a truncated example of what my commits looked like.
Create HTML for form
Style form
Add HTML5 validation
Fix unrelated JS bug
Add ajax submit to form with mock server results from PHP
Add JS validation to form
Send form results via email
Log form results to database
Style form validation and success/error messages
Each of these commits adds either a bare-minimum feature, a fix, or a small improvement to the feature or code. The semantic commit message style we use at Sparkbox requires our commits to be atomic.
If you are publishing a change log from your commit history, as Rob Tarr recommends, you may want to rewrite your local history before you push, to combine smaller feature commits into a single one like “Create working contact form.”
Why Atomic Commits?
Code Reviews are Easier
When you keep your commits small like this, it is easier for someone to review your code and see each incremental change.
Easier to Roll Back
Let’s say you make a feature change and mix in an unrelated bug fix. Later, someone decides that feature change isn’t all that great, and they want to roll it back. If your feature change commit included only that feature, rolling back would be simply a matter of reversing the commit, but if it were mixed with a bug fix, rolling back would be more difficult (especially if someone other than you has to do it).
Committing Parts of a Changed File
One of the problems I run into trying to keep my commits clean and atomic is that I inevitably make changes to my code that are unrelated to my current task. I see a method from yesterday that I forgot to document, and instead of writing it down as a todo and making the changes later, it’s easier to just make the change now.
When I get ready to commit my changes for the current task, I want to make sure I don’t mix the new documentation into that commit. If the documentation change is in another file, that’s easy. With Git, you stage only the files you want to commit first.
$ git st
M contact.html
M contact.php
$ git add contact.html
$ git st
A contact.html
M contact.php
$ git ci -m "Add required attribute to form fields"
But what if you have changes in the same file that you need to commit separately? I used to fire up SourceTree, which would allow me to selectively stage “hunks” of code. However, I recently went cold-turkey and uninstalled SourceTree to force myself to only use the command line.
From the command line, you can selectively stage “hunks” of code that are in the same file using git add --patch or git add -p. When you use the patch argument for staging, Git will display each hunk of changes in a file and ask you whether you want to add it or not. Simply choose yes or no (or some of the other advanced options), and then commit your staged hunks when you are done.
Here is a CoffeeScript file with a documentation change and an unrelated naming change. With git status, we see just one file has changed.
$ git status
M coffee/forms.coffee
To stage hunks of code in a file instead of the whole file, use the patch parameter with your git add command. It will show you the first changed hunk in the file and ask you if you want to stage it or not. We’ll choose “?” so we can see the full list of options, and then we’ll choose “n” because we DON’T want to stage this hunk.
$ git add -p
diff --git a/coffee/forms.coffee b/coffee/forms.coffee
index a9f4d6d..40a8ed8 100644
--- a/coffee/forms.coffee
+++ b/coffee/forms.coffee
@@ -15,6 +15,7 @@ window.xhrForms =
$this = $(@)
$this.mask $this.data("mask")
+ # Submit handler for our ajax forms
handleSubmit: (form) ->
$form = $(form)
url = $form.attr "action"
Stage this hunk [y,n,a,d,/,j,J,g,e,?]? ?
y - stage this hunk
n - do not stage this hunk
a - stage this and all the remaining hunks in the file
d - do not stage this hunk nor any of the remaining hunks in the file
g - select a hunk to go to
/ - search for a hunk matching the given regex
j - leave this hunk undecided, see next undecided hunk
J - leave this hunk undecided, see next hunk
k - leave this hunk undecided, see previous undecided hunk
K - leave this hunk undecided, see previous hunk
s - split the current hunk into smaller hunks
e - manually edit the current hunk
? - print help
Stage this hunk [y,n,q,a,d,/,j,J,g,e,?]? n
Then, it presents the next hunk. This time, we’ll choose yes.
@@ -40,7 +41,7 @@ window.xhrForms =
$msg = $(".form-notice", $form)
if !$msg.length
$msg = $("<div/>").addClass("form-notice").prependTo($form)
- $msg.removeClass "form-success"
+ $msg.removeClass "form-success form-error"
if data.status
$msg.addClass "form-success"
else
Stage this hunk [y,n,q,a,d,/,K,g,e,?]? y
If we check the status again, we’ll see that our file is partially staged.
$ git st
MM coffee/forms.coffee
Finally, we can can commit the code changes we just staged.
$ git commit -m "Remove form-error class too"
No Excuses
Sometimes interactive staging can be a bit more complicated, but in general, atomic commits with Git are pretty easy. There is no excuse to continue committing your bug fixes with your features. By keeping your commits atomic, you will add some sanity to your workflow, and your fellow developers will thank you.
ap·prise / verb To inform or tell (someone). To make one aware of something.
Apprise allows you to send a notification to almost all of the most popular notification services available to us today such as: Telegram, Discord, Slack, Amazon SNS, Gotify, etc.
One notification library to rule them all.
A common and intuitive notification syntax.
Supports the handling of images and attachments (to the notification services that will accept them).
It's incredibly lightweight.
Amazing response times because all messages sent asynchronously.
Developers who wish to provide a notification service no longer need to research each and every one out there. They no longer need to try to adapt to the new ones that comeout thereafter. They just need to include this one library and then they can immediately gain access to almost all of the notifications services available to us today.
System Administrators and DevOps who wish to send a notification now no longer need to find the right tool for the job. Everything is already wrapped and supported within the apprise command line tool (CLI) that ships with this product.
The table below identifies the services this tool supports and some example service urls you need to use in order to take advantage of it. Click on any of the services listed below to get more details on how you can configure Apprise to access them.
growl://hostname growl://hostname:portno growl://password@hostname growl://password@hostname:port Note: you can also use the get parameter version which can allow the growl request to behave using the older v1.x protocol. An example would look like: growl://hostname?version=1
Apprise have some email services built right into it (such as yahoo, fastmail, hotmail, gmail, etc) that greatly simplify the mailto:// service. See more details here.
The easiest way is to install this package is from pypi:
pip install apprise
Command Line
A small command line tool is also provided with this package called apprise. If you know the server url's you wish to notify, you can simply provide them all on the command line and send your notifications that way:
# Send a notification to as many servers as you want
# as you can easily chain one after another (the -vv provides some
# additional verbosity to help let you know what is going on):
apprise -vv -t 'my title' -b 'my notification body' \
'mailto://myemail:mypass@gmail.com' \
'pbul://o.gn5kj6nfhv736I7jC3cj3QLRiyhgl98b'
# If you don't specify a --body (-b) then stdin is used allowing
# you to use the tool as part of your every day administration:
cat /proc/cpuinfo | apprise -vv -t 'cpu info' \
'mailto://myemail:mypass@gmail.com'
# The title field is totally optional
uptime | apprise -vv \
'discord:///4174216298/JHMHI8qBe7bk2ZwO5U711o3dV_js'
Configuration Files
No one wants to put their credentials out for everyone to see on the command line. No problem apprise also supports configuration files. It can handle both a specific YAML format or a very simple TEXT format. You can also pull these configuration files via an HTTP query too! You can read more about the expected structure of the configuration files here.
# By default if no url or configuration is specified aprise will attempt to load
# configuration files (if present):
# ~/.apprise
# ~/.apprise.yml
# ~/.config/apprise
# ~/.config/apprise.yml
# Windows users can store their default configuration files here:
# %APPDATA%/Apprise/apprise
# %APPDATA%/Apprise/apprise.yml
# %LOCALAPPDATA%/Apprise/apprise
# %LOCALAPPDATA%/Apprise/apprise.yml
# If you loaded one of those files, your command line gets really easy:
apprise -vv -t 'my title' -b 'my notification body'
# If you want to deviate from the default paths or specify more than one,
# just specify them using the --config switch:
apprise -vv -t 'my title' -b 'my notification body' \
--config=/path/to/my/config.yml
# Got lots of configuration locations? No problem, you can specify them all:
# Apprise can even fetch the configuration from over a network!
apprise -vv -t 'my title' -b 'my notification body' \
--config=/path/to/my/config.yml \
--config=https://localhost/my/apprise/config
Attaching Files
Apprise also supports file attachments too! Specify as many attachments to a notification as you want.
# Send a funny image you found on the internet to a colleague:
apprise -vv --title 'Agile Joke' \
--body 'Did you see this one yet?' \
--attach https://i.redd.it/my2t4d2fx0u31.jpg \
'mailto://myemail:mypass@gmail.com'
# Easily send an update from a critical server to your dev team
apprise -vv --title 'system crash' \
--body 'I do not think Jim fixed the bug; see attached...' \
--attach /var/log/myprogram.log \
--attach /var/debug/core.2345 \
--tag devteam
Developers
To send a notification from within your python application, just do the following:
import apprise
# Create an Apprise instance
apobj = apprise.Apprise()
# Add all of the notification services by their server url.
# A sample email notification:
apobj.add('mailto://myuserid:mypass@gmail.com')
# A sample pushbullet notification
apobj.add('pbul://o.gn5kj6nfhv736I7jC3cj3QLRiyhgl98b')
# Then notify these services any time you desire. The below would
# notify all of the services loaded into our Apprise object.
apobj.notify(
body='what a great notification service!',
title='my notification title',
)
Configuration Files
Developers need access to configuration files too. The good news is their use just involves declaring another object (called AppriseConfig) that the Apprise object can ingest. You can also freely mix and match config and notification entries as often as you wish! You can read more about the expected structure of the configuration files here.
import apprise
# Create an Apprise instance
apobj = apprise.Apprise()
# Create an Config instance
config = apprise.AppriseConfig()
# Add a configuration source:
config.add('/path/to/my/config.yml')
# Add another...
config.add('https://myserver:8080/path/to/config')
# Make sure to add our config into our apprise object
apobj.add(config)
# You can mix and match; add an entry directly if you want too
# In this entry we associate the 'admin' tag with our notification
apobj.add('mailto://myuser:mypass@hotmail.com', tag='admin')
# Then notify these services any time you desire. The below would
# notify all of the services that have not been bound to any specific
# tag.
apobj.notify(
body='what a great notification service!',
title='my notification title',
)
# Tagging allows you to specifically target only specific notification
# services you've loaded:
apobj.notify(
body='send a notification to our admin group',
title='Attention Admins',
# notify any services tagged with the 'admin' tag
tag='admin',
)
# If you want to notify absolutely everything (reguardless of whether
# it's been tagged or not), just use the reserved tag of 'all':
apobj.notify(
body='send a notification to our admin group',
title='Attention Admins',
# notify absolutely everything loaded, reguardless on wether
# it has a tag associated with it or not:
tag='all',
)
Attaching Files
Attachments are very easy to send using the Apprise API:
import apprise
# Create an Apprise instance
apobj = apprise.Apprise()
# Add at least one service you want to notify
apobj.add('mailto://myuser:mypass@hotmail.com')
# Then send your attachment.
apobj.notify(
title='A great photo of our family',
body='The flash caused Jane to close her eyes! hah! :)',
attach='/local/path/to/my/DSC_003.jpg',
)
# Send a web based attachment too! In the below example, we connect to a home
# security camera and send a live image to an email. By default remote web
# content is cached but for a security camera, we might want to call notify
# again later in our code so we want our last image retrieved to expire(in
# this case after 3 seconds).
apobj.notify(
title='Latest security image',
attach='http:/admin:password@hikvision-cam01/ISAPI/Streaming/channels/101/picture?cache=3'
)
To send more than one attachment, just use a list, set, or tuple instead:
import apprise
# Create an Apprise instance
apobj = apprise.Apprise()
# Add at least one service you want to notify
apobj.add('mailto://myuser:mypass@hotmail.com')
# Now add all of the entries we're intrested in:
attach = (
# ?name= allows us to rename the actual jpeg as found on the site
# to be another name when sent to our receipient(s)
'https://i.redd.it/my2t4d2fx0u31.jpg?name=FlyingToMars.jpg',
# Now add another:
'/path/to/funny/joke.gif',
)
# Send your multiple attachments with a single notify call:
apobj.notify(
title='Some good jokes.',
body='Hey guys, check out these!',
attach=attach,
)
Want To Learn More?
If you're interested in reading more about this and other methods on how to customize your own notifications, please check out the following links:
This article describes a way to periodically move on-premise Cassandra data to S3 for analysis. We’re been using this approach successfully over the last few months in order to get the best of both worlds for an early-stage platform such as 1200.aero: The cost effectiveness of on-premise hosting for a stable, live workload, and the on-demand scalability of AWS for data analysis and machine learning.
Our end goal was to upload data to S3 organized as follows:
/adsb/messages/year=2018/month=11/day=06
To that effect, we performed the following steps, described in detail in this article:
Reorganize the data using Cassandra materialized views
Use Spark to read Cassandra data efficiently as a time series
Partition the Spark dataset as a time series
Save the dataset to S3 as Parquet
Analyze the data in AWS
For your reference, we used Cassandra 3.11 and Spark 2.3.1, both straight open source versions.
The data we want to analyze consists of millions of low-altitude air traffic messages captured from ADS-B receivers at a number of regional airports. The original table definition is:
CREATE TABLE messages ( icao text, gentime timestamp, alert boolean, alt int, callsign text, emerg boolean, gndspd int, ident boolean, isgnd boolean, lat float, lon float, sqwk text, station text, trk int, vspd int, PRIMARY KEY (icao, gentime) ) WITH CLUSTERING ORDER BY (gentime ASC)
Notice the primary key is (icao, gentime). Cassandra data models are designed with a specific data retrieval use case in mind. Our applications query data by aircraft identifier, so our partitioning column is the aircraft’s icao (a 6-character hex transponder code), followed by gentime (a timestamp) as the clustering column that orders records chronologically within each partition.
Problem: In order to periodically export a day/week/month’s worth of data, we needed to be able to retrieve records by date. Since such attribute did not originally exist, we had to:
Create and populate a new day column on the existing table.
Define the right Cassandra data structure to enable efficient data retrieval using the new day column.
Update our backend to populate the new field for incoming data.
Defining a new date column in Cassandra is straightforward:
alter table messages add day date;
However, populating it proved more complex than anticipated: We had to update several hundred million records and did not find a suitable CQL UPDATE statement to do so in place (i.e. deriving day from the gentime timestamp). After evaluating all alternatives, we ended up doing this:
Export the primary key for every record to a CSV file using CQL:
copy messages(icao, gentime) to ‘messages_pk_all.csv’ with header = false;
Import the modified file into the same table, which updates every record with just the new column value, leaving all other fields intact:
copy messages(icao, gentime, day) from ‘messages_pk_day.csv’;
The import took several hours, but worked flawlessly. In hindsight, it’s still far from ideal and required some research to guarantee the safety of the data, but given the lack of support in Cassandra, it’s as good a solution we could find.
Prior to creating our first Cassandra materialized view, we considered secondary indexes. However, the many caveats and words of caution from experienced users led us in the direction of materialized views.
CREATE MATERIALIZED VIEW messages_by_day AS SELECT * FROM messages WHERE day IS NOT NULL AND icao IS NOT NULL AND gentime IS NOT NULL PRIMARY KEY (day, icao, gentime) WITH CLUSTERING ORDER BY (icao ASC, gentime ASC)
Materialized view creation ran in background for ~12 hours as it duplicated and organized the data. This query reports any builds in progress:
Step 2: Read Cassandra data efficiently as a time series using Spark
Once we had the means to retrieve the data by date, we had to ensure that the Spark Cassandra connector issued the right queries to retrieve the data efficiently.
Spark configuration parameters for the Cassandra connector (passed when creating a SparkConf, via spark-submit, sparkmagic in Jupyter, etc):
spark.cassandra.connection.host: “host1,host2”, spark.cassandra.auth.username: “username”, spark.cassandra.auth.password: “password”, spark.cassandra.input.consistency.level: ALL
Import the right packages, define a dataset on the Cassandra table:
…or for a number of dates (discrete dates must be specified in an IN condition):
val messagesLastWeekDf = messagesDf.filter(messagesDf(“day”) isin (Date.valueOf("2018-11-06"), Date.valueOf("2018-11-05"), Date.valueOf("2018-11-04"), ... ))
To verify that your filter will be applied to the Cassandra table, print the physical plan with df.explain(). There should be predicates (i.e. actual values) being pushed down to the data source, as in these examples:
Important: Attempts to filter the dataset that don’t conform to the connector documentation will cause Spark to issue a full table scan to Cassandra (and thus defeat the purpose of reorganizing your data as done in Step 1).
Step 3: Partition the Spark dataset as a time series
As a reminder, our goal was to lay the data in S3 partitioned by date, example:
/adsb/messages/year=2018/month=11/day=06
So, rather than partitioning the dataset by the whole date string (e.g. "2018–11–06"), we needed to partition by its components (year, month, day). Hence, we had to first convert the day column into year, month and day columns:
val partMsgsLastWeekDf = messagesLastWeekDf.withColumnRenamed(“day”, “date”).withColumn(“year”, year(col(“date”))).withColumn(“month”, month(col(“date”))).withColumn(“day”, dayofmonth(col(“date”))).drop(“date”)
At long last, we had our data available in Spark with the necessary attributes and partitioned by the criteria we needed.
Dataset with three new columns (year, month, day), ready to be partitioned and written as Parquet
Step 4: Save your Spark dataset to S3 as Parquet
Setting up Spark to push data to S3 was pretty easy. Required AWS libraries are included in the Hadoop distribution Spark depends upon. All we had to configure were the following five entries in $SPARK_HOME/conf/spark-defaults.conf:
Performance of the above statement is surprisingly fast (in part due to Parquet files being much smaller than their CSV equivalents). Loading a day’s worth of records (~500,000 rows) completes in less than one minute.
We use SaveMode.Append to make it possible to load additional days without deleting existing data.
Note: The s3a URL prefix has desirable performance and capacity implications for large file operations such as Parquet.
Step 5: Analyze your data in AWS
From here, sky is the limit. You can spin an EMR cluster and query S3 directly with an arbitrary number of cluster nodes. Alternatively, you can also launch a Sagemaker notebook and point it to EMR Spark following similar steps to those outlined in my previous article. And, as of the time of writing, Boto3, the AWS SDK for Python, now makes it possible to issue basic SQL queries against Parquet files in S3.
Future articles will describe how 1200.aero is using these data to predict potentially hazardous situations for general aviation aircraft.
This repository contains sample Terraform configurations, Sentinel policies, and automation scripts that can be used with Terraform Enterprise.
infrastructure-as-code
This directory contains sample Terraform configurations to provision VMs into AWS, Azure, and Google Cloud Platform (GCP) as well as Kubernetes clusters into Azure Container Service (ACS) and Google Kubernetes Engine (GKE).
self-serve-infrastructure
This directory contains sample Terraform configurations to enable self-service infrastructure. In particular, it illustrates how developers can deploy applications to Kubernetes clusters provisioned by an operations team.
governance
This directory contains some sample Sentinel policies for several clouds which ensure that all infrastructure provisioned with Terraform Enterprise complies with an organization's provisioning rules.
operations
This directory provides artifacts that can be used by operations teams using Terraform Enterprise. In particular, it includes scripts that show how the Terraform Enterprise REST API can be used to automate interactions with Terraform Enterprise, set and delete variables in workspaces, and export, import, and delete Sentinel policies.
cloud-management-platform
This directory provides samples of how Terraform can be used to support cloud management platforms.
gitignore.tf Files
You may notice some gitignore.tf files in certain directories. .tf files that contain the word "gitignore" are ignored by git in the .gitignore file.
If you have local Terraform configuration that you want ignored (like Terraform backend configuration), create a new file in the directory (separate from gitignore.tf) that contains the word "gitignore" (e.g. backend.gitignore.tf) and it won't be picked up as a change.
In Apache Cassandra Lunch #54: Machine Learning with Spark + Cassandra, we will discuss how you can use Apache Spark and Apache Cassandra to perform additional basic Machine Learning tasks. The live recording of Cassandra Lunch, which includes a more in-depth discussion and a demo, is embedded below in case you were not able to attend live. If you would like to attend Apache Cassandra Lunch live, it is hosted every Wednesday at 12 PM EST. Register here now!
In Apache Cassandra Lunch #54, we will discuss how you can use Apache Spark and Apache Cassandra to perform additional basic Machine Learning tasks such as Naive Bayes classification and Random Forest classification. We discuss basic concepts of machine learning as a whole, the technologies used in the presentation, and present a Github repository with a demo project which uses these technologies and be run without any issues. The live recording embedded below contains a live demo as well, so be sure to watch that as well!
Basic Concepts of Machine Learning
Machine Learning is “the study of computer algorithms that improve automatically through experience and by the use of data”. It is seen as a part of artificial intelligence. Machine Learning is used in a large number of applications in today’s society in many different fields I.E. Medicine (drug discovery), image recognition, email filtering, and more. Machine Learning uses complicated mathematical concepts and systems (heavily focused on linear algebra), making it appear intimidating to many beginners. However, many open-source and easy-to-access tools exist for working with and using machine learning in the real world. Some examples include:
There exist a large number of reasons why Cassandra is a great database to use for machine learning applications. Below we list a couple of them:
Speed/Reliability:
Machine learning projects tend to involve processing very large amounts of data in order to properly train complicated models (millions of points of data are necessary for high accuracy in some applications)
Cassandra is very good at dealing with large amounts of incoming data.
Additionally, Cassandra is built upon a masterless architecture – there are no “master” nodes, and thus there is no single point of failure.
Data in Cassandra is replicated and multiple copies are stored across >1 nodes, ensuring the availability of data even when nodes drop from the cluster or unexpectedly shut down.
Scalability/Upgradability:
Cassandra can be scaled both horizontally and vertically:
Horizontal scalability allows the addition of many weaker machines as opposed to upgrading to more powerful hardware on a few machines, thus it is easy to scale up your performance with relatively little cost.
Machine Learning – Random Forest
In the following section, we briefly discuss the Random Forest, which is an ensemble learning method for classification, regression, and similar tasks, and involves building a set of decision trees with incoming data. The building block of a Random Forest is a concept known as a decision tree:
Decision Tree Learning is a predictive modelling approach used in statistics, data mining and machine learning
Given a set of data and labels to the data, we can build a decision tree to try and figure out how to separate the data into classes while testing on various attributes of the data.
The following diagram shows an example set of data off of which a decision tree is created based on selected attributes
Each node represents an attribute we test on.
Simple Decision Tree Example
A Random Forest, as the name may suggest, consists of a large group of decision trees made with different segments of the data. Suppose we have N pieces of data with M different features. We arbitrarily select to make n decision trees as part of the random forest.
For each initial input used to make the decision trees, we take a random sample of size N from the data but with replacement (intentionally adding in some duplicate pieces of data).
The reasoning for this is quite complicated.
Each decision tree also arbitrarily selects only m < M features of the data to make the decision tree based on.
This is known as Feature Randomness. It forces more variation amongst the trees, thus ultimately lowering correlation across trees and producing more diversification.
A given piece of data is then run through/classified by all of the decision trees and whichever class is predicted the most is the result of the prediction for the Random Forest model.
The following diagram visually shows the process a single piece of data goes through in order to be classified by a Random Forest classifier:
Diagram showing how one piece of data gets classified by a Random Forest Classifier
Machine Learning – Naive Bayes Classifier
The Naive Bayes classifier is a collection of simple probability based classification algorithms based on Bayes’ Theorem:
Bayes Theorem
In a classification problem, however, we are trying to obtain the probability that some object/thing is a member of a particular class given some large number of known values. If there is plenty of variation/possible values for the input data, then using Bayes’ theorem in complicated problems is not very practical if we are basing our results on probability tables. We write out Bayes theorem in terms of the probability of some piece of data being part of a class Ck given some known variables about the data x:
Bayes Theorem written out in context of a classification problem
If we have a complicated real-life problem with many portions to the known variables x, then we may not have a single data point for a given class Ck with exactly the values contained in x. Thus, we cannot properly obtain a value for p(x | Ck) using a probability table for such a complicated problem. To resolve this issue, we make the “naive” assumptions of conditional independence: assume that all features in x are mutually independent, conditional on the category Ck:
Naive Bayes Assumption (categorical independence of features in x conditional on the category Ck)
Although different features of most data sets are not conditionally independent, it turns out that making this assumption and building a classifier based on it gives pretty good results on real-life data sets. After going through some math, we come to the following conclusion given this set of assumptions:
Naive Bayes Assumption plugged into the initial Bayes Theorem problem for a classifier
From here, we can obtain values of p(xi | Ck) from a probability table and build a classifier based on this equation, which is the Naive Bayes classifier.
Technologies used in Demo
The following three technologies are the primary drivers of the demo project associated with this blog:
Apache Cassandra
Open-source distributed No-SQL database designed to handle large volumes of data across multiple different servers.
Cassandra clusters can be upgraded by either improving hardware on current nodes (vertical scalability) or adding more nodes (horizontal scalability).
Horizontal scalability is part of why Cassandra is so powerful – cheap machines can be added to a cluster to improve its performance in a significant manner.
Note: Demo runs code from DataStax, and they use the DSE (DataStax Enterprise) version of Cassandra (not open source).
Spark is also run together with Cassandra in DSE.
Apache Spark
Open-source unified analytics engine for large-scale data processing. It provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.
In particular, the demo project uses the Spark Machine Learning Library (MLlib).
Spark’s machine learning library
Features many common ML algorithms, tools for modifying data and pipelines, loading/saving algorithms, various utilities, and more.
Primary API used to be RDD (Resilient Distributed Datasets) based, but has switched over to being DataFrame based since Spark 2.0.
Project Jupyter
Jupyter is an open-source web application for creating and sharing documents containing code, equations, markdown text, and more.
Github repo which will be demoed in this presentation contains a few different Jupyter notebooks containing sections of Python code along with explanations/narration of the general idea of the code.
Jupyter notebooks can be used with programming languages other than python, including R, Julia, Scala, and many others.
The demo we go over in this week’s Cassandra Lunch was written by HadesArchitect at DataStax and contains a Docker compose file which will run three docker images:
DSE – DataStax Enterprise version of Cassandra with analytics(Spark) enabled.
The Github repo contains a lot of examples of Jupyter notebooks with different machine learning tasks/models being used, but we will be primarily looking at Random Forest.ipynb and Naivebayes.ipynb. The notebook Random Forest.ipynb focuses on running the Random Forest classifier to classify a wine into a classification score based on the physical and chemical properties of the wine. The notebook Naivebayes.ipynb does a similar classification task but using the Naive Bayes classifier instead of Random Forest. In order to run the demo project, one change needs to be made to the docker-compose.yml file in the main folder of the project. The line which starts with “PYSPARK_SUBMIT_ARGS” needs to be replaced with the following:
This change uses a more up-to-date version of the DataStax Spark Cassandra Connector and will allow PySpark in the example Jupyter Notebooks to run (successfully connect to Spark running in the DSE Cassandra docker container). Instructions for running the demo besides this change are included in the readme of the repository.
That will wrap up this blog on using Cassandra and Spark to perform additional basic Machine Learning tasks. As mentioned above, the live recording which includes a live walkthrough of the demo is embedded below, so be sure to check it out and subscribe. Also, if you missed last week’s Apache Cassandra Lunch #53: Cassandra ETL with Airflow and Spark, be sure to check that out as well!
Cassandra.Link is a knowledge base that we created for all things Apache Cassandra. Our goal with Cassandra.Link was to not only fill the gap of Planet Cassandra, but to bring the Cassandra community together. Feel free to reach out if you wish to collaborate with us on this project in any capacity.
We are a technology company that specializes in building business platforms. If you have any questions about the tools discussed in this post or about any of our services, feel free to send us an email!
Posted in Modern Business|
Comments Off on Apache Cassandra Lunch #54: Machine Learning with Spark + Cassandra Part 2
The spark-server is a Node.js REST interface for an Apache Spark REPL which can execute JavaScript, via node-java's JNI bridge.
It provides access to Spark contexts, which can be shared via sessions, and allows to execute statements within those contexts. It is mainly focused on Spark's DataFrame APIs.
Motivation: To create a proof-of-concept of a usable and multi-user Spark REPL web service. There are other projects, such as Livy (Hue subproject)
or Apache Toree, but each of them has it's own focus, and none of them is usable with JavaScript.
Current status: This project is to be considered as alpha code. Hopefully there will be some contributors which help to implement further features. Any help is welcome!
The application can either be installed via git clone https://github.com/spark-server/spark-server.git && cd spark-server && npm install or via docker pull sparkserver/spark-server.
Configuration
Preconditions
When the application is installed from Git directly, the following preconditions are in effect:
JDK >= 1.7 is installed and the JAVA_HOME environment variable is set correctly
Apache Spark >= 1.5 is installed and the SPARK_HOME environment variable is set correctly
Node.js >= 4 is installed
Application environment variables
The following variables can be set:
ASSEMBLY_JAR: The path to the Spark assambly jar (to be found in the lib folder of the Apache Spark distribution). When running via Docker, this gets automatically populated. Otherwise this need to be defined manually. (mandatory).
PORT_RANGE_START: The start of the application's port range (e.g. 15000). Default is 10000 if not specified.
PORT_RANGE_END: The end of the application's port range (e.g. 15500). Default is 10100 if not specified.
SPARK_PACKAGE_PATH: Set the path where the Spark package (*.tgz) is located locally. If set, Spark-Server will serve this file and automatically set the spark.executor.uri property, so that it can use a Mesos cluster to run the executors on. When running via Docker, this is automatically enabled.
API_PORT: The port on which the REST API server will listen.
BIND_INTERFACE: The network interface name which Apache Spark should use to determine the IP address to bind to. Default is en0.
Running
Running locally
Change to the folder to which you cloned spark-server to, and run node spark-server.js or npm start. This will start the app on port 3000 and bind on 0.0.0.0 as default. Make sure that the ASSEMBLY_JAR environment variable is set, and points to the Spark assembly jar.
Running via Docker
docker run -d -p 3000:3000 --name spark-server sparkserver/spark-server
If you also want to run Spark on the Mesos cluster, you have to specify the correct masterUrl property in the context configuration, such as mesos://zk://192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181/mesos.
API
You can find the generated RAML documentation of the REST API at http://spark-server.github.io/api/index.html, or on the spark-server instance under /v1/docs/api. The RAML file can be accessed via /v1/docs/api/raml.
Capabilities
The spark-server is able to run/execute JavaScript (including ES7-style async/await syntax, which is transpiled via babel.js) code together with Apache Spark code, which is bridged via node-java's JNI capabilities.
Downside of using the JNI bridge is that the JS code gets "uglier" than the original Scala or Java code (see examples below), but this is in the nature of using a bridge.
By default, the created Spark contexts use the FAIR scheduler to enable the parallel processing of jobs on the executors.
The scheduler pool is defined in the config/spark_pools.xml file.
Examples
The examples will use curl to show how the API can be used. It is assumed that the spark-server is running on http://localhost:3000.
Creating a context
We can create a new Spark context called mycontext by supplying some parameters in the request body. For an overview of possible parameters, see the API docs.
Now, we want to use the promisified actions together with the async/await syntax to enable parallel execution (i.e. when the context is used by multiple sessions in parallel, so that the execution is non-blocking to others):
curl -X POST -H "Content-Type: application/json" -d @examples/parallel.json "http://localhost:3000/v1/contexts/mycontext/sessions/0d3f34b951f9ef5129cb3f5870d9c70e3b9a2115/statements"
which returns
{
"context": "mycontext",
"sessionId": "0d3f34b951f9ef5129cb3f5870d9c70e3b9a2115",
"result": [
{
"year": 2012,
"make": "Tesla",
"model": "S",
"comment": "No comment",
"blank": ""
},
{
"year": 1997,
"make": "Ford",
"model": "E350",
"comment": "Go get one now they are going fast",
"blank": ""
},
{
"year": 2015,
"make": "Chevy",
"model": "Volt"
}
]
}
Also, we can try to mix JS with Spark code. Let's fetch the url stats of www.google.com from the Facebook Graph API asynchronously and calculate the comment-to-shares ratio with Spark SQL:
Apache Toree has one main goal: provide the foundation for interactive applications to connect and use Apache Spark.
The project intends to provide applications with the ability to send both packaged jars and code snippets. As it
implements the latest Jupyter message protocol, Apache Toree can easily plug into the Jupyter ecosystem for quick, interactive data exploration.
Installing as kernel in Jupyter
This requires you to have a distribution of Apache Spark downloaded to the system where Apache Toree will run. The
following commands will install Apache Toree.
One of the most common ways to use Apache Toree is for interactive data exploration in a Jupyter Notebook. You will
first need to install the notebook and get the notebook server running:
pip install notebook
jupyter notebook
The following clip shows a simple notebook running Scala code to print Hello, World!. Each of the code cells can be
run by pressing Shift-Enter on your keyboard.
A key component to Apache Toree is that is will automatically create a SparkContext binding for you. This can be accessed
through the variable sc. The following clip shows code accessing the SparkContext and returning a value.
When it comes to data science models they are intended to run periodically. As an example if we are predicting customer churn for next month, the model has to be run on the last day of each month. Manually running this model monthly is not an option. We can use a scheduler to automate this process. Apache Airflow is an ideal tool for this as it allows to schedule and monitor your workflows. In this article we will be talking about how to deploy Apache Airflow using Docker by keep room to scale out further. Being familiar with Apache Airflow and Docker concepts will be an advantage to follow this article.
Airflow consists of 3 major components; Web Server, Scheduler and a Meta Database. Web server is responsible for the user interface where the users can interact with the application. Scheduler is taking care of the job scheduling while Meta Database is storing the scheduling details. Even though Airflow has several executors, Celery executor is more suitable for scalability. With Celery executor 3 additional components are added to Airflow. They are Worker, Message Broker and Worker Monitor. Worker is responsible for executing jobs that are triggered by the scheduler. There can be multiple workers. These workers can be distributed across cluster instances. Number of workers can be decided upon the workload that has to be performed by the system along with machine capabilities. Message broker helps Celery to operate. A monitoring tool can be used to monitor Celery workers.
Apache Airflow with Celery Executor (Image by author)
With Docker, we plan each of above component to be running inside an individual Docker container. Web Server, Scheduler and workers will use a common Docker image. This common image is unique to the project and the Dockerfile to build that image will be discussed. All the other containers will use publicly available images directly.
For this tutorial PostgreSQL is used as the Meta Database, Redis is used for the message broker and Celery Flower is used to monitor workers. Since there are multiple containers it will be easy to use Docker Compose in order to deploy all the containers at once.
The image shows our project structure. All the project related files are placed inside the scripts folder. All the airflow related files are placed inside the airflow folder. Other files related to the deployment are in the outer most directory. env.list includes the environment variables required by the classification model. requrements.txt defines the packages to be installed in the python environment in order to run the model.
Dockerfile is used to create image to be used by Airflow web server, scheduler and workers. docker-compose.yml is used to define and start all the containers.
A simple model is proposed to classify famous iris datasets. I have added two DAGs with PythonOperator in the dags folder. One to train the model and one to get predictions with trained model. DAG to train model is defined with out a schedule interval. Which means it can be triggered once model training is required. Predictions are intend to retrieve daily. So the DAG which generates predictions is scheduled daily to run the model. File airflow.cfg contains the configuration attributes for Airflow.
To deploy Airflow with docker the best image to refer is puckel/docker-airflow.(update: Airflow has its official Docker image now) But this image can not be used as it is; due to few reasons. One reason is that it does not have all the packages installed that we are using in our project. If we need to change properties in airflow.config file, we have to pass them as environment variables. With a large number of variables this is not easy. So we will be writing a customized Dockerfile with the base image of puckel/docker-airflow. This image is going to be used in all our containers except message broker, meta database and worker monitor. Following is the Dockerfile.
FROM puckel/docker-airflow:1.10.9COPY airflow/airflow.cfg ${AIRFLOW_HOME}/airflow.cfgCOPY requirements.txt /requirements.txt RUN pip install -r /requirements.txt
Meta database
Lets define the service for meta database container first. Docker image postgres:9.6 is used for this. User credentials and database name should be given as environment variables. When setting up this container, database name(POSTGRES_DB=airflow) should be compatible with database connection string in Airflow configuration sql_alchemy_conn.
If you already have an existing PostgreSQL server and you wish to use it, An extra container is no need to be deployed on behalf of the meta db. Create a dedicated database for Airflow meta data in your server. Airflow will populate the database when it starting and it will be responsible to maintain the database afterwards. The server details can be given through airflow configurations. One way to set server details is modifying the sql_alchemy_conn in airflow.cfg file. Other way is to give it as an environment variable AIRFLOW__CORE__SQL_ALCHEMY_CONN. Following is the format of the connection string to be given for a PostgreSQL connection. Credentials to login your server username, password, host IP, database name should be passed as environment variables.
In this project we are focusing on scalability of the application by using multiple Airflow workers. For that we can use the Celery executor. Set executor = CeleryExecutorin airflow config file. The environment variable is AIRFLOW__CORE__EXECUTOR.
Celery is a task queue implementation which Airflow uses to run parallel batch jobs asynchronously in the background on a regular schedule. It needs a message broker like Redis and RabbitMQ to transport messages. Airflow does not have this part and it is needed to be implemented externally. Here we use Redis. We can use image Docker image redis:5.0.5
With our second container docker-compose.yml will look like this.
Here also we can use an external Redis service without creating this container. The credentials should be given by broker_url, celery_result_backend in airflow configurations file.
Airflow Web Server
Airflow User Interface (Image by author)
Our third container will be the Aiflow web server, which has the responsibility over the user Interface. For this container we are going to use previously created Docker image. Following is the docker container configurations for the service ‘webserver’.
build context : point to the created Dockerfile. The correspond image is built when container is starting
restart : re-deploy the container if it is stopped for any reason.
depends_on : web server needs to communicate with meta db and message broker containers
environment : These are the list environment variables requested by the base image puckel/docker-airflow. Keep them as it is. You are free to add more environment variables like ‘PYTHONPATH’ which are going to be used by your own program scripts.
env_file: list of environment variables can be given using file
volumes : dags and program scripts can be mounted as volumes. This is more effective than copying files into the Docker image. Once the files are updated changes will be automatically deployed in the web server. No need to build the image and redeploy the container.
ports : ports to be deployed on. web server is running on port 8080.
command : airflow command to start the web server. This is requested by the base image puckel/docker-airflow. Do not modify it.
healthcheck : test to check the health of the container
Airflow Scheduler
Our fourth container is Aiflow scheduler. Service configurations are very much similar to the Airflow web server except the command and dependencies. Command is the airflow command to start the scheduler. The scheduler is rely on web server container.
Next container is Airflow worker. Command is the airflow command to start the worker. The worker is rely on scheduler container. There can be multiple workers. Here I have added two.
Having multiple workers in the same machine also can reduce the execution time of the jobs. To get more efficient and optimal results properties parallelism ,dag_concurrency, worker_concurrency and max_threads in airflow config file should be adjusted with number of the workers. Some of these properties can be adjusted in the DAG level also.
Workers can be distributed in multiple machines within a cluster. An individual machine will be responsible for each worker while all the other containers can be deployed in one common machine. We can keep a separate docker-compose file to deploy the workers. The main docker-compose file will contain services for rest of containers. Following is an example docker-compose file for a worker container deployed in AWS EC2 instance in ECS cluster.
Now we are ready to deploy our Airflow project with multiple Docker containers. We can deploy all our containers at once by using docker-compose.yml file. The command is docker-compose up -d -- build Following are some useful Docker commands.
Start containers : docker-compose up -d -- build
Stop containers : docker-compose down
View Container : docker ps
Go inside a container : docker exec -it <container-id> bash
See logs of a container: docker logs <container-id>
docker-airflow | Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting.
Repo for building docker based airflow image. Containers support multiple features like writing logs to local or S3 folder and Initializing GCP while container booting.
Airflow version: Notation for representing version XX.YY.ZZ
Execution Mode: standalone(simple container for exploration purpose, based on sqlite as airflow metadata db & SequentialExecutor ) or prod(single node based, LocalExecutor amd mysql as airflow metadata db) and cluster (for distributed production long run use-cases, container runs as either server or worker )
Backend database: standalone- Sqlite, prod & cluster- Mysql
Scheduler: standalone- Sequential, prod- LocalExecutor and Cluster- Celery
Task queue: cluster- Redis
Log location: local file system (Default) or AWS S3 (through entrypoint-s3.sh)
User authentication: Password based & support for multiple users with superuser privilege.
Code enhancement: password based multiple users supporting super-user(can see all dags of all owner) feature. Currently, Airflow is working on the password based multi user feature.
Other features: support for google cloud platform packages in container.
Airflow ports
airflow portal port: 2222
airflow celery flower: 5555
redis port: 6379
log files exchange port: 8793
In server container: redis, airflow webserver & scheduler is running.
In worker container: airflow worker & celery flower ui service is running.
Kubeflow is a popular open-source Machine Learning platform that runs on Kubernetes. Kubeflow streamlines many valuable ML workflows i.e. it allows users to easily deploy development environments, scalable ML workflows with Kubeflow Pipelines, automated hyper-parameter tuning and neural architecture search with Katib, easy collaboration within teams and much more. With a such a large number of features also comes complexity. A full Kubeflow deployment contains many services and dependencies, making it difficult for users to customize, manage and install Kubeflow using the legacy kfctl CLI tool and a KfDef YAML file. For this reason, the upcoming Kubeflow 1.3 release has stopped using kfctl and instead is using standard Kustomize, making it easier to deploy Kubeflow with GitOps tools such as Argo CD.
Argo CD is “a declarative, GitOps continuous delivery tool for Kubernetes”. The Argo Project is already closely related to Kubeflow, as Kubeflow Pipelines uses Argo Workflows behind the scenes, making Argo CD a natural choice for deploying Kubeflow. A quick description of GitOps is the use of a Git repository as the source of truth for the desired state of a deployment. A Continuous Delivery tool such as Argo CD then monitors the Git repository for changes and updates deployment if needed.
The code and manifests needed to follow this guide can be found at https://github.com/argoflow/argoflow. The deployment should be compatible with any Kubernetes cluster both on-premises and in the cloud. Note - While some extra manifests are included to make on-premises deployments a bit easier, such as MetalLB and Rook Ceph, these are disabled by default.
The first step is to fork the ArgoFlow repository and clone it locally. Next, you will need to have yq version 4 installed to use the scripts that allows you to easily setup your fork and Kustomize v4.0.5 to install Argo CD with the provided manifests. Access to a Kubernetes cluster is of course also a requirement. This can be done locally with tools such as minikube, kind or MicroK8s, with VMs or in the cloud. However, setting up a Kubernetes cluster is out of the scope of this article.
For the sake of simplicity, Argo CD will be installed on the same cluster as Kubeflow. The release candidate of Argo CD 2.0 in HA mode is used as I like to stay up to date with software. A few optimizations were done to the Argo CD configuration along with including the v4.0.5 of Kustomize, the latest version at the time of writing.
After cloning your fork of ArgoFlow locally, you can install Argo CD by executing the following command from within the ArgoFlow folder:
kustomize build argocd/ | kubectl apply -f -
Next install the Argo CD CLI tool locally. To access the Argo CD UI, you can expose its service using a LoadBalancer, Ingress or connect to it using kubectl and port forwarding. The Argo CD getting started page has more instructions on how to do this.
Next, get the default password for the admin user by executing the following command:
You can now login with the Argo CD CLI tool and change the default password as follows:
argocd login <ARGOCD_SERVER> # e.g. localhost:8080 or argocd.example.com
argocd account update-password
You should now be able to login to the Argo CD UI with your new password.
Argo CD Web Interface
The manifests have been setup with some default values to help new users get started, but should be changed by most people. For example, users should change the password for the default user and setup an issuer in cert-manager to create proper certificates for the Istio ingress gateway. Another thing many people will likely need to change is the ingress gateway service, which is setup to use a LoadBalancer by default. In the spirit of keeping this article clear and to the point, I will not go into details how to change these settings here as it would involve a lot of links to files. The README in the ArgoFlow repository provides instructions for changing these aspects of the deployment.
Getting ready to deploy Kubeflow
Along with the Kustomize manifests for the different Kubeflow components, the ArgoFlow repository also contains Argo CD application specifications for each one of the Kubeflow components. To make it easier for people to select and deploy the components they want, a kustomization.yaml file located in the root repository is used to define which Argo CD applications should be deployed. In turn, the kubeflow.yaml file in the root of the repository is an Argo CD application for the previously mentioned kustomization.yaml file. Simply put, it is an Argo CD application that deploys other Argo CD applications.
Next, the Argo CD application specifications need to be edited so they point to your fork of the ArgoFlow repository. A script is included to make this easier. In the root of the repository, simply run the following command to update all the application specs to point to your fork of the repository:
./setup_repo.sh <your_repo_fork_url>
Bonus: Extending the Volumes Web App with a File Browser
A large problem for many people is how to easily upload or download data to and from the PVCs mounted as their workspace volumes for Notebook Servers. To make this easier a simple PVCViewer Controller was created (a slightly modified version of the tensorboard-controller). The PVC Viewer will work with ReadWriteOnce PVCs, even when they are mounted to an active Notebook Server. This feature was not ready in time for 1.3, and thus I am only documenting it here as an experimental feature as I believe many people would like to have this functionality. It is not (yet) part of upstream Kubeflow and might change in the future.
Demo of the PVC Viewer in action
If you would like to deploy this feature, you need to comment out the argocd-applications/volumes-web-app.yaml entry in the root kustomization.yaml file and uncomment the following two lines:
You should now start seeing applications appear in the Argo CD UI, and if all goes well they should be “Healthy” and “Synced”. Due to the large amount of resources in a Kubeflow deployment and some chicken & egg scenarios some applications might stay “Out of Sync” for a bit.
Kubeflow Component Applications in Argo CD
By clicking on an application Argo CD presents you with an overview of all the resources created by this application.
Overview of the Jupyter Web App resources in Argo CD
By clicking on any of the resources Argo CD will show you useful information such as the diff between the currently deployed YAML and what is declared in the Git repository or the logs of a pod.
Updating your deployment
To update or change your deployment, simply commit and push your changes to your repository and Argo CD will automatically detect and change the deployment in your Kubernetes cluster.
As the upcoming release of Kubeflow 1.3 uses standard Kustomize, this makes it easier to use tools such as Argo CD to manage its deployment. Furthermore, due to the size and complexity of Kubeflow, a declarative approach using GitOps and Argo CD should make it easier for users to customize and deploy Kubeflow suited to their needs.
This article explains what the basic features and differences between ETL and ELT are. I’m also going to explain in
detail what an ELT pipeline is and a relevant architecture for the same in Azure. So far, we have come a long way
dealing with ETL tools which basically are Extract, Transformation and Load technique used in populating a data
warehouse. ELT, on the other hand, is another way to load data into a warehouse that implements the process of
Extract, Load and Transform.
When we try to understand ETL, it is the technique that we use to connect to source data, extract the data from
those sources, transform the data in-memory to support the reporting requirements and then finally load the
transformed data into a data warehouse. In a typical ETL workload, the tool can connect to the source databases
periodically and extract the data from those sources. Further, these extraction methods are either full or
incremental loads, which means either the complete data from the source is loaded or only a changed portion of the
data.
The figure below provides a graphical illustration of an ETL workload. Notice that the data is extracted from the
sources and then transformed by the ETL tool in memory before being loaded into the warehouse.
Figure 1 – Overview of ETL
ELT is another technique, where the data is extracted and loaded into the warehouse directly without any
transformations. Once the extract and load process has been completed, the transformation logic is applied to the
warehouse where the data is stored. This transformed data is then again loaded back into the warehouse from where
reporting and other business decisions can be supported. In the later part of this article, I’ll sight why we would
need to use this technique over the traditional ETL method that we have been using for so long and differentiate the
features between ETL and ELT workloads.
In the figure below, the difference between ETL and ELT is clearly visible, as in this case, the data is extracted
and loaded into the warehouse first and then transformed.
Figure 2 – Overview of ELT
Why do we need ELT?
The traditional method of using the ETL architecture is monolithic in nature, often used to connect only to
schema-based data sources and they have very little or no room to process data flowing at very high speed. With the
businesses dealing with high velocity and veracity of data, it becomes almost impossible for the ETL tools to fetch
the entire or a part of the source data into the memory and apply the transformations and then load it to the
warehouse.
In modern applications, we tend to have a variety of different data sources ranging from structured schema-based SQL
sources to an unstructured NoSQL database. These different data sources rarely have any identical schema and each
time a new source is added it becomes difficult or more time consuming to implement the logic in a traditional ETL
tool.
An important factor that leads to the implementation of ELT systems is the adoption of the cloud data warehouses or
data lakes by the organizations. An example of a cloud data warehouse is Azure Synapse Analytics (formerly known as
Azure SQL Data Warehouse) or maybe Amazon RedShift. These cloud data warehouses have an MPP architecture (Massively
Parallel Processing) and can be provisioned in very little time. Snowflake is also an example of a cloud data
warehouse where all the infrastructure is managed, and customers need not worry about anything other than designing
the business logic.
Also, these cloud data warehouses support the columnar store which is very helpful while analyzing large datasets
with petabytes of data. In a columnar store, the data is stored by columns as opposed to the traditional method of
row-wise data stores. In addition to this, cloud data warehouses are provisioned to run in multiple nodes with a
combination of various RAMs and SSDs to support the high-speed processing of data. These are a few reasons which
make a cloud data warehouse suitable for transforming and analyzing data on the fly without affecting the query
performance.
Comparing ETL and ELT
ETL
ELT
Technology Adoption
ETL has been in the market for over two decades now and is relatively easier to find developers who have
vast experience in designing ETL systems.
On the other hand, ELT is a new technology that is more focused on cloud-based warehouses. Searching
suitable engineers to develop ELT pipelines are as easy as for ETL.
Data Availability
In an ETL workload, the data which is required only for analytics or reporting is being loaded into the
warehouse, leaving other unnecessary data in the source systems as is.
Whereas in an ELT system, we tend to load anything and everything into a warehouse or a data lake from
where it can be analyzed at a later point of time.
Calculated Fields and Transformations
Yes, in ETL, we can add or remove specific columns while transforming the data in the ETL tool. We can
also add calculated columns and load them to the warehouse.
In ELT, additional columns are directly added to the existing dataset in the warehouse. Usually, there
is no modification of the source columns.
Transformation Complexities
In an ETL workload, we can implement much complex data transformations as and when required. All these
transformations occur in-memory.
In an ELT workload, the more focus is given towards analyzing highly variable structured and
unstructured data that is arriving at a high pace rather than complexities.
Infrastructure
Most of the traditional ETL tools need to be installed on-premises which incur a lot of cost to the
analytics workloads.
ELT, on the other hand, is mostly cloud-based and doesn’t require to be installed on the premises.
Postproduction Maintenance
In an ETL pipeline, that is installed on-premises, maintenance is frequently required.
ELT, since it is cloud-based or serverless, no or very little maintenance is required.
Transformation Area
In an ETL pipeline, the transformations are applied in memory in a staging layer before the data is
being loaded into the data warehouse.
In ELT, the transformations are applied once the data has been loaded into the warehouse or a data lake.
In this case, usually, there is no requirement for a staging layer unlike in the ETL.
Support for semi-structured and unstructured data
Although an ETL tool can read data from semi-structured or unstructured data sources, it is usually
transformed in the staging layer and only stored as a proper structure in the warehouse.
ELT is designed to handle all types of data structures from semi-structured to unstructured data in the
data lakes which can be further analyzed.
Example of ETL and ELT Workloads in Azure
Now that we have some idea about the comparisons between ETL and ELT, let us go ahead and see how a typical ELT
workload can be implemented in Azure.
As you can see in the figure above, the on-premises data is first ingested into the blob storage which is a file
system in the cloud and from there it is computed and stored in the SQL Data Warehouse for further analysis. To read
more about this architecture, please follow the official documentation from Microsoft.
In the next figure, you can see how data is being imported from the various data sources and then ingested into the
SQL Data Warehouse using Data Factory. Further, a semantic model is created using Azure Analysis Services and the
data is visualized using Power BI.
In this article, we talked about the main differences between ETL and ELT architecture. Data processing is an important operation for an organization, and it should be chosen carefully. Although there are a few differences between ETL and ELT, for most of the modern analytics workload, ELT is the most preferred option as it reduces the data ingestion time to a great extent as compared to the traditional ETL process. This helps organizations to make faster decisions and improvise analytical capabilities.
Aveek is an experienced Data and Analytics Engineer, currently working in Dublin, Ireland. His main areas of technical interest include SQL Server, SSIS/ETL, SSAS, Python, Big Data tools like Apache Spark, Kafka, and cloud technologies such as AWS/Amazon and Azure.
He is a prolific author, with over 100 articles published on various technical blogs, including his own blog, and a frequent contributor to different technical forums.
In his leisure time, he enjoys amateur photography mostly street imagery and still life. Some glimpses of his work can be found on Instagram. You can also find him on LinkedIn
Containers let you freeze and restart an exact copy of a system that you plan to deploy, including the operating system and configuration files. This makes debugging easy and testing a snap, and it even changes the way that deploys and rollbacks happen in IT operations.
Container packages are not only complete, but they are also small and efficient enough to download and run in seconds. Cluster managers provide the load balancing and scale to ensure uptime even during a rollout.
Once you decide to use containers (or expand their use to a different area), you'll face a new challenge. The very growth and expansion in container technology brings a large set of choices, including which standards to use, how to store the old versions and deploy new images, and how to manage containers in production.
So how do you assemble the right mix of products and services to build, run, and manage containers efficiently? To answer that question, TechBeacon reviewed a range of container technologies, from architecture to cluster management, storage, security, and even training and support. Here are the essential tools every team should consider.
Container runtimes
Docker was the first major open-source container offering, and quickly emerged as a de facto standard. Now Kubernetes is evolving as the new standard for clusters and cluster management.
Kubernetes initially supported Docker and rkt (or "rocket") through custom code. But now, with the creation of the Container Runtime Interface (CRI), you have many ways to store virtual machines and at the same time communicate through that interface.
Docker
The first and still most popular container technology, Docker's open-source containerization engine works with most of the products that follow, as well as many open-source tools.
Docker Enterprise
This set of extensions not only adds features to Docker, but also makes it possible for Docker (the company) to add commercial support. If you need a support matrix to know exactly which versions of what software are supported—and a phone number to call if things go wrong—then Docker Enterprise might be for you.
CRI-O
The first implementation of the Container Runtime Interface, CRI-O is an incredibly lightweight, open-source reference implementation.
rktlet
The aforementioned rkt, redesigned and retooled to use the CRI as rktlet, now has a set of supported tools and community to rival Docker.
containerd
A project of the Cloud Native Computing Foundation, containerd was an early container format. More recently the developers of containerd built a CRI plugin that lets Kubernetes run containerd in the same way it runs rktlet or CRI-O.
Microsoft Containers
Positioned as an alternative to Linux, Microsoft Containers can support Windows containers under very specific circumstances. They generally run in a true virtual machine and not in a cluster manager like Kubernetes.
Cluster management and deployment
Once the team can create images and pass them around in development, comes the hard part: running and supporting containers in production. That means registering artifacts, deploying them to production as a system, and managing servers and collections of servers. The latter includes a collection of servers in the cloud, known as a "cluster."
Cluster management tools manage workloads, including moving instances from one virtual host to another based on load, and allocate resources such as CPU and memory.
Kubernetes
While there is no standard for cluster management, the Kubernetes open-source cluster manager, originally developed by Google, is far and away the most popular. Supported by Amazon's AWS, Google's Cloud Engine (GCE) and Microsoft's Azure Container service, Kubernetes is relatively portable, which helps prevent vendor lock-in.
Kubernetes can even run on a private cloud: OpenStack. Microsoft, Amazon, and Google all provide container services that run Kubernetes—with commercial support options available.
Istio and Envoy
Where Kubernetes provides load balancing and scalability, by default it assumes that the same service will run the same version number and that all services can talk to one another. Kubernetes does not provide the tools to debug services that call services—sometimes called "observability."
Envoy and Istio are open-source service mesh technologies that add a layer to provide security and observability. They can encrypt traffic inside of the cluster while observing it. Developed by Lyft, Envoy was the first service mesh for Kubernetes. Istio includes Envoy, sits on top of it, and adds several plugins, dashboards, and other features to extend it.
Apache Mesos
A tool for abstracting computing resources, Apache Mesos can run both Docker and rkt images side by side in the same cluster. DC/OS is a platform built on Mesos that functions as a data center operating system.
Docker Swarm
Docker's free product for cluster management, Swarm runs from the command line and comes bundled with Docker 1.12 and higher. Swarm does not support autoscaling or load balancing natively, but third-party extensions provide this functionality.
Docker Datacenter
More than just a commercial alternative to Kubernetes, Docker Datacenter is designed as a drop-in replacement that lets you containerize your entire data center, and it includes commercial support. The tool has LDAP integration and a web-based dashboard with control panel, registry, monitoring, logging, and continuous integration.
Of course, Docker Datacenter embraces and extends Docker's free, open-source products: Docker and Swarm. The tool adds the load balancing and scaling that Swarm is missing. And, of course, it works with Docker Enterprise.
Storage containers
Containers are designed to be interchangeable, like currency. That works exceptionally well for web services and microservices that can scale on demand. Storage and databases, on the other hand, need persistent locations to house data, or at least a standard interface layer. Organizations that want to move to an all-container infrastructure need storage, and companies now meet that demand.
BlockBridge
BlockBridge, the "elastic storage platform" company, offers storage as a container using Docker, with support for Kubernetes, OpenStack, and software-defined secure storage.
EMC / libstorage
The EMC / libstorage system offers a code library to provide container storage that's free and open.
Docker plugins for storage
EMC, NetApp, and others have created plugins to support storage, which Docker Inc. makes available for download.
Container security
Single sign-on, LDAP integration, auditing, intrusion detection and prevention, and vulnerability scanning—all are pain points for organizations moving to containers. Even traditional devices and software can be hard, if not impossible, to configure on container clusters. Fortunately, a handful of vendors is working to address this need.
Twistlock
You build Docker images out of components, such as an operating system, a web server, or a content management system. The problem is that unpatched or outdated software on an image could harbor security risks. Twistlock's vulnerability scanner addresses that by comparing images against a database of known threats. This is an automated audit against a database that's constantly updated. Other core features include more classic intrusion detection, and regulatory compliance systems.
Aqua Container Security
Like Twistlock, Aqua focuses on the ability to create, monitor, and enforce policy for containers, along with integration with continuous integration (CI), running security checks on every build.
StackRox
Co-founded by Sameer Bhalotra, a former security executive at Google and senior director for cybersecurity in the Executive Office of the President of the United States, StackRox provides Kubernetes cluster discovery. The software examines an entire cluster, comparing how the running containers behave compared to a company's security policies. StackRox allows those policies to be documented and evaluated automatically in code.
Aporeto
Aporeto encrypts every workload between containers, providing authentication and authorization. The software also allows you to define security policy programmatically, and to ensure it is enforced.
Operating systems
Most Linux operating system distributions are based on convenience and include big, preinstalled packages, just in case the user might want them. Docker, in contrast, is designed for lightweight virtualization—to run as many identical machines as possible with the least amount of overhead in terms of memory, disk, and CPU.
In response, vendors have developed container-optimized builds that attempt to balance the capabilities teams might need in a Linux distribution with the minimalism that containers demand. Here are a few of the most popular ones, along with some non-Linux options.
Alpine Linux
If you create a Docker image and do not specify an operating system, you'll be using Alpine Linux. That means a great number of sample and test Docker containers use it. It might be good to get to know the strengths and weaknesses of the operating system, which is bare-bones but small, fast, and relatively secure.
RancherOS
Containing only the Linux kernel and Docker itself, the RancherOS system image fits into just 22MB of disk space. RancherOS eliminates systemd, the service management system built into most versions of Linux, instead starting the Docker Daemon itself as the init, or bootstrap, system.
CoreOS Container Linux
Designed to work with CoreOS Linux tools and systems, CoreOS Container Linux is preconfigured to run Linux containers. It also comes with automatic updates turned on; operating systems update themselves without any handling.
Ubuntu Core
Canonical, the parent company of Ubuntu Linux, claims that Ubuntu is the most common OS for containers. Within the Ubuntu distribution is Core, the small, secure release designed for Internet of Things devices and containers. Core is designed to have high performance, a small footprint, and transactional updates, ensuring that updates that fail roll back successfully. Of course, using Ubuntu Core means you can purchase support from Canonical.
Red Hat Atomic Host
Organizations that run Red Hat Enterprise and want to use containers will want to have their hosts run the Red Hat Atomic Host operating system. These tools will let you host Linux containers in a minimal version of Red Hat Enterprise Linux.
Microsoft Nano Server
Nano Server is a small, remotely administered, command-line operating system based on Windows Server 2016. Designed to run solely as a container, Nano brings native container capability to Windows Server. Windows Pro 10 Enterprise is another Microsoft operating system that can host Windows containers.
VMware Photon
Weighing in at 220MB on disk, Photon is a larger container operating system than some others, although it's still only about one-hundredth the size of the latest version of Windows. This Linux container host is designed to integrate with VMware's vSphere virtualization products.
Container events and sources for support
Once you've committed to containers, the hardest part will be implementing and supporting them. From conferences to support forums to commercial support, here are the resources you need.
Cloud Native Computing Foundation
Once companies like IBM, Lyft, and Google launch an open-source technology, they need someone to take it over and maintain it. That's where the CNCF comes in, providing maintenance and governance for projects such as Kubernetes, Envoy, and containerd. CNFC also organizes events.
DockerCon
This is the event to attend if your company is pursuing an all-Docker architecture, with the support of Docker Datacenter, Swarm, and other products from Docker's business partners. DockerCon 2019, which concluded on May 2, had 11 tracks, ranging from lightning talks to hands-on workshops, vendor presentations, and case studies.
KubeCon+CloudNative Con
The premier event of the CNCF, KubeCon+CloudnativeCon events are held on multiple continents each year. In 2019, locations include Barcelona, Spain, and China.
StackOverflow
The largest, most popular online Q&A site for programmers, StackOverflow offers plenty of information on deploying your applications in containers. It also does so without intrusive ads or insisting you register to get the information.
While the concept behind containers is simple, the devil, as they say, is in the details. If your technical team uses containers strictly for builds and testing, your decisions are limited to choosing the correct operating system and container type. But once the system is creating an image for every build, why stop there?
Today, tools such as Kubernetes allow a production cluster built out of containers, with autoscale. Because the standards are open and every major cloud provider supports Kubernetes, using it for cluster management can prevent vendor lock-in.
That's our concise list of resources container resources, but we welcome yours. What did we miss? Feel free to add your tips and suggestions to this list by posting them below.
Keep learning
Ready to advance up the IT career ladder? TechBeacon's Careers Topic Center provides expert advice you need to prepare for your next move.
The job of data engineers is to help an organization move and process data but not to do that themselves.
Data architecture follows a predictable evolutionary path from monolith to automated, decentralized, self-serve data microwarehouses.
Integration with connectors is a key step in this evolution but increases workload and requires automation to correct for that.
Increasing regulatory scrutiny and privacy requirements will drive the need for automated data management.
A lack of polished, non-technical tooling is currently preventing data ecosystems from achieving full decentralization.
"The future of data engineering" is a fancy title for presenting stages of data pipeline maturity and building out a sample architecture as I progress, until I land on a modern data architecture and data pipeline. I will also hint at where things are headed for the next couple of years.
It's important to know me and my perspective when I'm predicting the future, so that you can couch mine with your own perspectives and act accordingly.
I work at WePay, which is a payment-processing company. JPMorgan Chase acquired the company a few years ago. I work on data infrastructure and data engineering. Our stack is Airflow, Kafka, and BigQuery, for the most part. Airflow is, of course, a job scheduler that kicks off jobs and does workflow things. BigQuery is a data warehouse hosted by Google Cloud. I make some references to Google Cloud services here, and you can definitely swap them with the corresponding AWS or Azure services.
We, at WePay, use Kafka a lot. I spent about seven years at LinkedIn, the birthplace of Kafka, which is a pub/sub, write-ahead log. Kafka has become the backbone of a log-based architecture. At LinkedIn, I spent a bunch of time doing everything from data science to service infrastructure, and so on. I also wrote Apache Samza, which is a stream-processing system, and helped build out their Hadoop ecosystem. Before that, I spent time as a data scientist at PayPal.
There are many definitions for "data engineering". I've seen people use it when talking about business analytics and in the context of data science. I'm going to throw down my definition: a data engineer's job is to help an organization move and process data. To move data means streaming pipelines or data pipelines; to process data means data warehouses and stream processing. Usually, we're focused on asynchronous, batch or streaming stuff as opposed to synchronous real-time things.
I want to call out the key word here: "help". Data engineers are not supposed to be moving and processing the data themselves but are supposed to be helping the organization do that.
Maxime Beauchemin is a prolific engineer who started out, I think, at Yahoo and passed through Facebook, Airbnb, and Lyft. Over the course of his adventures, he wrote Airflow, which is the job scheduler that we and a bunch of other companies use. He also wrote Superset. In his "The Rise of the Data Engineer" blog post a few years ago, Beauchemin said that "... data engineers build tools, infrastructure, frameworks, and services." This is how we go about helping the organization to move and process the data.
The reason that I put this presentation together was a 2019 blog post from a company called Ada in which they talk about their journey to set up a data warehouse. They had a MongoDB database and were starting to run up against its limits when it came to reporting and some ad hoc query things. Eventually, they landed on Apache Airflow and Redshift, which is AWS's data-warehousing solution.
What struck me about the Ada post was how much it looked like a post that I’d written about three years earlier. When I landed at WePay, they didn't have much of a data warehouse and so we went through almost the exact same exercise that Ada did. We eventually landed on Airflow and BigQuery, which is Google Cloud's version of Redshift. The Ada post and mine are almost identical, from the diagrams to even the structure and sections of the post.
This was something we had done a few years earlier and so I threw down the gauntlet on Twitter and predicted Ada’s future. I claimed to know how they would progress as they continued to build out their data warehouse: one step would be to go from batch to a real-time pipeline, and the next step would be to a fully self-serve or automated pipeline.
I'm not trying to pick on Ada. I think it's a perfectly reasonable solution. I just think that there's a natural evolution of a data pipeline and a data warehouse and the modern data ecosystem and that's really what I want to cover here.
Figure 1: Getting cute with land/expand/on demand for pipeline evolution
I refined this idea with the tweet in Figure 1. The idea was that we initially land with nothing, so we need to set up a data warehouse quickly. Then we expand as we do more integrations, and maybe we go to real-time because we've got Kafka in our ecosystem. Finally, we move to automation for on-demand stuff. That eventually led to my "The Future of Data Engineering" post in which I discussed four future trends.
The first trend is timeliness, going from this batch-based periodic architecture to a more real-time architecture. The second is connectivity; once we go down the timeliness route, we start doing more integration with other systems. The last two tie together: automation and decentralization. On the automation front, I think we need to start thinking about how we operate not just our operations but our data management. And then decentralizing the data warehouse.
Figure 2: The six stages of data-pipeline maturity
I designed a hierarchy of data-pipeline progression. Organizations go through this evolution in sequence.
The reason I created this path is that everyone's future is different because everyone is at a different point in their life cycle. The future at Ada looks very different than the future at WePay because WePay may be farther along on some dimensions - and then there are companies that are even farther along than WePay.
These stages let you find your current starting point and build your own roadmap from there.
Stage 0: None
You're probably at this stage if you have no data warehouse. You probably have a monolithic architecture. You're maybe a smaller company and you need a warehouse up and running now. You probably don't have too many data engineers and so you're doing this on the side.
Figure 3: Stage 0 of data-pipeline maturity
Stage 0 looks like Figure 3, with a lovely monolith and a database. You take a user and you attach it to the database. This sounds crazy to people that have been in the data-warehouse world for a while but it's a viable solution when you need to get things quickly up and running. The data appears to the user as basically real-time data because you're reading directly from the database. It's easy and cheap.
This is where WePay was when I landed there in 2014. We had a PHP monolith and a monolithic MySQL database. The users we had, though, weren't happy and things were starting to tip over. We had queries timing out. We had users impacting each other - most OLTP systems that you're going to be using do not have strong isolation and multi-tenancy so users can really get in each other's way. Because we were using MySQL, we were missing some of the fancier analytic SQL stuff that our data-science and business-analytics people wanted and report generation was starting to break. It was a pretty normal story.
Stage 1: Batch
We started down the batch path, and this is where the Ada post and my earlier post come in.
Going into Stage 1, you probably have a monolithic architecture. You might be starting to lean away from that but usually it works best when you have relatively few sources. Data engineering is now probably your part-time job. Queries are timing out because you're exceeding the database capacity, whether in space, memory, or CPU.
The lack of complex analytical SQL functions is becoming an issue for your organization as people need those for customer-facing or internal reports. People are asking for charts, business intelligence, and all that kind of fun stuff.
Figure 4: Stage 1 is the classic batch-based approach to data
This is where the classic batch-based approach comes in. Between the database and the user, you stuff a data warehouse that can accomplish a lot more OLAP and fulfill analytic needs. To get data from the database into that data warehouse, you have a scheduler that periodically wakes up to suck in the data.
That's where WePay was at about a year after I joined. This architecture is fantastic in terms of tradeoffs. You can get the pipeline up pretty quickly these days - when I did it in 2016, it took a couple of weeks. Our data latency was about 15 minutes, so we did incremental partition loads, taking little chunks of data and loading them in. We were running a few hundred tables. This is a nice place to start if you're trying to get something up and running but, of course, you outgrow it.
The number of Airflow workflows that we had went from a few hundred to a few thousand. We started running tens or hundreds of thousands of tasks per day, and that became an operational issue because of the probability that some of those are not going to work. We also discovered - and this is not intuitive for people who haven't run complex data pipelines - that the incremental or batch-based approach requires imposing dependencies or requirements on the schemas of the data that you're loading. We had issues with create_time and modify_time and ORMs doing things in different ways and it got a little complicated for us.
DBAs were impacting our workload; they could do something that hurt the replica that we're reading off of and cause latency issues, which in turn could cause us to miss data. Hard deletes weren't propagating - and this is a big problem if you have people who delete data from your database. Removing a row or a table or whatever can cause problems with batch loads because you just don't know when the data disappears. Also, MySQL replication latency was affecting our data quality and periodic loads would cause occasional MySQL timeouts on our workflow.
Stage 2: Realtime
This is where real-time data processing kicks off. This approaches the cusp of the modern era of real-time data architecture and it deserves a closer look than the first two stages.
You might be ready for Stage 2 if your load times are taking too long. You've got pipelines that are no longer stable, whether because workflows are failing or your RDBMS is having trouble serving the data. You've got complicated workflows and data latency is becoming a bigger problem: maybe the 15-minute jobs you started with in 2014 are now taking an hour or a day, and the people using them aren't happy about it. Data engineering is probably your full-time job now.
Your ecosystem might have something like Apache Kafka floating around. Maybe the operations folks have spun it up to do log aggregation and run some operational metrics over it; maybe some web services are communicating via Kafka to do some queuing or asynchronous processing.
Figure 5: Stage 2 of data-pipeline maturity
From a data-pipeline perspective, this is the time to get rid of that batch processor for ETL purposes and replace it with a streaming platform. That's what WePay did. We changed our ETL pipeline from Airflow to Debezium and a few other systems, so it started to look like Figure 6.
Figure 6: WePay’s data architecture in 2017
The hatched Airflow box now contains five boxes, and we're talking about many machines so the operational complexity has gone up. In exchange, we get a real-time pipeline.
Kafka is a write-ahead log that we can send messages to (they get appended to the end of the log) and we can have consumers reading from various locations in that log. It's a sequential read and sequential write kind of thing.
We use it with the upstream connectors. Kafka has a component called Kafka Connect. We heavily use Debezium, a change-data-capture (CDC) connector that reads data from MySQL in real time and funnels it in real time into Kafka.
CDC is essentially a way to replicate data from one data source to others. Wikipedia’s fancy definition of CDC is "… the identification, capture, and delivery of the changes made to the enterprise data sources." A concrete example is what something like Debezium will do with a MySQL database. When I insert a row, update that row, and later delete that row, the CDC feed will give me three different events: an insert, the update, and the delete. In some cases, it will also provide the before and the after states of that row. As you can imagine, this can be useful if you're building out a data warehouse.
Figure 7: Debezium sources
Debezium can use a bunch of sources. We use MySQL, as I mentioned. One of the things in that Ada post that caught my eye was the fact that they were using MongoDB - sure enough, Debezium has a MongoDB connector. We contributed a Cassandra connector to Debezium a couple of months ago. It's incubating and we're still getting up off the ground with it ourselves but that's something that we're going to be using heavily in the near future.
Last but not least in our architecture, we have KCBQ, which stands for Kafka Connect BigQuery (I do not name things creatively). This connector takes data from Kafka and loads it into BigQuery. The cool thing about this, though, is that it leverages BigQuery’s real-time streaming insert API. One of the cool things about BigQuery is that you can use its RESTful API to post data into the data warehouse in real time and it's visible almost immediately. That gives us a latency from our production database to our data warehouse of a couple of seconds.
This pattern opens up a lot of use cases. It lets you do real-time metrics and business intelligence off of your data warehouse. It also allows you to debug, which is not immediately obvious - if your engineers need to see the state of their database in production right now, being able to go to the data warehouse to expose that to them so that they can figure out what's going on with their system with essentially a real-time view is pretty handy.
You can also do some fancy monitoring with it. You can impose assertions about what the shape of the data in the database should look like so that you can be satisfied that the data warehouse and the underlying web service itself are healthy.
Figure 8: Problems at WePay in the migration to Stage 2
Figure 8 shows some of the inevitable problems we encountered in this migration. Not all of our connectors were on this pipeline, so we found ourselves between the new cool stuff and the older painful stuff.
Datastore is a Google Cloud system that we were using; that was still Airflow-based. Cassandra didn't have a connector and neither did Bigtable, which is a Google Cloud equivalent of HBase. We had BigQuery but BigQuery needed more than just our primary OLTP data; it needed logging and metrics. We had Elasticsearch and this fancy graph database (which we're going to be open-sourcing soon) that also needed data.
The ecosystem was looking more complicated. We're no longer talking about this little monolithic database but about something like Figure 9, which comes from Confluent and is pretty accurate.
Figure 9: The data ecosystem is no longer a monolith
You have to figure out how to manage some of this operational pain. One of the first things you can do is to start integration so that you have fewer systems to deal with. We used Kafka for that.
Stage 3: Integration
If you think back 20 years to enterprise-service-bus architectures, that's really all data integration is. The only difference is that streaming platforms like Kafka along with the evolution in stream processing have made this viable.
You might be ready for data integration if you've got a lot of microservices. You have a diverse set of databases as Figure 8 depicts. You've got some specialized, derived data systems; I mentioned a graph database but you may have special caches or a real-time OLAP system. You've got a team of data engineers now, people who are responsible for managing this complex workload. Hopefully, you have a happy, mature SRE organization that's more than willing to take on all these connectors for you.
Figure 10: Stage 3 of data-pipeline maturity
Figure 10 shows what data integration looks like. We still have the base data pipeline that we've had so far. We've got a service with a database, we've got our streaming platform, and we've got our data warehouse, but now we also have web services, maybe a NoSQL thing, or a NewSQL thing. We've got a graph database and search system plugged in.
Figure 11: WePay’s data ecosystem at the start of 2019
Figure 11 depicts where WePay was at the beginning of 2019. Things were becoming more complicated. Debezium connects not only to MySQL but to Cassandra as well, with the connector that we'd been working on. At the bottom is Kafka Connect Waltz (KCW). Waltz is a ledger that we built in house that's Kafka-ish in some ways and more like a database in other ways, but it services our ledger use cases and needs. We are a payment-processing system so we care a lot about data transactionality and multi-region availability and so we use a quorum-based write-ahead log to handle serializable transactions. On the downstream side, we've got a bunch of stuff going on.
We were incurring a lot of pain and have many boxes on our diagram. This is getting more and more complicated. The reason we took on this complexity has to do with Metcalfe's law. I'm going to paraphrase the definition and probably corrupt it: it essentially states that the value of a network increases as you add nodes and connections to it. Metcalfe's law was initially intended to apply to communication devices, like adding more peripherals to an Ethernet network.
So, we're getting to a network effect in our data ecosystem. In a post in early 2019, I thought through the implications of Kafka as an escape hatch. You add more systems to the Kafka bus, all of which are able to load their data in and expose it to other systems and slurp up the data of in Kafka, and you leverage this network effect in your data ecosystem.
We found this to be a powerful architecture because the data becomes portable. I'm not saying it'll let you avoid vendor lock-in but it will at least ameliorate some of those concerns. Porting data is usually the harder part to deal with when you're moving between systems. The idea is that it becomes theoretically possible, if you're on Splunk for example, to plug in Elasticsearch alongside it to test it out - and the cost to do so is certainly lower.
Data portability also helps with multi-cloud strategy. If you need to run multiple clouds because you need high availability or you want to pick cloud vendors to save money, you can use Kafka and the Kafka bus to move the data around.
Lastly, I think it leads to infrastructure agility. I alluded to this with my Elasticsearch example but if you come across some new hot real-time OLAP system that you want to check out or some new cache that you want to plug in, having your data already in your streaming platform in Kafka means that all you need to do is turn on the new thing and plug in a sink to load the data. It drastically lowers the cost of testing new things and supporting specialized infrastructure. You can easily plug in things that do one or two things really well, when before you might have had to decide between tradeoffs like supporting a specialized graph database or using an RDBMS which happens to have joins. By reducing the cost of specialization, you can build a more granular infrastructure to handle your queries.
The problems in Stage 3 look a little different. When WePay bought into this integration architecture, we found ourselves still spending a lot of time on fairly manual tasks like those in Figure 12.
Figure 12: WePay’s problems in Stage 3
In short, we were spending a lot of time administering the systems around the streaming platform - the connectors, the upstream databases, the downstream data warehouses - and our ticket load looked like Figure 13.
Figure 13: Ticket load at WePay in Stage 3
Fans of JIRA might recognize Figure 13. It is a screenshot of our support load in JIRA in 2019. It starts relatively low then it skyrockets and it never fully recovered, although there's a nice trend late in the year that relates to the next step of our evolution.
Stage 4: Automation
We started investing in automation. This is something you've got to do when your system gets this big. I think most people would say we should have been automating all along.
You might be ready for Stage 4 if your SREs can't keep up, you're spending a lot of time on manual toil, and you don't have time for the fun stuff.
Figure 14: Stage 4 adds two new layers to the data ecosystem
Figure 14 shows the two new layers that appear in Stage 4. The first is the automation of operations, and this won’t surprise most people. It's the DevOps stuff that has been going on for a long time. The second layer, data-management automation, is not quite as obvious.
Let’s first cover automation for operations. Google’s Site Reliability Engineering handbook defines toil as manual, repeatable, automatable stuff. It's usually interrupt-driven: you're getting Slack messages or tickets or people are showing up at your desk asking you to do things. That is not what you want to be doing. The Google book says, "If a human operator needs to touch your system during normal operations, you have a bug."
But the "normal operations" of data engineering were what we were spending our time on. Anytime you're managing a pipeline, you're going to be adding new topics, adding new data sets, setting up views, and granting access. This stuff needs to get automated. Great news! There's a bunch of solutions for this: Terraform, Ansible, and so on. We at WePay use Terraform and Ansible but you can substitute any similar product.
Figure 15: Some systemd_log thing in Terraform that logs some stuff when you're using compaction (which is an exciting policy to use with your systemd_logs)
Figure 16: Managing your Kafka Connect connectors in Terraform
You can use it to manage your topics. Figures 15 and 16 show some Terraform automations. Not terribly surprising.
Yes, we should have been doing this, but we kind of were doing this already. We had Terraform, we had Ansible for a long time - we had a bunch of operational tooling. We were fancy and on the cloud. We had a bunch of scripts to manage BigQuery and automate a lot of our toil like creating views in BigQuery, creating data sets, and so on. So why did we have such a high ticket load?
The answer is that we were spending a lot of time on data management. We were answering questions like "Who's going to get access to this data once I load it?", "Security, is it okay to persist this data indefinitely or do we need to have a three-year truncation policy?", and "Is this data even allowed in the system?" As a payment processor, WePay deals with sensitive information and our people need to follow geography and security policies and other stuff like that.
We have a fairly robust compliance arm that's part of JPMorgan Chase. Because we deal with credit cards, we have PCI audits and we deal with credit-card data. Regulation is here and we really need to think about this. Europe has GDPR. California has CCPA. PCI applies to credit-card data. HIPAA for health. SOX applies if you're a public company. New York has SHIELD. This is going to become more and more of a theme, so get used to it. We have to get better at automating this stuff or else our lives as data engineers are going to be spent chasing people to make sure this stuff is compliant.
I want to discuss what that might look like. As I get into the futuristic stuff, I get more vague or hand-wavy, but I'm trying to keep it as concrete as I can.
First thing you want to do for automated data management is probably to set up a data catalog. You probably want it centralized, i.e., you want to have one with all the metadata. The data catalog will have the locations of your data, what schemas that data has, who owns the data, and lineage, which is essentially the source and path of the data.
The lineage for my initial example is that it came from MySQL, it went to Kafka, and then it got loaded into BigQuery - that whole pipeline. Lineage can even track encryption or versioning, so you know what things are encrypted and what things are versioned as the schemas evolved.
There's a bunch of activity in this lineage area. Amundsen is a data catalog from Lyft. You have Apache Atlas. LinkedIn open-sourced DataHub as a patch in 2020. WeWork has a system called Marquez. Google has a product called Data Catalog. I know I'm missing more.
These things generally do a lot, more than one thing, but I want to show a concrete example. I yanked Figure 17 from the Amundsen blog. It has fake data, the schema, the field types, the data types, everything. At the right, it has who owns the data - and notice that Add button there.
Figure 17: An example of Amundsen
It tells us what source code generated the data — in this case, it's Airflow, as indicated by that little pinwheel — and some lineage. It even has a little preview. It's a pretty nice UI.
Underneath it, of course, is a repository that actually houses all this information. That's really useful because you need to get all your systems to be talking to this data catalog.
That Add button in the Owned By section is important. You don't as a data engineer want to be entering that data yourself. You do not want to return to the land of manual data stewards and data management. Instead, you want to be hooking up all these systems to your data catalog so that they're automatically reporting stuff about the schema, about the evolution of the schema, about the ownership when the data is loaded from one to the next.
Figure 18: Your data ecosystem needs to talk to your data catalog
First off, you need your systems like Airflow and BigQuery, your data warehouses and stuff, to talk to the data catalog. I think there's quite a bit of movement there.
You then need your data-pipeline streaming platforms to talk to the data catalog. I haven't seen as much yet for that. There may be stuff coming out that will integrate better, but right now I think that's something you’ve got to do on your own.
I don't think we've done a really good job of bridging the gap on the service side. You want your service stuff in the data catalog as well: things like gRPC protobufs, JSON schemas, and even the DBs of those databases.
Once you know where all your data is, the next step is to configure access to it. If you haven't automated this, you're probably going to Security, Compliance, or whoever the policymaker is and asking if this individual can see this data whenever they make access requests - and that's not where you want to be. You want to be able to automate the access-request management so that you can be as hands off with it as possible.
This is kind of an alphabet soup with role-based access control (RBAC), identity access management (IAM), and an access-control list (ACL). Access control is just a bunch of fancy words for a bunch of different features for managing groups, user access, and so on. You need three things to do this: you need your systems to support it, you need to provide tooling to policymakers so they can configure the policies appropriately, and you need to automate the management of the policies once the policymakers have defined them.
There has been a fair amount of work done to support this aspect. Airflow has RBAC, which was a patch WePay submitted. Airflow has taken this seriously and has added a lot more, like DAG-level access control. Kafka has had ACLs for quite a while.
Figure 19: Managing Kafka ACLs with Terraform
You can use tools to automate this stuff. We want to automate adding a new user to the system and configuring their access. We want to automate the configuration of access controls when a new piece of data is added to the system. We want to automate service-account access as new web services come online.
There's occasionally a need to grant someone temporary access to something. You don't want to have to set a calendar reminder to revoke the access for this user in three weeks. You want that to be automated. The same goes for unused access. You want to know when users aren't using all the permissions that they're granted so that you can strip those unused permissions to limit the vulnerability of the space.
Now that your data catalog tells you where all the data is and you have policies set up, you need to detect violations. I mostly want to discuss data loss prevention (DLP) but there's also auditing, which is keeping track of logs and making sure that the activities and systems are conforming to the required policies.
I'm going to talk about Google Cloud Platform because I use it and I have some experience with its data-loss solution. There's a corresponding AWS product called Macie. There's also an open-source project called Apache Ranger, with a bit of an enforcement and monitoring mechanism built into it; that's more focused on the Hadoop ecosystem. What all these things have in common is that you can use them to detect the presence of sensitive data where it shouldn't be.
Figure 20: Detecting sensitive data
Figure 20 is an example. A piece of submitted text contains a phone number, and the system sends a result that says it is "very likely" that it has detected an infoType of phone number. You can use this stuff to monitor your policies. For example, you can run DLP checks on a data set that is supposed to be clean - i.e., not have any sensitive information in it - and if a check finds anything like a phone number, Social Security number, credit card, or other sensitive information, it can immediately alert you that there's a violation in place.
There’s a little bit of progress here. Users can use the data catalog and find the data that they need, we have some automation in place, and maybe we're using Terraform to manage ACLs for Kafka or to manage RBAC in Airflow. But there's still a problem and that is that data engineering is probably still responsible for managing that configuration and those deployments. The reason for that is mostly the interface. We're still getting pull requests, Terraform, DSL, YAML, JSON, Kubernetes ... it's nitty-gritty.
It might be a tall order to ask security teams to make changes to that. Asking your compliance wing to make changes is an even taller order. Going beyond your compliance people is basically impossible.
Stage 5: Decentralization
You're probably ready to decentralize your data pipeline and your data warehouses if you have a fully automated real-time data pipeline but people are still coming to ask you to load data.
Figure 21: Decentralization is Stage 5 of the data ecosystem
If you have an automated data pipeline and data warehouse, I don't think you need a single team to manage all this stuff. I think the place where this will first happen, and we're already seeing this in some ways, is in a decentralization of the data warehouse. I think we're moving towards a world where people are going to be empowered to spin up multiple data warehouses and administer and manage their own.
I frame this line of thought based on our migration from monolith to microservices over the past decade or two. Part of the motivation for that was to break up large, complex things, to increase agility, to increase efficiency, and to let people move at their own pace. A lot of those characteristics sound like your data warehouse: it's monolithic, it's not that agile, you have to ask your data engineering team to do things, and maybe you're not able to do things at your own pace. I think we're going to want to do the same thing - go from a monolith to microwarehouses - and we're going to want a more decentralized approach.
I'm not alone in this thought. Zhamak Dehghani wrote a great blog post that is such a great description of what I’m thinking. She discusses the shift from this monolithic view to a more fragmented or decentralized view. She even discusses policy automation and a lot of the same stuff that I'm thinking about.
I think this shift towards decentralization will take place in two phases. Say you have a set of raw tools - Git, YAML, JSON, etc. - and a beaten-down engineering team that is getting requests left and right and running scripts all the time. To escape that, the first step is simply to expose that raw set of tools to your other engineers. They're comfortable with this stuff, they know Git, they know pull requests, they know YAML and JSON and all that. You can at least start to expose the automated tooling and pipelines to those teams so that they can begin to manage their own data warehouses.
An example of this would be a team that does a lot of reporting. They need a data warehouse that they can manage so you might just give them keys to the castle, and they can go about it. Maybe there's a business-analytics team that's attached to your sales organization and they need a data warehouse. They can manage their own as well.
This is not the end goal; the end goal is full decentralization. But for that we need much more development of the tooling that we're providing, beyond just Git, YAML, and the RTFM attitude that we sometimes throw around.
We need polished UIs, something that you can give not only to an engineer who’s been writing code for 10 years but to almost anyone in your organization. If we can get to that point, I think we will be able to create a fully decentralized warehouse and data pipeline where Security and Compliance can manage access controls while data engineers manage the tooling and infrastructure.
This is what Maxime Beauchemin meant by "... data engineers build tools, infrastructure, frameworks, and services." Everyone else can manage their own data pipelines and their own data warehouses and data engineers can help them do that. There’s that key word "help" that I drew attention to at the beginning.
Figure 21 is my view of a modern decentralized data architecture. We have real-time data integration, a streaming platform, automated data management, automated operations, decentralized data warehouses and pipelines, and happy engineers, SREs, and users.
About the Author
Chris Riccomini is a software engineer with more than a decade of experience at Silicon Valley tech companies including PayPal, LinkedIn, and WePay, a JPMorgan Chase company. Over his career, Riccomini has held titles as data scientist, staff software engineer, and distinguished software engineer. He has managed engineering teams in the distributed systems and payments space. Riccomini is active in open source. He is co-author of Apache Samza, a stream-processing framework, and is also a member of the Apache project management committee (PMC), Apache Airflow PMC, and Apache Samza PMC. He is a strategic investor and advisor for startups in the data space, where he advises founders and technical leaders on product and engineering strategy.
The world of data science is evolving, and it's changing rapidly. In the good old days, all your data was readily available in a single database and all you needed to know as a data scientist was some R or Python to build simple scripts. I, for one, remember setting up an R script, making it munch some data from a single table, and spit out some markdown reports, all glued together in a lonely CRON job. But we all must learn a precious lesson: data grows.
As companies grow, more and more data sources inevitably get added. Some of those data arrive in batches and others stream in through various channels: terabytes, petabytes of data accumulating so quickly, your head feels like it’ll explode. Some of you might've recognized this years ago when you moved into a new role as a data engineer, tasked with storing data safely and correctly. If you find yourself in this sticky situation, or if you’re just getting started as a data engineer, I have some good news for you: This article provides you with all the resources you need to learn data engineering. A lot of these resources are bundled in DataCamp’s career track for data engineers.
Data engineering is essential for data-driven companies, but what do data engineers actually do? The following analogy may help:
Think of data engineers like crop farmers, who ensure that their fields are well maintained and the soil and plants are healthy. They’re responsible for cultivating, harvesting, and preparing their crops for others. This includes removing damaged crops to ensure high quality and high yielding crops.
This is similar to the work data engineers do to ensure clean, raw data can be used by other people in their organization to make data-driven business decisions.
1. Become proficient at programming
Before we dive into the tools you’ll need, you have to understand that data engineers lay at the intersection of software engineering and data science. If you want to become a data engineer, you’ll need to first become a software engineer. So you should start brushing up on foundational programming skills.
The industry standard mostly revolves around two technologies: Python and Scala.
Learn Python
With Python programming, it's essential that you not only know how to write scripts in Python, but that you also understand how to create software. Good software is well-structured, tested, and performant. That means you should use the right algorithm for the job. These courses lay out a path to become a Python programming rockstar:
To deepen your knowledge of Python even further, take our new skill track on Coding Best Practices with Python. The knowledge you build in these courses will give you a strong foundation of writing efficient and testable code.
Learn the basics of Scala
A lot of tooling in the data engineering world revolves around Scala. Scala is built on strong functional programming foundations and a static typing system. It runs on the Java Virtual Machine (or JVM), which means it’s compatible with the many Java libraries available in the open-source community. If this sounds intimidating, we can ease you in with our course Introduction to Scala.
2. Learn automation and scripting
Why automation is crucial for data engineers
Data engineers must understand how to automate tasks. Many tasks you need to perform on your data may be tedious or may need to happen frequently. For example, you might want to clean up a table in your database on an hourly schedule. This comic by xckd says it best:
Tl;dr: If you know an automatable task takes a long time, or it needs to happen frequently, you should probably automate it.
Essential tools for automation
Shell scripting is a way to tell a UNIX server what to do and when to do it. From a shell script, you can start Python programs or run a job on a Spark cluster, for example.
CRON is a time-based job scheduler that has a particular notation to mark when specific jobs need to be executed. The best way to illustrate how it works is by giving you some examples:
Here’s an awesome website that can help you figure out the correct schedule: https://crontab.guru/. If you can’t wait to get started on shell scripting and CRON jobs, get started with these courses:
Later in this post, I’ll talk about Apache Airflow, which is is a tool that also relies on your scripting capabilities to schedule your data engineering workflows.
3. Understand your databases
Start by learning SQL basics
SQL is the lingua franca of everything related to data. It's a well-established language and it won’t be going away anytime soon. Have a look at the following piece of SQL code:
What’s so beautiful about this SQL code is that it's a declarative language. This means that the code describes what to do, not how to do it—the “query plan” takes care of that part. It also means that almost anyone can understand the piece of code I wrote here, even without prior knowledge of SQL: Return how many distinct IP addresses are used for all logins from each user.
SQL has several dialects. As a data engineer, you don't necessarily need to know them all, but it may help to have some familiarity with PostgreSQL and MySQL. Intro to SQL for Data Science gives a gentle introduction on using PostgreSQL, and Introduction to Relational Databases in SQL goes into more detail.
Learn how to model data
As a data engineer, you also need to understand how data is modeled. Data models define how entities in a system interact and what they’re built out of. In other words, you should be able to read database diagrams, like this one:
You should recognize techniques like database normalization or a star schema. A data engineer also knows that some databases are optimized for transactions (OLTP), and others are better for analysis (OLAP). Don’t worry if these data modeling topics don’t ring a bell yet—our Database Design course covers all of them in detail.
Learn how to work with less structured data
Sometimes you'll find yourself in a situation where data is not represented in a structured way, but is stored in a less structured document database like MongoDB. It definitely helps to know how to extract data from these. Our Introduction to MongoDB in Python course can help you with that.
4. Master data processing techniques
So far, I’ve only covered the fundamentals of knowing how to program and automate tasks, and how to leverage SQL. Now it’s time to start building on top of that. Since you now have a strong foundation, the sky's the limit!
Learn how to process big data in batches
First, you need to know how to get your data from several sources and process it: This is called data processing. If your datasets are small, you might get away with processing your data in R with dplyr or in Python with pandas. Or you could let your SQL engine do the heavy lifting. But if you have gigabytes or even terabytes of data, you’d be better off taking advantage of parallel processing. The benefits of using parallel processing are two-fold: (1) You can use more processing power, and (2) you can also make better use of the memory on all of the processing units.
The most commonly used engine for parallel processing is Apache Spark, which according to their website is a unified analytics engine for large-scale data processing. Let me decode that for you: Spark provides an easy-to-use API by using common abstractions like DataFrames to do parallel processing tasks on clusters of machines.
Spark manages to significantly outperform older systems for parallel processing like Hadoop. It’s written in Scala, and it helps that it interfaces with several popular programming languages like Python and R. Lesser-known tools like Dask can be used to solve similar problems. Check out the following courses if you want to learn more:
Data processing often happens in batches, like when there’s a scheduled daily cleaning of the prior day’s sales table. We call this batch processing because the processing operates on a collection of observations that occurred in the past.
Learn how to process big data in streams
In some cases, you might have a continuous stream of data that you want to process right away, known as stream processing. An example is filtering out mentions of specific stocks from a stream of Tweets. In this case, you might want to look into other data processing platforms like Apache Kafka or Apache Flink, which are more focused on processing streams of data. Apache Spark also has an extension called Spark Streaming to do stream processing. If you want to learn more about stream processing with Kafka or Flink, check out this gentle introduction.
Finally, with the scheduled data processing job in place, you’ll need to dump the result in some kind of database. Often, the target database after data processing is an MPP database. We'll see some examples of MPP databases in the upcoming section about cloud computing. They’re basically databases that use parallel processing to perform analytical queries.
5. Schedule your workflows
Workflow scheduling with Apache Airflow
Once you’ve built the jobs that process data in Spark or another engine, you’ll want to schedule them regularly. You can keep it simple and use CRON, as discussed earlier. At DataCamp, we choose to use Apache Airflow, a tool to schedule workflows in a data engineering pipeline. You should use whichever tool is best suited for your workflow. A simple CRON job might be enough for your use case. If the CRON jobs start adding up and some tasks depend on others, then Apache Airflow might be the tool for you. Apache Airflow has the added benefit of being scalable as it can run on a cluster using Celery or Kubernetes—but more on this later. Apache Airflow visualizes the workflows you author using Directed Acyclic Graphs, or DAGs:
The above DAG demonstrates the steps to assemble a car.
Tl;dr: You can use Airflow to orchestrate jobs that perform parallel processing using Apache Spark or any other tool from the big data ecosystem.
The ecosystem of tools
Speaking of tools, it’s easy to get lost in all the terminology and tools related to data engineering. I think the following diagram illustrates that point perfectly:
This diagram is very complete, but it’s not very helpful in our case. Instead of overwhelming you with an excess of information, let’s wrap up the past two sections with an illustration that should help you make sense of all of the tools that I’ve presented.
A lot of the more popular tools, like Apache Spark or Apache Airflow, are explained in more detail in our Introduction to Data Engineering course.
An observant reader might see a pattern emerging in these open-source tools. Indeed, a lot of them are maintained by the Apache Software Foundation. In fact, many of Apache’s projects are related to big data, so you might want to keep an eye out for them. The next big thing might be on its way! All of their projects are open source, so if you know some Python, Scala, or Java, you might want to have a peek at their GitHub organization.
6. Study cloud computing
The case for using a cloud platform
Next, I want you to once again think about parallel processing. Remember that in the previous section, we talked about clusters of computers. In the old days, companies that needed to handle big data would have their own data center or would rent racks of servers in a data center. That worked, and a lot of companies still do it this way if they handle sensitive data, such as banks, hospitals, or public services. The drawback of this setup is that a lot of server time goes to waste. Here’s why: Let’s say you have to do some batch processing once a day. Your data center needs to handle the peak in processing power, but the same servers would sit idle the rest of the time.
This is clearly not efficient. And we haven’t even talked about geo-replication yet, where the same data needs to be replicated in different geographical locations to be disaster-proof.
The impracticality of every company managing their servers themselves was the problem that gave rise to cloud platforms, which centralize processing power. If one customer has idle time, another might be having a peak moment, and the cloud platform can distribute processing power accordingly. Data engineers today need to know how to work with these cloud platforms. The most popular cloud platforms for companies are Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP).
Common services provided by cloud platforms
Cloud platforms provide all kinds of services that are useful to data engineers. AWS alone offers up to 165 services. Let me highlight a few:
Cloud storage: Data storage forms the foundation for data engineering. Each cloud platform provides its version of cheap storage. AWS has S3, Microsoft has Azure Storage, and Google has Google Storage. They all pretty much do the same thing: store a ton of files.
Computation: Each cloud platform also has its own low-level computation service. This means they provide a remote machine to do computations on. AWS has EC2, Azure has Virtual Machines, and Google has its Compute Engine. They’re configured differently, but they pretty much do the same thing.
Cluster management: All cloud platforms have their version of a managed cluster environment. AWS has EMR, Azure hosts HDInsight, and Google has Cloud Dataproc.
MPP databases: Massively parallel processing databases is a fancy term for databases that run over several machines and use parallel processing to do expensive queries. Common examples include AWS Redshift, Azure SQL Data Warehouse, and Google BigQuery.
Fully managed data processing: Each cloud platform has data processing services that operate the infrastructure/DevOps setup for you. You don’t have to worry about how many machines to use in your cluster, for example. AWS has something called Data Pipelines, and there’s also Azure Data Factory and Google Dataflow. Google has open-sourced the programming model of Dataflow into another Apache project: Apache Beam.
That’s just a small subset of relevant services for data engineers. If any of this has piqued your interest, be sure to check out the first chapter of Introduction to Data Engineering, which has a lesson on cloud platforms. If you want to get more hands-on experience, check out Introduction to AWS Boto in Python.
7. Internalize infrastructure
You might be surprised to see this here, but being a data engineer means you also need to know a thing or two about infrastructure. I’m not going to go into too much detail on this topic, but let me tell you about two crucial tools: Docker and Kubernetes.
When to use Docker
First, let’s see if you recognize this situation:
You probably know where I’m going with this. Mastering Docker can help you make applications reproducible on any machine, no matter what the specifications of that machine are. It’s containerization software that helps you create a reproducible environment. It allows you to collaborate in teams and ensures that any application you make in development will work similarly in production. With Docker, data engineers waste considerably less time setting up local environments. Take Apache Kafka as an example, which can be pretty overwhelming to set up locally. You can use a platform called Confluent that packages Kafka along with other useful tools for stream processing, and the Confluent documentation provides an easy-to-follow guide on how to get started using Docker.
When to use Kubernetes
What logically follows single containers is a whole bunch of containers running on several machines. This is called container orchestration in infrastructure jargon, and Kubernetes is the tool to use. You might rightfully be reminded of parallel processing and tools like Apache Spark here. In fact, you can use a Kubernetes managed cluster with Spark. This is one of the more advanced topics in data engineering, but even newbies should be aware of it.
8. Follow the trends
If you’ve made it this far, don’t get discouraged if you feel that you don’t have a full understanding of the data engineering landscape. It’s a huge field that’s constantly changing. What’s most important is to use the right tool for the job, and to not overcomplicate the big data solutions you build.
That said, it doesn’t hurt to keep up with recent developments. Here’s a handful of useful resources:
That's it! You're at the end of the road. At this point, you're practically a data engineer… But you must apply what you've learned. Building experience as a data engineer is the hardest part. Luckily, you don’t need to be an expert in all of these topics. You could specialize in one cloud platform, like Google Cloud Platform. You could even start your first pet project using one of Google’s public BigQuery datasets.
I hope you feel inspired by this blog post, and that the resources I provided are useful to you. At DataCamp, we’re committed to building our data engineering curriculum and adding courses on topics like streaming data and cloud platforms—so stay tuned!
When implementing Event Sourced system, at some point, you will need to answer two major questions:
Where to store events?
How to store events?
I will address the first question in a separate post. Right now I want to focus on the second one, and try to analyze different serialization strategies for events (and snapshots) to store and read them effectively.
Probably the majority of people would choose JSON — a pretty obvious preference. It’s easy to implement, supported natively by some DBs like MongoDB, PostgreSQL, and others. Any technology stack is able to parse or produce it. JSON format is heavily used and has many advantages. It is the best option as a default choice.
What if I tell you that there are other solutions? No, no, no, I’m not thinking about XML or YAML. Instead, there are plenty of binary serialization options. Let’s start with plain text vs. binary serialization comparison.
The first advantage of plain text serialization is that this format is human-readable. It’s quite convenient to select some events from DB and analyze them directly from the query result. A binary format requires an additional step, where bytes are transformed into something readable. Is it a real problem? Usually, when you need to debug something on production you will very quickly create a tool to do this for you automatically. Of course, when choosing binary format, such tool should be created upfront, but there’s no better motivation than an issue in production.
Plain text formats, especially JSON, have some problems with number precision, just follow these links: JSON IEEE 754, DoS.
The binary format is, obviously, more compressed, so storage usage will be lower. The actual profit depends on your events model. In my case, it was 60% to 70% savings on disk space. Someone could argue that storage is cheap and this point is irrelevant. In general, that’s true, storage prices usually go down, but at the same time, the amount of data we are gathering is growing even faster. Not to mention data replication, required by some databases like Cassandra. Also if you check the prices of fast and large SSD drives for DB purposes the actual costs are not so small in my opinion.
All benchmarks, at the time of writing this post, are pretty consistent, binary serialization is much faster than plain text serialization. This shouldn’t be a surprise since plain text formats require a lot of additional steps when serializing or deserializing something (check here and here for some details).
The last point is schema evolution support. This is an extremely important feature for serializing and deserializing events, because it will allow you to version your events without typical versioning pains (postfixes like _V1, migrations, version adapters, etc.).
Clearly, binary protocols are a pretty enticing option, but which one should we choose:
Java serialization
Kryo
Thrift
Protocol Buffers
Avro
…
Java serialization — let’s not waste time on this horrible mistake.
Kryo — very fast, very compact, but it works only on JVM, there is no point in limiting our infrastructure to only JVM applications. Maybe some crazy NodeJS developer would also like to read our events.
Thrift — from Facebook, almost the same when it comes to functionalities as Google’s Protocol Buffers, but subjectively Protobuf is easier to use. Documentation is very detailed and extensive. Support and tools for Java and Scala are on a very good level. That’s why I have chosen Protocol Buffer vs Avro (from Hadoop) for the final comparison.
The detailed comparison can be found in this great post by Martin Kleppmann, so there is no point in describing these technologies once again, but let’s focus on aspects like speed, size and schema evolution.
The difference is so small that it can be ignored. Both solutions are really fast.
Serialized payload size is more or less the same for both technologies.
A tie again. Avro and Protocol Buffers will provide you full compatibility support. Backward compatibility is necessary for reading the old version of events. Forward compatibility is required for rolling updates — at the same time old and new version of events can be exchanged between micro service instances.
Adding, removing or even renaming a field is not a problem. From my perspective, it will be easier to achieve in Protocol Buffers (aliases and default values in Avro might be sometimes quite challenging), but it is possible in both options.
What is the difference then? Well, the biggest difference between Avro and Protocol Buffers is how they deal with schema management. In Protobuf you need to create a schema first. Then, using tools like ScalaPB you will compile the schema and generate Scala/Java classes (as well as parsers and serializers for them).
And now the only thing you need to do is to translate your rich domain event to the Protobuf version.
Avro approach is at first glance much simpler. From your rich domain event you can generate Avro schema thanks to libraries like avro4s.
Of course, some custom mappings to understand types like UserId, Email, etc. will be necessary. Similarly to most of the JSON mappings frameworks.
What should you choose then? Avro, especially at the beginning, seems much easier to use. The cost of this is that you will need to provide both reader and writer schema to deserialize anything.
Sometimes this might be quite problematic. That’s why tools like Schema Registry were developed.
The next problem you might face with Avro is the overall impact on your domain events. At some point, Avro can leak into your domain. Some constructions like e.g. a map with keys that are not strings are not supported in Avro model. When the serialization mechanism is forcing you to change something in your domain model — it’s not a good sign.
With Protocol Buffers, schema management is much simpler — you just need schema artifact, which can be published as any other artifact to your local repository. Also, your domain can be perfectly separated from the serialization mechanism. The cost is the boilerplate code required for translation between domain and serialization layers.
Personally, I would use Avro for simple domains with mostly primitive types. For rich domains, with complex types and structures, I’ve been using Protocol Buffers for quite some time. Clean domain with no serialization influence is really worth paying the boilerplate code price.
Kafka serialisation schemes — playing with AVRO, Protobuf, JSON Schema in Confluent Streaming Platform. The code for these examples available at https://github.com/saubury/kafka-serialization
Apache Avro was has been the default Kafka serialisation mechanism for a long time. Confluent just updated their Kafka streaming platform with additional support for serialising data with Protocol buffers (or protobuf) and JSON Schema serialisation.
Kafka with AVRO vs., Kafka with Protobuf vs., Kafka with JSON Schema
Protobuf is especially cool, and offers up some neat opportunities beyond what was possible in Avro. The inclusion of Protobuf and JSON Schema applies at producer and consumer libraries, schema registry, Kafka connect, ksqlDB along with Control Center. It’s worth a few minutes of your time getting familiar with the new opportunities possible with a choice of serialising strategies for your streaming application.
So why bother with serialising structured data? Let’s start with an example data string … “cookie,50,null” What does this data mean? Is cookie a name, a place or something to eat? And what about 50 - is this an age, a temperature or something else?
If you were using a database (such as Postgres or Oracle) to store your data you would create a table definition (with nicely named columns and appropriate data types). The same is true for your streaming platform — you really should pick data format for serialising structured data. Bonus points for and being consistent across your data platform!
Until recently your choices for serialising structured data within Kafka were limited. You had “bad” choices (such as free text or CSV) or the “better” choice of using Apache Avro. Avro is an open source data serialisation system which marshals your data (and it’s appropriate schema) to a efficient binary format. One of the core features of Avro is the ability to define a schema for our data. So our data cookie,50,null would be associated with a snack Avro schema like this
Here we can see our data cookie,50,null is snack data (the most important type of data). We can see cookie is a string representing the name of the snack. Our schema offers us a lot of flexibility (our schema can evolve over time) plus ensures data integrity (for example, ensuring calories are integers ).
Although most of Apache Kafka users use Apache Avro to define contracts for their messages, it’s always been a bit of a “Java thing”. Classes automatically generated by the Apache Avro compiler favour JVM developers. You can certainly use AVRO in pretty much any language, however, Google Protocol Buffer (protobuf) is very popular for serialising, de-serialising and validating data in other languages (Python, Rust, Ruby, Go).
This isn’t a blog on the “best” serialisation strategy. However, let’s get familiar with how we can use new choices for serialising structured data
We’ll run through a few examples. Remember our initial set of yummy data looks like this saved in the file snacks.txt
Let’s remind ourselves how to encode our snacks using AVRO serialisation. We’ll use the include command line tool kafka-avro-console-producer as a Kafka producer which can perform serialisation (with a schema provided as a command line parameter). A producer is something that writes data into a Kafka broker.
And to read the data, we can use the kafka-avro-console-consumercommand line application to act as kafka consumer to read and de-serialising our AVRO data
This time we’ll use protobuf serialisation with the new kafka-protobuf-console-producer kafka producer. The concept is similar to to approach we took with AVRO, however this time our Kafka producer will can perform protobuf serialisation. Note the protobuf schema is provided as a command line parameter.
Finally we’ll use JSON Schema serialisation with the new kafka-json-schema-console-producer kafka producer. Note the json-schema schema is provided as a command line parameter.
Let’s have a look at a more complex modelling example to illustrate some of the possibilities with Protobuf schemas. Imagine we want to model a meal and describe the ingredients within the meal. We can use a protobuf schema to describe a meal such as a taco is composed of beef filling and cheese topping.
Our data for a taco and fish-and-chips meal could look like this
Which gives you an idea of how flexible data representation can be using protobuf in Kafka. But what happens if we need to make changes to these schemas?
You may have wondered where the schemas actually went in the above examples? The Confluent Schema Registry has been diligently storing these schemas (as part of the serialisation process when using kafka-blah-console-producer). That is, both the schema name (eg., SNACKS_PROTO-value), schema content, version and style (protobuf, Avro) have all been stored. We can peak at the stored schema with curl. For example, to explore the recently used protobuf schema for our snacks
As the saying goes — the only constant is change. Any good data platform needs to accommodate changes — such as additions or changes to a schema. Supporting schema evolution is a fundamental requirement for a streaming platform, so our serialization mechanism also needs to support schema changes (or evolution). Protobuf and Avro both offer mechanisms for updating schema without breaking downstream consumers — which could still be using a previous schema version.
Tacos and pizza’s sound great — but let’s have something to drink with our meal! We can now add some additional attributes to our schema to include meals. This is sometimes called schema evolution. Note we’ll continue to use the existing MEALS_PROTO topic.
One nice inclusion with the Confluent Control Center (the Web GUI included in the Confluent platform) is the ability to look at schemas, and the differences between schemas. For example, we can see version 1 and version 2 of the MEALS_PROTO-value schema
Schema difference with Confluent Control Center
Let us now build an application demonstrating protobuf classes. To generate protobuf classes you must first install the protobuf compiler protoc. See the protocol buffer docs for instructions on installing and using protoc.
We can compile our Python schema like this. This will take our schema (from meal.proto) and will create the meal_pb2.py Python class file.
protoc -I=. --python_out=. ./meal.proto
Excellent, with the meal_pb2.py Python class file you can now build protobuf classes and produce into Kafka with code like this
Have a look at producer-protobuf.py for a complete example of protobuf Kafka producer in Python.
Kafka continues to grow in capabilities, and having the options of AVRO, Protobuf, JSON Schema use within the Confluent Platform gives even more opportunities to build cool streaming applications
Apache Avro was has been the defacto Kafka serialization mechanism for a long time. Confluent just updated their Kafka streaming platform with additioinal support for serializing data with Protocol buffers (or protobuf) and JSON Schema serialization.
Protobuf is especially cool, and offers up some neat opportunities beyond what was possible in Avro. The inclusion of Protobuf and JSON Schema applies at producer and consumer libraries, schema registry, Kafka connect, ksqlDB along with Control Center.
Do I care about serializing structured data?
So why bother with serializing structured data? Let's start with an example data string ... cookie,50,null. What does this data mean? Is cookie a name, a place or something to eat? And what about 50 - is this an age, a temperature or something else?
If you were using a database (such as Postgres or Oracle) to store your data you would create a table definition (with nicely named columns and appropriate data types). The same is true for your streaming platform - you really should pick data format for serializing structured data. Bonus points for and being consistent across your data platform!
Until recently your choices for serializing structured data within Kafka were limited. You had "bad" choices (such as free text or CSV) or the "better" choice of using Apache Avro. Avro is an open source data serialization system which marshals your data (and it's appropriate schema) to a efficient binary format. One of the core features of Avro is the ability to define a schema for our data. So our data cookie,50,null would be associated with a snack Avro schema like this
Here we can see our data cookie,50,null is snack data (the most important type of data). We can see cookie is a string representing the name of the snake. Our schema offers us a lot of flexibility (our schema can evolve over time) plus ensures data integrity (for example, ensuring calories are integers ).
Although most of Apache Kafka users use Apache Avro to define contracts for their messages, it's always been a bit of a "Java thing". Classes automatically generated by the Apache Avro compiler favor JVM developers. You can certainly use AVRO in pretty much any language, however, Google Protocol Buffer (protobuf) is very popular for serializing, deserializing and validating data in other languages (Python, Rust, Ruby, Go).
This isn't a blog on the "best" serialization strategy. However, let's get familiar with how we can use new choices for serializing structured data
Let's remind ourselves how to encode our snacks using AVRO serialization. We'll use the include command line tool kafka-avro-console-producer as a Kafka producer which can perform serialization (with a schema provided as a command line parameter). A producer is something that writes data into a Kafka broker.
And to read the data, we can use the kafka-avro-console-consumer command line application to act as kafka consumer to read and de-serializing our AVRO data
This time we'll use protobuf serialization with the new kafka-protobuf-console-producer kafka producer. The concept is similar to to approach we took with AVRO, however this time our Kafka producer will can perform protobuf serialization. Note the protobuf schema is provided as a command line parameter.
Finally we'll use JSON Schema serialization with the new kafka-json-schema-console-producer kafka producer. Note the json-schema schema is provided as a command line parameter.
You may have wondered where the schemas actually went in the above examples? The Confluent Schema Registry has been diligently storing these schemas (as part of the serialization process when using kafka-blah-console-producer). That is, both the schema name (eg., SNACKS_PROTO-value), schema content, version and style (protobuf, Avro) have all been stored. We can peak at the stored schema with curl. For example, to explore the recently used protobuf schema for our snacks
curl -s -X GET http://localhost:8081/subjects/SNACKS_PROTO-value/versions/1
Let's have a look at a more complex modeling example to illustrate some of the possibilities with Protobuf schemas. Imagine we want to model a meal and describe the ingredients within the meal. We can use a protobuf schema to describe a meal such as a taco is composed of beef filling and cheese topping.
Our data for a taco and fish-and-chips meal could look like this
Which gives you an idea of how flexible data representation can be using protobuf in Kafka. But what happens if we need to make changes to these schemas?
As the saying goes - the only constant is change. Any good data platform needs to accommodate changes - such as additions or changes to a schema. Supporting schema evolution is a fundamental requirement for a streaming platform, so our serialization mechanism also needs to support schema changes (or evolution). Protobuf and Avro both offer mechanisms for updating schema without breaking downstream consumers - which could still be using a previous schema version.
Adding drinks to our meals
Tacos and pizza's sound great - but let's have something to drink with our meal! We can now add some additional attributes to our schema to include meals. This is sometimes called schema evolution. Note we'll continue to use the existing MEALS_PROTO topic.
Visualizing schema difference with Confluent Control Center
One nice inclusion with the Confluent Control Center (the Web GUI included in the Confluent platform) is the ability to look at schemas, and the differences between schemas. For example, we can see version 1 and version 2 of the MEALS_PROTO-value schema
Let us now build an application demonstrating protobuf classes. To generate protobuf classes you must first install the protobuf compiler protoc. See the protocol buffer docs for instructions on installing and using protoc.
Python compile schema
protoc -I=. --python_out=. ./meal.proto
This will create the meal_pb2.py Python class file. You can now build protobuf classes and produce into Kafka with code like this
Kafka continues to grow in capabilities, and having the options of AVRO, Protobuf, JSON Schema use within the Confluent Platform gives even more opportunities to build cool streaming applications
The code for these examples available at
https://github.com/saubury/kafka-serialization
Service design for networked business models - presentation at Servic…
Successfully reported this slideshow.
Service design for networked business models - presentation at Service Design Network Conference, Cardiff, 2013
1.
Service design for networked business models
Service Design Network Conference, Cardiff, 2013
Aldo de Jong, Co-Founder, Claro Partners
2.
Share
your
thoughts?
@claropartners
#sharingeconomy
#sdn13
3.
Claro
helps
corpora7ons
and
startups
to
navigate
disrup1ve
shi3s
in
society
and
business
Michael
Elisabeth
Mwenge
NL
USA
Congo
Rich
USA
Megan
UK
Mercè
Spain
Aldo
NL
Mandy
Lebanon
Abby
USA
Jiri
Belgium
Gunes
Turkey
4.
We
deliver
business
innova1on
and
service
design
in
the
context
of
disrup1ve
shi3s:
Ownership
Services
delivered
by
companies
Big
data
and
aggregated
resources
Internet
of
informa7on
and
people
Access
Services
enabled
through
networks
Small
data
and
personalised
experiences
Internet
of
Things
Request our point of view paper on each of
these topics at POV@claropartners.com
5.
The
basis
of
our
thinking:
two
6-‐month
global
open-‐innova1on
projects
USA
UK
DENMARK
SPAIN
INDIA
CHINA
JAPAN
BRAZIL
46
Stakeholder
interviews
99
Ethnographic
sessions
39
Expert
interviews,
plus
secondary
research
Business
Perspec7ve
+
People
Perspec7ve
+
Systems
Perspec7ve
Collabora7ve
and
individual
workshops
with
the
par7cipa7ng
companies
6.
The
burden
of
ownership
is
challenging
the
consump1on
economy
+
JOY
OF
OWNERSHIP
BURDEN
OF
OWNERSHIP
-‐
TIME
Acquisi1on
Use
Post-‐use
7.
The
sharing
economy
emerges
from
communi1es
of
exchange.
Trust
between
strangers
is
a
new
currency.
EXCHANGE
MY
STUFF
YOUR
STUFF
8.
Networked
business
models
are
disrup1ng
tradi1onal
ways
of
doing
business
GiffGaff
has
only
34
employees
and
the
average
response
7me
to
a
customer
problem
is
under
90
seconds
At
its
peak,
Encarta
had
62,000+
ar7cles.
Its
highly
centralised
control
contributed
to
its
failure
As
of
May
2011,
Wikipedia
had
3.5m+
ar7cles
in
English,
and
18m
in
all
its
261
languages
Microso3
has
35.000
engineers
and
designers
who
build
solu7ons
based
on
understanding
customer
needs
50.000
Quickbase
users
exchange
soXware
solu7ons
and
knowledge
with
people
like
them
9.
We
call
them
Par1cipatory
Service
Networks
(PSN)
A
system
where
value
is
co-‐
created
and
exchanged
in
a
distributed
way
by
a
network
of
par7cipants.
• Networked
business
models
• Value
exchange
networks
• Collabora8ve
consump8on
• P2P
services
• Bo>om-‐up
value
exchange
• Sharing
economy
• User-‐generated
content
etc.
10.
Landscape
of
Alterna7ve
Models
of
Ownership
&
Value
Exchange
claropartners.com
>
login
u:
pdfa
pw:
ownership
11.
The
access
economy
drives
new
business
models
t Rental
fee
Car
use
u
t Payment
Car
u
CONSUMER
CAR
USER
12.
Par1cipatory
Service
Networks
(PSNs)
are
networked
business
models
t Payment
Car
u
CONSUMER
CAR
OWNER
Car
use
u
t
Rental
fee
CAR
USER
13.
PSNs
are
difficult
for
tradi1onal
business
to
adopt
PARTICIPATORY
SERVICE
NETWORKS
TRADITIONAL
BUSINESSES
Centralised
value
crea7on
VALUE
CREATION
Decentralised
value
co-‐crea7on
Command
and
control
chain
CONTROL
Diffusion
of
control
High
on
the
company
side
CAPITAL
INTENSIVENESS
Low
on
the
company
side
Limits
to
scale,
speed
and
localness
AFFORDABILITY
Unaffordable
projects
now
possible
Aims
at
efficient
labour
use
LABOUR
EFFICIENCY
Redundancy
and
flexibility
of
roles
Resource
alloca7on
RESOURCES
Resource
aarac7on
Clear
role
responsibili7es
ACCOUNTABILITY
Diffusion
of
responsibility
Highly
designed
and
planned
DESIGNED
vs.
ORGANIC
Organic
characteris7cs
14.
Networked
business
models
calls
for
a
shi3
in
approach
to
service
design
1. Focus
on
the
individual
customer
2. Design
and
deliver
a
service
3. Own
a
unique
rela1onship
with
customer
Tradi1onal
service
models
Networked
business
models
1. Uncover
opportuni1es
in
a
network
2. Enable
exchanges
to
deliver
service
3. Iden1fy
your
role
in
the
ecosystem
15.
1. Uncover
opportuni1es
in
a
network
2. Enable
exchanges
to
deliver
service
3. Iden7fy
your
role
in
the
ecosystem
16.
1.
Uncover
opportuni1es
in
a
network
Individual
customer
What
do
they
need?
Network
What
do
they
need?
What
do
they
have?
17.
1.
Uncover
opportuni1es
in
a
network
Key
ques1ons:
• How
are
people
connected
within
the
network?
• What
is
their
mo8va8on
to
join?
• What
are
the
values
exchanged?
18.
1.
Uncover
opportuni1es
in
a
network
How
are
people
connected
within
the
network?
19.
ust
t of
A
e
circumstances the introduction of money can turn collaborative dynamics
1.
Uncover
opportuni1es
in
a
network
into more competitive ones where structural loops are lost and the
What
is
their
mo1va1on
to
join?
course, this competition can be integral
network becomes fragmented. Of
to the model, but at other times it can limit value exchange to a simple,
calculated transaction, especially in cases where the network is dependent
upon people’s creativity.
TANGIBLE
BENEFIT
SMALL
+
TOKEN-‐LIKE
SIMPLE,
QUICK
TRANSACTION
pes
or
m
ose)
The motivations to join a PSN are very different then the motivations to actively contribute. Initially,
20.
1.
Uncover
opportuni1es
in
a
network
What
are
the
values
exchanged?
RK
odes – or actors
pes of value
ns can be identified
change network.
AIM
mon, highs the type of
d is its reason for
pants is crucial for
ompany to apply
o take advantage
ness activates that
gh a PSN initially,
Connections
Knowledge
connecting to others
sharing knowledge with others
“By connecting with each other, we
heighten our identity as KoreanAmericans.”
“By connecting with each
other, we will find out things
we want to know about new
technologies.”
Competencies
Resources
using skills to create value
providing funds/resources to others
“By connecting to each other we
can trade children’s clothes to save
money.”
“By connecting with each other, we
can learn how to speak each other’s
language.”
A company can effectively map the value present both within their business and within their
wider network (of customers, partners, suppliers) into four loose groups: resources, knowledge,
competencies and connections.
21.
1. Uncover
opportuni7es
in
a
network
2. Enable
exchanges
to
deliver
service
3. Iden7fy
your
role
in
the
ecosystem
22.
2.
Enable
exchanges
to
deliver
service
Design
the
service
journey
Design
par1cipatory
services
Single
user
experience
Networked
experience
23.
2.
Enable
exchanges
to
deliver
service
Key
ques1ons:
• How
is
value
exchanged?
• How
to
design
for
a
networked
experience?
• How
to
enable
interac8ons
and
exchanges?
• How
to
encourage
contribu8ons?
24.
2.
Enable
exchanges
to
deliver
service
How
is
value
exchanged?
Short
descrip1on:
RelayRides
is
a
P2P
car
rental
service.
Private
car
owners
to
rent
out
their
cars
for
money,
to
drivers
looking
for
a
close,
and
affordable
way
to
rent
cars.
How
is
the
value
exchanged?
Sketch
out
the
network
Who
is
exchanging
value?
Car
users,
car
owners
and
RelayRides
What
value
is
exchanged?
Car,
money,
members,
insurance
CAR
OWNER
Car
use
u
t
Rental
fee
USER
25.
2.
Enable
exchanges
to
deliver
service
How
do
we
design
for
a
networked
experience?
t
Rental
fee
Car
use
u
USER
DISCOVER
JOIN
USE
EXTEND
26.
2.
Enable
exchanges
to
deliver
service
How
do
we
design
for
a
networked
experience?
DISCOVER
JOIN
USE
EXTEND
27.
2.
Enable
exchanges
to
deliver
service
How
do
we
enable
interac1ons
and
exchanges?
Trust
in
the
value
exchanged
Trust
in
the
plahorm
Trust
in
other
network
par7cipants
28.
2.
Enable
exchanges
to
deliver
service
How
to
encourage
contribu1ons?
CONTRIBUTION"
29.
1. Uncover
opportuni7es
in
a
network
2. Enable
exchanges
to
deliver
service
3. Iden1fy
your
role
in
the
ecosystem
30.
3.
Iden1fy
your
role
in
the
ecosystem
BRAND
BRAND
1:1
rela1onship
with
customer
Branded
experience
Facilitate
rela1onships
among
customers
Brand
facilitates
rela8onships
in
an
ecosystem
31.
3.
Iden1fy
your
role
in
the
ecosystem
Key
ques1ons:
• What
roles
does
the
network
need
to
func8on,
and
who
can
provide
it?
• How
to
start
and
grow
the
network?
BRAND
32.
3.
Iden1fy
your
role
in
the
ecosystem
What
roles
does
the
network
need
to
func1on,
and
who
can
provide
it?
3
OWNER
ROLES
Host
Community
creator
Community
orchestrator
5
NON-‐OWNER
ROLES
Crowd
gatherer
Builder
Network
enhancer
Conversa1onalist
Beneficiary
in
33.
3.
Iden1fy
your
role
in
the
ecosystem
How
to
start
and
grow
the
network?
Emerge
eg.
Look
for
the
right
condi7ons
to
create
a
plahorm
for
exchange
Seed
PSN
eg.
Start
the
network
in
key
places,
with
key
actors
and
the
right
condi7ons
to
grow
Adapt
eg.
Allow
the
network
to
fragment
or
specialise
if
it
needs
to
Nurture
eg.
Encourage
par7cipa7on
and
help
the
network
to
flourish
Weed
eg.
Discourage
or
filter
out
nega7vity
which
could
reduce
par7cipa7on
in
the
network
34.
Networked
business
models
calls
for
a
shi3
in
approach
to
service
design
1. Focus
on
the
individual
customer
2. Design
and
deliver
a
service
3. Own
a
unique
rela1onship
with
customer
Tradi1onal
service
models
Networked
business
models
1. Uncover
opportuni1es
in
a
network
2. Enable
exchanges
to
deliver
service
3. Iden1fy
your
role
in
the
ecosystem
36.
Value
proposi1on
template
Design
a
networked
service
Short
descrip1on:
Who
is
exchanging
value?
What
value
is
exchanged?
How
is
the
value
exchanged?
Sketch
out
the
network
37.
Share
back
@claropartners
#sharingeconomy
#sdn13
38.
A
toolkit
to
design
a
par1cipatory
service
network
leveraged
by:
39.
Explore
more:
join
us
for
the
Global
Service
Jam
in
Barcelona!
7-‐9
March"
Strategists, developers and designers all across the
world come together in one weekend to explore,
create and prototype a service."
In Barcelona this is mentored by Claro Partners,
their clients and other professionals."
Aldo
de
Jong|
Co-‐Founder
aldo.dejong@claropartners.com
+34
647
857
922
(m)
Send
us
an
email
at
jam@claropartners.com
to
be
no7fied
as
soon
as
registra7ons
open
barcelonaservicejam.org
Clipping is a handy way to collect important slides you want to go back to later. Now customize the name of a clipboard to store your clips.
Introduction
Here are present actual charts for performance comparison of serialization libraries for Scala for the corresponding article «Scala Serialization».
The legend for tests. «Site» is a data transfer object (DTO) with different types of data (lists, enums, regular fields). «Events» are the primitive representation of this DTO for Event Sourcing (many small objects). Data sizes (1k, 2k etc.) are the corresponding size of the DTO in JSON format.
Please notice, that unlike the article, here the performance tests are performed via JMH and using 2 threads. The configuration of a hardware is Intel® Core™ i7–5600U CPU @ 2.60GHz × 4 (2 core + 2 HT) with 16 GB RAM.
Changelog
Major upgrade to scala 2.12.4.
Add CBOR data format (via Jackson)
Upgrade versions of libraries: jackson (2.9.5), protobuf (3.4.0), boopickle (1.3.0), chill (0.9.2).
Downgrade: protobuf (3.1.0).
Fixed Chill threading via state per thread (no penalty for clone for it).
Remove circe (failed to compile).
Introduced benchmark for Circe library.
Downgrade versions (because using bazel rules): scrooge (4.6.0).
Upgrade versions of libraries: scala-library (2.11.8), jackson (2.9.1), protobuf (3.4.0),
scalapb (0.6.5), boopickle (1.2.5), chill (0.9.2).
Fixed issue with Chill by cloning input array (therefore it’s time of Chill + array clone).
Initial version of benchmark.
Libraries: scala-library (2.11.7), jackson (2.7.3), protobuf (3.0.0-beta-2), scalapb (0.5.31),
scala-pickling (0.11.0-M2), boopickle (1.2.4), chill (0.8.0), libthrift (0.9.1), scrooge (4.7.0).
Chill failed with ”Buffer underflow” exception on only-deserialization benchmarks,
because of bug.
ScalaPB translates Protocol Buffers to Scala case classes. The generated API is easy to use!
Supports proto2 and proto3
ScalaPB is built as a protoc plugin and has perfect compatibility with the protobuf language specification.
Nested updates
Updating immutable nested structure is made easy by an optional lenses support. Learn more.
Interoperate with Java
Scala Protocol Buffers can be converted to Java and vice versa. Scala and Java protobufs can co-exist in the same project to make it easier to gradually migrate, or interact with legacy Java APIs.
Scala.js support
ScalaPB fully supports Scala.js so you can write Scala programs that use your domain-specific Protocol Buffers in the browser! Learn more.
gRPC
Build gRPC servers and clients with ScalaPB. ScalaPB ships with its own wrapper around the official gRPC Java implementation. There are gRPC libraries for ZIO, Cats Effect and Akka.
](http://mybinder.org/repo/apache/incubator-toree){kind=icon}
Chris Riccomini is a software engineer with more than a decade of experience at Silicon Valley tech companies including PayPal, LinkedIn, and WePay, a JPMorgan Chase company. Over his career, Riccomini has held titles as data scientist, staff software engineer, and distinguished software engineer. He has managed engineering teams in the distributed systems and payments space. Riccomini is active in open source. He is co-author of Apache Samza, a stream-processing framework, and is also a member of the Apache project management committee (PMC), Apache Airflow PMC, and Apache Samza PMC. He is a strategic investor and advisor for startups in the data space, where he advises founders and technical leaders on product and engineering strategy.
