In this article, we are going to build a simple Extract, Transform, and Load (ETL) data pipeline using Apache Airflow, and Cassandra. Airflow is going to be the orchestration tool and we are going to load our data into Apache Cassandra. Apache Airflow is an open-source project that was developed at Airbnb in 2015, and Apache Cassandra was a database that was created at Facebook and was later open-sourced to the public. The focus is going to be on writing to Cassandra using the Cassandra-Airflow provider.

In this blog, we discuss using Spark Structured streaming via Datastax Enterprise version 6.8.15 to process crypto trade information made available on a Kafka topic. Data sent to the Kafka topic is generated from CryptoCompare API and streamed through Websockets and consumed using Akka.

StitchData is a no-code platform that simplifies the ETL process. In this post we are going to cover setting up StitchData for the first time and getting your first extract, transform, and load process running smoothly.

https://www.stitchdata.com/

Graph data readily lends itself to visualization – and notebooks make it easy to do just that. In this post, we will test out the Graph Notebook tool put out by the AWS team on JanusGraph.

Introduction

In the first part of this blog series, we looked at the underlying issue that supernodes cause in DSE Graph due to how partitioning in Cassandra works and solutions that have been put forward for DSE Graph 5.x and 6.0-6.7. In the second part, we took a look at what this actually looks like within DSE 6.7 via some hands-on experimentation with intermediary vertices. We saw that although intermediary vertices successfully break apart the wide partitions, there are some problems inherent in introducing an extra hop like that.

In this third and final part of the series, we will look at how we can take advantage of the changes made in DSE 6.8 in order to implement a better solution than what was possible in DSE 6.7.