In this blog, we run and schedule Spark jobs on Apache Airflow, we built a Spark job that extracts data from an API, transforms the result JSON data, and loads the data into an S3 bucket. We have decided to run Spark and Airflow locally, and we have configured Spark and Airflow to talk together using the Airflow UI. To ensure smooth communication between Spark and S3 bucket, we have set up an S3 access point, and a dedicated AWS IAM role so that data is sent to the S3 bucket directly from our Spark application.

In Cassandra Lunch #80, we discussed how DataStax Astra can be used to create and track a content management system. We will demo a small application that is a clone of Tik Tok using Astra’s Document API. The live recording of Cassandra Lunch, which includes a more in-depth discussion and a demo, is embedded below in case you were not able to attend live. If you would like to attend Apache Cassandra Lunch live, it is hosted every Wednesday at 12 PM EST. Register here now!

CTO of Onilab Alex Husar, writes about five benefits of using IoT in eCommerce and how Cassandra can be a solution to the data requirements of setting up IoT infrastructure.

Join Anant and DataStax as we walk you through the steps to get started with your cloud migration. We will be partaking in a 3 part online workshop to help lead you through the journey. We will be going over how to transform your legacy platform to a server-less cloud database and API. We will discuss planning, migrating, and fully realizing your cloud transformation. Part 1: Designing & Planning a Cloud Migration will begin at 12 pm EST on Tuesday, September 21st. Be sure to register to join us here.

In this blog post, the first in a series, we will be taking a look at Kubernetes. More specifically the differences between Local setups and setups on a cloud provider like AWS and Digital Ocean. We will take a look at how to spin up a basic cluster on both Local and Cloud.