Business Platform Team

Anant Corporation Blog: Our research, knowledge, thoughts, and recommendations about building and managing online business platforms.

Tag Archives: spark-jobs


Airflow and Spark: Running Spark Jobs in Airflow (Docker-based Solution)

Airflow and Spark: Running Spark Jobs on Airflow (Docker-based Solution)

In this blog post, we set up Apache Spark and Apache Airflow using a Docker container, and in the end, we ran and scheduled Spark jobs using Airflow which is deployed on a Docker container. This is very important because, with Docker images, we are able to solve problems we encountered in development. For example, problems that relate to a different environment, dependencies issues e.t.c, thereby leading to fast development, and deployment to production.

Continue reading
Airflow and Spark: Running Spark Jobs in Apache Airflow

Airflow and Spark: Running Spark Jobs in Apache Airflow

In this blog, we run and schedule Spark jobs on Apache Airflow, we built a Spark job that extracts data from an API, transforms the result JSON data, and loads the data into an S3 bucket. We have decided to run Spark and Airflow locally, and we have configured Spark and Airflow to talk together using the Airflow UI. To ensure smooth communication between Spark and S3 bucket, we have set up an S3 access point, and a dedicated AWS IAM role so that data is sent to the S3 bucket directly from our Spark application.

Continue reading
Airflow and Spark

Data Engineer’s Lunch #25: Airflow and Spark

In Data Engineer’s Lunch #25: Airflow and Spark, we discuss how we can use Airflow to manage Spark jobs. The live recording of the Data Engineer’s Lunch, which includes a more in-depth discussion, is also embedded below in case you were not able to attend live. If you would like to attend a Data Engineer’s Lunch live, it is hosted every Monday at noon EST. Register here now!

Continue reading
Apache Spark jobs in scala for cassandra data operations

Apache Cassandra Lunch #46: Apache Spark Jobs in Scala for Cassandra Data Operations

In Apache Cassandra Lunch #46: Apache Spark Jobs in Scala for Cassandra Data Operations, we discuss how we can do Apache Spark jobs in Scala Cassandra data operations. The live recording of Cassandra Lunch, which includes a more in-depth discussion and a demo, is embedded below in case you were not able to attend live. If you would like to attend Apache Cassandra Lunch live, it is hosted every Wednesday at 12 PM EST. Register here now!

Continue reading

Join Anant's Newsletter

Subscribe to our monthly newsletter below and never miss the latest Cassandra and data engineering news!