In Data Engineer’s Lunch #37: Pipedream: Serverless Integration and Compute Platform, we will discuss Pipedream, a serverless integration and compute platform that is free for individual developers to use. The live recording of the Data Engineer’s Lunch, which includes a more in-depth discussion, is also embedded below in case you were not able to attend live. If you would like to attend a Data Engineer’s Lunch live, it is hosted every Monday at noon EST. Register here now!
Pipedream Website: https://pipedream.com/
Pipedream is a “production-scale serverless platform to connect APIs, remarkably fast”. It gives users the ability to perform the following tasks:
As stated on Pipedream’s website, “pipedream supports use cases from prototype to production and is trusted by 150k+ developers from startups to Fortune 500 companies”. Some of these companies include the following:
Pipedream’s platform is capable of processing billions of events and is “priced for use at scale”. A sample list of some of the wide variety of use cases for Pipedream includes:
Additionally, Pipedream’s website contains a community forum where users can post questions and ask about / contribute to upcoming features. The community forum can be found here: https://pipedream.com/community/
In Pipedream, the primary entities that a user builds are called workflows. A workflow is:
Some example trigger steps are shown in the image below:
In Pipedream, two primary metrics are kept track of to determine how much a particular user gets charged. The first of these two metrics is known as the number of Invocations. Invocations are:
The second metric that is kept track of by Pipedream for the purpose of charging a user is Compute Time. Compute time is calculated as the total time a particular user’s workflow/event source runs user code and various prebuilt steps. There is a minimum of 100ms block per execution of a workflow, therefore if a process takes <100ms then it will count as a 100ms compute time block.
Depending on the type of user (free vs paid), these two metrics may limit a user in how much they can use the service unless they choose to pay more. The primary two plans for a single user, along with their compute time and invocation limits, are as follows:
So for example: even if you only use very few invocations daily, but each invocation takes up a very large amount of compute time, it is possible that you will be limited by compute time on a free plan.
More information about various plans Pipedream has for both individual users and organizations can be found here: https://pipedream.com/pricing
As stated on Pipedream’s website, “Pipedream maintains an open-source component registry on GitHub”. This GitHub repository can be found here: https://github.com/pipedreamhq/pipedream/
Using these prebuilt, open-source components allows users to avoid writing boilerplate code for common API integrations. Note that components can be used as no-code blocks in workflows, or they can be customized with your own code and then used. Here is a brief list of some existing popular open-source components/integrations in Pipedream:
For contributing to this repository, Pipedream’s documentation has a page which describes how to both develop for Pipedream and how to properly contribute to the open-source component repository by submitting a pull request: https://pipedream.com/docs/components/guidelines/#overview
For this example, we will go through a two-step workflow and then bring up an example from the Quickstart guide showcasing a significantly more complex workflow. For our first workflow, we created a workflow that looks like this:
From the above image, we can see that the workflow is composed of two steps:
A Twitter account was attached in Pipedream and now Twitter API blocks can be used with that particular account. This particular trigger step is set to run on a schedule (every few minutes) and if any new likes on tweets are done by the user, then the workflow will trigger. The second step of the workflow takes these newly liked tweets and sends the text of those tweets to the appropriate email.
For a more complicated example, you can follow Pipedream’s quick start guide, part of their documentation, linked here: https://pipedream.com/docs/quickstart/
The image below shows a sample workflow we built following the Quickstart guide, with some modifications:
For a complete rundown on this 8 step workflow, check out the Youtube video at the bottom of this blog.
And that’s a wrap on today’s blog on Pipedream: Serverless Integration and Compute Platform. For more content similar to this week’s DEL on Pipedream: Serverless Integration and Compute Platform, check out https://blog.anant.us.
Cassandra.Link is a knowledge base that we created for all things Apache Cassandra. Our goal with Cassandra.Link was to not only fill the gap of Planet Cassandra but to bring the Cassandra community together. Feel free to reach out if you wish to collaborate with us on this project in any capacity.
We are a technology company that specializes in building business platforms. If you have any questions about the tools discussed in this post or about any of our services, feel free to send us an email!