The Captain's Compass is our initiative to answer frequently asked questions, shed light on crucial topics, and give you a comprehensive understanding of where Shipyard stands in the ever-evolving data community.
Today, we dive deep into a popular question that we often encounter during our demos, concerning the similarities and differences between Shipyard and Apache Airflow. Many people ask: "Airflow has been around for a long time. Can Shipyard do the same things that Airflow can do?"
Comparing Shipyard and Airflow
Both Shipyard and Airflow are orchestration software that enable you to construct pipelines from extraction to BI layers to reverse ETL transformations. At a general level, Shipyard can indeed do essentially the same things as Airflow. Both products serve their respective use cases and hold unique positions in the market.
That being said, I'd like to explore the differences between the two, not to establish superiority but to highlight how these differences make each product suitable for different scenarios. During our demos, we even suggest teams opt for Airflow instead of Shipyard if we believe it better fits their needs.
Key Differences Between Shipyard and Airflow
One significant difference between Airflow and Shipyard lies in hosting. Airflow allows self-hosting, meaning it can reside in your own environment. This advantage provides you with the autonomy to host the service yourself. There are also external tools, such as Astronomer, that offer hosting services for Airflow.
Conversely, Shipyard is cloud-based and can currently only be utilized in the cloud, which might be a limitation for some teams that prefer or require on-premise solutions.
Another differentiating factor is the user interface. Shipyard offers a point-and-click interface that some might find more intuitive, while Airflow leans towards a more code-based user experience. Both styles have their own merits and suit different types of users.
Further setting them apart, Shipyard provides low-code templates or blueprints that allow you to quickly set up your workflows with major data players, saving you the effort of writing code from scratch. While Airflow does offer some pre-built components, they do not match the depth of what's available in Shipyard.
Lastly, both Shipyard and Airflow permit you to use your own scripts as part of your data pipeline. However, Airflow requires additional wrappers around your code to ensure proper pipeline functioning. On the other hand, Shipyard facilitates a smoother transition - if a Python, Bash, or Node script runs on your local machine, you can simply copy it over to Shipyard and expect it to work the same way.
Wrapping Up
In this edition of Captain's Compass, we've navigated through the distinct aspects of Shipyard and Airflow, shedding light on how these differences influence their applicability to various scenarios.
Be sure to check out our substack of articles that our internal team curates weekly from all across the data space. Ready to try Shipyard? Get started with our free Developer Plan now.
