Shipyard is an Airflow alternative that gives your team the ability to create powerful workflows in a matter of minutes.
Get Started FreeStarting at $50/Month.
Pay for what you use.
Starting at $100/Month for 24/7 Servers, Databases, etc.
5 Minutes
3+ Weeks of DevOps
Hosted via AWS
Self-Hosted
Any
Python
Create & Edit Workflows Visually
Read-Only
Unlimited
Depends on infrastructure
Automatic Retries, Email Alerts, In-Depth Logging
Available with manual configuration.
Containerized
Shared Resources
50+
-
-
-
-
-
-
Up to 20 clones per month
Business, Technology
1 User
3 months
3 months
-
Up to 20 clones per month
Business, Technology
1 User
3 months
3 months
-
Up to 20 clones per month
Business, Technology
1 User
3 months
3 months
-
Shipyard is a cloud-native workflow orchestration platform designed for the modern data team. With its intuitive UI and powerful infrastructure, Shipyard can easily automate the toughest tasks in a matter of minutes. Solutions can be created modularly and shared across the organization to help spread the influence and reusability of a data team's work.
Apache Airflow is a well-known open-source project for workflow orchestration, released in 2014 by the Data Team at Airbnb. While flexible in its potential use cases, its rigidity and adherence to python-specific code prevent make it more difficult to work with. Infrastructure knowledge is required to ensure a solid foundation for your workflows.
Airflow forces you to manage your files through constant uploads to an external server. If you want to keep your code in sync with Github, you'll have to manage that process through cron jobs.
Shipyard connects directly to your Github repos, allowing you to sync your code to a specific branch or tag at runtime. Easily test out changes on development branches and rest easy knowing that you're always running the most up-to-date code.
Workflow as code solutions like Airflow require users to constantly edit their code to make any changes. In order to create a workflow, you have to set up a proprietary DAG file. To create a template, you have to create a function with Airflow specific logic. To pass connections and other secrets from your DAG to your scripts, you have to force your script to parse through Airflow's storage methods.
This setup forces your team to constantly mix Airflow logic with business logic, increasing the switching costs down the road.
With Shipyard, we believe that if you can run your script locally, you should be able to run it in our platform without any changes. Avoid lock-in by focusing on coding solutions that solve the problem, rather than solving the platform you're running on.
Need to run two scripts at the same time using different versions of Pandas? On Airflow, this type of structure is nearly impossible. The typical recommendation is to use the same package versions and to keep these in sync across your multiple Airflow servers.
With Shipyard, every task and workflow is automatically run in a container, using the package dependencies you specify, without the need to write a Dockerfile. You can orchestrate tasks that use different packages and different languages without worrying about one affecting the other. Stop worrying about managing virtual environments and rest easy that your code will run.
Most Data Analysts, Data Scientists, and Data Engineers aren't infrastructure experts, but Airflow expects them to set up everything on their own. Credentials aren't encrypted by default. User authentication and management has to be set up manually. Parallelization requires knowledge of Dask, Celery, or Kubernetes.
Working with Airflow means that you're on your own when it comes to ensuring your data is secure and that business-critical workflows won't run into resource limitations. Why take those risks if you don't have to?
With Shipyard, we let Data Teams focus on what they're best at - solving problems with data. Our infrastructure is ready to go as soon as you sign in. Tasks scale automatically and run in parallel. Everything runs in isolation, so no single job can take down the whole system. User management and audit logs are baked into the platform. All of your data is encrypted and routinely wiped after runtime.
If your team is serious about security and stability, we make it seriously easy to get it right.
In Airflow, you can create re-usable templates through Operations. The only problem? They require someone to know the Operation exists and to be capable of writing a script to use it.
Shipyard focuses on making templates usable without needing to know code. Each template you create maps a script's inputs to a basic form. End users only need to fill out that form and set triggers to start using the template.
Every template your data team creates is available to the rest of the organization. Teams can monitor template usage and make changes that bulk update every task built using the template. You can finally scale the efforts of your data team by getting their work into the hands of the entire organization.
Airflow recommends that every script you write be atomic - where data doesn't need to be shared between the tasks. This structure results in monolithic scripts that run through a series of cascading steps. When errors inevitably occur, it's difficult to know what exactly went wrong. If you need to re-run your task, you have to start over from the beginning. If part of your solution needs to be used elsewhere, you'll end up with code snippet nightmares.
If you choose to not make your scripts atomic, you'll be stuck adding additional code to store or download your files and variables externally, adding even more points of failure to the process.
With Shipyard, every task can be broken down into its core functionality, with data seamlessly transferred from one step to the next. Write 3 separate components to download data, transform the data with business logic, and upload it to an external service. Reuse these components as many times as you like across the platform.
On it's surface, Airflow gives you visibility into all the jobs your organization is running. But what happens when you need to make quick changes to a workflow? You'll have to copy the code from the UI, make changes locally, spin up Airflow locally for testing, then upload the new files to your servers.
Shipyard focuses on a seamless UI experience that helps teams experiment rapidly. Need to change up your workflow? Just drag and drop it around. Need to make a change to your code? Edit it directly in the UI. Want to test slight variations to your algorithm? Duplicate a task and edit the inputs. Making changes couldn't be easier.
While Airflow can look enticing due to it's open-source price tag of free, hidden costs start cropping up with any extended use. If your scripts won't run, you'll have to dig through thousands of articles in hopes of a solution. If you find a bug with the platform, you'll need to wait months for the community to fix it (or spend money dedicating engineering resources yourself). If all of your workflows stop working for any reason, you'll have a mad dash to troubleshoot your infrastructure. Why put that level of risk and frustration on your team?
Shipyard prides itself on world-class support, with a team of dedicated experts to help your through any issues you might run across. Can't get your script to run? We can help. Need a specific feature? We can build it. With the click of a button, our team is available to get you on the right track as soon as possible.
Our FREE migration service makes switching from Apache Airflow to Shipyard easy.
Need some help moving your existing DAGs and scripts from Airflow?
Good news: we’ve got you covered.
Migrations from Airflow are completely FREE with a subscription to the Startup plan or higher.
Just sit back, relax, and let us do the heavy lifting by moving all of your automated workflows to Shipyard. We'll work with you every step of the way to create a seamless transition.
Starting at $50/Month.
Pay only for what you use.
$100 minimum for 24/7 Servers, Databases, etc.
5 Minutes
2-3 Weeks of DevOps
Create an Account
Sign In
Write Code or Fill out Blueprint Form
Click Run
Spin up Server(s)
Install Packages/Languages on Server(s)
Install Airflow on Server(s)
Spin up Database
Link Database to Airflow
Secure Web Traffic to all Airflow Resources
Modify Airflow Configs for Encryption, User Accounts, etc.
Create DAG File
Upload DAG + Scripts to Server
Open Airflow
Click Run
Python, Bash, SQL
Python
Support for Node.js, R, Ruby, Go, and more by using Bash.
Bash, Docker
Web UI
Web UI, CLI
Diagram and Gantt
Diagram and Gantt
Yes
Yes
Yes
Yes
Script output only. Mirrors what you would see running the script locally.
Mixes Airflow platform logs and script output.
Names + automatically generated UUIDs at every level
Names only
View Tasks by Project or Template
None
Unlimited with Dynamic Scaling.
Restricted by infrastructure you choose.
Up to 4 hours
Restricted by infrastructure you choose.
Limits per Workflow, but unlimited across the platform.
Restricted to the infrastructure you choose.
Each task runs in independent containers.
All DAGs run in one environment.
All metadata for Upstream, Downstream, and Current Stream Tasks
Only High Level Timing and IDs
Yes, automatically.
Only if running on a single server.
Yes, with files.
Yes, with XCOM.
Yes
Yes
CLI built in any language
Python Function that can only be used with Airflow.
Map arguments to form inputs.
Publish in platform.
Save function files on server(s)
Fill out a form in the UI.
Import as a function in your script or DAG. Requires you to look up documentation for the template.
See all templates and their dependent tasks in one spot.
Manually parse through DAGs. Must know template names.
49 - Open-source
38 - Open-source
21
15
Yes. Drag and drop visual interface.
No. Complex Python DAG code.
Yes. Edit Code directly in the UI.
No. Read only.
Yes. Drag and drop file uploads in the UI.
No. Upload scripts and DAGs to the Airflow Server(s).
Converging, Branching, or Successive in any order.
Converging, Branching, or Successive in any order.
On Success Only.
On Success, On Failure, On Either.
Add in the UI. Installed at runtime.
Must be pre-installed to server(s).
Hourly, Daily, Weekly, or Monthly with time select. Set in the UI.
Cron schedule syntax.
Set in the DAG.
Click Run Now in UI.
Click Run Now in UI.
No
No
Different timezone per project.
One timezone for entire platform. Default of UTC.
All variables and code encrypted by default.
Only connections, if you install Fernet.
All generated or downloaded files are immediately removed after runtime.
Files and variables are stored indefinitely, unless script specifies otherwise.
Built-In
Must be configured
Every Action
Logins and CLI Actions
User based. Granular usage permissions per Project.
Role based. Granular permissions per DAG.
Google Supported
None
Set in the UI. Provided as environment variables at runtime.
Must be set on the Airflow server or referenced as Airflow connection objects.
Up to 3x with customizable delays.
Unlimited with customizable delays.
Built-in email alerts for failures.
Requires configuration and manual additions to each DAG.
Chat & Email during business hours.
None.
Run scripts individually on any computer.
Install and run local instance of Airflow for testing.
Yes
Yes
Up to 20 clones per month
Business, Technology
1 User
3 months
3 months
-
Up to 20 clones per month
Business, Technology
1 User
3 months
3 months
-
Up to 20 clones per month
Business, Technology
1 User
3 months
3 months
-
