The Apache Airflow alternative that doesn't require DevOps

Shipyard is an Airflow alternative that gives your team the ability to create powerful workflows in a matter of minutes.

Start Your Free Trial

Shipyard

Shipyard is a cloud-native workflow orchestration platform designed for the modern data team. With its intuitive UI and powerful infrastructure, Shipyard can easily automate the toughest tasks in a matter of minutes. Solutions can be created modularly and shared across the organization to help spread the influence and reusability of a data team's work.

vs.

Apache Airflow

Apache Airflow is a well-known open-source project for workflow orchestration, released in 2014 by the Data Team at Airbnb. While flexible in its potential use cases, its rigidity and adherence to python-specific code prevent make it more difficult to work with. Infrastructure knowledge is required to ensure a solid foundation for your workflows.

See Why Data Teams are switching from Airflow

No More Platform-Specific Code

Workflow as code solutions like Airflow require users to constantly edit their code to make any changes. In order to create a workflow, you have to set up a proprietary DAG file. To create a template, you have to create a function with Airflow specific logic. To pass connections and other secrets from your DAG to your scripts, you have to force your script to parse through Airflow's storage methods.

This setup forces your team to constantly to mix Airflow logic with business logic, increasing the switching costs down the road.

With Shipyard, we believe that if you can run your script locally, you should be able to run it in our platform without any changes. Avoid lock-in by focusing on coding solutions that solve the problem, rather than solving the platform you're running on.

Built-in Security and Stability

Most Data Analysts, Data Scientists, and Data Engineers aren't infrastructure experts, but Airflow expects them to set up everything on their own. Credentials aren't encrypted by default. User authentication and management has to be set up manually. Parallelization requires knowledge of Dask, Celery, or Kubernetes.

Working with Airflow means that you're on your own when it comes to ensuring your data is secure and that business-critical workflows won't run into resource limitations. Why take those risks if you don't have to?

With Shipyard, we let Data Teams focus on what they're best at - solving problems with data. Our infrastructure is ready to go as soon as you sign in. Tasks scale automatically and run in parallel. Everything runs in isolation, so no single job can take down the whole system. User management and audit logs are baked into the platform. All of your data is encrypted and routinely wiped after runtime.

If your team is serious about security and stability, we make it seriously easy to get it right. Learn more about our infrastructure and data security.

Isolated Executions

Need to run different versions of Python? Or run two scripts at the same time using different versions of Pandas? On Airflow, this type of structure is nearly impossible. The typical recommendation is to use the same package versions and to keep these in sync across your multiple Airflow servers.

With Shipyard, every task and workflow is automatically run in a container, using the package dependencies you specify, without the need to write a Dockerfile. You can orchestrate tasks that use different packages and different languages without worrying about one affecting the other. Stop worrying about managing virtual environments and rest easy that your code will run.

Easy-to-Use Templates

In Airflow, you can create re-usable templates through Operations. The only problem? They require someone to know the Operation exists and to be capable of writing a script to use it.

Shipyard focuses on making templates usable without needing to know code. Each template you create maps a script's inputs to a basic form. End users only need to fill out that form and set triggers to start using the template.

Every template your data team creates is publicly available to anyone using the platform. Teams can monitor template usage and make changes that bulk update every task built using the template. You can finally scale the efforts of your data team by getting their work into the hands of the entire organization.

Modular Workflows = Reusable Code

Airflow recommends that every script you write be atomic - where data doesn't need to be shared between the tasks. This structure results in monolithic scripts that run through a series of cascading steps. When errors inevitably occur, it's difficult to know what exactly went wrong. If you need to re-run your task, you have to start over from the beginning. If part of your solution needs to be used elsewhere, you'll end up with code snippet nightmares.

If you choose to not make your scripts atomic, you'll be stuck adding additional code to store or download your files and variables externally, adding even more points of failure to the process.

With Shipyard, every task can be broken down into its core functionality, with data seamlessly transferred from one step to the next. Write 3 separate components to download data, transform the data with business logic, and upload it to an external service. Reuse these components as many times as you like across the platform.

Make Rapid Changes in the UI

On it's surface, Airflow gives you visibility into all the jobs your organization is running. But what happens when you need to make quick changes to a workflow? You'll have to copy the code from the UI, make changes locally, spin up Airflow locally for testing, then upload the new files to your servers.

Shipyard focuses on a seamless UI experience for a reason; it lets teams experiment rapidly. Need to change up your workflow? Just drag and drop it around. Need to make a change to your code? Edit it directly in the UI. Want to test slight variations to your algorithm? Duplicate a task and edit the inputs. Making changes couldn't be easier.

Support When You Need It Most

While Airflow can look enticing due to it's open-source price tag of free, hidden costs start cropping up with any extended use. If your scripts won't run, you'll have to dig through thousands of articles in hopes of a solution. If you find a bug with the platform, you'll need to wait months for the community to fix it (or spend money dedicating engineering resources yourself). If all of your workflows stop working for any reason, you'll have a mad dash to troubleshoot your infrastructure. Why put that level of risk and frustration on your team?

Shipyard prides itself on world-class support, with a team of dedicated experts to help your through any issues you might run across. Can't get your script to run? We can help. Need a specific feature? We can build it. With the click of a button, our team is available to get you on the right track as soon as possible.

Starting at $50/month.
Pay for what you use.

Try an Apache Airflow alternative that gives you power and speed without the headaches.

Try Shipyard Free for 14 Days

Shipyard vs Apache Airflow for Workflow Orchestration

See how we compare side by side

Compare
Platforms

Shipyard

Airflow

Pricing

Starting at $50/Month.
Pay only for what you use.

$100 minimum for 24/7 Servers, Databases, etc.

Hosting

Native AWS Cloud Infrastructure

On Prem or hosted through Astronomer, Google Cloud Composer

Initial Setup Time

5 Minutes

2-3 Weeks of DevOps

Setup Required

Create an Account
Sign In
Write Code or Fill out Blueprint Form
Click Run

Spin up Server(s)
Install Packages/Languages on Server(s)
Install Airflow on Server(s)
Spin up Database
Link Database to Airflow
Secure Web Traffic to all Airflow Resources
Modify Airflow Configs for Encryption, User Accounts, etc.
Create DAG File
Upload DAG + Scripts to Server
Open Airflow
Click Run

Native Language Support

Python, Bash, SQL

Python, Bash, SQL

Additional Language Support

Support for Node.js, R, Ruby, Go, and more by using Bash.

Only through Docker Containers.

Interfaces

Web UI

Web UI, CLI

VISIBILITY

Workflow Order Visualizations

Diagram and Gantt

Diagram and Gantt

Runtime Visualizations

Yes

Yes

Status Visualizations

Yes

Yes

Logs

Script output only. Mirrors what you would see running the script locally.

Mixes Airflow platform logs and script output.

Reference IDs

Names + automatically generated UUIDs at every level

Names only

Task Organization

View Tasks by Project or Template

None

RUNTIME

Parallel Processing

Unlimited with Dynamic Scaling.

Restricted by infrastructure you choose.

Runtime Length

Up to 4 hours

Restricted by infrastructure you choose.

Resource Utilization

Limits per Workflow, but unlimited across the platform.

Restricted to the infrastructure you choose.

Execution Environment

Each task runs in independent containers.

All DAGs run in one environment.

Runtime Context Variables

All metadata for Upstream, Downstream, and Current Stream Tasks

Only High Level Timing and IDs

File Sharing

Yes, automatically.

Only if running on a single server.

Variable Sharing

Yes, with files.

Yes, with XCOM.

Live Logging

Yes

Yes

TEMPLATES

Template Structure

CLI built in any language

Python Function that can only be used with Airflow.

Template Creation

Map arguments to form inputs.
Publish in platform.

Save function files on server(s)

Template Usage

Fill out a form in the UI.

Import as a function in your script or DAG. Requires you to look up documentation for the template.

Template Monitoring

See all templates and their dependent tasks in one spot.

Manually parse through DAGs. Must know template names.

Built-In Templates

49 - Open-source

38 - Open-source

3rd Party Integrations

21

15

WORKFLOWS

Workflow Editing

Yes. Drag and drop visual interface.

No. Complex Python DAG code.

Code Editing

Yes. Edit Code directly in the UI.

No. Read only.

Code Uploading

Yes. Drag and drop file uploads in the UI.

No. Upload scripts and DAGs to the Airflow Server(s).

Workflow Paths

Converging, Branching, or Successive in any order.

Converging, Branching, or Successive in any order.

Conditional Logic

On Success Only.

On Success, On Failure, On Either.

External Package Installation

Add in the UI. Installed at runtime.

Must be pre-installed to server(s).

Schedule Triggers

Hourly, Daily, Weekly, or Monthly with time select. Set in the UI.

Cron schedule syntax.
Set in the DAG.

Manual Triggers

Click Run Now in UI.

Click Run Now in UI.

Webhook Triggers

No

No

Timezones

Different timezone per project.

One timezone for entire platform. Default of UTC.

SECURITY

Data Encryption

All variables and code encrypted by default.

Only connections, if you install Fernet.

Data Security

All generated or downloaded files are immediately removed after runtime.

Files and variables are stored indefinitely, unless script specifies otherwise.

User Authentication

Built-In

Must be configured

User Audit Logging

Every Action

Logins and CLI Actions

User Permissions

User based. Granular usage permissions per Project.

Role based. Granular permissions per DAG.

SSO Login

Google Supported

None

Credential Management

Set in the UI. Provided as environment variables at runtime.

Must be set on the Airflow server or referenced as Airflow connection objects.

RESILIENCY

Retries

Up to 3x with customizable delays.

Unlimited with customizable delays.

Alerts

Built-in email alerts for failures.

Requires configuration and manual additions to each DAG.

Support

Chat & Email during business hours.

None.

Local Testing

Run scripts individually on any computer.

Install and run local instance of Airflow for testing.

Documentation

Yes

Yes

ACCOUNTS

Clones

Up to 20 clones per month

Categories

Business, Technology

User Limit

1 User

DESIGN INTERACTIONS

Lottie Animation

3 months

Design Interactions

3 months

DESIGN INTERACTIONS

Clones

Categories

-

ACCOUNTS

Clones

Up to 20 clones per month

Categories

Business, Technology

User Limit

1 User

DESIGN INTERACTIONS

Lottie Animation

3 months

Design Interactions

3 months

DESIGN INTERACTIONS

Clones

Categories

-

ACCOUNTS

Clones

Up to 20 clones per month

Categories

Business, Technology

User Limit

1 User

DESIGN INTERACTIONS

Lottie Animation

3 months

Design Interactions

3 months

DESIGN INTERACTIONS

Clones

Categories

-

Sit back and let us do the heavy lifting

Our FREE migration service makes switching from Apache Airflow to Shipyard easy.

Need some help moving your existing DAGs and scripts from Airflow?

Good news: we’ve got you covered.

Migrations from Airflow are completely FREE.

Just sit back, relax, and let us do the heavy lifting by moving all of your automated workflows to Shipyard. We'll work with you every step of the way to create a seamless transition.

Your search for an Apache Airflow alternative is over.

Shipyard has all the power of Airflow with less setup, less code management, and world-class support.

Try Shipyard Free for 14 Days