The increasing popularity of dbt has resulted in a huge demand for additional tooling to make dbt easier than ever to run. While we previously announced our ability to automate and execute dbt Core in the Cloud, not every team has a workflow that runs off of dbt Core executions.

The dbt Cloud platform, built directly by the team at Fishtown Analytics (now dbt Labs), helps teams quickly create and automate the execution of their data models in one tightly-integrated package. This service is a great way to keep build models interactively, execute them, and access the final documentation in one location.

However, we've frequently heard from teams that it's difficult to get dbt Cloud connected with other parts of their data stack in an elegant fashion. Luckily, that's no longer the case.

Introducing the dbt Cloud Blueprints

As of today, Shipyard now has 4 new Blueprints to help you quickly work with dbt Cloud. This potent combination of Shipyard + dbt Cloud, will enable your team to continue building models directly within the dbt Cloud UI while connecting the execution and output of these models to your entire data ecosystem.

By providing a dbt Cloud Service Token, your Account ID, and a specific Job ID you can mix and match Blueprints to:

• Execute a dbt Cloud job on a schedule or via a webhook
• Check a job run for it's status
• Download all artifacts and logs associated with a job run

Store dbt Cloud Logs Externally

When you run your jobs in dbt Cloud, the artifacts and logs are accessible via the UI, but not other systems that you may be using. Wouldn't it be nice to transport these logs to any tool of your choosing for further analysis?

By leveraging the "Download Artifacts & Logs" Blueprint, Shipyard makes it easy to download these files and quickly connect them to another Blueprint that uploads the files directly to your data storage of choice (S3, GCS, Box, etc.). Whether you want long-term access to the console output, debugging logs, documentation files, or generated SQL scripts, Shipyard makes it a breeze to access and store this data elsewhere.

Run other Data Processes before and after dbt Cloud

With the current state of dbt Cloud, your team's data models live in a silo from the rest of your data operations. When your models start running, you can't be 100% sure that the correct data has been loaded beforehand. When your models finish running, there's no way to kick off other workflows built with the data. This puts your team in a situation where you're potentially running suboptimal scheduling-based workflows, resulting in longer runtimes and potentially inaccurate data.

By using the "Execute Job" and "Run Sync Status" Blueprints in Shipyard, you can build a workflow that runs dbt after your data loads and immediately run machine learning models and reporting after the models complete successfully. If your dbt models fail, Shipyard will automatically prevent downstream jobs from running so you can troubleshoot the issue directly in dbt Cloud.

Connect All Of Your Data Tools

With the introduction of the dbt Cloud Blueprints, you can quickly connect the execution of dbt Cloud to any script that you write in Python, Bash, or Node.js. You can also connect it to other common processes that need to be run against external databases (Snowflake, Redshift, Bigquery, etc.) or data storage systems (AWS S3, Google Cloud Storage, etc.) using our Blueprint Library.

Shipyard automates and connects ANY data tool that you might be using - not just dbt Cloud. With the Shipyard platform, your team has a greater flexibility to create a modular pipeline where each step shares data with each other, rather than relying on flimsy schedule-based pipelines between siloed systems.

Here's a few examples of how you can connect dbt Cloud to other services to create a seamless solution for your Data Team.

• Trigger dbt Cloud jobs after a successful Fivetran data load
• Refresh Tableau extracts after building your dbt Models
• Kick off Reverse ETL jobs with Hightouch after your dbt Models run successfully
• Run ML models after your dbt models finish running
• Send Slack alerts with dbt test results

Get Started Today

The dbt Cloud Blueprints are now available to all subscribers and can be tested with any account. Shipyard is making it easier than ever to automate your dbt Cloud. Sign up for our free Developer Plan to get started automating your dbt Cloud executions.

We're looking forward to seeing how users will take advantage of this new blueprint to implement dbt Cloud in production quickly and deploy data solutions across their modern data stack.

About Shipyard:
Shipyard is a modern data orchestration platform for data engineers to easily connect tools, automate workflows, and build a solid data infrastructure from day one.

Shipyard offers low-code templates that are configured using a visual interface, replacing the need to write code to build data workflows while enabling data engineers to get their work into production faster. If a solution can’t be built with existing templates, engineers can always automate scripts in the language of their choice to bring any internal or external process into their workflows.

The Shipyard team has built data products for some of the largest brands in business and deeply understands the problems that come with scale. Observability and alerting are built into the Shipyard platform, ensuring that breakages are identified before being discovered downstream by business teams.

With a high level of concurrency and end-to-end encryption, Shipyard enables data teams to accomplish more without relying on other teams or worrying about infrastructure challenges, while also ensuring that business teams trust the data made available to them.

For more information, visit www.shipyardapp.com or get started for free.