The increasing popularity of dbt has resulted in a huge demand for additional tooling to make dbt easier than ever to run. While we previously announced our ability to automate and execute dbt Core in the Cloud, not every team has a workflow that runs off of dbt Core executions.
The dbt Cloud platform, built directly by the team at Fishtown Analytics (now dbt Labs), helps teams quickly create and automate the execution of their data models in one tightly-integrated package. This service is a great way to keep build models interactively, execute them, and access the final documentation in one location.
However, we've frequently heard from teams that it's difficult to get dbt Cloud connected with other parts of their data stack in an elegant fashion. Luckily, that's no longer the case.
Introducing the dbt Cloud Blueprints
As of today, Shipyard now has 4 new Blueprints to help you quickly work with dbt Cloud. This potent combination of Shipyard + dbt Cloud, will enable your team to continue building models directly within the dbt Cloud UI while connecting the execution and output of these models to your entire data ecosystem.
By providing a dbt Cloud Service Token, your Account ID, and a specific Job ID you can mix and match Blueprints to:
- Execute a dbt Cloud job on a schedule or via a webhook
- Check a job run for it's status
- Download all artifacts and logs associated with a job run
Store dbt Cloud Logs Externally
When you run your jobs in dbt Cloud, the artifacts and logs are accessible via the UI, but not other systems that you may be using. Wouldn't it be nice to transport these logs to any tool of your choosing for further analysis?
By leveraging the "Download Artifacts & Logs" Blueprint, Shipyard makes it easy to download these files and quickly connect them to another Blueprint that uploads the files directly to your data storage of choice (S3, GCS, Box, etc.). Whether you want long-term access to the console output, debugging logs, documentation files, or generated SQL scripts, Shipyard makes it a breeze to access and store this data elsewhere.
Run other Data Processes before and after dbt Cloud
With the current state of dbt Cloud, your team's data models live in a silo from the rest of your data operations. When your models start running, you can't be 100% sure that the correct data has been loaded beforehand. When your models finish running, there's no way to kick off other workflows built with the data. This puts your team in a situation where you're potentially running suboptimal scheduling-based workflows, resulting in longer runtimes and potentially inaccurate data.
By using the "Execute Job" and "Run Sync Status" Blueprints in Shipyard, you can build a workflow that runs dbt after your data loads and immediately run machine learning models and reporting after the models complete successfully. If your dbt models fail, Shipyard will automatically prevent downstream jobs from running so you can troubleshoot the issue directly in dbt Cloud.
Connect All Of Your Data Tools
With the introduction of the dbt Cloud Blueprints, you can quickly connect the execution of dbt Cloud to any script that you write in Python, Bash, or Node.js. You can also connect it to other common processes that need to be run against external databases (Snowflake, Redshift, Bigquery, etc.) or data storage systems (AWS S3, Google Cloud Storage, etc.) using our Blueprint Library.
Shipyard automates and connects ANY data tool that you might be using - not just dbt Cloud. With the Shipyard platform, your team has a greater flexibility to create a modular pipeline where each step shares data with each other, rather than relying on flimsy schedule-based pipelines between siloed systems.
Here's a few examples of how you can connect dbt Cloud to other services to create a seamless solution for your Data Team.
- Trigger dbt Cloud jobs after a successful Fivetran data load
- Refresh Tableau extracts after building your dbt Models
- Kick off Reverse ETL jobs with Hightouch after your dbt Models run successfully
- Run ML models after your dbt models finish running
- Send Slack alerts with dbt test results
Get Started Today
The dbt Cloud Blueprints are now available to all subscribers and can be tested with any account. Shipyard is making it easier than ever to automate your dbt Cloud. Sign up for a free 14-day trial to get started automating your dbt Cloud executions.
We're looking forward to seeing how users will take advantage of this new blueprint to implement dbt Cloud in production quickly and deploy data solutions across their modern data stack.
Shipyard is a serverless data workflow platform that helps Data Teams launch, monitor, and share their solutions 10x faster. Driven by a mission to simplify every company’s data operations, they are creating an ecosystem where organizations can break down data silos and move beyond dashboards towards a future of fully automated, data-driven actions. The founding team draws on their previous experience at top agencies and media companies, handling high-throughput digital advertising and inventory data for Fortune 500 companies. For more information, visit www.shipyardapp.com or get started with a 14-day free trial.