Below is an ordered list of changes that we have made to Shipyard. Changes include but are not limited to: new features, feature enhancements, bug fixes, or other usability updates. Some changes may be omitted, such as changes that do not affect the user-facing experience.
The purpose of this changelog is to publicly document the updates we are continuously making to improve the Shipyard platform. We hope this is useful to our users to know exactly what parts of the product are being actively worked on.
Have a feature request, a bug report, a suggestion or an interesting use case? We'd love to hear it! Let us know at email@example.com!
Features On Deck
All timing of feature development is estimated and subject to change
- Version Control (May)
- Paramaterized Webhooks (June)
- API Access for Exporting Logs (July)
- API Access for Bulk Editing/Creating Fleets (July)
Hit a "Panel Invalid" error while building a Fleet? We've all been there. But now the message will be a bit more helpful and the invalid Vessel and panels will be highlighted in red.
Ever find it weird that you can see billable runtime on Fleet Run and Voyage tables but not on the actual log? Yeah, we did too. Now you can.
- Vessels were rebelling against their Blueprint parents and refusing to adopt the same guardrails. The Blueprints won out. Every Vessel built with a Blueprint (from this point forward) will be set to same guardrails as the original Blueprint. If you decide to change the Vessel guardrails anyway, that decision will also be respected.
- We did a little refactoring and miiight have left a variable undefined. That's been fixed, so next time you visit the YAML Editor it shouldn't keel over on you.
- A few clients had trouble viewing their Usage charts but that's been cleared up.
- Update the code input on Code Vessels to have syntax highlighting.
- Add a badge with a count of required fields that need completing to Blueprint Vessels.
- Add more troubleshooting tips for when a Vessel errors.
- Fix error occurring when Panda's version is not provided in a Python Vessel.
- Fix issue with some jobs not scheduling appropriately.
- Enhanced Developer Plan. It now includes 10 hours of runtime each month.
- Fix Voyages not always terminating when requested.
- Add New Zealand to the billing options.
- Always show Vessel tools on the selected Vessel in the Fleet Builder.
- Allow Vessels in Fleets to have the same name as Project Vessels.
- Fix error occurring when editing a Project Blueprint Vessel.
- Update Delete Fleet and Go to Vessel buttons to make their function more obvious.
- Prevent new Fleets overwriting existing Fleets with the same name.
- Fix selected Vessel not being highlighted in Fleet.
- Fix error occurring when saving Fleets larger than five Vessels.
- Prevent the "double scheduling" error.
- Separate tabs in Logs for Historical and Upcoming Fleet Runs.
- Disable Run Now on an edited, pre-saved Fleet
- Prevent the modal to select a Project from appearing unnecessarily in the Fleet Settings
- Add YAML Fleet editor to the UI.
- Add error messages for incompatible Pandas versions.
- Add a new Project modal to the Fleet builder flow.
- Fixed issue where changing order of Blueprint Inputs would not prompt a save.
- Fixed an issue where users could not click on Vessels after creating a Project.
- Triggers have been removed from Project Vessels for more Fleet-centered workflow construction.
- Add help messages to the logs pages with possible resolutions.
- Make unique Blueprint downloaded file name in a voyage.
- Blueprint "Dependent Vessels" table now has links to both the Vessel and conditionally to the Fleet if the Vessel is a part of one
- Project-level Vessels can now be added to and removed from Fleets without having to build them in the Fleet Builder
- Duplicate Vessel names are now prevented in the Project
- Users can now stop Voyages even if they are currently in a "Retry" state.
- Pagination has been removed from the Vessels table on a Fleet's Log page for better visibility.
- The "Edit Vessel" on the Vessel Logs page is now more prominent and easier to find.
- Handlebars have been added to Blueprint inputs for reordering in the UI.
- Subscriptions will now correctly reflect the number of users active in the account when the organization upgrades to a paid account.
- Users can create a Project directly from the Fleet Builder now.
- A "New Fleet" button has been added to the main sidebar of the application.
- A success notification will alert users when a Project has been created.
- Upon first logging in, users will be redirected to the new Fleet Builder page.
- Blueprint descriptions will be displayed automatically when a Blueprint Vessel is added to a Fleet.
- Users with one Organization will now be directed to the Projects page instead of the Organizations page..
- The Shipyard Demo Organization has been renamed to Example Organization with clearer links to it and has been made less obtrusive.
- Added a mini map view of Fleets with zoom in/out buttons for easier navigation.
- Prevented an error on save after removing and adding Blueprint Vessels to a Fleet.
- Allow Debian Linux system packages to be installed on Node and Python Vessels.
- Display the Usage tab in the Admin sidebar.
- The Organization Blueprints dropdown is now expanded by default for easier discovery.
- A "create" button has been added to all pages that report no data exists (e.g. "No Vessels").
- An error is now thrown when users attempt to signup without accepting the Terms of Service.
- The Admin sidebar items now remain highlighted after selection.
- Blueprint code will not be deleted when making updates to the Requirements tab.
- Allow Fleets to have a single Vessel instead of requiring two or more.
- Added a Need Help section to the sidebar with different options for reaching out for support.
- Switched the Fleets and Vessels tab on the main Project landing page. Additionally, moved the New Fleet option above New Vessel in the New dropdown.
- Updated the "Use Blueprint" button to redirect to the Fleet builder with a Vessel selected for building from that Blueprint.
- Added a confirmation before deleting a Vessel in the Fleet builder.
- Added functionality to accept coupons during subscription sign up.
- Prevented Usage Dashboard graphs from occasionally showing no data.
- Updated Snowflake Blueprints to suppress SQLAlchemy error messages.
- Prevented an error that occurred when changing a Fleet name immediately after saving.
- Removed project ID field from the project creation screen.
- We now accept international payments from Canada, Australia, United Kingdom, and any country in the EU! The billing portal has been updated to reflect these changes.
- Updated the Fleet Builder tab to allow the creation of multiple Vessels all from one location. You will no longer need to create individual Vessels first then string them together. In fact - you can't! If you're wanting to connect Vessels, you'll need to build them as part of your Fleet initially. See more about these changes here
- Added the ability to Duplicate a Fleet. See more about these changes here
- Added the ability to Duplicate a Vessel directly from the Fleet Builder screen.
- Added an Airtable Blueprint to download data from a base/table/view as a CSV.
- Split the functionality of saving a Fleet from scheduling a Fleet. This update will make large Fleets save quicker, while the scheduling happens in the background.
- Improved the Snowflake Blueprints to have specific error codes and better messaging for common problems.
- Fixed an issue where Blueprints in the Library Blueprint organization could not be duplicated.
- Improved internal logging to better track every notification email sent by our platform.
- Prevent Vessel changes from being saved if the Vessel was removed from the Fleet.
- Fixed an issue that sometimes prevented error emails from sending for larger Fleets.
- Updated the Fleet Builder tab to allow editing of multiple Vessels all from one location. See more about these changes here
- There's a new feature for Guardrails to exclude specific exit codes. This was custom-built to help situations when working with asynchronous APIs where you may not need to keep retrying because a final result of an error is returned and that will never change. See more about these changes here.
- Arguments are now more flexible. You can provide only a flag, a flag and a value, or just a value.
- Github connections now show the organization name as part of the repository name, like
org_name/repository. This makes it easier to discern which org a repository belongs to if there is a shared repository name.
- Fixed issue where refreshing on the Fleet's Triggers tab would not show saved schedules.
- Add multiline support for Alphanumeric and Password Blueprint Variable types. This will allow for easier support for multiline Alphanumeric inputs like JSON or SQL and for multiline password inputs like SSH keys.
- Added Usage Dashboard data for Trialing and Developer Tier Organizations.
- Fixed an issue where Voyages in a Fleet Run could incorrectly be removed.
- Fixed an issue where the Usage Dashboard would sometimes show no data.
- Fixed Duration ticker not running while viewing a log.
- Fixed leaving the Triggers tab warning for Fleets.
- Added free developer plan to the application to allow continued, limited usage of the platform after the trial.
- Users can now be invited to belong to multiple organizations.
- Expanded the clickable area for small + and - buttons.
- Removed "Time to Start" from the UI and replaced with Billable Runtime.
- Prevent native Python package from being entered in the Python requirements section.
- Fixed incorrectly set Org in dropdown in header.
- Fixed generic error message for a duplicated Vessel name.
- Fixed incorrectly disabled "Stop Voyage" button for Voyages in a Fleet Run.
- Added new Guardrail for Runtime Cutoff that limits the amount of time a Voyage can run.
- Added a warning alert on mobile indicating the site is not meant for use on mobile devices.
- Fixed an issue where duplicating a Vessel then immediately duplicating it again would cause an error.
- Updated the Markdown styling for Blueprint previews.
- Fix issue where users could not navigate away from "Something went wrong." page.
- Updated the Blueprint cards grids to be more responsive.
- Updated the historical runtime charts for Fleets and Vessels.
- Add all newly invited users to a "Shipyard Demos" Organziation with read only access.
Fixed issue where you could not remove Vessels that were a part of prior Fleet Runs and the Vessel is no longer part of the Fleet.
Fixed issue where creating a new Project would not allow for Project access for the requesting user.
- The admin permission panel has been updated to include new permissions. The application throughout has been updated to more correctly reflect a User's permissions.
- Fixed bug where the Fleet's Last Statuses were not the most recent runs.
Update the Fleet Builder to use a new library to enable new enhancements coming soon.
Update Trigger permissions with access for each of On Demand (run now), Schedules, and webhooks.
- Added Userflow widget for feature walkthroughs in the app.
- Update the Fleet Log Gantt chart to be interactive.
- Updated the Save and Next Step button bars to be sticky.
- Update Projects dropdown on sider to be just a single Projects link.
- Update Blueprint Library card names from truncating.
- Rename "Vessels" tab on Fleets to "Fleet Builder".
- Switched order of tabs on the Blueprints page.
- Fixed Edit Vessel button not showing up on log page.
- Added the ability to add image URLs to Blueprints that will render during the selection stage when building a Vessel.
- Added a restriction where a user cannot create a Fleet unless there are at least two (minimum required) Vessels in a Project.
- Updated the Logs table to show explicit "Scheduled" indicators.
- Fixed issue where update Select variable default option would not save.
- Fixed issue where Blueprint Library Blueprints were not available to use when building a custom Vessel.
- Added the most recent eight status on the Vessels and Fleet Runs tables.
- Moved the Fleet Run Voyages under the Fleet Run in routing.
- Removed Shipyard support users from billing in the Usage Dashboard and in Stripe.
- Added in Blueprint Library usage read only ability to all Organizations.
- Blueprint Password input values are no longer sent to the frontend.
- Added auto refreshing to the Fleet Run page.
- Added Timezone information to the Triggers tab for Vessels and Fleets.
- Fixed bug where writing to subdirectories of a Git clone would fail with invalid permissions.
gpgbinary available to all Voyages.
- A new simplified pricing structure with a pricing calculator.
- Native Node Vessels. You can now run Node Vessel with dependencies isntalled for you.
Added six environment variables available to your Voyage at runtime:
SHIPYARD_VESSEL_START_TIMEindicates the time at which the Vessel started running.
SHIPYARD_FLEET_START_TIMEindicates the time at which the Fleet started running. It will be empty if the Voyage is not part of a Fleet.
SHIPYARD_VESSEL_SCHEDULED_TIMEindicates the time at which the Vessel was scheduled to run.
SHIPYARD_FLEET_SCHEDULED_TIMEindicates the time at which the Fleet was scheduled to run. It will be empty if the Voyage is not part of a Fleet.
SHIPYARD_BLUEPRINT_IDindicates the ID of the Blueprint of the Vessel. It will reflect the Code type if the Vessel is not build from a Blueprint.
SHIPYARD_BLUEPRINT_NAMEindicates the name of the Blueprint of the Vessel. It will reflect the Code type if the Vessel is not build from a Blueprint.
Updated the maximum number of allowed retries to 24.
Added new Webhook triggers to Vessels and Fleets.
In addition to Schedules, you can now trigger a Voyage or Fleet Run by POSTing
to a webhook URL generated from the UI.
Read more about this update here.
Added a new Usage Dashboard that shows progress within a billing period and
estimated cost for paying customers.
You can access this dashboard from the Admin section of the Org, if available.
Removed Triggers steps from the Vessel create flows.
You can now add Triggers once the Vessel has been created.
This is in preparation for adding Webhook Triggers.
When creating a custom Vessel, you will be shown directions to the Blueprint
Library if your Org does not have any Blueprints to choose from.
When your Org's trial or subscription has lapsed, you are shown a splash screen
indicating such instead of just error message notifications.
- Fixed a backend bug concerning improper error handling of a billing error.
- Updated the Vessels DAG view for a Fleet to have a better initial layout.
Fixed styling issue on the Blueprint Library discover page.
Fixed bug surrounding correct form data on Code steps.
Fixed bug where and Organization after their trial period could not sign up
for a subscription.
- Fixed broken link for "Manage Git Integrations".
Added functionality to know what Vessel you are editing in the dropdown
for a Vessel node on the Fleet edit DAG page.
Updated error messages for non-unique entity names.
Updated the messages in empty tables due to filtering.
Updated Project name in breadcrumbs where it was showing a generic 'Project' text.
Updated backend formatting of notification emails.
Added two environment variables when the Vessel is built using a Git connection:
SHIPYARD_REPO_NAME indicates the name of the Git repository defined in the Code section.
SHIPYARD_CLONE_LOCATION indicates the local directory of where the root of
the Git repository was cloned to.
Make consistent wording around "Use this Blueprint".
Fleets now have conditional paths between Vessels.
There are three options for a path to be followed:
Success - The upstream Voyage was successful.
Errored - The upstream Voyage was not successful.
Completed - Either Success or Errored.
For a downstream Voyage to run, all upstream Voyages and the path conditions
must be met.
- Fixed bug that allowed for invalid Blueprint Variable reference names.
- Fixed backend issue where Fleet run infrastructure was not being correctly cleaned up.
- Fixed idempotency issues with an external backend service.
- Add the ability for self service password resets.
- For Git connected Voyages, the git commit hash is now printed to output.
- Update Blueprint Variable previewing mechanics.
- Update Schedules forms to be explicit about empty Schedules.
- Fixed bug to throw "unsaved changes" dialog when clicking Logs or Vessels tab.
- Fixed bug where removed Blueprint Variables would error in a Voyage.
Added the ability to use GitHub repositories as the source of your code in Blueprints
You can find the option to use this under the Git radio button on the Code tabs.
- Updated our backend to better log web based errors.
- Add a new error boundary and error page to our frontend.
- Fixed Organization billing Uploaded Data values.
- Fixed non active Organizations from scheduling Voyages.
We overhauled the creation and edit flow for both Vessels and Blueprints to make the process quicker and more seamless. Noticeable improvements are:
- The "Write" tab is now the default tab for providing code.
- The "Script Name" and "File to Run" fields now mirror each other when you're on the write tab.
- If you have unsaved changes, you'll receive a warning when navigating away.
- You can't click "Save" if you don't have any changes - making it easier to identify when you've made updates.
- You can only click "Run Now" or "Duplicate" after first saving your changes.
- The "Inputs" step for Blueprints has a clearer preview section to show you what your Blueprint will look like.
- Clicking back out of a creation flow will now take you to the last page you came from.
python3-devas a default package that is installed in the container
that runs your code.
Add a Playground project to every new Organization upon sign up.
Note the the project will default to the US/Pacific timezone.
- Implement new pricing tiers: Freelance, Startup, and Business.
- Add a panel to the sign up page to let users know what they are getting.
- Redirect user to a new confirmation URL on successful sign up request.
- Allow building a Vessel from a Blueprint from the Blueprint's page.
- Update password strength requirements and their display to the user.
- Redirect users to the Projects page if they are a member of just one Organization.
- Fix errors with a blank default value on the Select Blueprint variable type.
Added a distinct action arrow on empty collection pages to help the user figure
out what to do.
- Fix configuration handling across the application.
- Removed deprecated queueing infrastructure.
New banner in the header indicating how many days left you have in your trial.
There are no changes if you are already a paying, subscribed customer.
The entire cards in the New Vessel flow are clickable to select Code or Custom
The Active and Inactive Vessels tables have been merged into one.
The notion of an Acitve or Inactive Vessel no longer exists.
A Vessel will always run on all of its Triggers.
The Active, Test, and Inactive Blueprints tables have been merged.
The meaning of an Active, Test, or Inactive Blueprint has been updated to mean
visibility for the Users in your Org.
The translation is as follows:
Active > Visible to Everyone. Every User can create Vessels from these Blueprints.
Test > Editors. Users with Blueprint Create and Update permissions can create
Vessels from these Blueprints.
Inactive > Hidden. No User can create Vessels from these Blueprints.
- Connection to Stripe Customer Portal in the Sider once you have a subscription.
- New User permission for restricting access to the Billing section of Orgs.
- Add ReCAPTCHA v3 to Sign Up page.
- Improvements to backing infrastructure that your code runs on.
- Minor bug fix with backing queue connectivity.
- Self Sign up and new Organization creation.
- Allow for manually stopping a running Voyage.
- Backend automated Org billing.
- Improvements to backend queueing technology.
- Improvements to internal APM technologies.
We've been hard at work over the past few months to deliver a new system for Fleets that's modular and flexible enough to meet the needs of the modern data team.
🎉 New Features:
Fleets have been redesigned from the ground up to tackle more complex problems with a greater level of flexibility and visibility.
- Edit your Fleets visually. Drag and drop Vessels and paths to make a workflow that's as complex or simple as you need.
- Fleets can now run on converging, branching, and sequential paths.
- Vessels in a Fleet all share the same local file storage while continuing to run in independent containers. This allows upstream Vessels to pass data to downstream Vessels without the need for external storage and without the risk of package and language conflicts.
- Fleets now have their own logs, allowing you to assess historical status and performance.
- Fleets now have their own notifications. Receive alerts whenever a Fleet runs into errors preventing it from running in its entirety.
- Vessels can now run independently AND as a part of a Fleet, allowing you to maintain overlapping triggers without having to duplicate Vessels.
- Browse through over 40 pre-made Blueprints that integrate with more than 15 external services, databases, and cloud storage vendors.
- Add a Blueprint to your organization so you can quickly create Vessels and Fleets that download, upload, and manipulate data across your tech stack.
- Vessels no longer have the option for "Internal Triggers" to create a Fleet.
- The "Build Vessel" button has been exchanged with a "+ New" button that allows you to make a New Vessel or a New Fleet.
- The sidebar navigation has been updated to include:
- A link for the Blueprint Library
- A divider to split options for the Organization or for the Platform.
🎉 New Features:
You can now follow along with actively running Vessels - LIVE!
Clicking the "Run Vessel" button will automatically redirect you to the newly created log.
If a Vessel is scheduled to run soon, or is currently running, the log page will now automatically refresh to show you live details. Details include:
Current Vessel Status
Live Updating Duration
Live Output Indicator
When you put all of these changes together, it looks like this:
This new feature will provide better visibility into the progress of Vessels without requiring users to constantly refresh the page.
- The output section of the Log page has been refreshed to allow for:
- Word highlighting
- Line counting
- Lines without wraparound
- All of our automated email notifications have slick new look! These will provide greater clarity and insight to help you get started tracking down and resolving issues with your Vessels.
- Welcome Email
- Success Email
- Error Email
- Blueprint descriptions now support Markdown to help you write more in-depth guides for users. Use all of the traditional features of Markdown like headers, bullets, and code snippets to get your point across.
🔧 Bug Fixes:
- Fixed an issue where the error message for unnamed Vessels would shift the spacing of all the fields on the Information Tab.
- Timestamps on the Log page no longer overlap on smaller screens.
- Retries have an updated look! We've refined the design to give you better visibility into:
- When the next retry is going to occur
- What retry number Shipyard is currently on
- Visibly see when something is currently retrying
- If your Vessel uses a Custom Blueprint, you'll now see a link to go directly to that Blueprint. This will make navigation throughout the application a bit easier.
🔧 Bug Fixes:
- Trying to access an organization that you don't have permissions for will result in an explicit error message and be redirected to their list of organizations. Previously, this landed you on a broken page.
- When you refresh the page, projects on the sidebar will no longer disappear.
- Fixed an error where the Blueprint field would just say "Blueprint".
🔧 Bug Fixes:
- Fixed an issue where Blueprint names couldn't be duplicated across the application.
🎉 New Features:
You can now create a new organization directly through the UI!
While we still don't support multi-organization users, let us know if this use case would be helpful for you.
- Selecting your Timezone when creating a project is now easier, with more common Timezones listed at the top.
🔧 Bug Fixes:
- Guardrail values now correctly display the saved values.
- Guardrails from Blueprints now transfer to Vessels set up with that Blueprint.
- Removed the ability to add the current Vessel as an Internal Trigger.
🎉 New Features:
- User Management page for viewing, managing, and deleting users
- User Invite System for growing your organization and collaborating with outside parties.
- User Permissions Panel to set granular access settings per user.
_**_For more information about these features, see our blog post.
🔧 Bug Fixes:
- Fixed Google SSO issue that some users were having.
- Prevented code from disappearing when you moved back during the Wizard setup.
🔧 Bug Fixes:
- Fixed issue where a Log's output would sometimes not refresh correctly.
- Fixed issue where project name had to be unique across all organizations. Project names still have to be unique within your organization.
- Added a page to make it more clear when Blueprints have no dependent Vessels.
🔧 Bug Fixes:
- Fixed issue where Intercom button was occasionally overlapping buttons.
- Updated Python from 3.5.3 to 3.7.3