{"id":238056,"date":"2019-01-27T07:17:00","date_gmt":"2019-01-27T12:17:00","guid":{"rendered":"https:\/\/wordpress-756359-3782526.cloudwaysapps.com\/?p=238056"},"modified":"2024-09-21T04:53:27","modified_gmt":"2024-09-21T04:53:27","slug":"2019-08-27-deployment-tooling-dpl-v2-preview-release","status":"publish","type":"post","link":"https:\/\/www.travis-ci.com\/blog\/2019-08-27-deployment-tooling-dpl-v2-preview-release\/","title":{"rendered":"Announcing dpl v2 developer preview release"},"content":{"rendered":"\n<p>Over the last few months, we have been rewriting the current codebase of&nbsp;<a href=\"https:\/\/github.com\/travis-ci\/dpl\">dpl<\/a>, our deployment tooling, and the result is a new major version: dpl v2.<\/p>\n\n\n\n<p>Almost every line of code has been touched, code quality, test coverage and test quality greatly improved, and many of the supported service providers and volunteer contributors have been involved. We are excited about this huge community effort to improve and modernize&nbsp;<code>dpl<\/code>&nbsp;and give you the best deployment experience.<\/p>\n\n\n\n<p>The diff stat on the main\u00a0<a href=\"https:\/\/github.com\/travis-ci\/dpl\/pull\/1003\">pull request<\/a>\u00a0for this work gives a vague impression of its extend:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"627\" src=\"https:\/\/www.travis-ci.com\/wp-content\/uploads\/2024\/07\/dpl-pull-request-diff-stat-1024x627.webp\" alt=\"\" class=\"wp-image-241349\" srcset=\"https:\/\/www.travis-ci.com\/wp-content\/uploads\/2024\/07\/dpl-pull-request-diff-stat-1024x627.webp 1024w, https:\/\/www.travis-ci.com\/wp-content\/uploads\/2024\/07\/dpl-pull-request-diff-stat-300x184.webp 300w, https:\/\/www.travis-ci.com\/wp-content\/uploads\/2024\/07\/dpl-pull-request-diff-stat-768x470.webp 768w, https:\/\/www.travis-ci.com\/wp-content\/uploads\/2024\/07\/dpl-pull-request-diff-stat.webp 1221w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>Out of these ~16,000 lines, less than 7,000 are implementation code, the rest is documentation, tests, etc.<\/p>\n\n\n\n<p>Today, we are releasing dpl&nbsp;<code>v2.0.0-alpha.1<\/code>&nbsp;as a developer preview release, and would love for you to try it out.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"developer-preview-release\">Developer preview release<\/h2>\n\n\n\n<p>You can opt in by adding this to your&nbsp;<code>.travis.yml<\/code>&nbsp;file:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>deploy:\n  - provider: &#91;your-provider]\n    edge: true\n    ...<\/code><\/pre>\n\n\n\n<p>This version will not become the default until we are are able to release a stable version, but as always, early feedback is greatly appreciated. We would love to welcome you in the&nbsp;<a href=\"https:\/\/travis-ci.community\/c\/deployment\/v2\">community forum<\/a>&nbsp;to share yours.<\/p>\n\n\n\n<p>During the process, close to 30 pull requests have been ported and merged to the new codebase. If you have open issues that we have not been able to address please give us a nudge.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"dpls-new-maturity-model\">Dpl\u2019s new maturity model<\/h2>\n\n\n\n<p>We have implemented a maturity model for dpl providers (per service \u201cplugins\u201d) that will help us communicate expectations about the stability of a provider programmatically, e.g., in your build logs, our README and the help output of the command line tool (e.g. if used outside of Travis CI).<\/p>\n\n\n\n<p>The model includes 4 levels:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>dev<\/code>&nbsp;&#8211; the provider is in development (initial level)<\/li>\n\n\n\n<li><code>alpha<\/code>&nbsp;&#8211; the provider is fully tested<\/li>\n\n\n\n<li><code>beta<\/code>&nbsp;&#8211; the provider has been in alpha for at least a month and successful real-world production deployments have been observed<\/li>\n\n\n\n<li><code>stable<\/code>&nbsp;&#8211; the provider has been in beta for at least two months and there are no open issues that qualify as critical (such as deployments failing, documented functionality broken, etc.)<\/li>\n<\/ul>\n\n\n\n<p>Given the criteria above, most providers are currently in&nbsp;<code>alpha<\/code>: they are fully tested ports of previous functionality, but we have not seen any real world production deployments yet. Few of them are in&nbsp;<code>dev<\/code>&nbsp;since we need an end-to-end integration test, deploying to the respective service.<\/p>\n\n\n\n<p>You can check your favorite deployment provider in the README under&nbsp;<a href=\"https:\/\/github.com\/travis-ci\/dpl#supported-providers\">supported providers<\/a>.<\/p>\n\n\n\n<p>This is how you can help us move things forward:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If you know of real-world deployments that a provider has made to production as part of your builds. We will look into ways of automating this, but for now it would help us move providers out of&nbsp;<code>alpha<\/code>.<\/li>\n\n\n\n<li>If a provider you are familiar with is one of the few providers that are in&nbsp;<code>dev<\/code>. You could help us get an actual test deployment going in an automated way.<\/li>\n<\/ul>\n\n\n\n<p>Please let us know in the&nbsp;<a href=\"https:\/\/travis-ci.community\/c\/deployment\/v2\">community forum<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"breaking-changes\">Breaking changes<\/h2>\n\n\n\n<p>As this is a new major version it is also the (rather rare) opportunity for us to correct early decisions that have turned out not that great.<\/p>\n\n\n\n<p>Dpl v2 contains two changes that are breaking, at least in a very technical sense. We do not expect this to actually break many deployments, but sometimes things might behave slightly differently, so we call them \u201cbreaking\u201d.<\/p>\n\n\n\n<p>These are:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If you need your Git working directory to be cleaned up after the build, add&nbsp;<code>cleanup: true<\/code><\/li>\n\n\n\n<li>If you need your Git history to be erased when using the GitHub Pages provider, add&nbsp;<code>keep_history: false<\/code><\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"cleaning-up-the-git-working-directory\">Cleaning up the Git working directory<\/h3>\n\n\n\n<p><code>skip_cleanup<\/code>&nbsp;is now deprecated, and&nbsp;<code>cleanup<\/code>&nbsp;is&nbsp;<code>false<\/code>&nbsp;by default. The default used to be&nbsp;<code>true<\/code>, so you had to opt out using&nbsp;<code>skip_cleanup<\/code>&nbsp;\u2026 and has been used&nbsp;<em>a lot<\/em>. Cleaning up the working directory from any left over build artifacts only made sense for few providers, so we have changed this default.<\/p>\n\n\n\n<p>If you need the Git working directory to be cleaned up after your build you now need to opt into that using:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>deploy:\n  - provider: &#91;your-provider]\n    cleanup: true<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"keeping-git-history-with-the-github-pages-provider\">Keeping Git history with the GitHub Pages provider<\/h3>\n\n\n\n<p>Also, for the GitHub Pages provider the original default behavior was to&nbsp;<code>git push --force<\/code>&nbsp;a new orphan branch onto your target repository and branch (e.g.&nbsp;<code>gh-pages<\/code>), erasing your Git history on that branch.<\/p>\n\n\n\n<p>As this is not usually the desired behavior, an option&nbsp;<code>keep_history<\/code>&nbsp;was added later in order to not break that default.<\/p>\n\n\n\n<p>We have taken the opportunity to change this default: the provider&nbsp;<code>pages<\/code>&nbsp;will now keep the existing history of the target repository and branch by default.<\/p>\n\n\n\n<p>If for whatever reason you need your deployment to erase that history, starting with dpl v2 you will need to opt into that:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>deploy:\n  - provider: pages\n    keep_history: false<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"functional-changes\">Functional changes<\/h2>\n\n\n\n<p>Our goal with this rewrite was to keep existing functionality, not change it. But there are still functional changes that are worth mentioning:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Failed deployments now always error the build when using dpl v2<\/li>\n\n\n\n<li>The&nbsp;<a href=\"https:\/\/github.com\/travis-ci\/dpl#npm\">npm<\/a>&nbsp;provider now supports specifying a custom registry, such as&nbsp;<a href=\"https:\/\/myget.org\/\">MyGet<\/a>&nbsp;or&nbsp;<a href=\"https:\/\/github.com\/features\/package-registry\">GitHub<\/a>.<\/li>\n\n\n\n<li>The&nbsp;<a href=\"https:\/\/github.com\/travis-ci\/dpl#openshift\">OpenShift<\/a>&nbsp;provider has been outdated for a while, and is now revived. Open Shift\u2019s API has changed, and we have worked with the OpenShift team to replace it with a brand new implementation that works with OpenShift v3.<\/li>\n\n\n\n<li>A new strategy for deploying to&nbsp;<a href=\"https:\/\/github.com\/travis-ci\/dpl#github-pages\">GitHub Pages<\/a>&nbsp;via their new HTTP API has been added.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"failed-deployments-now-always-error-the-build\">Failed deployments now always error the build<\/h3>\n\n\n\n<p>Until now failed deployments did not always fail or error the build when using certain providers. This meant that you may not have been notified if your deployment failed.<\/p>\n\n\n\n<p>Starting with dpl v2 a failed deployment always errors the build.<\/p>\n\n\n\n<p>In one of the very few scenarios were you&nbsp;<em>do<\/em>&nbsp;want a failed deployment to be silently ignored, you can opt out of erroring the build using:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>deploy\n  - provider: &#91;your-provider]\n    allow_failure: true<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"the-fun-and-challenges-in-maintaining-dpl\">The fun and challenges in maintaining dpl<\/h2>\n\n\n\n<p>Dpl is an open source project written in Ruby, started in the early days of Travis CI. It is community-driven, but maintained by Travis CI developers and it has been adapted and used by and on many other platforms, such as&nbsp;<a href=\"https:\/\/docs.gitlab.com\/ee\/ci\/examples\/deployment\/\">GitLab<\/a>.<\/p>\n\n\n\n<p>Dpl supports 40 (<em>forty<\/em>) deployment providers, many of which we only learned of when they were contributed by you over the past 6 years that this project existed.<\/p>\n\n\n\n<p>All of this accumulated domain knowledge is worth its weight in gold. It allows you to deploy to any of these services without having to learn much about the underlying tooling, and go through the hassle of setting things up on Travis CI manually.<\/p>\n\n\n\n<p><a href=\"https:\/\/github.com\/BanzaiMan\">Hiro<\/a>&nbsp;has done a stellar job at dealing with this constantly incoming stream of knowledge and change. We like to look at his position as sitting at the receiving end of a firehose of unicorn fairy dust chaos for years.<\/p>\n\n\n\n<p>However, we have not always done the best job in keeping the code quality high, tests consistent and complete, and all functionality in sync with changes on the respective services.<\/p>\n\n\n\n<p>Maintaining centralized code in a diverse community environment is extremely educating and fun, but also can be challenging. Contributions sometimes come from developers not very familiar with Ruby, and very often from developers with different styles and approaches. Will still accept them, of course, if they add onto existing or new functionality.<\/p>\n\n\n\n<p>Quite often we have accepted pull requests in a hurry though, and not always have we had the time and resources to help refactor, clean things up and smooth over consistency.<\/p>\n\n\n\n<p>So we decided it was time for a rewrite.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"technical-highlights\">Technical Highlights<\/h2>\n\n\n\n<p>As this is the announcement of a developer preview release we thought it might be interesting to you to include some more details on a rather technical level.<\/p>\n\n\n\n<p>Our goal with this rewrite was to keep existing functionality, but improve code quality, maintainability, and consistency.<\/p>\n\n\n\n<p>Here\u2019s what has changed under the hood, in a nutshell:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Dpl now uses a rich&nbsp;<a href=\"https:\/\/github.com\/svenfuchs\/cl\">command line parser<\/a>, which provides a powerful DSL for specifying options and generates gorgeous help output (modeled after Rubygem\u2019s help output).<\/li>\n\n\n\n<li>Autogenerated README using this help output, now guaranteed to always be 100% inline with the options the tool actually accepts, no more human oversight or manual editing needed.<\/li>\n\n\n\n<li>Declaring all options explicitly (as the parser requires) has uncovered ~25 options (out of about 300 in total) that the code uses, and that were not mentioned in the README or docs. There were even more options that our code would accept and pass through to underlying libraries (such as Octokit) without us knowing or the code complaining.<\/li>\n\n\n\n<li>Unified, consistent code style, and much reduced implementation code in the individual provider classes. Rich DSL for specifying runtime requirements, shell commands, log message etc., separating shell commands and log messages from the implementation.<\/li>\n\n\n\n<li>Unified, consistent test style in unit tests, testing at an input\/output acceptance level as much as possible, rather than unit testing individual methods using lots of stubs and mocks. (Also, adding tests for those providers that so far are untested). Now tests all options a given provider supports (coverage so far was rather incomplete). With that the number of unit tests has more than doubled.<\/li>\n\n\n\n<li>Now features three different levels of tests: unit test suite using RSpec, as common with almost all of our tooling. A suite of runtime dependency installation tests that exercises installing various dependencies that usually would be installed at runtime (e.g. Ruby gems, Node.js, Python, or Go command line tooling, etc.). These are run as a second&nbsp;<a href=\"https:\/\/docs.travis-ci.com\/user\/build-stages\/\">build stage<\/a>&nbsp;on Travis CI. And finally, an end-to-end integration test suite that actually does a deployment to almost all service providers, to make sure our integration is stable, and we catch any major changes to the respective service quickly (we have missed a few in the past). These are run periodically using a&nbsp;<a href=\"https:\/\/docs.travis-ci.com\/user\/cron-jobs\/\">cron build<\/a>.<\/li>\n\n\n\n<li>The existing concept of passing in a context object is now put to good use by having a Bash and a Test context. This stops the test suite from executing shell commands that potentially could wreak havoc on a development machine (installing things all over the place, changing existing config files etc).<\/li>\n\n\n\n<li>Huge bash commands (e.g. for manually installing CLI tooling via shell scripts) now live in separate files (\u201cassets\u201d), making them a lot more readable due to proper syntax highlighting etc. (only 4 at the moment).<\/li>\n\n\n\n<li>Switches around how we deal with runtime Ruby gem dependencies. All code now lives in the main&nbsp;<code>dpl<\/code>&nbsp;gem, and Ruby gems required by a give provider class are now installed and required at run time (not load time) using Bundler\u2019s&nbsp;<a href=\"https:\/\/bundler.io\/v2.0\/guides\/bundler_in_a_single_file_ruby_script.html\">inline<\/a>&nbsp;strategy. This simplifies things a lot, and removes the need for us to build sub gems, which was our previous strategy of dealing with incompatibilities between requirements of various provider implementations.<\/li>\n\n\n\n<li>This also makes it possible to run all unit tests in a single Ruby process again, which dramatically helps with development: Test run time on Travis CI has dropped from ~10 minutes to under a minute (most of that time is spent installing Rubies and bundling gems, the test suite itself passes in ~15 seconds).<\/li>\n<\/ul>\n\n\n\n<p>All of this is good because:<\/p>\n\n\n\n<p>Requiring all options to be specified gives us insight into what options we actually support, in a programmatical way. That means we can now auto-generate the README, and auto-validate the&nbsp;<code>deploy<\/code>&nbsp;section in the&nbsp;<a href=\"https:\/\/github.com\/travis-ci\/travis-yml#build-config-specification\">build config format specification<\/a>, as well as the documentation.<\/p>\n\n\n\n<p>Using a rich command line parser allows reducing the implementation code significantly, and enforces consistent behavior in dealing with unknown options, missing options, malformatted values, etc.<\/p>\n\n\n\n<p>Separating shell commands and user facing log messages from the implementation code achieves several minor improvements. It makes it easy to get an overview of the commands used by a given provider class, compare messages and spot inconsistencies in wording. It also makes the implementation a lot more focused on the logic which helps with pattern recognition.<\/p>\n\n\n\n<p>Providing a meaningful DSL means implementors have to worry less about our specific implementation requirements, and can focus more on the logic and behavior they need to provide. This is important to us because it enforces consistency in a very diverse and somewhat chaotic community environment, and thus reduces complexity and effort in communication by enforcing standards. (It also requires much better documentation for contributors, which we hope we have addressed by putting a lot of work into the&nbsp;<a href=\"https:\/\/github.com\/travis-ci\/dpl\/blob\/master\/README.md\">README<\/a>,&nbsp;<a href=\"https:\/\/github.com\/travis-ci\/dpl\/blob\/master\/CONTRIBUTING.md\">CONTRIBUTING<\/a>&nbsp;guide, and&nbsp;<a href=\"https:\/\/www.rubydoc.info\/github\/travis-ci\/dpl\">rubydocs<\/a>.)<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"shoutout-to-our-amazing-contributors\">Shoutout to our amazing contributors<\/h2>\n\n\n\n<p>We would like to give a big shoutout to those of you that have invested their time and energy to work with us on making&nbsp;<code>dpl<\/code>&nbsp;better.<\/p>\n\n\n\n<p>A big shoutout to:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Ahmad Nassri (npm)<\/li>\n\n\n\n<li>Alex Foran (Catalyze)<\/li>\n\n\n\n<li>Alex Jurkiewicz<\/li>\n\n\n\n<li>Alexander Borsuk<\/li>\n\n\n\n<li>Andrew Z Allen<\/li>\n\n\n\n<li>Ashen Gunaratne<\/li>\n\n\n\n<li>Austin Siford<\/li>\n\n\n\n<li>Ben Abrams<\/li>\n\n\n\n<li>Ben Parees, Parag Dave, Chris Morgan (OpenShift)<\/li>\n\n\n\n<li>Charles Milette<\/li>\n\n\n\n<li>Christian Elsen, Vinicius Kiatkoski Neves (AWS)<\/li>\n\n\n\n<li>Coury Richards<\/li>\n\n\n\n<li>Darko Meszaros, Csaba Tamas, Sophia Kuhl (AWS)<\/li>\n\n\n\n<li>Dennis Walters, Kevin Johnson, Daniel Valfre (Engine Yard)<\/li>\n\n\n\n<li>Diego B\u00farigo Zacar\u00e3o (Transifex)<\/li>\n\n\n\n<li>Domenic Denicola<\/li>\n\n\n\n<li>Dustin Ingram<\/li>\n\n\n\n<li>Dwayne Forde<\/li>\n\n\n\n<li>\u00c9tienne Michon (Scalingo)<\/li>\n\n\n\n<li>Filip \u0160<\/li>\n\n\n\n<li>Gershom B (Hackage)<\/li>\n\n\n\n<li>Gil Tselenchuk, Vijay Sharma (TestFairy)<\/li>\n\n\n\n<li>Jacek Juraszek<\/li>\n\n\n\n<li>Jannis Leidel<\/li>\n\n\n\n<li>Jeff Waugh<\/li>\n\n\n\n<li>Jeffrey Yasskin<\/li>\n\n\n\n<li>jgollhardt<\/li>\n\n\n\n<li>Karol Danko<\/li>\n\n\n\n<li>Karthik Balaji<\/li>\n\n\n\n<li>kewubenduben<\/li>\n\n\n\n<li>Kingdon Barrett, Anton O (Hephy)<\/li>\n\n\n\n<li>Leo Unbekandt (Scalingo)<\/li>\n\n\n\n<li>Lo\u00efc Albertin<\/li>\n\n\n\n<li>Mads Marquart<\/li>\n\n\n\n<li>Martin Junkern<\/li>\n\n\n\n<li>Michael Francis<\/li>\n\n\n\n<li>Nicco Kunzmann<\/li>\n\n\n\n<li>Patrique Legault<\/li>\n\n\n\n<li>Peter Newman<\/li>\n\n\n\n<li>Radek Lisowski<\/li>\n\n\n\n<li>Sarah Dayan<\/li>\n\n\n\n<li>Simon Hengel<\/li>\n\n\n\n<li>Sviatoslav Sydorenko<\/li>\n\n\n\n<li>Szczepan Faber<\/li>\n\n\n\n<li>Valeria Tran<\/li>\n\n\n\n<li>Yoran Brondsema<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Over the last few months, we have been rewriting the current codebase of&nbsp;dpl, our deployment tooling, and the result is a new major version: dpl v2. Almost every line of code has been touched, code quality, test coverage and test quality greatly improved, and many of the supported service providers and volunteer contributors have been [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_breakdance_hide_in_design_set":false,"_breakdance_tags":"","footnotes":""},"categories":[16],"tags":[7,23,17],"class_list":["post-238056","post","type-post","status-publish","format-standard","hentry","category-news","tag-community","tag-deployment","tag-product"],"_links":{"self":[{"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/posts\/238056","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/comments?post=238056"}],"version-history":[{"count":2,"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/posts\/238056\/revisions"}],"predecessor-version":[{"id":241352,"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/posts\/238056\/revisions\/241352"}],"wp:attachment":[{"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/media?parent=238056"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/categories?post=238056"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/tags?post=238056"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}