I recently started spending a likely-unhealthy amount of time watching TikTok videos. This is only of interest (possibly) to blog readers because it led to undertaking this little piece of video art.
That’s right. Over the course of a month or so, I dedicated free time to test driving a non-trivial implementation of FizzBuzz. It involved not just FizzBuzz, but a SongPlayer, a song structure, and a console app to display the process. I also spent way too much time figuring out how long Fizz and Buzz and FizzBuzz should be played, in order to best sync to the music.
If you’re at all interested, I have the whole thing in a Github repo. This should at least explain why I haven’t been continuing the battleship game walking skeleton…
There are a lot of CI/CD systems out there. If you’re in .NET land, Azure DevOps has some pretty excellent tooling out there. Jenkins is another that has some extremely robust capabilities. There are oodles more I’ve never even brushed up against, including rolling your own on a virtual machine somewhere in the cloud (or even on your own machine).
But wait. What the heck is a CI/CD pipeline? Why do you need one? If you’re asking either of those questions, please take a look at this recent post of mine.
It integrates beautifully with GitHub (which is my repo of choice)
It’s platform agnostic (works equally well for a variety of tech stacks)
Step One is to get yourself a Travis account. This is absurdly, almost suspiciously easy: you sign up with your GitHub account.
This means you may need to allow Travis as an authorized connection or app in your GitHub account, because Travis will be accessing your repo in order to detect changes and to clone the repo so it can build the application.
You’ll end up at your dashboard, which eventually will display your most recent build and a quick link to your repositories–for now, you shouldn’t see anything.
Step Two is to link a repository, which means you need to access your profile settings. Click your avatar in the top right of the screen, then the “Settings” link.
Step Three is to find and switch on your repository so Travis knows to check it for builds. Your public repos are all listed under the “Repositories” tab, which should be the default view (private repos required a paid subscription with travis-ci.com, currently).
It can take several minutes for your repositories to show up in Travis the first time. You may have to click the “sync account” button. It also may take refreshing the browser window itself–I once waited almost ten minutes to see a repo list and as soon as I refreshed the browser it all showed up.
Find the one you want to activate, and toggle it on.
Now Travis is going to be scanning that repo through GitHub APIs and looking for a .travis.yml file to get build instructions. Which means…
Step Four is to create a .travis.yml file in the root of your repository. The dot-travis file is, as you might guess from the dot starting the name, a configuration file used by the Travis-CI build system to know what language you’re using, what build environment to use, what commands to run, and any deployment steps. It’s sort of a big deal, and if there’s no .travis.yml file found in the code pushed to GitHub, Travis simply won’t do anything.
Again, you want to create an empty file in the root of your repository–not necessarily the root of your code base. Find the .git folder, and save your yml file there. I do this step typically in VS Code or Notepad++, depending what OS I’m on.
Step Five is to define the build configuration. This will vary significantly depending on which language you’re developing in and what flavor of that tech stack you prefer, and the folks at Travis do a really great job documenting what goes in the .travis file.
The .travis.yml file I will use for this project including some helpful (to me, anyway) comments about what each line means…
# Mono is used to build .NET on Linux--we don't need it with Core.
# Needed to run commands in the Xenial CLI
# The version of Ubuntu to run the Travis virtual machine -- needed for .NET Core
# Your SDK version, not your run-time version
script: # These commands are executed in the Travis VM just like you would on your local machine
# Use a "cd" command to move the Travis command line prompt into your solution directory
# - cd /home/travis/build/<yourTravisAccount>/<yourRepoRootDirectory>/.../<yourSolutionDirectory>/
- cd ./BattleshipTDD/
# Use these to build the project without tests
# - dotnet restore
# - dotnet build
# if your project has tests, you can skip "restore" and "build" and just use "test" command -- "test" will run the other two automatically
- dotnet test
Step Six is to commit the new file and push it up to your repo, where Travis should catch the change and trigger a build.
Step Seven is to review the build in Travis. Even before I was done prepping the terminal photo above for the post, I got an e-mail from Travis saying my build passed–hooray!
When we look at the dashboard we get some key items right away. The repository name and a build badge are at the top, and then the specific build info–what branch was built, the commit message, commit ID, the build number, how long it ran for are all front and center.
If we keep scrolling down, you can see the actual build logs, starting with info on the build environment–handy for debugging problems, like when your build is fine locally, but fails when Travis tries it.
For instance, if one tries to run a .NET Core app in the default Travis Linux OS, it won’t work–hence needing to specify Xenial to make sure we were running on Ubuntu. Being able to compare the build environment with the suggestion on a help document was key to figuring this out.
Scrolling down further we get to the actual build process. Each command you specify in the .travis.yml is listed out separately–you can see on line 246, even the “cd ./BattleshipTDD/” got it’s own result output (and yes, I have seen this fail, especially on complicated solution structures where I was missing a directory level in the path).
The test output is particularly helpful–my one test passes here, but should it fail it outputs the exception message just like any other test runner, letting you know what failed and why.
At the very bottom, the build status gets reported–if any of the commands exited with anything except a 0 code, the entire build is marked as a failure.
Some examples of unhappy build logs…
Failing tests give you the output of your testing framework–no guessing what happened
Wrapping It All Up:
What we’ve done here is linked your GitHub to Travis-CI, which will build the project and run tests against it whenever changes are pushed to master. This by itself may not provide a lot of value (I mean, if you’re not running your tests before pushing code, we need to talk) but it does lay an important foundation.
From here, we’re able to automate deployment, headless browser testing, dockerizing, packaging, updating a badge on your repo to tell the world your code is sound, and more. And it happens without having to remember it–push the code, and Travis checks it. It’s one heck of a safety net, if nothing else. It lets you focus more on the code than on the boring devopsy* stuff.
*I mean, I don’t exactly think devops is boring stuff, but no shame if you do–most programmers get into code to write code, not manage deployments and QA etc etc etc.
So what is CI/CD, what does it do for you, and do you need it?
I did a quick search on the interwebz before deciding to write this and what I found at a glance fell into 2 categories:
People selling a service
People hyping a trend
Since my target audience is, largely, a solo dev who doesn’t really have the context needed to parse either of those kinds of posts, I wanted to break it down a little more succinctly.
But First, Some Vocab:
CI (continuous integration):
What it Is: The act of integrating the code being worked on into the main code base on an on-going basis, rather than waiting until a feature is complete and you’re ready to release it. Typically, merging code or pushing local changes triggers a service (a CI system) that will pull the new code, build it, run unit tests, and let you know if you managed to break your application with the latest changes. Code repos like GitHub can be configured to block pull requests that have a failing CI system build. Often, the CI system will archive the built application so it can be used later in deployment.
Problem it Solves: At root, it helps you limit the damage you can do to your code at one time. Rather than checking out a feature branch, working on it for weeks, then trying to merge it back into master and finding out your feature code is incompatible with the master branch with 231 merge conflicts (or failed tests), you can catch issues almost immediately–when they’re much easier to fix. With multiple people in a code base, this gets even more important; everyone gets aligned far more often and reduces the amount of effort needed to keep things moving in the same direction.
CD (continuous delivery/deployment):
Is it Delivery or Deployment? It depends partly on who you’re talking to (I personally don’t view them as different things and will use them interchangeably) and what the context is. If my code is going to some other department that will do the rest of the dev-opsy stuff, it makes more sense to call it continuous delivery–you’re delivering it to someone. But, if your team is responsible for getting it to the users via a webservice or some other process, then it’s more of a deployment of the software. Context is the key, friends.
What it Is: Either way, this is the process of continually putting finished software where it needs to go. This could be a staging environment, a testing environment, or directly to the users if you’re a company like Facebook (or you’re just trying to show off to family like me). A common approach is the CD system will wait for the CI process to successfully complete, and then once it knows the code is good it will either take the output of the CI process (generally referred to as a “build artifact”) and deploy it based on the configured settings, or it creates a whole new artifact for the deployment.
Problem it Solves: In the not-so-distant past, once software shipped it was done, ready or not. Think about all the games you’ve played with crazy bugs and glitches in them that you just don’t see in modern games–games these days can get patched without inconveniencing player or developer. That’s continuous deployment at work. Finish some code, ship some code. It also lets you put a basic application in front of someone, say a client, to get instant feedback–and then just keep adding functionality.
Is It For You?
So if you’re new enough to all this, is CI/CD something worth setting up? I mean, it depends…but my short answer is yes.
It’s not as arduous to setup as it may sound, especially when you look at guides put together by people trying to sell you things. If you do it as one of the first steps in your process, it’s fairly easy to get done–and it really is a velocity accelerator. You won’t get caught by surprise trying to merge in new features (“When did THOSE tests start failing?”) and don’t have to fumble around moving files in order to make sure things work or to show off a weekend of work.
Of course, there’s also the fact that those are the kind of skills that separate someone who can write code and someone who can delivery value to an employer or client. While writing software is a challenging a rewarding pastime, the best program in the world isn’t really any good until someone can put it in front of users.
I’m assuming that you know the basics on how to create a code base in your language of choice, in your IDE or text editor of choice. This series is aimed at folks who can stand up a console and/or web app on their own, with really high-level examples in C#. The goal of the series is to help you start leveraging CI/CD automation, not learn how to set up your dev environment.
I’m also assuming you’re working with Github. There are plenty of other ways to do git, and you may be able to use your service of choice in some places–but that’s on you, my guide will be based on Github. If you want a run down on this before we get rolling, this is a pretty solid article.
Okay, so why skeleton code? What does that even mean? Let’s get into it.
If your goal is to write some code as a kata, and don’t really intend on deploying it anywhere or configuring a pipeline, you don’t really need to be simple. You can go hog wild, and test drive as much as you want as fast as you’re able.
But if, as this whole exercise states, the goal is to set up a project that can continuously integrate and deploy by automated means, the absolute last thing you want to do is put together a bunch of code with moving parts. The more moving parts to your code, the more things that can go wrong. Add in all the moving parts to configuring a CI/CD pipeline, and you’ll very soon wish you had fewer things to troubleshoot.
Okay, How Simple?
For the “business logic” you should have one small passing test. How that is architected is up to you–I’m doing this in C# and you can eyeball the repo for yourself. The standard convention is a class library project and a unit test project, with one test class for each class in the library project.
You’ll also want a super bare-bones web app. Some languages/frameworks (like ASP.NET Core) build quite a bit into their “default” templates, so chances are pretty good you can get a basic “This is my webpage” type app without having to write any code at all. In my case, I changed the HTML on the index page just to amuse myself.
And that’s it. The webpage should appear when you run it locally, and the tests should pass when you run them. Otherwise, you don’t actually want your code to do anything yet. The less that can go wrong with your code, the easier it will be to setup the pipeline.
Once you have that code set up, with tests passing and a webpage viewable, go ahead and push it up to your repo. Then you’ll be ready for the next step, linking your repo to Travis-CI.
If you’re a professional developer, and your team isn’t using a ticket system, you’re probably familiar with card boards. You may even be a Trello wizard–I personally have spent more time in Microsoft’s Team Foundation (now called Azure DevOps), but it’s the same idea. A way to track what needs to be done and how long it’s taking.
Since this is a (relatively) simple one-off project, I just threw together a Trello board. Here’s the 800-mile-view:
My backlog is the total sum of what I want to get done/cover in my post series. I have code items (labeled in orange), infrastructure items (labeled in green), and posts to write (labeled in yellow). I have them roughly grouped in order by things I want to do for a specific post.
Current Effort is what might be called a sprint–it’s the group of cards I think are obtainable to get done in the next week or two. To continue my trip analogy this is just the next leg of the trip–maybe how much ground I want to cover before stopping for dinner. It would be the point where I would feel comfortable putting in a pull request, a point where if need be I could say “I have to put this aside for three months” and be able to pick it up as a complete module of the larger project. The acceptance criteria for each “effort” is to have one or more blog posts scheduled.
In Prog is just that–in progress. Stuff that I’m still working on, since I anticipate doing little bits throughout the week.
Done should be another self-explanatory item. It’s all cards I’ve completed–not just during one effort, but over the course of the whole project.
My first step was to fill the backlog. Everything I needed. For instance, I knew I needed a simple code library to provide some sort of functionality. I also knew there’d need to be some sort of web project that can display that functionality…but I wouldn’t need to link them together yet in order to write a blog post about them, or to get the solution to build in a CI system like Travis.
I decided my first effort would look like this:
Write about this plan, and my high-level thoughts on planning solo projects
Create the Trello board to map the whole thing
Write about the Trello board and my planning process
Setup a simple code library I could test in a CI system
Setup a simple web project just to have it ready
Write about the code skeleton
My second effort will focus on:
Setup Travis-CI to build and test my solution
Write about setting up Travis
Write some tests against the web project’s UI
Get them to run in Travis
Write a post about UI tests
Wire the web project to the code library
Update UI tests to reflect the dynamic logic
Write about wiring up the two
After that, I’m not sure exactly. Logically, deploying the program is the next step, but I’m not going to think that far ahead. Sticking with the road trip mentality, I know where I’m going tonight, I know roughly where I’m going to be the night after that…anything further isn’t super useful to plot in detail. I just don’t know what will happen.
Honestly, I’m already second guessing how big that second effort looks. I’m a big fan of bite size pieces. But, since I don’t have a better idea and all of that is grouped logically together, I’m just going to roll with it until a better idea presents itself.
I’ll repeat this process till I get all the backlog cleared. If something needs to get added to the backlog, or needs to go back to the backlog, I’ll do it. If I learn after the next effort that 3 posts is way too much to try and fit into a week, I’ll trim the next one to account for it.
It feels pretty Agile to me, but as a millennial I hate to put labels on things, y’know…
One of the major challenges I had (and still have, if we’re being honest) is looking at a project I want to do and actually delivering it as a solo dev. As time has gone on, I’ve learned that’s not exactly a problem exclusive to solo devs. One of the things that I have really appreciate at Pillar/Industry X.0 is how we approach this problem…and some of it really translates well to hobby code and side projects.
There are two key points I want to harp on:
Know where you’re going (what your desired end result is)
Do only the next thing (focus on what makes sense at the time)
Know Where You’re Going
My personal problem with a lot of side projects is I get an idea, and I start writing code for it. I don’t think about what the end result is going to be, usually telling myself “Don’t get too far ahead of yourself, just flesh it out a bit and it will become clear as you go along.” While I think this is a pretty decent way to write a novel, I’ve come to realize it’s a terrible way to write software.
There is a not-so-subtle difference between planning too much and planning too little. There’s no real benefit in planning out every single thing in advance, from the size and color of your standard buttons to how to promote your new app. But the general idea–web app or mobile? Is this going to live in Azure or AWS or Heroku or what? Am I using GitHub or BitBucket? Should I be continuously deploying this?–helps guide the architecture, the tech stack, and helps define the tasks.
Writing the code is important (sort of the point), and is broken up in it’s own way. But when does it make sense to setup a build pipeline? The repository? Do I need a production environment? When can I actually see changes when I deploy them, anyway? By keeping these questions in mind, by seeing a bigger picture than just features in the code, it helps guide the overall workflow.
Do Only the Next Thing
Having those tasks ranked and in rough chronological order helps you stay focused. Now you can say, “Self, we are going to test drive this feature here. And when it’s done? We’ll setup a build pipeline so it’ll merge into master on it’s own.” Once that pipeline is working, “Self, now we’ll add this other feature here, because when it’s done we can add it to a UI project!”
By modularizing the tasks, you’ll avoid the trap I always seemed to fall into–“I have a web app with no logic behind it and deploying it is a nightmare” or “This code library I spent a month on is useless till I get it plugged into something and I am just too tired to figure out what to do now.” Being able to focus on checking things off a list is satisfying and keeps things moving, especially when you’re navigating a full time day job.
You know you need to cross the country to get from one Portland to the other, but if you focus on that you’ll never make it. You need to focus on getting to the next stop, rather than the whole trip.
The Great Pipeline Post Project
I had this great idea to blog about setting up a CI/CD pipeline, both for much-needed practice and to relay some lessons learned. Spoiler alert: this is a much bigger deal than it seems at first glance. I sat down to get started and quickly got overwhelmed by the sheer volume of things to work through, especially as they occurred to me randomly, while I was sitting in front of Visual Studio.
That’s where this post came from. What if, instead of just breaking down how to setup a pipeline and all the steps under that umbrella, I started with how I planned this mini project? If my target audience would benefit from seeing all the pieces of CI/CD tied together it’s likely they’d benefit from seeing the pre-work too.
Over the next few posts we’ll start by exploring the planning that goes into standing up a new project and move on into the actual work needed. My goal, the whole time, will be to keep it on track and demonstrate how the joint efforts of knowing the plan and doing on step at a time keep things moving at a steady pace.
I talk about writing software being like magic. It’s how I view the craft in a lot of ways–using arcane words and phrases combined with strange tools to conjure amazing things out of a series of 1’s and 0’s on a circuit board. But when I describe my ultimate career goal as “being a wizard” I mean something, or someone, very specific.
I’m not talking David Copperfield or Penn and Teller. They are very skilled magicians, not wizards. That sort of magic is based on memorizing an illusion, of doing a thing in a certain way because of the reaction you know you’ll elicit. One does a very specific thing, or specific set of things, in a well organized way, over and over and while it gets more polished with practice, it will always be the same.
The words “programming is magic” conjure up Gandalf for me, from the Lord of the Rings universe. For the majority of those adventures, it wasn’t actually magic that made Gandalf the go-to guy–it was never “Gandalf has a spell for that.” What made Gandalf the truly formidable foe and an invaluable ally was the fact he knew things. He understood the old lore, the behavior of people and animals, how the elements of the world fit together and how they could either help or hinder a particular goal.
Gandalf knew the configuration of Middle Earth. Maybe not the *exact* configuration for every situation, but he knew what was missing, or what was the blocker. His magic was always a tool to divine knowledge or to force something into place, to adjust the configuration of a situation. Never magic for the sake of magic, happening in isolation.
“Gandalf knew the configuration of Middle Earth.”
Just like the veteran programmer on the team who always seems able to spot the problem
In a recent post about implementing Azure’s Application Insights in a webjob, I mentioned in passing the need to have a storage account in order to actually run the webjob. I didn’t cover then how to go about creating that because THAT post was already taking close to a minute and a half just to scroll…so here we are.
In addition to being a really convenient and relatively cheap way to store files “on the cloud,” Azure will require storage accounts to be associated with many resources (like webjobs, machine learning workspaces, and virtual machines) so those resources have a place to store their logs, temp files, another other such critical things.
What, you thought Microsoft would just foot the bill for that kinda thing, or build it into the resource price? Come on now.
Then I get a few weeks into my internship, and there’s this new idea: a webjob. Essentially, a console app that runs on the cloud for background tasks. They’re awesome…until they start doing weird things while deployed. Azure gives you a couple hundred lines of debug console for free, but that’s just not a sustainable solution. This was the point my boss said some vague words about “App Insights” and pointed at an example the other developer had worked out a while back (modified from a guide neither of us were able to locate).
It took me quite a while to figure out how to even configure Application Insights in the webjob, and even longer to really zero in on how to use it intelligently. So I wanted to combine some of that hard-earned knowledge in a very long walk thru. We put together a toy app inside the webjob template, and configure it to run and send telemetry to an Application Insights resource.