David Ruttka

I make computers do things

Git Pushit

| Comments

I wanted to do something silly this morning, so here’s my newest git alias.

git config --global alias.pushit '!f() { git push; start http://druttka.github.io/pushit ; }; f'

This will do a git push and then open a browser. The browser will either play “Push It” by Salt-N-Pepa or “Push It” by Static-X.

You Are Not in a Relationship With Your Tools

| Comments

Note: I haven’t forgotten or abandoned the Committed series.

Yesterday, I was beginning to prepare an internal talk, and I referred to some old materials I used when doing similar talks for previous teams. These talks go back years, and the concepts in them go back even further. Still, I see a lot of resistance to one of the ideas in particular.

tl;dr – Languages, platforms, libs, frameworks, and tools are not looking for a monogamous relationship. You don’t need to marry one and use it exclusively, nor do you need to leave one to experiment with another. The more the merrier. If you really must crush on your tools, at least be poly-amorous.

Rewind: The Technology Council

I cannot make this up. I was once in a position where the grand plan was to create a sub-team of four(ish?) engineers called The Technology Council. To be fair, I can’t remember if that was the official name, or just what those of us in the trenches started calling it.

These engineers would not touch production code much any more. Their responsibility would be to create a white list of libraries and frameworks that the other developers would be allowed to reference. Removed from the actual problems, they’d be able to see more clearly which tools would solve them. Of course, it goes without saying that once a single tool was chosen for Problem X, it would suit Problems Y and Z just fine.

Revolt

I expressed my displeasure with this idea. How could developers stay up to date or experiment with new approaches if they had to wait months for council approval? Then, when I eventually resigned, I was told I could be on the council if I’d stay. Apparently there was some confusion, and it was believed that I just wanted to be one of the deciders.

Before I left, I gave my team an internal talk called (prepare to facepalm) “Enterprise Agility – Surviving the Waterfall and Staying Sane to Boot.” Despite the title, I still stand behind most of the ideas in that talk, especially since they were heavily inspired by a (much better) talk I had just seen at CodeStock1.

One of the slides in my talk was titled “A Word About Golden Hammers.” Here’s one of the quotes I had in my notes.

What I am against is the thought that there is one solution to all problems, or that all problems can be solved by one solution. Every problem is different. The key, and the most important thing of all, is to understand not only what you’re using (framework, libraries, etc), but why that’s the right tool for the job. – Anthony Ferrara

The New Hotness vs. Legacy Rot

Have you recently heard anyone mocking someone else’s use of jQuery 1.6? Are you simultaneously locking into Angular so hard that you think your blog is a SPA? If so, you will be the jQuery 1.6 of five years from now2.

jQuery solved some very real problems, and arguably still does. There was a time when if you didn’t have a $(function(){...}), your reputation as a professional web developer would be in serious question!

Huh? –> This is really cool –> I use it everywhere –> Hmm, sometimes it’s not so great. – Mike Hadlow

Don’t let angular.module(...).run(...) be your $(function(){...}).

More Than You Need

These days, people cast dispersions $('#fancyDiv'), touting the power and simplicity of vanilla.js’s document.getElementById('fancyDiv').

These people often simultaneously take a dependency on the entirety of Bootstrap because they want their buttons to be blue. What follows is a lot of headaches and tweaks trying to comply with Bootstrap’s opinions.

I feel like there is always the “one true way”. The way the tool wants me to do it. … Need another way? That’s gonna be a lot of work. The “one true way” makes me feel like I am driving a slot car. There is nowhere else to go. – Rob Maher

More ironically, what follows is a lot of headaches are tweaks trying to make things look less like Bootstrap and more like what the UI/UX folks really wanted. This becomes a lot harder than the couple of lines of less or sass that would have just made that button blue.

Weapon of Choice

Mastery of a single weapon or superpower will allow you to tackle a certain set of scenarios with grace and ease, but there’s a reason that teams like The Avengers and Justice League exist. It’s rare that any one skill or power will enable you to be the hero of all conflicts.

Instead of a single weapon, what you could instead have is a utility belt. For your superpower, you could choose a curious mind and sharp, critical thinking. Then you could reason about which gadget will best get you out of each jam.

Always be yourself. Unless you can be Batman. Always be Batman. – unknown source, no less true

Disillusionment

If you decide to start using X more than the Y you were using, you aren’t “leaving” the former for the latter. At least, you don’t need to do so. If you do so, no one cares.

Please do tell us what you’re learning and exploring in X. This is how we learn and grow. What I’d like to see recognized more is that adopting X does not imply that Y gets thrown into the bin. Y’s strengths are not removed by your new discovery of X. It could very well still have value in certain applications.

Wrapping This Up

The other day I was asked what tools and languages I’m using right now to get my job done. I very honestly replied that I’ve been working with a wide variety of things including but not limited to C#, node, PowerShell, and bash. The people I work with closely and I believe in familiarizing ourselves with many options, knowing their strengths and weaknesses, and utilizing the right tool for each task. We are building components that do their job and do it well, and then composing those to create the big picture solution. I hope this concept will become more mainstream in the industry. I hope I will never have to explain again why I know both PowerShell and Node, or why I’ve started learning golang along with them.

1 – Phil Japikse’s talk was called Lessons Learned: Being Agile in a Waterfall World
2 – Or, at our accelerated rate of change, five weeks from now.

Education and Experience

| Comments

Note: This is dated July 23, but it really happened and most of it was drafted on July 8. I guess we see where finishing up blog posts sits on my personal backlog.

We’ve got some interns. I paired with one of them this morning, and we performed a mild refactoring in the area of dependency injection. What we did isn’t the point of this post. The point of this post is that he asked me later in the hallway (quoted as close as I can recall):

How did you learn all this stuff and make it so your code doesn’t look like college student code?

tl;dr – Experience.

jklolThe Evolution of A Software Engineer

I’ve said for quite some time that the purpose of formal education in our industry isn’t actually to teach you what you need to know in “the real world.” Its true value is in teaching you how to learn the things you’ll need to know. It’s important to realize that going to college and paying for a piece of paper isn’t the only way to gain this skill, just one way.

Learning On the Job, Apprenticeships, Mentoring

The truth is that once you leave the school setting, there’s a world of crap you weren’t taught. Like how to use source control properly. Like team specific conventions. Like navigating an existing codebase larger than you’ve ever seen and fixing a bug without breaking anything else. Like doing that when there are no docs, no tests, and none of the original authors hanging around to answer questions.

So one part of my answer to the intern was a collection of things that fall under the “experience” heading.

  1. Find a mentor and/or engage in pair programming
  2. Pay attention to what works well for you and what doesn’t
  3. Request feedback through channels like code review
  4. Read the code that your team is writing
  5. Use the magic of open source to do all of the above with the entire world, not just the people at your workplace!

I could do a whole post on pairing itself, or perhaps a whole series on pairing itself. All of these have value, though. The lowest common denominator is share and spread awareness of what people are doing, and pay attention.

Constant Learning

Our toolchains are very volatile. We need to be constant learners if we hope to work on interesting projects, or even to be employable until retirement age. I’ve seen colleagues who started their careers in VB6 have a hard time finding new jobs when inevitable layoffs or collapses occur because they never took the time to learn anything else.

The other part of my answer to the intern was a collection of things that fall under the “continuing education” heading.

  1. Find some books.
    1. You want some like Clean Code and Think Like A Programmer – general, conceptual, technology agnostic.
    2. You want some others that dive deep into the technology stack you’re using, like C# in Depth or JavaScript the Good Parts.
  2. Subscribe to some blogs or podcasts that cover the things you’re directly working with.
  3. Subscribe to more that cover areas you’re interested in, but not yet working with. It helps to get your head around those concepts and the vocabulary.
  4. Subscribe to still more that cover areas you’re not touching at all. See how the other half thinks, and reflect on whether what they’re doing could apply to where you are.
  5. Subscribe to some aggregators that collect a daily dose of ideas, new tech, tutorials, etc. Stay aware of what’s out there.
  6. Attend conferences, either physically or virtually, to get a sense of what’s going on beyond your own experience.
  7. For all of the above, do not blindly accept everything anyone says. Think.

The question usually becomes, “Isn’t that a lot of reading and listening to do? When you do actually WORK?” The answer is that you don’t have to read it all. You’re looking for headlines to keep you aware of what’s happening, skimming to evaluate what’s important, and you choosing which ones are worth a full read. You control your own destiny. I have 79 sources in the “Development” category of my RSS reader, and I spend 15 minutes, maybe 30 max, each morning.

To steal (and perhaps repurpose) a talk title,

It’s not what you read, it’s what you ignore. – Scott Hanselman

And that’s all I have to say about that.

Committed to Good Commits: Validation

| Comments

This is part of a series. The full list of posts can be found in the introduction.

A note before I start. I heard through the grapevine that a previous post in this series gained some popularity, and that some people held the opinion that these things are too obvious to be posting about.

Why are you sharing this? People should already know this! – Derpicus, Elite Priest of Arrogancia

Like I said in the introduction, these things are unfortunately not that obvious to many people. If every person across every team that you’ve ever worked on was on board with this stuff, you are either incredibly lucky or incredibly young. Until then, we should not feel that it is “too obvious” to discuss. Those with less exposure need to hear it. Perhaps even we who think we have a handle on proper use of source control still have more to learn from others even wiser than ourselves.

Now, where were we…

Oh, ya.

Validation

Usually, when I delivered this content as a talk, the “Validation” slide came later in the progression. I’m switching it up a little for the blog post series because without proper validation, a lot of hard work can suddenly be all for naught.

Don’t. Pollute. The. Stream.

Source control should be a safe place. Bad things are going to happen from time to time, but it should be the exception, not the rule. There are ways to be confident that you’re going to keep things clean and happy for yourself and your team.

Here are a few simple things that you can do before you push (or check-in, or whatever your source control calls putting your changes into a place where they’ll directly affect your team).

I consider this to be the minimum bar of respect and consideration for others on my team, and I feel bad if I realize I skip them1.

  1. Make sure you’re at least syntactically sound (compile, or lint, or what have you)
  2. Confirm that the app can actually start running
  3. Run whatever tests you have against your changes2
  4. Check status / diff to confirm that only the changes you intend to include are included
    1. Do you have test / debug code that should be reverted?
    2. Do you have local config changes that should be reverted?
    3. Did your tooling do anything ridiculous when you weren’t looking? (cough cough ReDerper3 cough)

Why?

Because no one wants to pull the latest changes and be blocked by a broken build, or by being unable to start, or by seeing work they did two weeks ago suddenly disappear because you resolved conflicts backwards.

Why not?

Some of the common points of resistance I’ve heard…

  1. Not in the habit. Ok, make it habit. Put some form of accountability in place that encourages people to keep things clean. I’ve heard of donating $1 each time you break it, or having to take over deployments until the next time someone breaks it. I’ve heard of dashboards and sirens and traffic lights. I’ve heard of going the other way and creating rewards for those who break it LEAST. Depending on your team culture, there’s a lot you could do here.
  2. Too much time. We’ll get to some specific ways to make all of this faster below, but even without those, think about the time you’re going to lose if you just slam code into the repo without paying attention. Time determining which of the recent pushes introduced the problem. Time your coworkers lose unable to build while waiting for it to get fixed. Time chasing bugs that really just come down to a hacky line you put in for local testing and forgot to take out because you didn’t diff.

Making This Easier

Here are some things that I’ve found can reduce some of the friction around validation.

  1. Automate all the things. If a task runner is watching your changes and constantly running tests, notifying you if they break, you can fix it before it becomes a bigger problem. gulp watch is your friend!
  2. Code review. Looking back over a unified diff helps catch a lot of those “config oops” moments, typos, and weird things that tooling does to “help us” .
  3. We’re working on topic branches right now. If I push something awful, only the people working on that topic are affected (at least, that is, until the topic is merged). Note that this doesn’t give us an exemption from validating our changes, but in case we forget…
  4. Automate even more things. Michael and I just finished setting up a process for our team that can run various validations whenever new code is pushed to any branch. We also have a dashboard that gives us visibility of the status of each branch at a glance.

What else?

You tell me. I’m sure there are a lot of things we could be doing even better.

[1] You should, too.

[2] Be sure you’re running integration/functional tests against an environment that actually, you know, has those changes.

[3] Mostly kidding. ReSharper has some great features. It’s just…sometimes…well, I think it might drink a little is all.

Committed to Good Commits: Messages

| Comments

This is part of a series on good commits.

Imagine that you had a narrative view of your project’s history, telling the story of its evolution and the intent behind each change. Imagine that our project had a Twitter timeline.

Now stop imagining, and start making better commit messages.

This is what many project histories look like:

Author    Date         Message
---       ---          ---    
druttka   6/13 10:58        
jrob      6/13 10:52   did stuff. ps why don't ruttka add messages? 
druttka   6/13 09:15        
jrob      6/13 08:45   things are done.
druttka   6/13 07:32        

What if it looked like this instead?!

Author    Date         Message    
---       ---          ---    
druttka   6/13 10:58   Fix serialization of foo  
jrob      6/13 10:52   Add support for bar
druttka   6/13 09:15   Support baz
jrob      6/13 08:45   Add Foo to IBarProvider
druttka   6/13 07:32   Reduce Wizzbang execution time

What if you could then see additional detail for the serialization fix? By looking at the expanded log statement, you might see more detailed information like

Fix serialization of foo

Needed to zingify the wizzbangs

Amazing!!

The applications for more explanitory messages are endless. Some examples include

  • Code review. Get an idea of what the author intended before you see what they implemented.
  • Diff debugging. Get an idea of what changed when, and why.
  • Reviewing old fixes. A similar issue appears later, and you can go find when it was fixed before to see how.
  • Communication. Maybe your team doesn’t have a regular communication loop, and viewing the history is how you can stay up to date on changes you weren’t directly involved in.

Common Excuses (and Counters)

I usually hear three reasons for missing or poor commit messages. Let’s see what they are and try to remove the barriers.

  1. A person claims they don’t know what to say. Just say what you did. Start with a brief synopsis on the first line. Be imperative, and include an issue number if appropriate. Something like “Implement new logging mechanism”. If there’s more context to be given about why or how this change was made, that can be provided on following lines.
  2. A person just isn’t in the habit. Enforce it. TFS has check-in policies. Git opens a text editor for you if you don’t provide a message with -m, and aborts the commit if you leave the message empty.
  3. A person claims that it’s a waste of time. That’s quite relative. Compare that with the time you already spent investigating, planning, implementing, and (hopefully) testing the change itself. Also consider the time you’ll save diffing revisions later, when you could just grep the log.

Note: Will recently did a post on good commits, too. Read it.

Committed to Good Commits: Introduction

| Comments

A few years ago, I did a talk called Committed to Good Commits at TechMixer University, CodeStock, and HUNTUG. I said I’d probably never give the talk again. It gets kind of awkward. Not many people show up when there are other talks going on about the new shiny hotness, and those who do show up already agree with everything I say. We just end up nodding our heads and smiling at each other.

For a while, I’ve considered turning the talk into a blog post series. Today is the day!

Like the talk, this series will try to be version control agnostic. While examples will be in Git, the concepts should apply equally well in TFS, SVN, SourceSafe, or what have you.

Much of this will be opinion, but these are practices and principles that have worked for me. That said, we will cover

  1. Commit Messages / Comments
  2. Validation
  3. Work Item Association
  4. Atomic, Frequent Commits
  5. Synchronization / Integration
  6. Team Signals
  7. Branching Strategies

No One Talks About This

Along the same lines, a few weeks ago, a younger coworker noted to me in passing that he never learned anything about source control in any of his computer science classes. – Jeff Atwood

When I first gave this talk at TechMixer U in 2011, a college senior expressed a similar situation. The class had been exposed to CVS, but only the most basic operations. When I was in college, I used “poor man’s version control” by putting the last good version on a floppy disk. Then I got wiser and kept n-number of “last good” versions in zip files with timestamps for names!

We Need To Talk About This

We can do better. While caring about version control and using it properly is ignored in schools and conferences, it is critical!

Source control is the very bedrock of software engineering. Atwood again

But if you don’t have source control, you’re going to stress out trying to get programmers to work together. Spolsky

People Are Starting To Talk About This

The community is starting to ask a lot more about this, and care a lot more about this. I linked to these three questions when I gave the talk, but I’m sure there are plenty more (plenty better?) examples by now.

  1. http://programmers.stackexchange.com/questions/109523/how-do-i-convince-my-fellow-devs-to-want-to-add-comments-to-source-code-commits
  2. http://programmers.stackexchange.com/questions/107884/to-branch-or-not-to-branch
  3. http://programmers.stackexchange.com/questions/98580/whats-the-work-flow-with-2-people-on-a-project

It’s Not Automatic

This stuff does take some discipline. For some people, it’s only after dealing with a painful situation due to poor version control practices that they choose to care. For others, knowing that the pitfalls exist is enough to spark them toward preventative measures, and they care earlier. In either case, it’s important to realize that bad things will happen even to the most careful developers, and that’s ok. It’s how we learn. I’ve certainly been sloppy myself, but try to get better all the time.

I’m going to try not to preach much in this series. We’re going to look at real issues, and some thought processes toward practices that can ease the pain points. It will then be up to each person / team to decide what processes are or are not appropriate for their workflows.

Git Clone: The Quickening!

| Comments

I’ve been using Visual Studio Online for a bit, so the example in this post will specifically show that, but the concept applies anywhere.

If you have a project up there named foo, the git clone URI is something like

https://you.visualstudio.com/DefaultCollection/_git/foo

It’s a real pain to type all that. So I did this in my .gitconfig

[alias]
qc-ds = "!f() { p=${1}; d=${2-$1}; git clone https://me.visualstudio.com/DefaultCollection/_git/$p $d; }; f"

Usage

To clone myproject into myproject,

git qc-ds myproject

To clone myproject into somedirectory,

git qc-ds myproject somedirectory

Breakdown

qc stands for quick clone.

ds is an abbreviation of the me part of my VSO account. It’s meaningful to me. Make yours whatever you want.

The rest creates a function and executes it. The first argument is assigned to p (“project”). The second is optional and is assigned to d (“directory”). It defaults to p if not supplied. Then it’s just filling in the blanks.

Caveat

This gets a little weirder if you have multiple repositories in a project, e.g. a second repo named bar inside of the foo project

https://you.visualstudio.com/DefaultCollection/foo/_git/bar

I know how I’m going to tackle this, but I will leave it as an exercise for the reader.

And of course, if you’re on GitHub, BitBucket, or whatever…the repo URIs are usually pretty standard. Just :s it!

Why Git Makes Things Better

| Comments

I still plan on writing some posts about how we migrated from TFVC (TFS) to Git. For now, I want to share a few stories of things that happened over the first couple of days that we were officially on Git, and express why I’ve wanted to do this for so long.

Collaboration

Rewriting History to Remove Unwanted Binaries

| Comments

We’re in the middle of a TFVC (TFS) to Git migration that I’ll probably blog about more completely later. Right now, I want to cover one thing that we’re cleaning up in the process.

These Aren’t The Packages You’re Looking For

We got into a bad situation where we were checking in our packages folder instead of letting NuGet restore handle it.

  • Some of this was because we were using older build templates in VSO and NuGet restore wasn’t working how we expected
  • Some of this was because we have private packages that aren’t on any feed that VSO can currently access

Rewriting History

If we’re going to have a new project in VSO, and we’re going to be creating a new repo based on the TFVC history, we might as well move up to the new templates and pretend those packages were never there.

The naive approach would be to just do the git tf migration, then do a follow-up commit that deletes all the packages. They’d still be in the history, the repo would still be oversized, and the index would still take the hit of having all those binaries hanging around.

Here’s a command that would do the trick for the Newtonsoft.Json.6.0.1 package alone.

DANGER! WARNING! WE’RE DOING THIS TO A NEW REPO, DURING MIGRATION, BEFORE IT BECOMES SHARED HISTORY. READ THE MANUAL AND CAVEATS BEFORE YOU DO THIS TO EXISTING, SHARED REPOS read it here

git filter-branch -f --index-filter "git rm -rf --cached --ignore-unmatch packages/Newtonsoft.Json.6.0.1" HEAD

Breakdown

tl;dr this is going to roll through all of our commits, and for each one, remove the Newtonsoft.Json.6.0.1 directory and everything inside of it from both the working tree and index. It NEVER EXISTED.

  • packages/Newtonsoft.Json.6.0.1 is the most self-explanatory part. This is the sub-directory we’re going to pretend was never added.
  • -rf should be similarly self-explanatory for anyone with a bit of *nix background. Recursive. Force.
  • --ignore-unmatch sparked a bit of discussion between Michael and me. What it boils down to is that this instructs git rm to exit with a success code even if no files match the pattern. Otherwise it would exit as a failure if no files were matched and removed.
  • git rm says to remove the files.
  • --cached removes it from the index but leaves the working tree alone.
  • This is all wrapped in quotes because it’s a parameter to git filter branch
  • git filter-branch on the left is going to rewrite history for every commit
  • --index-filter is better explained in the docs*
  • HEAD on the right side says we want to save the rewritten history in HEAD.
  • -f forces. I will cover why we added this, but not just yet.

But That’s Just One Of Them

We want to do this for all the public packages, and we want to do it for none of our private ones. We’ll need to get a list.

From the packages directory

git log --diff-filter=A --summary . `
    |? { $_ -ne $null -and $_ -match 'create mode \d+ (.*)?(/lib/.*)' } `
    |% { $($matches[1]) } `
    | select -unique | sort `
    |% { "git filter-branch -f --index-filter ""git rm -rf --cached --ignore-unmatch $_"" HEAD" }

Breakdown

  1. The first line dumps all of the adds that happened in the current directory (packages)
  2. The second line filters out blank lines, and matches the regex of created files, capturing the path as a group.
  3. The third line pulls out the matched path
  4. dlls, pdbs, nupkgs themselves, who knows what else might have been added, but our filter-branch + rm above is going -rf on it anyway. Dedupe them, and sort just for convenience.
  5. Write the command we want to execute to the output stream. This doesn’t execute it, it just outputs what you want to execute.

The output looks something like this

git filter-branch -f --index-filter "git rm -rf --cached --ignore-unmatch packages/bar" HEAD
git filter-branch -f --index-filter "git rm -rf --cached --ignore-unmatch packages/foo" HEAD

You could go so far as to not wrap that in quotes and dump it to the output stream. You could just execute it directly. But then, you wouldn’t have accounted for the private packages, so you’d want to have some kind of $safePackages = ("x","y","z") and add a |? to strip them. Your choice, but until we’ve run this through its paces, I kind of like the safety of having the chance to review what’s going to be done.

The Final Stroke, And Why We -f It

Each filter-branch creates a .git-rewrite directory and a bunch of temporary history. While this exists, the next filter-branch will not execute and ask you to clean it up. We are going to be doing this a whole bunch of times and then clean it up after all of them are done, so we just -f through it.

Then, at the end, you do want to clean it up and run a gc. How?

rm -rf .git/refs/original/ && git reflog expire --all &&  git gc --aggressive --prune

Congrats! Now before you push your fancy, newly migrated to Git repo, it’ll be a lot lighter without all those unnecessary binaries. You could certainly consider applying this procedure to all kinds of other directories where you’ve been putting garbage into source control.

* Hat tip to an old post by David Underhill that got us started.

** More thanks to Josh for responding quite quickly to my call for his favorite way to find subdirectories that used to exist but got deleted. I ended up going a different way, and just looking for everything that was ever added, whether it exists or not. And using PowerShell. He says he’s considering blogging his solution soon!!