In case you missed the big news in the industry this week, a GitLab employee accidentally deleted a ton of production data and took their platform down for hours. It was only when everything was on fire and they were in deep trouble that they turned to their backup systems… only to find that none of them actually worked.
Backup Prod Data Regularly
Not exactly a groundbreaking statement, right? Everybody knows this. If there was a “working in corporate IT 101” manual it would have a chapter on this concept. It’s common sense.
Even still, a lot of people and companies – like GitLab – tend to “set and forget” their backups. They probably created their backup mechanism years ago, tested it at the time, confirmed that it worked, and then scheduled it to run every night at 1am EST or something. Then, since it was out of sight and out of mind, they promptly forgot about it and moved on to other things. After all, they never had a need to check on it right? Nothing had broken down. Until yesterday.
A Guide To Good Backup Process
The secret to ensuring that your backup process is effective and functional is to integrate it into your daily work. One of the best ways to do this is to use it to set up a new dev’s local environment. Have them configure and install the IDE and related tools, and then have them pull down the most recent backup and restore from it to set up their local database. What’s that, you say? It has PII and sensitive data? You’re probably right, which is why your backup process should, as appropriate, create 2 copies: 1 that strips the data (for local dev env) and 1 that doesn’t (for prod restore).
Great, so you’ve confirmed that your backups work for a local environment, but what about production? The next step in a good process is simple too: artificially destroy your production environment regularly. Set up fail-over tests at off hours (and compensate your amazing site reliability / IT team appropriately for conducting these tests in off hours too). I recommend once per quarter as a starting point: at 2am on Sunday drop your production database (but don’t delete it, just take it offline so you can bring it back if you find out that your backup system isn’t working). Let your staff work to restore a recent backup and bring the site back online. Announce the outage in advance to your users, and update people on social media or via email when it begins and ends.
There is much to be learned and gained from this intrusive and destructive process. For one, you will force your dev team to create a good “the site is down” experience since your customers will otherwise see infinitely spinning web pages or terrible error dumps. Another is that you can time the average outage and thus discern how long you’ll be down if your production database ever actually takes a spill. Finally, your disaster recovery staff will be fresh on their skills and able to fix your real outages quickly and predictably. There are many tangible and hidden benefits derived from just a few hours of planned outage per year.
GitLab Did One Thing Right
The final step in your solid, functional backup process which you test quarterly and use to spin up new dev hires is to document the hell out of everything. When you do these planned outages, have the disaster recovery staff document, step by step, the actions taken to fix it. When you have real live outages, document those too and share the knowledge with the public.
GitLab got this part right, and are being heralded as a great example and learning experience in the industry instead of spited for mysterious downtimes and no communication. I promise you that this week, many disaster recovery people are doing extra backup tests that they wouldn’t have thought to do otherwise – all as a direct result of the GitLab incident. Making your disasters and their recoveries public creates goodwill in the community, provides a learning experience, and shows people that you can be trusted.
GitLab took a bad situation and created the best possible outcome, both for themselves and the entire community. For that they should be thanked, not mocked. After all, we are all human and we all make mistakes. Knowing this, you’ll be really glad that you practice making mistakes every quarter when your production database actually goes down in flames.
In part 2 of my series on dev team interactions, I’d like to talk about conducting good code reviews. Most dev teams will find themselves in a situation where code reviews are necessary, and in my experience many do them very poorly. I’ve even worked in companies that had such a negative code review culture that people left the review sessions upset, even considering quitting. With a few easy adjustments, you can quickly learn to conduct excellent and positive code reviews with your team.
The Ground Rules
A code review is a process. Like any good process, clear rules need to be established and followed to ensure a consistent experience. Here are mine:
- Attack the code, never the person. Criticizing code is OK, but people are not code. It’s never OK to criticize the person and make them feel bad. Focus strictly on the code output and never make it personal.
- Don’t laugh or make negative jokes. A person who has their work on display – often on a projector in front of others – is feeling self-conscious as it is. Don’t snicker at their work. Avoid joking about their decisions. I assure you they are trying to do the right thing.
- Set a strict time limit as a means of focus. Make the code review 15 minutes, 30 minutes, or even 1 hour. Stick to this schedule. This forces you to prioritize the important stuff and ensures (intentionally) that you can’t review absolutely everything the person has done. Don’t take unlimited time to scrutinize every single line of code written. Ever had someone comb through your code line by line, making commentary as they go? “And how did that make you feel?”
- Thank the person for sharing their code with you. A code review is an intimidating, scary thing – especially for developers on the junior side of the spectrum. Set the tone correctly by being appreciative of their time and sharing. Make it known that they are valued and you appreciate their work up front. This will help them feel relaxed and learn to enjoy code reviews, which in turn will cause them to want to share more of their code willingly in the future as well.
The above rules serve only as an example from my personal experience. You should create rules that work for your company and work culture. Use real and practical standards that matter to your team, not just theoretical ideologies that someone on your team read in a book. Always remember: a process is only as good as the people that follow it, so try to be consistent with whatever rules you decide on. And if they aren’t working well in practice, change them up!
Seek Understanding, Not Power
The general goal of your code review attendees should be to seek understanding, not explanations. Avoid an us-them conflict or standoff. This can be easily accomplished with a subtle shift in communication approach. Rather then asking aggressive questions that demand the reviewee explain the code to your team such as “why would you do it that way?” or “how could that possibly work?” you need to ask questions to promote understanding the code instead. Once your team and the reviewee correctly understand the code being reviewed, you can proceed to discuss a different approach without conflict or hurt feelings.
When you see some code that makes you think “what the hell were you smoking?” (explain) you should instead ask “Can you tell me why you wrote the method this way?” (understand). These statements are similar but the former gives the power to you while the latter empowers the code reviewee to share their knowledge and thought process in a non-defensive way. This approach to questioning takes a bit of practice, but is very powerful. Some other examples of how you’d change common code review aggressive thoughts and statements from explaining to understanding:
- “This class name is wrong” becomes “This class name doesn’t conform to our standards, was that intentional?” You’ll probably find that they say it wasn’t, and agree to fix it on their own volition.
- “I see a bug in your code!” becomes “I think there’s a null reference exception on line 28 for variable X, do you see how it happens?”
- “This is terrible” becomes nothing – keep your mouth shut. There’s no value in such a statement other than to make the person feel bad. If the code truly is terrible, express these concerns to the person (and maybe their manager) privately for further investigation and resolution.
People are not robots, and will never conform perfectly to your shop’s coding standards. That’s OK. Pick your battles, and call out only the major violations. Let the little things (like naming and spacing) slide. If the dev is hitting 90% of the standards, the rest of the team can pick up on the 10% deviation without much worry or effort. That is to say, the code won’t be that different than what they expect to see.
If you seek understanding consistently, you’ll find that the person drops their defenses and ego, and instead feels encouraged by your positive approach and attitude. They’ll even start pointing out and suggesting fixes to their code themselves – right on the spot – which is the golden sign of trust and confidence. The best kind of code reviews are the ones that a person points out issues themselves to fix, rather than the team having to do it for them.
Avoid Opinion Wars
When reviewing code, there will be two general categories of issue:
- Objective, fact-based issues
- Subjective, opinion-based issues
Focus explicitly on objective issues, and disregard all subjective issues. An objective issue is an exception or oversight in conforming to well-defined coding standards. A subjective issue is you not liking the way the person solved the problem, or feeling that what they did is not what you would have done. You’re right because you’re different people. Striving to make others conform to your thought process as the one true standard is egotistical, destructive, and stressful. In the long run you will be unhappy because there will always be a gap between others and yourself. This is natural so go with the flow. Allow members of your team to be individuals and write code with their own flair and flavor. It’s OK, it will be OK, and you will be OK. I promise.
Code is for computers, but programmers are humans. Be kind to each other and always remember that the thing on the other end of the code review is a real live person with feelings and emotions. Treat them with respect in what you say and do. Recognize that they are valuable and thank them for their hard work. Together you will create a great team that others will love being a part of!
As a developer working for a company, you probably work on a team. The interactions on these teams are sometimes pleasant, and other times hostile. What’s interesting to me is that a lot of the time, a hostile interaction could have been a pleasant one if only approached differently. Hostile teams are created by the actions of the people on them, not by the situations they encounter. One such hostile action is blame.
Blame is assigned externally – it comes from other people. You cannot control blame when it is directed at you. Blame tends to surface under tense circumstances, such as when the build has broken or a project has failed. It often rears its ugly head in the form of pithy and simplistic statements. “Jane broke the build again!” or “this is Mohinder’s fault!”
Blame is often facilitated – if not encouraged – by a poor workplace culture. When the patterns and practices of the management team demand that heads roll when mistakes are made, blame is the obvious solution. It allows you shift the burden of responsibility to someone else, forcing them to defend themselves from an attack while you retreat to safety.
Blame Never Helps Anything
Blame may solve the immediate problem of your head being on the proverbial chopping block, but it never solves the actual problem at hand. If anything it tends to distract from the issue as the discussion changes from “what went wrong” to “who caused this?” A challenging situation becomes even more complicated as strong negative feelings start to emerge.
Another effect of blame is that people feel alienated and attacked. Team members disengage, afraid to speak up or make changes because they may be attacked for doing so. In this way it damages the innovation and progression of your people, product, and entire company. Blame holds your entire organization hostage.
It’s not hard to see that a workplace which embodies blame culture is really the root problem in-and-of itself.
Unlike blame, accountability is assigned internally – it comes from within. It is a positive force of empowerment that requires courage and compassion. Accountability allows people to take control over themselves and their own actions. It destroys “fault” culture while building trust and communication channels with other team members. Statements like “that was my fault team, sorry about that” are welcomed and encouraged, but not necessary.
Accountability can only thrive if the workplace culture is tolerant and accepting. In a blame culture, accountability cannot exist. The mere act of owning up to a mistake assigns the blame to you, and scrutiny and / or punishment is sure to follow. A good work culture is intolerant of blame, instead focusing on people learning from their mistakes and growing as professionals.
A Blameless Workplace
My personal take on accountability as a manager is that “everybody makes mistakes, and as long as you make new and interesting mistakes – rather than repeating them – you have nothing to worry about.”
Mistakes are normal; we all make them regularly. A workplace culture that understands this fact and incorporates it into professional and personal development is one that succeeds with vibrant and engaged teams. Allow people to make mistakes, and follow-up only if they do not learn and grow from those mistakes. Shut down any and all blame conversations immediately. Communicate the difference between assigning blame to someone for their mistakes, and taking ownership of those mistakes themselves. Foster a workplace culture that is blameless.
At Stack Overflow, we do this via a tool that we call the Wheel of Blame. When something goes wrong, anyone in a company chat room can say “not my fault” to spin the wheel. The wheel arbitrarily picks a person from those present in the room and blames them. With blame quickly and definitively assigned, we then move on to fixing the problem.
Despite the Wheel assigning blame – which clearly goes against the blameless culture which I described earlier – this tool is very powerful. It’s a clear and obvious signal to seasoned and new employees alike that we don’t care about blame. When someone is randomly assigned, it’s almost always the case that the problem wasn’t actually their fault. More often than not they had nothing at all to do with the current situation. An ironic and hilarious side effect of the Wheel is that when it does randomly assign the blame to the person who is accountable to the issue, it results in uncontrollable laughter and joking. This alone makes the Wheel worth using: it lightens the mood and communicates our rules around blame in a tribal way (there is no policy written anywhere).
To shift a culture from blame to accountability, begin by shutting down blame discussions. Explain that blame is not helpful, and that the team needs to immediately focus on resolving the issue at hand. Once the problem is solved, consider doing a retrospective which explains what went wrong and who was involved. Circulate this retrospective to any and all interested parties with the clear intention of spreading information, not assigning blame. This allows people to learn and grow from what happened. Continuing this pattern will slowly shift the team away from blame. The trick here is that you don’t have to work on accountability: it will naturally follow when people stop fearing blame.
I encourage you to work towards a culture of accountability on your development team, starting today. Sometimes this transformation can be very difficult to accomplish; particularly when the blame culture starts at the executive level. Nonetheless, do what you can to change the behavior in the part of the company that you influence. Your peers and reports will thank you for it.
Well, I’ve utterly failed to blog at regular intervals, writing only three posts in 2016. Ouch. To be fair, one of those posts is insanely famous (the one about NPM and left-pad.js), but still, I’ve really let my readers – and myself – down.
So, I resolve to write a blog post every single week of 2017, starting today. This will probably mean that I write slightly shorter posts, and maybe even multi-part series posts. My traditional style has been “come upon something that is really bothering me or is really tricky, and proceed to blog about it in great detail writing thousands of words for all to benefit from” which doesn’t really scale well. Instead I plan to take the approach of “write about a new or interesting topic each week, and see what people like and what they don’t like” which will hopefully be better.
This week’s post is about compassion, especially in the field of programming. Let’s use an example that is both recent and practical, if a tad emotional.
3 days ago our dog Ruby died. She was 16 and lived a good, long life. Having said that, her death was unexpected and rather painful to go through. My wife and I were up with Ruby all night comforting her as she slowly passed away from sudden heart failure. The last breath she exhaled will forever be seared into my brain. It feels like it’s all I can think about right now.
My wife and I have been grieving for the past 72 hours in our own ways. This has surfaced as mostly a mix of depression, tears, quick but sad chats about the things we miss about Ruby, and sorrow every time we look somewhere in the house for her only to find that she is no longer there. It sucks.
What’s important to realize is that I am not unique or special in this circumstance. Animals die all of the time, and you don’t feel sad about them because you never met them. They are out of sight, out of mind. Yet they are out there, and the passing of good dogs and cats – as well as other households pets – is happening every single day. Even as you read this.
These events occur in the scope of individual’s lives. There is no national news story about the passing of Ruby to make us all aware of it and help us all feel the effects of it. There never will be. This event is confined to the lives of me, my wife, and our extended family. We suffer through it silently (other than my writing about it and a few social media posts).
This silent suffering is the point that I want to drive home. Others out there are silently suffering too. It might be the death of a family pet, or it might be something else entirely. It’s not a contest of who has it the worst. I want you to simply realize that crappy things are happening to people, even if they are not in your domain and thus you are not aware.
Odds are that someone in your life or workplace is suffering with something that troubles them. Right now for me it’s the passing of my dog. For a close friend of mine it’s stress about work and finding the next permanent job. For another person I know it’s about being a brown man in America and how scared they feel about the future. We all battle demons every single day.
With this in mind, I ask you to be compassionate everywhere, but especially at work. You work with people, not cogs of some machine (even if management may think of them in that way). These people have real feelings and some of them are surely fighting battles you’ve never even fathomed.
Your responsibility as a good member of the IT field – and as a good citizen of this planet – is to demonstrate compassion. Be nice to everyone you work with until you are given a reason not to be. Consider that they might be having a bad day if you feel they’re lashing out or being an asshole, and forgive them. Try to sympathize with what frustrates them. Work to understand them beyond their role at the company and their job output.
We are all in this together. Now more than every we need to be compassionate, respectful, kind, and considerate to others. It is my opinion that our industry has a long history of being hostile to certain people and demographics – such as women – and we must each work individually to change that.
My last request to you: please think hard about what you’ve read here as you begin your first week of work in 2017. I know I will.
Picture this situation: you woke up this morning to find that there’s no water coming through your valves and taps. No sink water. No shower water. Having no plumbing experience, you call around for a plumber.
Plumber #1, let’s call him Mario, tells you he can’t be bothered to come check out your issue because it’s minor and he’s very important and too busy for it. You explain that you really need a plumber, and he explains he’ll do it for 1.5x what everybody else costs, and only if you have lunch and coffee ready for him when he arrives. You have no water, keep in mind, so making coffee is an extra special effort.
Plumber #2, let’s call him Luigi, agrees to show up and assess the damage. He walks in your door, doesn’t bother to shake your hand or introduce himself, and immediately gets to work. A few minutes of inspection later and Luigi is telling you that all of your plumbing is wrong. None of the pipes are plumbed how he’d plumb them, so they’re wrong. He needs to rip out the whole plumbing system and do it all again from scratch, which will cost you thousands of dollars. He has a reputation for being skilled: his past references love him, but this is going to cost a lot of money and take a lot of time. It’ll be at least a week before you have running water again.
Plumber #3, let’s call her Peach, asks you some detailed questions about the problem over the phone. “How long have you lived in the home?” “Have you had any prior plumbing issues?” “Has this happened before?” “Do you have any other sources of water?” Based on this analysis, she tells you she has a good idea of what the issue is and agrees to come over later that day, since she’s in the area anyway. She shows up, immediately discovers that the pesky neighbour kids shut off your water main, turns it back on, and then gives you some advice: “it’s not something you need to do right now, but you should upgrade to a locked main switch that operates with a key so that this won’t happen again! Call me if you want something like that installed.”
Now, which one would you hire?
Back to Programming
Let’s take this [terrible] analogy back to programming. Plumber #1 is a rock star developer who demands special treatment and pampering. Plumber #2 has a huge ego, a serious case of not-invented-here syndrome (they didn’t plumb the house so it’s wrong!), and might also be a rock star. Plumber #3 is pragmatic, friendly, sensible, analytical, and balances cost of time and money with urgency. For me personally, #3 is the only one I’d happily hire.
It seems so obvious that egos and not-invented-here cost a lot of money and do a lot of damage in that analogy. It’s less evident in the workplace. Some companies foster the ego and demand to hire rock stars, believing that the “ten ex” developer will outperform others they could hire. What they don’t realize is that the cost of this hire almost certainly outweighs the benefits.
There’s no place for ego in programming. An egotistical programmer does incredible amounts of damage to those around them. They do cultural damage to your company, emotional damage to their peers, and financial damage to your budgets. Those rewrites aren’t cheap!
Don’t be a rock star. Rock stars are cold, hard to approach, notoriously unfriendly and self-absorbed, make no time for others and teaching them (because everything’s about ME!), and are frequently dramatic and rude (to get in the “news” – after all, any publicity is good publicity).
Seriously, "rock stars" are arrogant narcissists. Plumbers keep us all from getting cholera. Build functional infrastructure. Be a plumber.
— Molly Sauter (@OddLetters) June 17, 2016
I don’t know about you, but I want to work with Peach. I prefer a humble, thoughtful, talented peer that can teach me while maintaining respect and a relationship of trust. That’s the kind of workplace that I get up every morning excited to go to. Peach is probably not perfect, and has her flaws, but the lack of an ego and attitude means she’s willing to learn and grow – and that’s the key to success in this industry.
If this analogy hit a little close to home, it’s time for a little introspection. The beauty of the human condition is that we are always on a path, growing and changing. Who you are today is not who you are forever. Think about which plumber you resemble above, and then adjust. Next time you’re about to say “that’s all wrong!” or “we have to write it ourselves!”, pause and think: there’s probably a decent a reason that the code exists in its current state. Is now really the time to undertake a very expensive rewrite / greenfield project? Probably not.
Let’s say you were walking down a street one day and noticed an ad for help wanted. It is posted in the window of a bakery. It reads:
Need a baker for FT work. Must be familiar with modern baking methods such as ovens, barbecuing, and deep fryers. 5+ years experience with the Super 6 commercial baking oven required (aside: came out in 2014). Nice to haves include experience with butcher’s blocks, chopping meat, and making candles.
Given that most of us understand the high-level job of a baker pretty well, it’s easy to see how totally ridiculous this job listing is. Let’s break it down:
Need a baker for FT work.
Doesn’t say too much, but it makes sense and describes your need accurately.
Must be familiar with modern baking methods such as ovens, barbecuing, and deep fryers.
Most bakers would be familiar with ovens, but no baker in their right mind would ever “bake” with a barbecue or deep fryer. At this point we begin to question whether or not this bakery has any idea of what it’s doing at all.
5+ years experience with the Super 6 commercial baking oven required (aside: came out in 2014).
Well, they want me to have 5 years of experience (or more) with an oven that has existed for only 1 year? That’s just silly. Now I think that they’re sort of out to lunch and don’t really want to work here.
Nice to haves include experience with butcher’s blocks, chopping meat, and making candles.
A baker is a very specialized position. There’s rarely going to be one who has specialized in not just baking but also butchery and candlestick making (catch my sly joke there?). A place asking for all 3 is really asking for 1 person to do 3 jobs for the price of 1 job. This isn’t fair and further dissuades the candidate.
This job posting sounds funny because it’s plain to see just how insanely off-kilter it is. However, recruiters are out there right now posting tech job listings that are just as ludicrous. The problem is that they haven’t bothered to learn much about the field that their job is focused around. So, they end up sounding as silly as the bakery owner does above.
Recruiters: this is how silly you sound to developers when you post a listing full of jargon that you don’t understand. Most of us don’t want to ever work on both C# and Java at the same time, and the odds of finding both an iOS and Android pro in one human being are slim-to-none.
All that is required for a recruiter to gain a significant competitive advantage over their peers is a little knowledge. Knowledge that could be learned in weeks. Knowledge of a standard tech stack, and the ability to Google technical acronyms like MVC and EF when they show up. If you just took a little time to make sure that your job listing was not a walking contradiction – full of statements that counter each other, technologies that don’t work together, and demanding 5+ years of experience on things that have existed for less than 2 years – you’d have a huge leg up on your peers.
Developers are rational, logical people. We will engage in the jobs that make sense. If your listing is irrational or illogical, we will avoid you like the plague. We will also make fun of your job listing to our friends, especially when you
spammed e-mailed us with it directly.
So, if you’re a recruiter reading this, and you want to get ahead, it’s very simple. Learn the tech that your listing is based around at a high level (in my opinion a basic proficiency of your profession), or failing that, consult a developer to make sure that your listing doesn’t sound crazy!