In case you missed the big news in the industry this week, a GitLab employee accidentally deleted a ton of production data and took their platform down for hours. It was only when everything was on fire and they were in deep trouble that they turned to their backup systems… only to find that none of them actually worked.
Backup Prod Data Regularly
Not exactly a groundbreaking statement, right? Everybody knows this. If there was a “working in corporate IT 101” manual it would have a chapter on this concept. It’s common sense.
Even still, a lot of people and companies – like GitLab – tend to “set and forget” their backups. They probably created their backup mechanism years ago, tested it at the time, confirmed that it worked, and then scheduled it to run every night at 1am EST or something. Then, since it was out of sight and out of mind, they promptly forgot about it and moved on to other things. After all, they never had a need to check on it right? Nothing had broken down. Until yesterday.
A Guide To Good Backup Process
The secret to ensuring that your backup process is effective and functional is to integrate it into your daily work. One of the best ways to do this is to use it to set up a new dev’s local environment. Have them configure and install the IDE and related tools, and then have them pull down the most recent backup and restore from it to set up their local database. What’s that, you say? It has PII and sensitive data? You’re probably right, which is why your backup process should, as appropriate, create 2 copies: 1 that strips the data (for local dev env) and 1 that doesn’t (for prod restore).
Great, so you’ve confirmed that your backups work for a local environment, but what about production? The next step in a good process is simple too: artificially destroy your production environment regularly. Set up fail-over tests at off hours (and compensate your amazing site reliability / IT team appropriately for conducting these tests in off hours too). I recommend once per quarter as a starting point: at 2am on Sunday drop your production database (but don’t delete it, just take it offline so you can bring it back if you find out that your backup system isn’t working). Let your staff work to restore a recent backup and bring the site back online. Announce the outage in advance to your users, and update people on social media or via email when it begins and ends.
There is much to be learned and gained from this intrusive and destructive process. For one, you will force your dev team to create a good “the site is down” experience since your customers will otherwise see infinitely spinning web pages or terrible error dumps. Another is that you can time the average outage and thus discern how long you’ll be down if your production database ever actually takes a spill. Finally, your disaster recovery staff will be fresh on their skills and able to fix your real outages quickly and predictably. There are many tangible and hidden benefits derived from just a few hours of planned outage per year.
GitLab Did One Thing Right
The final step in your solid, functional backup process which you test quarterly and use to spin up new dev hires is to document the hell out of everything. When you do these planned outages, have the disaster recovery staff document, step by step, the actions taken to fix it. When you have real live outages, document those too and share the knowledge with the public.
GitLab got this part right, and are being heralded as a great example and learning experience in the industry instead of spited for mysterious downtimes and no communication. I promise you that this week, many disaster recovery people are doing extra backup tests that they wouldn’t have thought to do otherwise – all as a direct result of the GitLab incident. Making your disasters and their recoveries public creates goodwill in the community, provides a learning experience, and shows people that you can be trusted.
GitLab took a bad situation and created the best possible outcome, both for themselves and the entire community. For that they should be thanked, not mocked. After all, we are all human and we all make mistakes. Knowing this, you’ll be really glad that you practice making mistakes every quarter when your production database actually goes down in flames.
I’m going to discuss an important topic that affects everybody in tech: diversity.
No, this won’t be some preachy post about how diversity is great and how you should be a better human being. Rather, I’m going to tell you about the things I’ve experienced working on diversity – particularly the interesting events of the last few days that happened internally at Stack Overflow.
It’s no secret that the tech industry is not that diverse. It’s mostly dominated by white males, with a few women and minorities making appearances. Those who do enter the industry as a minority often feel marginalized and excluded.
Of about 40 engineering employees at Stack Overflow, maybe 8 are not white men. We can clearly do better.
As you may have learned in my last post, earlier this year I was promoted to Engineering Manager at Stack Overflow (formerly Stack Exchange (formerly Stack Overflow)). One of my first projects was to work on diversity at our company. As a part of this goal, I was given the opportunity to create a new diversity page to showcase our efforts.
There’s a volunteer group that we have internally called the Diversity & Inclusion Panel (DIP). This group works to make Stack Overflow a more open, diverse, and welcoming environment.
There were some awkward feelings that came with taking the lead on the diversity page project. For one, I was a straight white male running a project about diversity, which might raise some eyebrows or make people feel weird/jaded. To mitigate those feelings, and also because I value their opinions, I took the approach of including the DIP in the creation and review of the page. My rationale was that by including a group of self-selected folks who are passionate about diversity, we could reach a solid page design that wasn’t biased by building it myself. To this end the DIP were consulted multiple times and ultimately had the final say on the copy of the page.
Diversity Page Goes Live
After a few months of iterating with designers and the DIP, last week we rolled out the new page. Here’s what it looked like in all its glory (please don’t critique it in the comments – if you do, you missed the point of this post):
I was excited that we had made progress on such an important and sensitive topic. I announced the roll-out to the DIP group with little feedback or response, which at the time I didn’t think much of. I also broadcast it on Twitter, with very few others from Stack Overflow doing as such themselves. Again, I thought nothing of the lack of engagement.
A few days go by with the new page rolled out where things are mostly business-as-usual from my perspective. And then, last weekend, something really interesting happened.
The Blog Post
An employee at our company wrote a blog post titled “Discussing diversity terrifies me” on their personal blog. They have since taken the post down – a decision they made themselves. The post described this person’s feelings about our new page and how it made them feel uncomfortable in ways that they couldn’t fully explain. This person also explained how they felt terrified to discuss diversity with anyone, ever – even their best friends. This was because of prior experiences at prior companies where the discussion went poorly.
I came across the post on Saturday night at about 9pm. It did not call out me or the page as doing something wrong; it was a way for this person to express some thoughts that were very important to them, but that they otherwise felt they could not bring up.
I Felt Wronged
Upon reading it, I was really pissed off. I couldn’t believe that this person, given numerous opportunities to shape the page as part of the DIP, had remained silent during the project. I couldn’t believe that they didn’t come to me or anyone else at work, and instead took to talking about their issues on their blog, in public, which at the time I felt served to embarrass me in front of the company. After all, they were given tons of opportunities to talk about it internally, so why didn’t they? I felt slighted and wronged. I selfishly felt that this was all a personal attack on my work. I wasn’t happy.
The Company-Wide Revelation
On Monday afternoon I had a private conversation with the person who wrote the blog post. We had a really constructive chat about how the person felt, and ways to improve upon the new page and make it more sincere and reflective of our true values and current workplace diversity. It ended on a positive note. Later Monday evening, I sent out an anonymous survey to solicit broad feedback from folks on how to improve the page. The plan was to iterate to a better version ASAP. Satisfied with trying to improve the situation, I went to bed.
On Tuesday I found myself in a few conversations about our new page. The people engaging me were mostly straight white males (not unlike myself) who all said something similar: “talking about diversity scares me to death” and “it feels like one wrong word and I’m fired.” These conversations continued until about 1pm, when a forward-thinking co-worker realized that we all needed to get on the same page. They brought the diversity page controversy up in our company-wide chat room for literally anyone and everyone to discuss. Many people jumped in and what followed, for lack of better description, was quite an epic conversation. It involved a lot of people from a lot of departments – both people who were visibly diverse and those who were not – and even included 4 executives. The conversation rolled on for a few hours and culminated in a few important outcomes:
- The person who wrote the blog post felt very uncomfortable talking about diversity at all, with anyone
- Their blog post gave a voice to many others who felt the same way but were afraid to speak up
- The diversity page was making people within Stack Overflow uncomfortable
As a result of the company-wide chat, the new diversity page was taken offline.
I Felt Even More Wronged
Conveniently, all of this discussion happened while I was at the doctor’s office for a check-up. As a result of the timing I didn’t get to participate at all. When I returned from the doctor and read the chat transcript, I became very, very upset and angry. I felt betrayed by the company and wronged by almost everybody who participated in the conversation. In hindsight, I think that I probably would have said some things that I’d now regret had I been around to participate in the chat. Ignorance is sometimes bliss.
I was boiling over. Many unfair, selfish, and angry thoughts crossed my mind as I read the chat transcript. I thought things like “f*** it, I’m never touching another diversity project ever again” and “how could these people do this to me?” and even, for a brief moment, “what am I doing working somewhere where I’m undermined and treated unfairly?”
I called it quits a few hours early on Tuesday and had a few drinks of the alcoholic nature. I was fuming, and ranting to my wife, and otherwise upset about how I had been so wronged by the people that I was trying to help. I took some ibuprofen (for the impending hangover and headache) and passed out, unsure of what Wednesday would bring.
Worth noting, I don’t recommend self-medicating with alcohol when you’re upset – it’s a bad strategy.
This Wasn’t About Me At All
I awoke Wednesday morning with a clear mind and new perspective on the events of the past few days. I realized some really important things:
- Absolutely, literally, none of this was about me. I was being selfish and making it about how I felt, when in reality it was about how we all felt.
- We had created a diversity page that did a poor job of accomplishing our goals on diversity.
- Taking the page offline was the right thing to do, because it was making a lot of people in our company uncomfortable. If it had this effect internally, it surely had a similar effect externally. In this case, the page existing was perhaps worse than having no page at all.
- The person who wrote the blog post was completely in the right, and arguably did the right thing. The alternative was that if they hadn’t spoken up on their blog, nobody may have. This would leave everyone silently feeling uncomfortable.
- The person who wrote the blog post felt uncomfortable speaking up about diversity at work. They talked about it on their personal blog to keep it outside of the company. This was an important revelation as it showed me that we have a serious issue at our company: people feel as if they don’t always have the right to speak up about difficult subjects like diversity. One person described it as “a manager’s word being law and difficult to contest.”
- Nobody had been malicious at all. We were all trying to do the right thing, assumed the best of intentions in each other, and were simply expressing our honest feelings on the topic. Feelings are always valid.
Diversity At Stack Overflow
Through all of this I realized some of why diversity is such a hard topic for people to discuss: no matter what position you are in, you probably feel like you’re not entitled to participate in the conversation. When nobody feels like they’re able to talk, silence soon becomes apathy.
On the topic of diversity issues, apathy rules all. It’s a safe and easy play to do nothing at all. Taking a stance means meeting resistance and having conversations where everyone feels invalidated. Unfortunately, doing nothing enables and furthers the issue.
One of the mistakes that I made was not considering the emotional response of the new diversity page that we created. How the page makes people feel is the only measure of success. The people in our company felt that the page was insincere and made them uncomfortable.
Here’s the truth: despite what the page said, we’re not great at diversity. We have about 40 engineering team members, and only 4 of them are women. At most, 8 or 9 total are of a visible minority. We certainly have a diverse team in the sense that we’re located all over the world, but that isn’t how everybody defines diversity, and maybe isn’t even the most important form of diversity since those geographically diverse people aren’t necessarily marginalized.
It’s really easy to feel as if your company sucks at diversity when any discussion is met with frustration and hurt feelings. However, I am really proud of the fact that at Stack Overflow we are able to have these conversations. We had a company-wide discussion on how the page made everybody feel, and even though that discussion got extremely heated and many people felt many strong emotions, not one person made it personal. There were 4 executives in the conversation and nobody quit or got fired.
I’m thankful that one person had the courage to write a blog post and speak up about their overarching fear of discussing diversity. What scares me as a manager is the idea that people might feel as if there’s no channel to talk about things that make them uncomfortable. If people don’t express their concerns, they suffer silently. It was the actions of one person that saved us all from a much worse outcome.
Ultimately, we made a company diversity page that’s like every other company’s diversity page, at a company that prides itself in not being like any other company. We’ve identified that people at Stack Overflow sometimes feel like they are unable to speak up on difficult topics – both inside and outside of the company – and that’s something that we need to work on.
As I said earlier, the easy and safe play is to do nothing. We could leave things in their current state of no longer having a diversity page, but it would also be the wrong thing to do.
What we must do now is continue the conversation. We need to talk honestly about diversity and discuss the feelings that we have around it. We need to channel those feelings into a better diversity page that accurately reflects our company and how we feel on the subject. We need to admit that diversity is hard, but that we’re working on it. Most importantly, we need to assume the best intentions of each other and make progress together in all of our diversity initiatives.
I hope that in reading this, you feel inspired to start or continue the conversation about diversity in your workplace. Like many companies, we have a long way to go. It’s challenging, but as a co-worker said, “the hardest conversations are often the ones most worth having.”
In February of 2015, I was promoted to Engineering Manager at Stack Overflow. This has made a lot of people very angry and been widely regarded as a bad move.
There are tons of things I’ve learned so far, some of which I’ve learned the hard way. There’s also a world of difference between managing code, and managing people who code. Your day to day work routine changes completely. You define success differently. You feel a little bit like you just rebooted your career and are starting over at the bottom of the skills ladder. It’s intimidating.
I’m going to discuss my experiences and insights over the last 6 months as a new manager. This one goes out to all of the developers out there who wonder what it’s like to be a manager, or are considering taking the leap into the realm of Pointy Haired Boss.
Trust is the most important thing
To be a successful manager, you must earn the trust of your team members. People won’t work for someone that they can’t trust. Combine this with the fact that a large reason that people leave their jobs is their boss (even above money, benefits, commute, or vacation), and it’s clear that this is your biggest priority as a manager.
Trust is earned, not given. A manager can’t talk people into trusting them; you have to show them that you can be trusted. In my experience, the time it takes to earn trust varies from employee to employee. I had the advantage of having already worked with my employees for almost a year, so some people trusted me on day 1 as a manager based on prior interactions. Others took a few weeks or even months to come around. The important thing to realize is that you can’t rush developing trust. Patience is key.
To earn your team’s trust, you must demonstrate good professional and personal character:
- Hold everything told to you by employees in confidence. Don’t gossip or talk about employee issues with other employees.
- Have an “open door” policy: make it clear that people can talk to you about anything, at any time that you’re free.
- Be a person of your word. Say what you mean, and do what you say.
- Don’t promise anything that you can’t deliver on. It’s better to say “I’ll see what I can do, but no promises” about a raise, promotion, or bonus than to say “sure we can work that out! No problem!” and then not deliver.
- Never lie to anyone. Being a liar in the eyes of your employees is the fastest path to failure. They will not only distrust you, but actually start avoiding you as well.
- Be direct with your people about things that you need to address, especially when it’s uncomfortable to do so. Don’t dodge subjects. Don’t avoid elephants in the room. These problems grow and only get worse if left unresolved.
- Never be passive-aggressive. Don’t gossip about people. Don’t talk about person A to person B behind their back. It’s important to have an outlet to vent, but make that outlet your spouse or family in complete confidence – not your employees.
- Praise publicly, punish privately. Never alienate someone in front of the team.
- Be as objective and fair as possible at all times. Don’t give anyone special treatment or status.
All of these things (and more) are essential characteristics of a good manager.
Code scales, humans don’t
The great thing about code is that when written properly, it scales very well. The same code can easily handle both the 1 user case and the 100,000 users case if it’s efficient and optimized. As a developer this is what we aim for, and it’s easy to measure.
A manager, however, doesn’t scale. The work is done on humans, not computers, and human interactions don’t scale. Just imagine trying to reach consensus on a subject in a meeting of 20 people. It’s much easier to do in a room of 10 or fewer.
No manager can effectively manage 20 direct reports. In fact, people experienced in the field feel that 4-10 employees is the maximum. It’s important to recognize that you can’t get stretched too thin; this will cause you to pay too little attention to your employees and make them feel resentful and unimportant. It will also prevent you from getting any work done. If your team grows too large, split them up and get another manager on board to share the work.
Make people a priority
As a developer, people were an interruption and distraction to my work. As a manager, people are my priority. You must always prioritize people. Not just your people, either. It’s not uncommon for people from other teams to want to talk to you for an outside perspective – make those conversations a priority as well. This means being flexible: if they want to call you and chat at 5pm on a Friday, try and make it happen. If they work in another time zone and want to meet at 7am on a Tuesday, get up early and do it.
Another important aspect of prioritizing people is having a regular 1 on 1 with each of your employees. The 1 on 1 is your employee’s safe place to talk. They will tell you about what’s going on, bring up personal or professional issues, or give you feedback on things that you can do better. I have a recurring 45 minute meeting with each of my employees every 3 weeks. They have been incredibly valuable to me so far.
Make an effort to be organized and use your time effectively. Do everything that you can to avoid rescheduling, being late for, or missing a 1 on 1. If you do this regularly, your culture and morale will suffer greatly. Nothing says “you’re not important” more clearly than missing or rescheduling a 1 on 1 for no good reason.
Embrace uncomfortable conversations
As a developer I sometimes found myself gossiping about, or venting to, other developers about people and policies that were annoying me at work. As a manager, your job is to help people address their problems with each other directly. Encourage people to approach the other party to address their concerns, rather than gossip or vent to co-workers. Be sure to lead by example and practice this consistently yourself.
When someone comes to you with an issue they’re having with someone else, get both parties into a private room with you. Mediate a constructive conversation where they directly discuss their feelings and resolve their concerns. These situations can be awkward and uncomfortable, but you must embrace them and make them a part of your culture. Do this consistently until people start doing it without involving you at all. Make sure that you don’t let problems go unresolved; they only get worse with time.
Success is really hard to measure
The great thing about being a developer is that you usually have a specific problem to solve. “The application needs to do this new thing,” “Fix this bug,” and so on are well scoped problems. When you finally complete the work you get the satisfaction of having accomplished something. You might even get some accolades in a team meeting or town hall.
Management is not a race but a long-distance marathon. You will not be crossing any measurable finish lines anytime soon. As a manager, your work is in affecting the often-intangible aspects of human beings: their thoughts, feelings, soft skills, and overall growth. These types of progress are on-going and arguably never-ending. We are never truly finished growing, learning, and bettering ourselves.
Instead of “fix this bug,” a manager’s task is “help Person improve their interpersonal communication style” or “grow Person’s soft skills so that they can be promoted in a year” or even “figure out a way to help Person get to work on time more often because they keep sleeping in.” None of these are break-fix problems, and thus it’s rare to feel like you’ve succeeded or finished something you’re working on. I find that feelings of success come from the feedback of others: an employee thanking me for some great advice or feedback, someone telling me they’re noticing a positive change on my team, or even a “keep up the good work!” from someone outside of your team. Nothing feels better than knowing you’re making a positive impact in the day to day of your people, but it again requires patience to see results.
Nobody sees you do work
As a developer, your work is done overtly. You’re a better developer for being outspoken about what you’re working on, and creating transparency so that anyone – from other developers to the CEO – can see what you’re working on and what’s getting accomplished. It’s easy to show that you’re doing some real work.
As a manager, almost all of your work is done behind the scenes. As a result, being a manager becomes this “out of sight, out of mind” thing where some people will perceive you as doing very little work. This is because when I talk to an employee about their communication style and issues they’re having interacting with others, I do it privately. It’s not as if I can send a status report e-mail to the team that says “I had a chat with Person about their communication issues, high fives everyone!” As a result, nobody other than the person I talked with directly knows that the interaction happened. Thus, to others, it might appear that I never addressed the issue.
The way that a manager shows that they’re doing work is in observable team member improvements. It’s a slow process (like many things management), but it will become obvious that you’re doing good work when the team starts to notice that Person is interacting in generally more positive ways, or on time for work more often. As your people grow and develop, your team will become stronger and more effective; people may remark on your team’s positive changes as well.
Ultimately, your success is the success of others
If you’re in it for the praise, you’re gonna have a bad time. Management is about trust, having direct and sometimes difficult conversations, and doing your work behind the scenes but in direct and fair ways. It’s a very empowering, challenging, and rewarding job that I’m enjoying immensely.
One of my favourite quotes from Futurama was pretty much written for managers, so I’ll leave you with this:
“When you do things right, people won’t be sure you’ve done anything at all.”
I was working on my fireplace this past weekend. Specifically I had just finished ripping down the old surface to the red brick, and then preparing the brick surface with a layer of thinset for tiling. I spent all of Saturday cutting tiles and then placing them on the fireplace surround and hearth. Even with help it took 11 hours to do, and about 8 hours of it was measuring and cutting tiles.
While I was doing this work, which is just mindless enough that your mind wanders but requires just enough attention that it doesn’t wander freely, I began to recite a common trades mantra. Measure twice, cut once.
This quip – a practical saying – saturates the construction industry. Whether you’re a DIYer like me, or a professional tradesperson, it’s important to measure everything twice and do the work once. This saves you a lot of pain and time down the road, since you can double check your angles and distances and get everything right the first time.
The reason that this practice is important is as simple as considering a tile. Let’s say that I need a 3/4″ width tile, but I measure incorrectly and cut it to 1/2″. There’s no way for me to turn that 1/2″ piece back into a 3/4″ piece, so I just wasted that tile. I need to toss it out (if it can’t be used elsewhere) and cut a new tile to the correct measurement. In short, measuring twice saves you time and money.
As I stood above my trusty wet saw, cutting tile, after tile, after tile, my mind began to wander into the realm of programming. I began to realize something interesting. In my opinion, many IT departments have a policy of measuring twice and cutting once, with the supposed benefit of cost and time savings. One might even call this sort of approach waterfall or agile, where estimates are gathered in detail (measured) long before the work is done (cut).
I believe that this is a fallacy that ironically leads to even more work. Every single developer that I’ve ever met in my career, including myself, cannot accurately estimate anything. We sometimes get close, because we can relate the task at hand to a similar task we accomplished previously, but in general I find that a new task is very much an unknown and the time spent to gather an estimate is pointless since it’s wrong anyway. By measuring twice and cutting once, we waste a ton of time.
I believe that developers should measure once, quickly, for a rough estimate, and then cut. The reason that I believe this is due to a fundamental difference between programming and other kinds of work that is managed with processes and estimates.
Code is not a tile or piece of wood. It is a highly flexible, malleable, mutable, digital thing. If a developer cuts a feature short, they can add on to it later, expanding it seamlessly to the required size. If they overestimate a feature’s length, they can easily chop off the excess and move on to the next feature. There is no significant cost in quick, roughly estimated measurements for programming work.
Immediately your team will regain a ton of time in which they can do their development work. They won’t have to attend hours of planning meetings or requirements gathering sessions. They will just work to get things done as fast and accurately as they can.
The only tradeoff is a lack of estimates that management-types can cite and depend on. I would challenge that any estimates derived are very commonly wrong and useless regardless. More-so, if you do not trust your developers to do the right thing and use their time effectively, why do you keep them employed?
To me, a lot of the process models around development that are popular (waterfall, agile) are derived from the measure twice, cut once methodology. This approach is super practical to physical goods since inaccurate measurements are expensive, but this does not apply to development work. These meetings to gather estimates in the hopes of controlling costs ironically bloat budgets and help to deliver less code and extend goal dates and deadlines. You take people that are hired to code, and tie them up in meetings where they have to try and justify what they’re going to code by the hour. They don’t know how long it will take, but they will have a better idea after a few hours of coding – if you’d just give them a few hours of no meetings to code.
If you’re working on tiling your fireplace, measure twice and cut once. If you’re working on code, take a rough guess at the measurement and get to work!
Almost exactly 1 month ago today I found myself on a video call with Joel Spolsky. It feels insane to write that, even now, as it was a banner moment in my career. For me it was the equivalent of meeting a movie star who I had idolized since I was old enough to know what movies were. There had always been this Joel Spolsky guy throughout my career that I regularly read about and whose opinions on software development agreed with mine, and suddenly I was talking with him face to face. It was awesome.
Reaching this conversation was not the easiest thing I’ve done in my life. It took a few weeks and in all honesty it was a bit trying to find time to have so many interviews. How many interviews, you ask? Prior to Joel I talked with 5 or 6 other amazing people (Marc Gravell and Nicholas Larsen included). Somehow I managed to impress each of them enough to reach the end boss: Joel Spolsky.
The conversation lasted about an hour. It felt like 5 minutes to me, probably because of the excitement. Joel and I discussed the pros and cons of various software methodologies, mistakes each of us had made in our careers, some of the challenges of Dache (my open source software), and a few other topics. Then he said something awesome: “We’d love for you to come and work with us at Stack Exchange!” So much adrenaline hit me at this moment that I could have lifted a car with my bare hands. It was surreal.
A few days from now I officially start working with Stack Exchange. I feel very fortunate and excited for the opportunity. So far the Stack Exchange team have proven themselves an insanely skilled and professional organization that treat employees as human beings instead of expendable resources. I’m already loving the culture and interactions with my coworkers.
IMPORTANT NOTES: First and foremost, the commentary here consists of my views alone and not those of Stack Exchange or any other entity. This post is merely to reflect upon the interview process and discuss the aspects and traits of my career and knowledge which I feel helped me get the job. This is not a tell-all or any sort of shortcut or easy way out. If you want a job at Stack Exchange, you will have to endure the same technical and soft skills challenges that I did – the details of which I will NOT be disclosing. 🙂
So, with that disclaimer out of the way, here are my thoughts on how I got a job at Stack Exchange, and how you can too:
Ego is the mind killer, so kill the ego. Most developers that I’ve met (including some prior versions of myself) have massive egos. Egos so big that the room can barely hold them. Egos that even put the illustrious Kanye West’s attitude to shame. This is natural given that we spend all day creating things from scratch (which is a god-like quality) that often generate significant revenue for companies. We start to feel very powerful and even fawned over. We learn the entirety of the software and hardware vertical of our current job’s domain, and then make a Superman-like flying leap to conclude that we know EVERYTHING about ALL software and hardware.
Thinking you know everything is the easiest way to suck as a programmer. If you believe that you know everything, you stop trying to learn new things (since you already know them, duh). So, while you’ve mastered ASP.NET MVC 3 at your current gig, 4 and 5 came out… the catch is that your company never upgraded because it’s too risky, and so you never learned them (or cared to). Now a year or two later you’re so far behind the current development stack that you can’t even see it with binoculars. And did I mention those little things called Ruby and PHP and even Java that you’ve never written a single line of? And how about MongoDB, Couchbase, Azure, EC2, and the literally thousands of other platforms and programming languages? I should hope that by now you realize that you know a little bit about a handful of languages and hardware configurations amidst a sea of thousands… By percentage, you’ve conquered maybe 0.1% of all that there is to know about development. So have I, and that’s OK.
Don’t be a rock star developer. This is something that I did for a few years and it only hurt my career. Many companies employ a strategy of intentionally furthering the developer ego in order to make them feel valuable (often without handing out appropriate compensation). Being a rock star sounds cool, but really it’s a nasty strategy that can cultivate incredibly destructive developer attitudes. Companies seek out and hire rock stars, and rock stars have a sense of entitlement. They develop huge egos and run their mouths at meetings, interrupting and talking over others. They seem to love circular logic (they’re right because you’re wrong because they’re right). They feel that all other team members exist simply to serve their every whim… and everyone loathes working with them.
The thing is – while you may actually be a 1 in 1 million bona fide rock star developer – nobody cares. Talk is cheap and in my experience people will say nearly anything and everything to portray an image of who they want you to believe they are. If you are really good at what you do, telling people doesn’t do anything other than make them despise you… Hearing about how amazing your proprietary code is gets annoying – especially when it’s the 5th time this week you’ve said it. A good developer doesn’t need to brag about how good they are: their work speaks louder than any boastful words could. People around them will naturally do the bragging for them. Hallway conversations to the tune of “wow, he’s really smart” or “she knocked that out in hours when we thought it’d take days” will be fostered, and that isn’t a bad thing. Let people talk about you all they’d like, but maintain a sense of humility and reality. You might be the best developer on your team or even at your company, but you are still a human being and this is still a job. Nobody has been hired to serve your ego (even if their job is to serve you). Being a good developer doesn’t make you better than anyone as a person; it just makes you successful in your career. Never lose sight of the fact that thousands of other developers are great at their jobs too. What separates you from the pack is being a great developer AND modest. It is a very rare combination in my experience, and the complete package that many companies are striving to hire. So, while it’s cool that you’re the very best developer that there ever was, stop believing your own hype and telling everyone who will listen. They don’t care, and you shouldn’t either.
Know that you know enough to know that you don’t know enough. Know what you do know, and know what you don’t know, and never be afraid to say “I don’t know” when it’s the truth. A developer who isn’t afraid to say “I don’t know” in front of a room full of people is a rare gem. By being honest you create trust and credibility. You also foster positive relationships with your peers and company. Nobody will remember that you’ve never heard of Angular.js or Couchbase, though they’ll always appreciate that you didn’t waste their time or money by pretending that you did. You can’t trust a developer who doesn’t know what they don’t know.
Know your data structures and algorithms. High level programming languages such as C# abstract so much away from the modern developer that many of us have no idea what’s actually happening “under the hood” when it executes. It’s cool that you can sling LINQ everywhere, but do you know the computational complexity of what you’ve done? Do you know what a hash table is and why it is useful? Could you sort a list of things efficiently? Can you think of a scenario where a stack is the best option? Note that you don’t need to memorize things like sorting algorithms (hell, I couldn’t if I tried), but a working knowledge of data structures such as trees, hash buckets, lists, queues, and stacks combined with the rudimentary knowledge of things like sorting, searching, and caching is a very valuable skillset. It’s the difference between a good programmer and a great one. Anyone can write C#, but only those who understand even the low level operations of each deliberate method call will write good, clean, efficient code. You owe yourself a fundamental understanding of how data structures like stacks and heaps work, as well as by-value vs by-reference memory addressing. These core concepts apply to ALL programming languages. Too many developers ignore the complexity of their algorithms and just call pre-made methods without understanding the implications. Educate yourself on data structures and algorithms and suddenly you’ll be ahead of the pack.
Know why your code works and why your code doesn’t work. Have you seen this image circulating on sites like Reddit?
Despite being funny, the popularity of this image pisses me off. It claims that the essence of programming is having no freaking clue why your code does or does not run. I feel that this is unacceptable. A great developer strives for the WHY in every single thing that they do, not just HOW to quick-fix it. Code doesn’t compile? WHY? Race condition across threads? WHY? In asking “why” you further your knowledge and expand your skillset with the functional, rational “how” which allows you to become a better programmer. Most great programmers don’t repeat the same mistakes over and over, though they of course make mistakes… They just make new and interesting ones!
I remember the days of slinging shoddy code and then copy-pasting lines from blogs and sites like Stack Overflow until my code seemed to work (though I wasn’t sure if or why it did). Those days are long behind me. When my code doesn’t compile, 99% of the time I immediately know how to fix the issue. When my code has a bug, I usually know exactly how to track it down and resolve it, and in doing so I often learn how to avoid it in the future. Having no idea why your code works is like being a lawyer who has no idea why their client is not guilty: fairly useless, overpaid, and an amusing spectacle at times. Make sure that you know why your code works and why it doesn’t. In my opinion this is a basic competency of being a professional developer. It’s OK to create bugs and make mistakes – it’s not OK to make the same mistakes repeatedly.
At this point I feel that I’ve done a reasonable job of representing my skillset and core competencies. These are the things I showed to the Stack Exchange team in my interviews. I didn’t necessarily have the exact answer (or even most efficient answer) to each of their technical questions, but I was modest and never afraid to ask for help or say “I don’t know” when I got stuck. My answers involved efficient data structures and algorithms, and I was able to explain why the data structure was the best choice in solving the problem. When I created bugs I was mostly able to identify them and indicate how to fix them. In doing all of this, I demonstrated competency and confidence as a developer and fortunately ended up with my dream job.
I think that most devs would agree when I state that the definition of success in the corporate world of development places less emphasis on “good” code and more emphasis on “working” code. Working code is code that can be released to production on or before the deadline, regardless of performance or even bugs in most cases. As a developer, you ultimately feel as if you’ve failed when you toil for nights on end to meet steep deadlines and churn out crappy code. As a business, however, you’ve succeeded when you hit the deadline. My experience tells me that the typical metric upon which development teams are measured is often not quality of code or unit tests or even performance, but instead ability to meet deadlines and deliver solutions to clients. You’ve failed when you do not meet the deadlines and thus piss off the clients/customers. Your job has become a veritable boolean result with the outcomes of true and false. Deadline met? True. Deadline missed? False.
Doesn’t it feel awful to be measured in such a binary way? All or nothing, success or failure, deliver or delay. These are the only outcomes according to the people who write and sign your paychecks.
Why does this happen? A little introspective thought brings light to the subject, at least for me. The reason for these types of metrics becomes obvious when you consider their source. You work for a company who pays you (often with I.T. seen as a cost-center or “money pit”) to accomplish things which the company can then sell to clients. You’re an expensive tool by which they accomplish their means. Though these companies often see software as a profit source, they see the means by which they get the software as an expense and cost. Kind of strange, really.
The problem begins at the very core of the organization; the structure of the company is the starting point for guaranteed failure. In my experience, the dichotomy that forms in most companies is “I.T.” versus “The Business” in a bout-to-knock-the-other-guy-out. The minute you create this relationship of opposing fronts, you’ve already guaranteed development failure. With competing and contrasting goals (the business wants to sell stuff ASAP while I.T. wants to build stuff properly which takes longer) it is not possible for trust to exist within the organization. The Business will not believe a word that I.T. says when it comes to estimates, deadlines, or things that need to happen to ensure stability of the product in the future (technical debt). I.T. will not trust The Business to make rational decisions when it comes to features, development timelines and ensuring product quality. The result is a boxing match where each side is trying to force the other into compliance. Now you have conflict. Conflict dismantles good companies.
The Business is used to tracking their sales teams by metrics like “how many calls did you make/receive today?” and “how many sales did you make?” and “did you make X sales by Y arbitrary date?” where Y could be the end of each month. These are things that they understand, and thus like to control. Ask your favourite sales person for their opinion on the metrics by which their success is measured, and I am confident you’ll find that most will sum it up as “the faster I sell things, and the more things I sell, the more successful I am.” This makes sense from an empirical, see-the-figures-on-paper-on-my-desk-in-my-executive-office point of view, but I bet that the sales person in this case is not loving their life. A constant push to sell more, make more money, and do more. Any success in the future just raises the bar for the success which must follow. It’s a losing scenario. Eventually, they either get promoted out of the trenches of sales or they move to another company, resetting the bar which has been set too high. This buys them another year or two of raising that bar, until they ultimately repeat the process again.
Sales people who are put under the gun in such situations often resort to employing any tactic that they can to reach their goals… One of these strategies is saying anything at all to sign the client up. “Sure, the software can create rainbows and unicorns, just sign on the dotted line!” they say. It’s unfortunate, because customers who are hooked into these contracts tend to be very unhappy with the product when they find out that the software does not, in fact, create rainbows or unicorns. Or even a colour wheel and horses. It doesn’t even come close.
In the above case, The Business fails to measure the things that, long-term, make you the most money: client satisfaction and relationships. A good sales person (they definitely exist) is one that keeps the client happy with rational discussions and promises, and who is very transparent about what can and cannot be done and why. A great sales person is one the client loves so much that they’ll keep using your product, even when a better product exists, simply because they fear losing the relationship. This client is a client for life (or at least a long while) and makes you a lot of money. But how do you measure “happiness” and “relationships” long term? It’s a hard problem. Dating sites have been trying to solve it for over a decade. The Business will likely not dedicate the time and resource to do so themselves. So, they measure phone calls, sales, and other crappy metrics to ensure that the sales team are doing their job.
Here’s where we get back to the topic: developers and failure. The Business, who in most cases pays I.T. to create things to sell, employs these same arbitrary measurements when grading development teams. They often only know how to see success as a measured outcome of facts, and so they create the only measurements that they can empirically apply: features and deadlines. Does the dev team build all of the features and hit the deadline? Great. Do they not? Not great. These measurements themselves are acceptable (even good), but the combination of them (lots of features on short deadlines) is the problem.
The Money Talks
Where it gets tricky is in the realization that “show me the money” is how business ultimately tends to run. The sales people very overtly make the money, so they are seen as successful and important people in the company. The dev team also makes money, but is perceived to cost money, and they are seen as a cost-center that must be carefully weighed and measured to avoid excessive spending. What this leads to is an unhealthy practice of allowing sales people the freedom to employ any tactics necessary to land sales and make the money. In a business such as The Business as described, your life as a developer begins to suck.
To close the deal, the sales person will often promise the client almost anything about the software that you develop. They may promise new feature X by the end of the month, they may even promise 10 new features by the middle of the month. Whatever makes the client sign on. Then, the client says let’s rock and your quality of life drops sharply.
The very next thing that happens is The Business casually tries to confirm what seems obvious and even mandatory to them. “So your team will have these 10 things ready to go by the 15th, right?” they say. “This is a million dollar client, and it would be horrible if we lost them because you couldn’t deliver!” and now the pressure is on to do the nearly-impossible in virtually no time.
The dev team might try to politely push back and say that this is practically impossible, but The Business sees the dollars on the dotted line and will not listen. Flip the kill switch. Forego the QA time, all developers must focus on all of these features, day and night, so that the deadline can be met. Why? Because that’s how the team is measured. If the team doesn’t hit that deadline, they’ve failed and the million dollar deal is lost with the dev team seemingly at fault. Developers don’t want to work extra? Order in pizzas and promise them time-in-lieu as soon as the deadline is over with. Note that they will likely never actually see this time-in-lieu because right after this deadline will be the next one, with similar outlandish expectations and even tighter deadlines. And after that, another one. And another one. And the cycle will probably never end.
The Mad Production Dash
So, as the developer, you develop it as fast as you can. The code starts to resemble Frankenstein as you tack on bits and pieces to make it work ASAP. You subdue your ego and uneasiness about the quality of code by commenting // HACK: undo this crap later everywhere. Somehow that makes you feel better as it creates the slight glimmer of hope that eventually you’ll have enough time to come back and undo this monstrous pile of garbage. But you never will get that time, because the next deal is coming down the pipe. And so the code becomes worse. Your development effort completes 1-2 days before the arbitrary sales deadline, and after your QA team flips their lids on having 48 hours to test 1000+ hours of work, they do “critical path testing” to make sure it at least does something correctly and certify it as “good enough.”
The team releases to production early in the morning of the deadline day, and though it takes 5 hours because there are 17 untested things to fix on-the-fly (and realistically they have no option to abort the release or roll back because the consequences will be dire), they eventually shove the hacked up code out the door and declare it done. The Business shows their appreciation in the form of a short, impersonal e-mail that doesn’t name any person of achievement specifically. The development team is feeling underappreciated and pissed off.
What does the future hold for such a company? The code will probably spiral into bug-filled oblivion until it can’t do anything correctly or in any reasonable amount of time. Despite the weeks and months during which the development team pleaded with the business for time to clean up the technical debt, they are brushed off because taking time off of features loses clients and thus money. Then, as it starts to come crashing down in production, they suddenly beg the developers for a quick fix. “Do whatever it is that needs to be done!” they plead as they see their sales going down the drain. And now, because it is on fire and burning to the ground, the dev team is finally given a moment to pay back some of the technical debt that has been accrued during this vicious cycle. Repeat.
When a dev team has no say in the deadlines of the work they must do, they will usually fail. And when they are set up for failure from the start, they will likely get tired of being blamed for the problems without ever being given the time to devise the solutions. This leads to bad work culture, high turnover, and low productivity.
The way to guarantee dev team success is obvious at this point. It’s really as simple as trust between I.T. and The Business. They must keep each other in the loop as stakeholders. The Business has no product without I.T. and I.T. has no job without The Business’s clients. It’s a mutually beneficial relationship and it should be treated as such, rather than mutually parasitic.
A good company’s sales team will often consult with I.T. prior to promising any dates and deadlines when unknowns are involved. It is practical to ask the people responsible for a task how long it will take them to complete a task. This is much like how you might ask a waitress how long it will take for the food to arrive or a painter how many days they need to paint your home. This is a positive and productive discussion. Hallway conversations should become commonplace: “Hey dev team, I’ve got a client who wants to sign on but not until we build X, how long will that take?” The reply is as easy as “We’ll discuss it as a team and send you an estimate with some assumptions to confirm with the client” and just like that there’s a great working relationship that practically guarantees success. The team knows what work is coming, and also knows how long they have to complete it.
The Correct Measurements
If a dev team continues to fail in an environment where trust exists, then that team is likely not competent. They either cannot estimate correctly or cannot deliver within their own estimates. Sometimes devs suck at estimating because they’ve been making estimates under the oppressive sales gun for so long that they’ve effectively forgotten how to give themselves a fair amount of time. The blame for this remains entirely on the dev team, and they (or The Business) must repair the situation quickly and effectively to maintain the mutually beneficial relationship based on trust. As The Business owes I.T. input into the deadlines, I.T. carries the burden of being fair, accurate, and responsible with those deadlines.
Assuming that The Business now has a competent, skilled dev team, the question turns to the customers. If the customers do not like the estimates given to them, this may cost the company sales. Perhaps the customer wanted the impossible and The Business is giving them a dose of reality. Perhaps The Business does not want such a needy customer and they’re in a situation to be able to afford to tell them no thanks. Perhaps The Business realizes that the client’s request is reasonable but the timeframe of the estimate feels a bit long. In that case they can ask I.T. why. If the answer is not sufficient and justifiable, then perhaps the dev team is still not competent. No dev team should be let loose without checks and measures on productivity, but those metrics should be reasonable.
Ultimately, if you want to guarantee the failure of a development team, simply promise features to clients and customers without ever asking for (or trusting) the input of the team that is actually going to build those features. It’s just like telling the waitress that your food must be on the table in 10 minutes, without first asking the cooks how long it takes to safely and properly make it.
If this situation sounds familiar, try talking with The Business about it. Try to help them see it from your point of view. Ask them “how successful would you be if I demanded that you sell 20 new clients by Friday?” and perhaps some light bulbs will start to go on. Ultimately, we as developers often know nothing about sales and have no business dictating their measurable work expectations. They similarly have no business dictating ours, but a relationship of trust can be built to allow us all to work together and accomplish great things.
Well, it’s been nearly 2 months since my last post… I’m learning that if you want a blog to be successful, you have to carve time out of your busy life and make it happen. So, with renewed focus, I re-enter the fray.
The Joel Test is a curious and honest thing. It has been around since the year 2000 and was invented by a guy named Joel Spolsky, as the name might imply. In short, it’s a very brief questionnaire that evaluates the quality of your software development team, and implicitly their happiness as well.
There are 12 questions. A score of 12 is perfect, 11 is good, 10 or lower and your team has serious problems. The questions are simple and in the format of yes/no. The Joel Test is as follows:
1. Do you use source control?
2. Can you make a build in one step?
3. Do you make daily builds?
4. Do you have a bug database?
5. Do you fix bugs before writing new code?
6. Do you have an up-to-date schedule?
7. Do you have a spec?
8. Do programmers have quiet working conditions?
9. Do you use the best tools money can buy?
10. Do you have testers?
11. Do new candidates write code during their interview?
12. Do you do hallway usability testing?
For those who have never heard of hallway usability testing, I offer Joel’s definition: “A hallway usability test is where you grab the next person that passes by in the hallway and force them to try to use the code you just wrote. If you do this to five people, you will learn 95% of what there is to learn about usability problems in your code.”
The Joel Test has been in and out of my life for quite some time. Every now and then I remember that it exists, and then I evaluate my current employer on it. My current company is suffering in the programming department; we are losing people like crazy and are having a hard time hiring good folks, despite offering reasonably competitive salaries and benefits… So, realizing that we have some issues, I conducted The Joel Test on a whim. Here’s how we did:
1. Do you use source control? Yes.
2. Can you make a build in one step? Yes.
3. Do you make daily builds? No.
4. Do you have a bug database? Yes.
5. Do you fix bugs before writing new code? No.
6. Do you have an up-to-date schedule? No.
7. Do you have a spec? No.
8. Do programmers have quiet working conditions? No.
9. Do you use the best tools money can buy? No.
10. Do you have testers? Yes.
11. Do new candidates write code during their interview? No.
12. Do you do hallway usability testing? No.
We scored 4. 4 out of 12. Recall that 10 or worse means that you have a serious problem. Yeah.
So, what did I learn? I learned that part of the reason that we are losing people is that our programming practices and team environment lack something to be desired, and thus people are not happy. This isn’t really news to me, however, since the people that have quit recently have all cited these things as some of the main reasons.
What’s the next play? We need to improve the aspects of our environment and team that failed the test. The question then becomes: will management improve these things, or will they choose to ignore our suggestions because most of them (using the best tools that money can buy, quiet working conditions) cost money, and the rest of them are process changes? In fact, I’d go so far as to say that for my current company, they’d be significant process changes… And if there’s one thing that I’ve learned in my career as a programmer so far, it’s that businesses love to stay in familiar territory, clinging to what they already know and feel experienced in, and are generally afraid of significant change.
As a result of the change not being delivered, the motivated individual realizes that change must then come from within… People quit to change the negative environment for themselves (by joining a more positive one).
I encourage you all to conduct The Joel Test in your environments, and to try and improve your team with the results. What have you got to lose? Oh yeah, more developers. 😉