Stack Overflow

100 Million Uniques Per Month On 9 Web Servers

David Haney | @haneycodes
Big Data Jax | February 2016

About David Haney

Engineering Manager Of Core Team At Stack Overflow

Ninja

Home Brewing Enthusiast

Home Brewing

Amateur Runner

Running

Hockey Fan

Hockey Game

Amazing Fashion Sense

Fashion

Beat XCOM 2 Commander Ironman

XCOM 2

Learning New Stuff Every Day

Learning

A Brief History of Stack Overflow

2008: Stack Overflow Founded

Joel Spolsky Jeff Atwood

2010: Stack Exchange Network Launched

Stack Exchange

2011: Stack Overflow Careers Launched

Stack Overflow Careers

2014: Stack Overflow In Portuguese & Japanese Launched

Jin Yang

2014: Native iOS & Android Apps Launched

Android App

2015: VCs Invest $40 Million In Stack Exchange

Send More VC

Our Mantras

Developer: Smart & Gets Things Done

Developer

Team: Move Fast & Fail Fast

Team

Company: Pluralistic Management

Company

Our Development Methodology

Managers Support Developers

  • Managers set company goals, projects
  • RFC system to get developer input on projects (1-5 page doc)
  • Developers trusted to make all technical decisions
  • Developers can recruit any person from any team as needed
  • Managers ensure that project moves forward, remove obstacles

We Hate Meetings & E-mail

  • No project planning, no sprints, no hour estimates
  • No stand-ups or scrums
  • No status report e-mails
  • All scheduled meetings discouraged, must have good reason
  • Weekly "status report" meeting to update team on projects
  • Developers typically attend 2-3 hours of meetings a week

Remote Work Culture

  • If one employee works remotely, whole company must also
  • All employees issued headset and webcam upon hire
  • All meetings conducted in Google Hangouts and chat rooms
  • Remotes have access to both, prevents isolation / feeling excluded
  • Work day spent in chat rooms & Hangouts, can see everyone's conversations / project talk
  • ...Even people in offices next to each other in NYC
  • Weekly Beverage Bash: a Hangout where people drink and chat
  • Coffee Roulette: paired with random employee for remote coffee

Remote Work Culture

Remote Work

A Distributed & Diverse Team

  • 35 developers, 6 sysadmins, 6 designers
  • 75% of Engineering is remote
Remote Workers

Default Public

  • We love public artifacts, make the internet a better place
  • Encourage developers to blog about their work, speak at conferences, be evangelists
  • Developers interact with users on meta sites for feature requests
  • Nearly all conversations held in company chat rooms and Hangouts
  • Exceptions: manager-employee chats, interviews, etc.

Treat People Well

  • Make your own hours, flexible work time
  • Results-oriented work environment (ROWE)
  • Work from anywhere you want
  • Great vacation, unlimited sick days
  • Excellent health and travel benefits
  • Employees are completely trusted

Treat People Well

Happy Developers

Rapid Iteration

  • Never commit code that isn't ready for production (keep branched)
  • Automated build tools, one click production deploys
  • Release code to production 1-15 times per day
  • Small bites: very few changesets per release
  • No QA: if release breaks, roll back, fix, release again
  • Code reviews triggered voluntarily by author

Culture Is The Most Important

  • Culture eats strategy for lunch
  • Trust and transparency eats command and control for dinner
  • Our practices create a great work culture of empowered, happy people
  • People love what they do, and leave if they don't
  • Very low turnover, people leave to form/join start-ups
  • Very few complaints, which makes my life easier :)

Today We're Pretty Popular

StackOverflow.com

Stack Exchange Network

How Did We Get Here?

By Getting The MVP Out The Door

Original Design

This is the very first Stack Overflow design (never live)

It's often true that developers are not designers

...However, this was actually done by a designer

By Iterating Our Code Rapidly

  • Stack Overflow launched (first to market)
  • It was fairly simple and had few features
  • We fixed bugs and added new features
  • It grew into what it is today

By Trying New Ideas & Failing Fast

  • We sold on-site Q&A to businesses
  • It went poorly
  • We tried OpenID
  • It went poorly
  • We tried new, exciting tech that none of us understood
  • It went poorly

By Not Being Afraid Of Failure

  • Avoid the "no" or "that's impossible" culture
  • Try new ideas that have merit
  • Get MVP live fast (so you can fail fast)
  • Recognize failure, pull the plug and keep moving

Failure Is Important

"I've made billions of dollars of failures at amazon.com. Literally. None of those things are fun, but they also don't matter. What really matters is that companies that don't continue to experiment - companies that don't embrace failure - they eventually get into a desperate position, where the only thing they can do is make a 'Hail Mary' bet at the very end of their corporate existence. I don't believe in bet-the-company bets."

-Jeff Bezos, Amazon

Developers Are The Linchpin

"If you think it's expensive to hire a professional to do the job, wait until you hire an amateur."

-Red Adair

Cheaper

Hire Smart Devs Who Get Things Done

  • Must understand data structures and algorithms
  • Identify best data structure or algorithm for problem
  • Experience helps but isn't required
  • Mythical man-month applies
  • Your hiring process should include code tests

A Few Notes On Hiring

  • Use neutral tone and writing for job descriptions
  • Fix your interviewing & hiring pipeline biases
  • Avoid assessing "cultural fit" - it usually means "people like me"
  • Avoid credentialism, great developers come from all walks of life
  • Don't miss out on a huge number of great developers!

Now For The Technical Stuff

(Recruiters, nap time starts here)

Programming Stack

Performance

Performance Is The Key To Success

  • Millions of users amplify your mistakes
  • Inefficient code destroys companies
  • Everything we do is focused on performance

Our Stats

  • ~25 active production servers
  • 15% or less CPU utilization
  • 21 ms average question render time

Our Architecture

Architecture

See our performance page

Web Tier Performance

Web Tier Performance

SQL Tier Performance

SQL Tier Performance

Tune Your Databases

Efficient Data Structures Use Less Space

  • BIGINT (8 bytes) vs INT (4) vs SMALLINT (2) vs TINYINT (1)
  • CHAR vs VARCHAR vs NVARCHAR
  • Don't optimize prematurely (BIGINT identities)

Indexes

  • Clustered and non-clustered
  • Non-clustered indexes have a cost and benefit
  • Slower create, update, delete, faster read
  • Index size related to column size (smaller is better)

Queries & Query Plans

  • SET NOCOUNT ON
  • Covered queries are key
  • Query plans are a direct result of indexes
  • Analyze query plans proactively and reactively
  • Seek (good) vs Scan (bad)
  • Page size related to column size (smaller is better)

Train Your Developers

  • Your app (usually) depends on a database
  • A little knowledge often means huge gains
  • Tons of SQL misinformation and not-so-awesome DBAs
  • "Never use select *"
  • "Everything must be read uncommitted"

SQL Is Not Always Best

Use The Right Database

  • Using SQL Full-Text Search?
  • Use Lucene or Elasticsearch instead
  • Writing to table with app A and deleting after app B reads it?
  • That's called a queue
  • Storing key-value-pairs?
  • That's the definition of NoSQL
  • Have a ton of big data to crunch?
  • Use Hadoop

Use Them All At Once

  • SQL as your source of record and for data queries
  • Elasticsearch for analyzing/searching text
  • Queues for push-pop messages
  • NoSQL for free-form key-value data
  • Hadoop to map-reduce query your big data

Don't Optimize Prematurely

  • First to market usually wins
  • Get the MVP out the door
  • Solve problems as they arise

Cache Everything

Caching

  • The storing of calculated output for future method calls
  • "Short-circuiting a method"
  • Trade CPU cost for memory cost
  • Cached data is never real-time
  • Only cache things that are read more than written

In & Out Of Process Cache

  • In process: same process as application, shared memory
  • Out of process: own process with own memory
  • In process fastest but doesn't scale
  • Out of process scales but network & serialization cost
  • Cache small, widely-used, mostly static lookup values in process
  • Cache dynamic data with multiple variations out of process

Performance Profiling

Proactive & Reactive

  • Profile your code proactively in dev, reactively in prod
  • Identify "hot paths" and fix inefficient code / queries
  • Use tools like WinDBG when app crashes (get a dump file)
  • Use tools like MiniProfiler when app is slow
MiniProfiler

Write Thread-Safe, Highly Concurrent Code

Thread-Safety Is Important

  • Avoid race conditions, crashes
  • Avoid mixing/overwriting/exposing data across threads
  • Thread-safety bugs really, really hard to find and fix
  • Local scope and state guarantees thread safely
  • Immutable objects are thread-safe by nature
  • Use locks for atomic cross-thread operations

Atomic Operations Are Slow Operations

  • Lock (code and DB) sparingly, blocks other threads
  • Lock (code and database) sparingly
  • Lock when must always read accurate data (online banking)
  • Also lock when caching expensive method call output
  • Locks are thread-safe but not concurrent

Cache Is Highly Concurrent

  • Store data in cache
  • Reading cache is cheap and non-blocking
  • Cache everything and use local variable scope

We Use Machine Learning

We Know Who & Where You Are

ML User Data

What Do We Do With This Data?

ML Defines A Tag Network

Tag Network
Tag Network Methods

We Can Use It To Answer Questions

When Did Each Tech "Peak?" (% of Jobs)

Tag Network Peak

Is Tech Growing Or Shrinking?

Tag Network Growth

How About Popularity Over Time?

Tag Network Time 2008
Tag Network Time 2009
Tag Network Time 2010
Tag Network Time 2011
Tag Network Time 2012
Tag Network Time 2013
Tag Network Time 2014
Tag Network Time 2015

Query Our "Big Data" Yourself!

SEDE

We're Hiring

Come and work with us

We hire people who are smart & get things done

Diverse teams build better products. We strongly believe that diversity of experience contributes to a broader collective perspective that will consistently lead to a better company and better products. We are working hard to increase the diversity of our team wherever we can and we actively encourage everyone to consider becoming a part of it.

The End

Questions Welcomed

David Haney | @haneycodes