Big Ruby Conf 2014 Notes

These are the notes I took while attending Big Ruby 2014 in Grapevine, Texas.

Testing the untestable

• Start with testing as though code is in a black box
• Heroku managed this by actually just git push/deploy/build to test build packs
• heroku run bash will put you in a terminal on your dyno
• If heroku can deploy 6 times and have at least 1 passing test suite, they consider it successful
• Heroku test speeds went from 5 minutes per test suite to 44 tests in 12 minutes.
  • He has worked at companies that had Rails tests going on for 30 minutes per suite
• parallel_rspec is a plugin to look into for speeding up tests
• Testing allowed heroku to be aggressive in refactoring.
  • They use to code Min Viable Patching because they wanted to add/modify the least amount of code per patch
• build out smaller black boxes and unit test those smaller components
• Untestable scenarios
  • Mock for determinism (webmock for small pages, VCR for larger network traffic)
• codetriage.com
• If things seem too big, start with integration tests
• Test things that would hurt if they break.

Business intelligence

• Supporting decision making
• Data is generally an afterthought to devs, it’s just a storage for objects
• How do you use the data that’s stored to make business decisions
• The naive approach
  • “We can run reports straight from the transactional db”
  • The complex joins are slow
  • affects production performance
• Figure out the business needs instead of just feature requests
• They might not know about data that is valuable like (we know how loyal customers are by the last login date)
• Turn inferences into facts
• De-normalize data for the facts. It’s ok for data to be stored in many places if the facts are only stored once.
• Tables can communicate more than graphs
• “A” recipe for success
  • Use Ruby - TDD, deployment, “data munging”, many Powerful ORMs
  • Mongodb
    • Flexible schemas
    • Map/Reduce
    • New Aggregation framework — Check this out
  • Postgres for transactional
• Minimize on-demand calculations
• Store facts, not attributes (denormalized)
• SQL is the recipe and the cake is the report. store the cake too.
• Not storing objects, storing events (events don’t change)

Job processing

• Queues and Processors
• Queues
  • Durability, Availability, Throughput (pick two)
    • Durability - Message sent, how likely is it going to be received
    • Availability - How fault tolerant
    • Throughput - How many jobs can we push through
    • Cost?
  • RabbitMQ - Fast, durable, reasonably fault tolerant (entirely how you configure it)
  • Amazon SQS - highly Durable, highly Available, reasonable throughput, Not fast
  • Redis - High throughput, fairly durable, availability is hard, really fast
  • Not all jobs are the same, Tapjoy has roughly 2 different types of jobs
    • X00,000 / minute (events) - RabbitMQ
    • 1,000,000 / hour (general jobs) - SQS
• Processors
  • Def
    • A way to retrieve jobs
    • A way to match that job to code
    • A way to run that code
  • Concurrently with RabbitMQ and Fibers
    • Ruby - Forks, Fibers, Threads
    • Rabbit - Async, Stateful connections (retrieve and say finished on same state), no batch processing
    • Fibers - Coroutines on the main stack, Fibers are great for async, no worries about cross thread talking
      • If CPU bound, it’s not worth it. however, IO bound is ok.
      • Pain points
        • SystemStackError - fixed, but size varies, frame for the stack under ruby (e.g., Can’t save an AR object in Fiber)
        • Needs Fiber-aware network libraries
        • Code needs to know its on a fiber
    • Fibers aren’t good for gen purpose jobs, RabbitMQ is complex, He says he wished he looked harder at Celluloid.
  • SQS Job Processor
    • SQS - HTTP API, Synchronous, allows for batch processing
    • Why not threads
      • Global Interpreter Lock
      • Thread Safety
      • Memory Bloat
    • Building the processor
      • Threads to fetch
      • For for each batch of messages
      • child has a pipe to parent to share stats
      • However, pipe processing is slow (tapjoy gets 6 children before CPU upper bounds)
      • Resolv is the Ruby DNS in Ruby instead of C to prevent GIL, use this instead of C DNS if more than 1 thread
    • Look into learning to use GDB (also useful is the LLVM equal, lldb)
    • Sometimes the bug is not your bug
    • Chore, resque compat serialization, per-job config, jobs for each server, queue agnostic and Concurrency agnostic
      • Not released yet :(

Shopify sharded Rails

• Sharding - data over more than 1 db
• Rails assumes 1 db
• Auto-Incr doesn’t work
• Normally can get by on tune and cache for years
• Why Shard
  • Smaller indexes
  • Better localization, data stays warmer longer
• No Joins between shards, normally there is a ‘stop and shard’ moment, shopify doesn’t do this.
• Denormalized everything to have a shop_id since everything is scoped through shop
• Noeqd, like snowflake for ids
• Make sure ids are javascript safe
• shopify ids are auto_incr + N * auto_incr
• Rebalancing (move across shards) lock the shop and move

Living Social lightning talks

• Check out Pry
• text-table gem for viewing data in the console
• Using thor to grep to view logs
• “Telling a programmer there is already a library for that is like telling a song writer there is already a song about love”

Castle on a cloud

• Look up ChatOps video
• common tasks are automated through hubot commands
• IAM is basically like LDAP. Can limit within a bucket too.
• look into graphite for graphing. I think it’s a python app
• VPC as a VPN
• GitHub is a MySQL shop and use RDS for even heroku-based apps

Working effectively on a distributed team

• As Remote Worker
  • Be visible
    • make sure people see your work
    • speak up in meetings
    • Use video, if you have to talk to someone, make it a video
    • Be visible with your calendar (we currently don’t really do use)
  • Be yourself
    • Livingsocial has an ‘off topic’ room in campfire that allows people to show personalities. Being silly and being fun
      • You need the water cooler time even online. work can be / is social
      • People need to see you in addition to your work
  • Be disciplined
    • Not just not being lazy, also be effective with the team (eg not working during odd hours)
    • Setup a dedicated workspace
  • Be available
  • Be flexible
  • Be free (e.g., if you want to move around, move around. just do your work)
• Use Video - it well worth the additional effort, even if it’s 6fps
• Screen Hero for paired programming (ss and audio)

Refactoring with science

• Github use to say “never break the API” now because of usage data, they can change and contact users
• Github deploys branches to production, not merge into master, then deploy
• This allows for easier rollbacks by just deploying master again to revert
• dat-science allows for basically A/B Testing of code
• dat-analysis allows for visualizing the dat-science results

Building a service

• Before writing a service. They write a spec
• Curl-ish, Description, URL params, Request Body, Response (201 created, 409 exist… etc)
• Then they write the client. It allows them to understand error handling
• Writing configuration
• Rack::Test is good for testing API services
• Dev would be best in docker or vagrant, but Union metrics (Austin dev team) just ‘wings it’
• Ops is important in SLA
• Benefits
  • small deploys, experiments (eg with different languages) because it only has to talk HTTP
  • Experiments are double edged. happy programmers due to trying new things. but you might get boxed into something
  • client and servers are built in parallel for faster development. stubbing the server allows the client to be visualized quicker

Legacy codebase

• It’s not a problem until it’s a problem
• Think of working with legacy code like an archaeologist
  • You have to dig, research, piece together… all to figure out what happened and why these choices were made
  • Survey, Excavation, Analysis
• Survey
  • Take inventory with tests, code comments, new relic, benchmarks
  • Ask dumb questions and flag myths. you’ll be able to figure out the culture which leads to insights in the code
  • Tricks -> Techniques -> Process -> Methodology -> Dogma (seems like you should watch out for this)
• Excavation
  • If you have a goal to fix something, fix something. If something else comes out of the excavation, make an issue and get to it after your goal
  • “First, do no harm”
  • Use warn when making changes while leaving current implementation in place
• Analysis
  • Keep up the documentation in blog posts, wiki articles, readme or even tests
  • Document everything. Even the most trivial things like initializing from scratch. Also keep it up to date
  • Make a map. Search for gems that will make a rails UML diagrams
• The fear of looking at and working with a legacy codebase is all in your head
• Ask why is it the way it is
• It’s not bad code in the sense that it’s not working (it probably is working) but it just never went to refactor after green

Active interaction

• Fat controllers, skinny models make god models
• Rails says to focus models on NOUNS
• Consider making models with verbs
• active_interaction

Key models

• RDB - Relations, transactions, schemas, ability to extend and ad-hoc queries
• KVS - Schema-less, single-access reads, write-heavy (append only), easier to scale (restricted api), it’s just a hash

Managing fleets (could be Postgres)

• Clint is from Missouri. Not St. Luis, Not KC. Columbus.
• In the beginning was a sinatra app talking to aws using sequel
• The simplest thing that could possibly work, but no less
• Now 5 apps in sinatra, Fog to talk to aws, and still sequel
• They use Sinatra to map to other sinatra apps based on url
• 50 worker types, 100s of workers all doing the heavy lifting
• each machine only has OS, Postgres, and Wal-E
• Monitoring is from the outside in. Using State Machines and stateless workers
• This is kind of like in a game. Loop over: observe environment, do something based on env
• their queue is while resource, feel, do, requeue. over and over again
• Feeler - collects data on the resource. Append data to a observation table. Finds out state as well
• Think - Eval the state the resource is in. (eg if it’s uncertain, do x)
• Stateless (background) workers talk to aws, heroku api, postgres because they all require network connection
• Failure
  • notify and push to another queue
  • it either resolves or needs a human
  • the notifications lead to the development of playbooks based on the type of incident notification.
  • They have codified their playbook for different incidents that can resolve them (like restarting)
  • “Have you tried turning it off and on again?” - can resolve a lot of aws issues because images will pop up on other machines
  • If the codified resolver can’t do it, it says “I need an adult” and sends it a human
• State Machines are great
• Use Stateless workers
• Expect things to break

Understanding others

• Leveling up people skills
  • Everyone is different
• Developers
  • Grinders - product/feature-obsessed. Iterative. “Move fast, break things”
    • Explain how process X is better to communicate
    • Use TDD
  • Tour Guides - Works across the system/stack. Helps others
    • Ask how is something the way it is.
    • 50/50 split between legacy and sharing knowledge. write it down and spread the knowledge
  • Geniuses - Thinkers, worriers. Focused on quality and experimentation
    • Ask how can we iterate towards it being better
    • Make sure they don’t want to completely throw out the current system
  • T-shaped people - Focused on one area of expertise.
    • they might be so-so at everything else except that one thing.
    • share that particular knowledge. teach their weaknesses
  • Fun leader
    • People skills with some tech knowledge
    • They can’t fix every people problem. let them cool down.
• Learn about your self and know your own tendencies
  • Know what you need to recharge yourself. Know how to humor people
  • Know when your personality type is in conflict with the other
• Empathy - Understand the other and their current situation and next move
  • Can be difficult online.
  • Why would someone say that?
    • Hanlon’s razor: stupid over malice
    • Occam’s razor: simple over complex
• There is another person on the other end of the text box
• “An adult is someone who has been around longer than one hype cycle”
• “Upstart is 1-3 years” and wants to rewrite the world