Learning Experiences from Going Viral (134k users in 1 day!)

Over the last 2 years, I’ve gone viral twice, most recently with a Reddit post that went way beyond my expectations. I had just wrapped up a months worth of work experimenting with some new data on savings rates, and I wanted to show off to /r/DataIsBeautiful. Data is Beautiful is a subreddit where analytical redditors post interesting graphs and charts. The nice thing about Data is Beautiful is that they allow original content, so that people like me (/u/shnugi_) can self post.  Most subreddits ban self posting, because it invites a deluge of spam.

I thought that I would get maybe a few hundred people to check out my data and give some constructive feedback. Well, this happened:

Went viral via reddit post.

134,752 sessions in a single day. My highest day ever.

The post I had made had attracted hundreds of comments and a Reddit score of 9898. Over ONE HUNDRED THOUSAND PEOPLE visited that day.

Dark Side of Going Viral – My Server Was Not Ready

Reddit #19 r/all

This is a screenshot my SO sent me of the post hitting #19 on r/All. It might have gone higher, but we were busy trying to fix the server.

I had a plan to be able to support a similar size of visitors as the Lifehacker feature (21k sessions in one day). My plan totally worked, until I completely blew past 21k sessions.  After that, my server was throttled for the next few hours.  My SO is much better than me at techy things, so I called in the reserves for help. We tried a bunch of things like resizing my Digital Ocean droplet, increasing PHP and MySQL memory limits, and then finally increasing the number of concurrent connections on Apache. The last one fixed the problem for me and my site resumed to normal speed, while also supporting over 1200 active users at the same time.

What I learned

  1. Have an even bigger plan next time.
  2. Load test my server to test the bigger plan, so that it isn’t like a fire drill every time it happens.
  3. Don’t be afraid of sharing. Typically, I shy away from self promoting, but it’s good to have some positive feedback when I do self promote that people like what I make.

Hopefully, someone finds this helpful.  For me, I know both the times that I’ve gone viral I’ve been highly under-prepared.  Good luck to all the other content creators out there!

38% of US Households Spend More Than Income


According to data from the 2015 Consumer Expenditures Survey by the Bureau of Labor Statistics, nearly 38.5% of US households spent more than they earned. Overall, 49.4 million out of 128 million US households are estimated to have had expenditures that exceeded their after tax income (table below). Another 21.1% (27.1 million) of US households saved less than $10,000 per year. One interesting fact is that 8.9% (11.3 million) US households are able to save at least $50,000 per year which is roughly equal to the median US household income.

Hover or click on the graph for more information.

The original Consumer Expenditures Survey considers retirement contributions as an expense, and even with adjusting those into savings 37.5% of households still spent more than they earned. Recently, there have been many studies reiterating the lack of savings that Americans have on had for emergencies. This data aligns with those earlier views on the poor health of the average American’s finances.

Source Data and Methodology

These results are calculated using the 2015 Bureau of Labor Statistics (BLS) Consumer Expenditures Survey (CEX) microdata. The microdata has survey results for a sample of 30,000 US households which are used to estimate the spending and income of the total US population. The data is weighted on a variety of factors by the Bureau of Labor Statistics so that the households sampled model reality. I used the pre-built SAS macro for the 2015 data to merge the interview with the diary files and aggregate expenditures by survey household unit. The interview and diary don’t link on a 1:1 ratio, so I allocated the diary expenditures across each survey household in the interview files. Each survey household was allocated a variable amount from the diary expenditures based on the household unit’s income, expenditures reported in the interview files, and population weight. I also took steps to ensure that the rolled up results still matched the published statistics on the BLS.

Annual Savings Amounts Table

Households Unadjusted : 38.5% of US households spent more than they earned in 2015. This was calculated using the CEX total annual expenditure by household. The CEX lumps all retirement contributions (401k, pensions, TSP) into expenditures. Household annual savings calculated as : [Estimated Pre-Tax Income] – [Estimated Taxes] – [Estimated Expenditures]

Households Adjusted : Adjusted for retirement contributions, 37.5% spent more than they earned in 2015. This was calculated using CEX total annual expenditure minus household retirement contributions by household. Social Security is still included in the expenditure values. Household annual savings calculated as : [Estimated Pre-Tax Income] – [Estimated Taxes] – [Total Estimated Expenditures] + [Total Estimated Retirement Contributions].

Annual Savings  Households (unadjusted)  Households (adjusted)
 <-$150k       1,994,856            1,867,628
 -$150k to -$140k          232,678                242,359
 -$140k to -$130k          355,854                303,275
 -$130k to -$120k          405,725                363,365
 -$120k to -$110k          429,066                411,577
 -$110k to -$10k0          553,061                513,501
 -$10k0 to -$90k          508,511                477,909
 -$90k to -$80k          593,815                581,473
 -$80k to -$70k          662,847                646,504
 -$70k to -$60k          919,974                932,389
 -$60k to -$50k       1,294,006            1,215,488
 -$50k to -$40k       1,851,218            1,833,831
 -$40k to -$30k       3,044,652            2,888,244
 -$30k to -$20k       5,260,480            5,229,834
 -$20k to -$10k       9,618,222            9,421,512
 -$10k to $0    21,733,884          21,187,798
 $0-$10k    27,128,611          26,860,088
 $10k to $20k    17,256,672          17,137,557
 $20k to $30k    11,193,070          11,312,717
 $30k to $40k       7,325,112            7,590,525
 $40k to $50k       4,709,618            4,965,180
 $50k to $60k       3,124,669            3,342,971
 $60k to $70k       1,994,672            2,250,973
 $70k to $80k       1,521,623            1,584,928
 $80k to $90k          929,844            1,064,559
 $90k to $100k          770,054                830,612
 $100k-$110k          632,352                659,814
 $110k to $120k          542,022                618,322
 $120k to $130k          450,592                484,008
 $130k to $140k          278,440                347,594
 $140k to $150k          219,223                217,597
 >$150k          901,936            1,053,230

If the interactive chart didn’t load for you, here is an image of the chart.
US Households by Annual Savings ($)

Related Topics

Savings Rate Ranking : This uses the adjusted savings rate calulation listed above to compare savings as a percentage of income.
Net Worth Rank by Age : This uses Survey of Consumer Finances data to calculate the net worth percentile rank depending on the age of the head of household.
Income Rank by Age : This uses Survey of Consumer Finances data to calculate the income percentile rank depending on the age of the head of household.

My 4 Favorite Personal Finance Calculators

There are a lot of personal finance related calculators out there, but there are only a handful that I would recommend using on a regular basis. Here are a few of my favorite tools that have easy to use options and clear results.

Rent vs Buy Calculator : New York Times

NY Times Rent vs Buy Calculator
This rent vs buy calculator balances all the costs you could thing of related to buying a home versus the monthly rental costs. One of this calculator’s great features is that it accounts for the opportunity cost of the mortgage down payment. The opportunity cost is the cost of what you could have earned from that money if you hadn’t bought the house. The calculator also builds in costs to account for rising rent prices, home appreciation, and inflation. Also, the controls are easy to adjust how long you plan on staying and your home price budget. I know it sounds like a lot of options, but if you aren’t sure about one of them just leave it at the default value.

Vanguard Retirement Income Calculator

Vanguard Retirement Calculator
This is my go to retirement calculator. I use it as a simple and quick check to make sure that my savings rates are high enough to meet my retirement goals. The calculator is completely free and doesn’t require registration or anything like that. It automatically does not include Social Security, so you have to manually key in a number for that. In general, I like to pretend that Social Security will be tiny by the time I retire. I just put in $1,000 a month at most ($2,000 if you’re married) for a very conservative estimate of how much SS would actually pay out.

When can I retire? : Networthify

When can I retire?
This calculator has pretty similar results to the previous Vanguard one, but it’s tilted more for figuring out how quickly you can retire. I really like the chart on this one, and that it emphasizes controlling spending in order to retire more quickly.

Savings Rate Calculator : Physician on Fire

Savings Rate Calculator
This is a great spreadsheet to help you understand that components of your savings rate in order to calculate it. It’s not as spiffy as some of the other tools, but it’s pretty straightforward and has a very detailed breakdown of how to tally up your income and savings.

Validating the Net Worth Percentile Calculator

With all the recent discussion on fake news. I want to take some time to go over some basic fact checking. Recently, it came to my attention that a certain website was publishing a number that said the average millennial had a negative net worth–this is false by the way. The author crudely extrapolated off of a handful of data points from the SCF and a WSJ article.

My Net Worth Percentile Calculator as well as the others all run off of the publicly available micro-data from the Federal Reserve’s Survey of Consumer Finances or Bureau of Labor Statistic’s Consumer Expenditure. I take great care to ensure that the results of the calculators would meet the same standards as the work I do in my day job as an analyst.

Comparison to Published SCF Stats

Here, are side by side comparisons from the Survey of Consumer Finances. I am using this Changes in U.S. Family Finances from 2010 to 2013: Evidence from the Survey of Consumer Finances to do the comparisons.

On page 12 these are the value’s calculated for median net worth by age from the SCF:

Click on the links to open the net worth percentile calculator for that age range.

Age Median Net Worth 2013
Less than 35  $                                 10,460
35-44  $                                 47,050
45-54  $                              105,600
55-64  $                              169,640
65-74  $                              229,800
Greater than 75  $                              194,700

As you can see, the totals from the calculator track very closely with the published statistics from the US Treasury.  The numbers do not match exactly, probably due to some differences in clean up that the US Treasury does for their official numbers versus what I do with the the raw data that they publish.

Charting Income Distributions of Rodan + Fields and It Works!

I’ve had a rash of co-workers and Facebook friends posting about MLM’s like Rodan and Fields, Nerium, and It Works! So, I’ve been reading Lazy Man and Money’s blog posts about MLM’s, and decided to chart their self reported income disclosures visually. Visually looking at a graph can be very different than looking at a table of the same data.

MLM Income Disclosures

Each MLM must publish income disclosures to be compliant with governmental FTC regulations. These disclosures can tell you how much people are earning at each level of the company. For example, the top level at Rodan and Fields is the RFx Executive Consultant. According to Rodan and Field’s discolsures, 0.1% of all active distributors are at that level. That translates to one in a thousand distributors is an RFx Executive Consultant. The average annual earnings for the top 1 out of a 1,000 consultants was $661,474 in 2015.

In the charts below, are the distributor levels for R + F and It Works! with the percentage of distributors who in each level with their average earnings next to it.

Rodan + Field 2015 Income Distribution

Source: Rodan and Fields 2015 Income Disclosure

It Works! 2015 Income Distribution

Note: The It Works! disclosure is reported in monthly values, whereas the previously mentioned R + F values were reported as annualized values.

Source: It Works! 2015 Income Disclosure

A special thanks to Jason Davies for posting his example of a D3 stacked population chart that I modified for this.

How much of the US could the top 1% buy?

If the top 1% of the nation put all of their money into land they would own 40 states.  In contrast the bottom 80% of the nation could only afford to own all the real estate in 3 states.  All the real estate in the contiguous US is estimated to be worth nearly $23 trillion. The top 1% of US households is estimated to be worth $19.2 trillion.  This means that only 1% of the US population could literally afford to buy almost the entire country if it were for sale.

These numbers are based on the top 1% by net worth, instead of income.  Sources and tables are at the bottom of the page.  If you’re curious how close you are to the top 1%, try out the net worth percentile calculator or the income percentile calculator.  They are based on a different data set, but the general trends are the same.

I filled in the maps starting in the west coast and moved east.  I removed states that the 1% couldn’t afford in their $19.2 trillion budget starting with Florida. The 8 states in the contiguous US that weren’t selected are Tennessee, West Virginia, Virginia, North Carolina, South Carolina, Georgia, Alabama, and Florida.

The bottom 80% had a combined wealth of $5.1 trillion, which could pay for California, Oregon, and Washington. I wanted to stay consistent and start from the west coast, so California just based on it’s size and wealth is driving down the number of states.
Continue reading

Top American Cities to launch a Financial Career

Finance is one industry that is very strong across most American cities with many job options that pay extremely well compared to median salaries. Read more to find out about the cities with the largest job markets and the fastest growing job markets. Also explore the county by county data in the interactive map at the bottom, where you can explore the regional differences in job growth rates, raises, income, and number of people employed in companies in the financial sector.

Top 5 Cities by Total Number of People Employed in Finance

 City Employed in Finance 2015
NYC (5 boroughs) 324854
Chicago (Cook County) 146597
Los Angeles (LA County) 133268
Phoenix (Maricopa County) 122541
Dallas (Dallas County) 117263

These cities generally align with the major financial areas that you typically think of for finance. NYC has Wall Street. Chicago has commodities and is the heart of trade for the center of the country. Los Angeles, Phoenix, and Dallas are corporate centers with many financial companies supporting. While these cities currently have the most jobs, they aren’t all growing. So these cities might have healthy financial sectors today, in the future other cities lower in the list may grow past them.  Many of the cities in the top 5 such as Chicago and Los Angeles have seen substantial drops in financial sector employment over the past 10 years. Keep reading to find out where the up and coming cities are.

Financial sector jobs have seen declines in many major areas.

Financial sector jobs have seen declines in many major areas.

Top 5 with Growing Finance Sectors

Texas dominates the counties with the largest nominal increases in employment in the financial sector with 3 of the top 5 counties with job growth over the past 10 years. With greater job growth comes greater opportunity to move companies for increased pay and responsibilities that you might not be able to get in more stagnant markets.

City 10 Year Growth in Employment
Dallas (Dallas County, Texas) 14740
Phoenix (Maricopa County, Arizona) 14671
San Antonio (Bexar County, Texas) 14558
North Dallas Suburbs (Collin County, Texas) 8265
Des Moines (Dallas County, Iowa) 7900

Top 10 Counties by Growth Rate

While Texas dominates the raw growth by number of jobs, the fastest growing counties by annual % job growth are spread all over the country in smaller cities. It may be good to launch your career in a smaller market that is growing fast. Sometimes it’s better to be a big fish in a small but growing pond than a big fish in a large shrinking pond.

City Annual Increase % (10 yr avg)
Des Moines (Dallas County, Iowa) 10.08
Denton (Denton County, Texas) 8.71
Birmingham (Shelby County, Alabama 8.36
Bloomington (McLean County, Illinois) 5.2
Columbia (Richland County, South Carolina) 3.9
North Dallas Suburbs (Collin County, Texas) 3.3
Austin (Travis County, Texas) 2.95
San Antonio (Bexar County, Texas) 2.82
Kansas City (Johnson County, Kansas) 2.26
Salt Lake City (Salt Lake County, Utah) 1.93

 

 

Interactive map after the jump.

Continue reading

Possible changes to Scottrade after the merger to TD Ameritrade

A month ago TD Ameritrade and Scottrade agreed to merge for $4 billion. In this merger TD Ameritrade will be buying Scottrade, and they expect to see $450 million in benefit by merging.  That $450 million will be realized through cost cutting and revenue growth.

What could this mean for you, if you have a Scottrade account?

TD Ameritrade generally has higher rates than Scottrade by as much as 43% (see below for transaction pricing differences).   Most likely after the merger Scottrade’s pricing will be increased to match TD Ameritrade.  This means that those price increases will be passed on to you to help the company “earn” the benefit from merging.  In addition, if there are TD Ameritrade branches that are nearby Scottrade branches, the merged company will most likely close one of those branches.  Read through the comparison of pricing to see if the cost differences could impact you.  If the differences are substantial for the types of trades that you do, you may want to consider moving your accounts to other brokerages.

Scottrade Pricing vs TD Ameritrade Pricing

Scottrade and TD Ameritrade do not charge inactivity fees or minimum monthly transaction fees.  So at minimum after the merger, nothing should change there.

Stocks and ETF Trades

Scottrade TD Ameritrade Difference
Online $7 $9.99 +43%
Broker $32 $44.99 +41%
Phone (IVR) $32 $34.99 +9%

One benefit that TD Ameritrade has is that they offer a wide array of around 100 ETF’s that you can trade for free.  Scottrade used to have a handful of no trading fee ETF’s but that program was discontinued.

Mutual Funds

Scottrade TD Ameritrade Difference
No Load, No Transaction Fee (NTF) $0 $0 0%
No Load $17 to sell $49.99 to sell 194%
Load $32 to sell $0 to sell -100%

 

Options and Contracts

Scottrade TD Ameritrade Difference
Online $7 + $0.70 per contract $9.99 + $0.70 per contract +43%
Broker $32 + $0.70 per contract $44.99 + $0.70 per contract +41%
Phone (IVR) $32 + $0.70 per contract $34.99 + $0.70 per contract +9%
Option Exercises Assignments $17 $19.99 +18%

 

The merger has not yet been approved by the FTC, but is it likely to be approved as there are other competing brokerages that would be larger than the combined company.

Find out how much are Southwest Points Worth

Southwest advertises the rate for redeeming Southwest Rapid Rewards Points at 70 points per dollar. That is equal to 1.4 cents per point.  Now, if you’ve flown with Southwest, you’ve probably noticed that the points to dollars conversion varies from flight to flight.

I’ve pulled together some data from 500 flights on Southwest’s flexible date calendar to calculate how much the Rapid Rewards points are actually worth. These flights are domestic and set to take place in November 2016 thru January 2017.

Overall Southwest points are actually worth:
61.72 points per dollar. Lower is better for you.
$0.0162 dollars per point. Higher is better for you.

This is the average rate, so you can find many Southwest flights that offer much more generous point conversions.  In my personal experience, Southwest offers some of the best conversion rates for points, and there are no black out dates.  There are times that you can find a Delta Skymiles flight that is deeply discounted for points, but overall Southwest has more consistently generous reward flights.

Looking at the Flights Visually

southwest-dollars-per-points

Looking at the the dollar per point values, only 2.2% of flights fall below the 1.4 cents ($0.0143) rate.  16.73% fall right around the advertised rate.  A tremendous 81.07% offer a better dollar to point conversion that advertised.

southwest-points-per-dollar

Looking  at the data by points per dollar gives a similar graph.  The categories are slightly different than the dollar based one, which is why the distributions are slightly different.

Methodology

To get these numbers, I took the price of a flight in dollars and compared it to the price of the same flight in points.  I adjusted the prices for the $5.60 fee per flight that is charged for any reward flights. For reference, here are some other point valuations.  The numbers may vary because of the flights they checked and dates of those flights:

  • NerdWallet : $0.011 dollars per point; 90.9 points per dollar
  • The Points Guy : $0.015 dollars per point; 66.7 points per dollar

On the eve of the election: Trump vs Clinton Spending

It’s the night before the 2016 election, and I’m having trouble focusing on the next day.  I’ve channeled my sleeplessness into creating this graph to compare Donald Trump and Hillary Clinton’s campaign spending by week.  The data goes up thru the last reported FEC update of October 19.

Overall Clinton reported spending $464 million dollars compared to Trump’s $251 million.  The majority of this disparity came from early in the campaign season when Trump had not yet won over the Republican National Committee.  In the graph, Trump’s spending roughly matches Clinton’s towards the most recent weeks.

Election Spending Thoughts

  • Donald has only been able to outspend Hillary a handful of weeks during the election season.
  • The week of 10/15/2016 is a partial week that cuts off spending on the 19th.
  • There are 6 times where one of the candidates spent more than $20 million in one week. 4 times for Trump. 2 times for Clinton.  That is an insane amount of money.
  • The raw data is from here: Clinton’s Data,  Trump’s Data

vote_today