The topic of diversity is as difficult as it is important to discuss; I've valued reading this thread, even though it keeps drifting off topic, and I'd like to thank everyone who has contributed.
I sympathize with @Dominique losing it a little bit: @Robert, your comments leave me uncomfortable as well. In particular, the sentence "Please show me all of these great people that I am missing [...] I can make a fortune off that knowledge." is kind of problematic in a couple different ways. You know the history of race and economics in this country, right? No offense, but, truthfully - based on the comments you've made on this thread - I don't see myself introducing you to, well, anyone. It's not for lack of knowing people. You seem to have concluded that because you don't know certain types of people, they must not exist. You might consider that they do very much exist... but they might not be so stoked to know you, or at least the "you" you are presenting in this thread (perhaps in real life you're not so polemic). Employee #1 for us was a first-generation American from NM - parents moved to US from Bolivia. Learned Linux on his own in high school in the early 90s, then worked for 10 years in tech in the bay area before joining us as the architect of all our back-end systems. You seem self-confident, but you're honestly not presenting as well on this thread as you might think.
@Demelza - your handling of this thread has been great; you've stayed logical, respectful and clear in your writing at all points. I admire what you've done here and look forward to further comments from you on threads with other topics as well.
I'd like to add my own observations back on the original topic - whether employee referral programs specifically are a cause of low diversity in tech. From what I've observed in my company, I believe the answer is "no." I do believe the results of referral programs are reflective of the wider lack of diversity, but unfortunately I do not believe the programs themselves are a causal mechanism, at least in my personal experience. I sort of wish they were: it would be easier to address the problem.
@Demelza - I think your argument about referral programs is exactly right regarding it being a sampling of the social connections existing employees already have made. However, it's not schools or neighborhoods that referral programs (in my experience) draw from - it's past work experiences. Across the US, I do see segregated schools, segregated neighborhoods, and segregated social organizations.
What I do not see, however, is segregated tech companies, just low-diversity tech companies. In some ways, that would almost be better than what we actually have in the tech industry: a mostly black tech company? That would be kind of cool - and I think unusual - to see. A mostly black church? Everyone thinks that is normal. Of course, there's likely an exception to what I'm saying, but what I'm trying to communicate is that I believe the lack of diversity in tech is pretty uniform - it's not just the industry as a whole, it's at the company level too.
If referral programs drew largely from people's school-based social connections, I could see the argument for the program itself being the underlying cause of the lack of diversity: the existing employees pull only from their schools, and if their schools are segregated off from broader America, then that segregation is going to be artificially reflected in the industry. However, I think it's worse than that: the referral programs pull from the industry itself - it's not the cause of the lack of diversity, it's reflective of the state of industry.
As a result, I do not believe eliminating referral programs would change much, unfortunately, at least at our company. We do not use referral programs to recruit junior people - we go directly to schools. Our junior team reflects very much the diversity of local schools (we're in Boston) like Harvard, Brown, and BU. These schools have systematically and consciously diversified their student body, and our junior tech team reflects that: we have benefited from those institutions' hard work. By selecting them to recruit from, we didn't have to do too much work beyond that to have a pretty diverse crew.
Every tech team needs a diversity of work experience too, however - we need people with 10 years and 15 years experience working on the problems we face. By definition, this segment of our team can only come from the industry. In a full-employment situation like we have for senior web developers across the US, referral programs are invaluable - senior web developers are not actively looking for work: they have it. We have to find them - unlike the junior team, they aren't looking for us.
If I thought tech companies were internally segregated, then I could again see how the referral program was the problem. However, I have not observed that. Our senior tech team largely comes from referrals. It is (naturally) much smaller than the junior team, and it is definitely less diverse. However, it's not uniformly white and the non-white members of our team all came through referrals from white members. Further, several white members of the senior team came from non-white members. So I don't see the referral mechanism artificially decreasing the ethnic diversity of our team: I do see it just reflecting the fact that if you take the population of individuals with 10-15 years web development experience, it seems like it's mostly white. I think most of our senior web team (regardless of their personal ethnicity) have worked for most of their careers at tech companies that were mostly white.
I am encouraged by our junior team - in 10 years, they will be the senior team at another startup, and they are far more diverse than our current senior team. So, I am optimistic, at least from my very limited experience at our small startup. I'm impatient, I'm concerned, but I do have hope that time may be on the side of ethnic diversity in our industry.
What I am horribly depressed by is gender diversity. Ethnic diversity has been very achievable today for us at the junior level without having to work too hard. In contrast, gender diversity has absolutely eluded us. We make the final interview decisions diversity-blind, but we recruit with different funnels - the goal is to do whatever it takes to make all recruiting stages prior to the final interview as diverse as possible. Forget thinking about the interview or "qualifications": I couldn't even make the *top* of the recruiting funnel acceptably gender diverse, no matter what artificial tricks I tried. Time does not appear to be on our side - if we do nothing, the industry appears to be headed towards being just as male in 10 years as it is today.
My pet theory on how to deal with this is to diversify startup CEOs and have them work on the problem. I don't necessarily know how to do this - I'm mostly just trying to make it someone else's problem I guess. We really started 5 years ago with great intentions, and I'm very disappointed how insanely short we've fallen.
Anyway, I don't think eliminating referral programs will change much and I think startups that do stay away from them will lose one of the most valuable tools they have to recruit senior developers. I don't think the programs are the problem - I think their results are symptomatic of the real problem (i.e. we didn't promote diversity enough 10 years ago, and now the people with 10 years experience aren't particularly diverse).