Newsletter Updates for May 2015

If you haven’t already been listening in to the O’Reilly Data Show Podcast hosted by Ben Lorica, then by all means do not walk, run to check it out! Episode linked above is with Anima Anandkumar @UC Irvine, recommended previously here, re: tensor analysis. Similarly, recent collaboration among David Gleich @Purdue, Austin Benson @Stanford, and Lek-Heng Lim @ UChicago, using tensor analysis to resolve hard problems in higher dimensional Markov chains, resulted in “Spacey Random Walks”. Punchline on slide #17. I detect a trend…

Confs

Three months and so much travel since my previous post: to paraphrase Ricardo Alberto Fernando Ricardo y de Acha, I’ve got some ’splaining to do :)

Highlights include:

Spark Summit East 2015, NYC

Really enjoyed the track chair gig for Data Science. My two favorite talks, both highly recommended:

• Un-collaborative filtering: Giving the right recommendations when your users aren’t helping you by Leah McGuire @Salesforce
• Next-Generation Genomics Analysis Using Spark and ADAM by Timothy Danford @AMPLab

That was a busy conf, indeed! You can tell since no time was carved off for sacred pilgrimages to Wassail NYC cider bar, Mast Brothers cold brew chocolate, or Blue Hill Farm. Instead my flight left the following day for…

QCon São Paulo

My first time to Brazil. I’m hugely impressed by the developer community there in SP. Most inspirational quote from the conf: “A team is like a symphony, not a factory.” by Randy Shoup.

Great sessions on Spark, Docker, and much much more. Fortunately, we did have time to sample the local cuisine… e.g., Italian food made with tropical ingredients, or my favorite meal of the week, Pirarucu steamed in banana leaves, with a local favorite recipe for pumpkin soup. Then back to the US, before one could even say “alfajores” – assisted on a course at Stanford, then back to NYC…

Pirarucu – Brasil a gosto, São Paulo

CodeNeuro East 2015

Organized by Jeremy Freeman and crew. Truly excellent: top neuroscience researchers in the world gather for a hackathon (i.e., coding together). Perhaps a few savvy Finance people lean in too, eagerly drooling over results that apply for their high-dimensional, time-series, non-linear correlations work as well… in what VCs insist on calling an “ecosystem”. Or something. See these excellent notes. My favorites:

• Michael Dewar : streamtools real-time analytics from NY Times
• Olga Botvinnik : flotilla - Py package for iterative machine learning analyses
• Eiman Azim on motor circuit function … with actual electronic circuits to emulate what Columbia discovered through ablation studies about cerebellum connections
• Brendan Lake teaching computers to scribble characters like humans, if you want some really interesting use cases for Deep Learning

We worked together the following day, between tutorials, to build an online platform for submitting algorithms to run against standard neuroscience data sets. This hackathon literally was research. If you want to understand more about Big Data being used expertly in life sciences, attend CodeNeuro!

Lower East Side, Manhattan – from New Museum roof

BBBT.us

Many thanks to Marilyn Waldman, Claudia Imhoff, and crew for a fantastic Spark tutorial at CU Leeds Business School! Followed by excellent convo via webcast with several hundred of the top BI analysts in the world. Then on to Boston for…

Big Data TechCon, Boston and Next.ML, MIT

Matei and I were multiplexing between these two conferences so much, UberX in oscillation overthruster mode, that we didn’t even see each other. Even so, lots of great Spark talks in Boston that week! Meanwhile, kind hotel staff redirected me toward The Barking Crab for dinner, and a good friend introduced L.A. Burdick. Also, there was lots of excellent hard cider in the area. Along with all that Big Data conf talk stuff. Then a red-eye flight took off for Europe…

Boston Seaport, by water taxi

A Coruña, Spain

Many thanks to all the work by Amparo Alonso-Betanzos, David Martínez-Rego, and colleagues for organizing the Spark tutorial at A Coruña. I’d never visited Galicia before, a place with lots of rain and people with red hair playing bagpipes amidst rolling green hills (no en UK – pero en España) … a place where there are software companies next to world championship surfing competitions and albariño vineyards (no en Santa Cruz, California – pero en España) … a place where they speak a language close to Portuguese – not so unfamiliar, right after São Paulo! Excellent other talks, along with a Spark tutorial by Juantomás García. What incredible people, Computer Science excellence at the university, and oh such good sea food. My keynote was broadcast on Spanish television – that’s a first! Use of runways in this corner of Spain is quite abbreviated, so we catapulted next for…

Frente a la Torre de Hercules

Strata + Hadoop World EU 2015, London

See a good summary of the conf online. Spark Camp this time had 12% of the conference attending, whereas before we’d been trending steady at just over 8%. Many thanks to all who participated. Great to meet so many people enthusiastic about using Apache Spark! Also got to host the Hadoop and Beyond track. Which, oddly enough, was mostly about Spark. My two favorite talks:

• Taming the firehose: Build analytics over 45 billion tweets using Elasticsearch and Spark by Anirudh Koul and Shashank Singh @Microsoft … I’d never seen a two-person session talk handled with such amazing orchestration, almost like an old-time radio play. Brilliant.
• Say goodbye to batch by Tyler Akidau @Google … an expert but highly accessible overview of streaming analytics, from one of the leads on MillWheel, along with a deep-dive into one of the most sophisticated streaming platforms

Plenty of hard ciders sampled while in London. Whenever visiting near West London, I make a point to drop by Princess Victoria. Also, got to see some friends at UCL and the Barclays incubator program. Then back to the US via customs in my former part-time home city, Vancouver…

Chicagoland

GOTO, Chicago

Dean Wampler, et al., have been hosting great Spark events in the Windy City. Always a treat to visit. We had a good turnout for the Spark tutorial at GOTO, one of my favorite software conferences. Walking alongside the Tribune Tower after my tutorial, I noticed that its walls contain rocks from other famous buildings all over the world. With labels, like an inverted museum. Check it out when you visit. There are also rumors of a cider bar being built, ahem, soon. Then a flight back to the Bay Area…

Future Learning 2020 Summit, Stanford

Serendipity. Got invited by a dear friend, Donna Kidwell @Webstudent to join this conf about the future of education, a collaboration among Future Learning Lab, H-STAR, EdCast, etc. Many thanks to Oddgeir Tveiten and crew.

Most discussions focused on experiences with MOOCs – from highly successful examples, e.g., Intro to Robotics by Peter Corke @QUT, or Trust Academy @Salesforce by Masha Sedova. However, the overall themes transcended MOOCs, asking the question of what comes “After Gutenberg”, how peer evaluation is transforming education at scale, and … Peter Norvig’s point that when so much learning material is available (e.g., via Google) the problem becomes a matter of how do you get people to want to interact with it? Generally, the social context of learning becomes key.

Along similar lines, Michael Shanks stressed that – in contrast to traditional academe that tends to decontextualize learning – more contemporary advances are focusing on how to contextualize, locally. That’s part of the essence of interdisciplinary work, e.g., Data Science. Also, delighted to meet Keith Devlin (our new neighbor) with amazing work in math education using Minecraft, etc. (sound familiar?) And was very fortunate to meet teaching superhero David Conover, who uses game design to teach topics like IoT in an at-risk high school in Austin. Brilliant.

Of course, we’ll be inviting the whole lot to propose talks for Strata! Speaking of advances in learning platforms, check out the recent beta site and related article Embracing Jupyter Notebooks at O’Reilly by Andrew Odewahn. Par example, Data visualization with Seaborn … that is integrating use of IPython/Jupyter, Docker, Thebe, etc. Brilliant++.

Data Science

Another recommend: a new series of ML-related interviews by David Beyer, beginning with my friend and colleague Reza Zadeh @Stanford – on the evolution of ML, deep learning, Stanford ICME, and Apache Spark.

If you haven’t seen the news, Nature banned use of p-values … #finally  Note that Fisher did not intend for p-values to be (ab)used that way. So I consider these tests to be truly excellent, for identifying intellectual limits. Related: The Nine Circles of Scientific Hell.

On the subject of pseudoscience – appalling to see recent and ongoing unscientific gaffes by people who should know better, e.g., Neil deGrasse Tyson about GMOs. In great contrast, antidotally, I’d point toward this gem – and please read it at least twice: Die, selfish gene, die by David Dobbs. So glad to see Dawkins getting served … #finally

Neil, Richard: just because you’re published, doesn’t mean you’ve become superior thinkers. Please keep your day jobs, respectively #thankyouverymuch

Speaking of ’splaining to do… another gem is Visualization Explanations @Setosa. Extra points if you grok the callback in their name.

If some of these items are what you’ve been talking about recently, you may just be a Data Science instructor … or interested in becoming one? Check out Become a Data Science instructor @ Galvanize (Seattle, SF, Boulder)

Spark

También se recomienda: una excelente introducción en español a la Scala y Apache Spark por Isra Gaytan. The Latin World has been busting a move lately on Spark, #justsayin

A Coruña – from Playa de Oza

Adatao published an excellent article about how they anticipated the inflection point for Spark adoption. Check out the cost curves. Brilliant.

Big news from AMPLab: Keystone.ML is released as open source, to make the process of constructing complicated machine learning pipelines easier. Good stuff from the eponymous Evan Sparks, et al.

I’ve got some graph analytics talks coming up… and was excited to see a streaming/incremental SSSP impl for Spark.

Also, speaking of the Apache Spark Developer Certificate, we’ve got another new neighbor: ORM + DataStax partner on C* cert. Collect all three!

Meanwhile, the big BIG news is Spark Summit 2015 coming up next month in SF. Use the discount code SparkSummitPC25 for 25% off registration. Not retroactive, but nice try :) Followed by Spark Summit EU in Amsterdam, this autumn. Spark it up!

IoT

Solid Conference is coming up again soon! Highly recommended. As an appetizer, check out this excellent article by Cameron Turner @The Data Guild: Caltrain Quantified - An Exploration in IoT which we could hear in our previous backyard every morning starting at about zero-dark-thirty. Now that former backyard has become the new GoogleX building, and SciFi tech experiments compete with the trains for attention.

For IoT in practice, I’m totally stoked to see: Surfers on acid… What an excellent application. And, culturally not far off that mark, here’s an interesting take on marine plastic: Net+Positiva.

Ag + Data

I’ve really been enjoying Biocoder News in quarterly installments, some of my favorite new articles in the world. Period.

On that note, I’m thoroughly ecstatic to announce that I’m moving to O’Reilly Media full-time. Even so, I'll stay involved with Spark and Databricks, assisting on Spark Summit, etc. We’re moving the family to a tiny farm, an old apple orchard that really needs some tending. Perfect as a research station for Ag+Data.

The Tiny Farm – redwoods 30m tall, planted 65 yrs ago by previous owner

In highly related news, check out How to Grow a Forest Really, Really Fast, about fantastic work by Shubhendu Sharma. I’m eager to try this out.

One of the top intellects of the early 21st century, Paul Stamets, had some excellent coverage: He Holds The Patent That Could DESTROY Monsanto And Change The World!  See also: BioMason  and Ecovative Fungi, FTW – and mycorrhiza in particular, as Mohamed Hijri explains quite succinctly.

Ag-related tech approaches in SV have become largely derailed by asinine priorities dictated by Monsanto – more about taking over hedge funds on commodity trading globally, than about feeding anyone. Perhaps the best analysis that I’ve read recently – and certainly one of the best books that I’ve ready recently – is the highly recommended The Third Plate by Dan Barber. I learned about that via Gastropod – where Cynthia Graber and Nicola Twilley consider food through the lens of science and history. Brilliant.

There’s been a terrible drought / water crisis in Brazil – largely exacerbated by transnational corporate interests. This was weighing on my mind as our flight landed in São Paulo. I got to speak with friends there who are working on Ag+Data analysis, very good to see.

As predicted: Finance is driving California water into the dust… take a moment to consider the jump in almond production versus the temporally co-located jump in variance for snow pack levels. That’s the tip of the iceberg for the near-term shape of major political battles brewing in California. To wit, some of our local mafia have become known under the more apt monicker of Oligarch Valley. While Fox News, et al., promotes the Israeli approach of desalinization at scale, many people who can actually think for themselves question the impact of that approach, and recognize what an utter environmental disaster it could produce. This is not an area of judgment where one gets to call #oops as an excuse, regardless of which side the local mafia may be taking.

Industry Insights

“Software eats the world” is a catchphrase used by A16z. While I slightly agree with the title from this Datanami article, How Machine Learning Is Eating the Software World, its conclusions are pretty much the opposite of what we’ve observed with Apache Spark use cases in the field. Don't get me wrong – Reynold is a good friend, and IMO one of the most talented people working in distributed systems today. However, I have a hunch that the reporter munged the line.

Two key reasons why organizations adopt cloud-based notebooks are (1) to reduce their need for DevOps people to run clusters; and (2) to reduce the need for programmers to assist business people with queries for insights Big Data. Done and done. In other words, domain experts trump all in Data Science applications, while application developers (in relatively large supply, but relatively expensive) and expert systems engineers (in relatively short supply, extremely expensive) both become less of an existential bottleneck for new ventures. I’ll let you do the math on that one.

Some of the themes that I’ve been researching and illustrating over recent years include: Functional Programming for Big Data, Approximation Algorithms, Tensor Factorization, etc. Recognize that each of these point toward less emphasis on developers leveraging APIs, and meanwhile more emphasis on domain experts leveraging simple-to-use frameworks. That’s the bottom line of Apache Spark. Meanwhile, I have no doubt that A16z will continue to rake in loads of money – some of their partners are well-connected billionaires – just perhaps not as a consequence of their thesis. That ship is already sailing. Off, perhaps, toward the oh-not-so Great Pacific Garbage Patch.

Diversity

Speaking of VCs in SV… TechCrunch analysis recently found that female founders nearly doubled in 5 years. Par example, check out the recent Women in Data: Their Work and Achievements.

Meanwhile, I thoroughly enjoyed this gem by Karin Rubin: How women are conquering SP500… My feelings about the overall ethics of algorithmic trading are arguably mixed. However, if it’s going to happen, why not guide it based on diversity, since that demonstrates a #winning strategy?

Fun Stuff, friends in the news…

Check out Lumo Interactive Projector by Meghan Athavale and crew. It’s an interactive floor projector, transforming a floor into games that kiddos can design themselves.

Also, this bit Our Coming Robot Overlords about David, Amanda, and Zeno Hanson – friends back in Texas.

And, what William Barker called one of his most honest interviews, ever.

Upcoming Events

• 2015–06–04: Graph Analytics in Spark @BloomReach, Mountain View
• 2015–06–07: Apache Spark: Intro Workshop for Developers with Andy Petrella, @Scala Days, Amsterdam
• 2015–06–08: Data Science at scale and graph analysis with Dean Wampler, Amsterdam Spark Meetup
• 2015–06–09: GraphX: Graph analytics for insights about developer communities, @Scala Days, Amsterdam
• 2015–06–15: Spark Summit 2015, SF
• 2015–06–15: Apache Spark Community Event, @GalvanizeU SF
• 2015–07–23: Microservices, containers, and machine learning, @OSCON, PDX
• 2015–08–03: ICT Labs Summer School, @SICS Stockholm
• 2015–08–14: Scala By The Bay, SF
• 2015–09–30: Strata + Hadoop World NY, NYC

Will just leave you with…

This article. Wonderful, on so many levels.

SV Synopsis: Fundamentalism in Technology

I am grateful for perspectives gained because our family lives in Silicon Valley. Many options here to work at novel ventures, and on fascinating projects… Opportunities to drop by Stanford or Berkeley for some remarkable guest lecture by a visiting expert… The wonders of an almost perpetual Maker Faire as one walks through the neighborhood on any given evening… Tech camps that our daughters can attend locally, as they wish… And, generally speaking, the lack of any real need to engage in ridiculous commutes…

As an open source evangelist and as an investor, I've felt grateful to learn from a veritable parade of interesting projects. However, I am troubled by the incidence of a particular problem. Far too often one runs headlong into what I could characterize as a close approximation of cocaine-fueled misogynistic narcissism. The condition is subtle, but systemic here. Even recently, I have witnessed this up close – along with the regrettably pervasive and predictable non-reactions to it. Increasingly, zero tolerance appears to be the only effective response. Or perhaps the tech industry percolates out elsewhere, far from SF and its inertia?

Without mentioning names, two well-known billionaire-club investors in Silicon Valley personify this character sketch. Evidence of panspermia ad absurdum festers in the "cultures" that they promote. Personal jihads seemingly to self-perpetuate their fundamentalist ideals.

A nagging question lingers… Why work alongside an ilk of people with whom I would never encourage my daughters to mingle? Granted, I believe quite strongly in the need to talk with just about everyone, to keep dialogue open, to reject the notion of "enemy". Even so, there are absolutes. Practical realities of livelihood aside, as a parent what kind of examples do my professional actions and affiliations set?

In addition, a question that investors ask over and over when considering whether to fund a new company is "Will the team scale?" Any measure of the toxins described above almost guarantees that the answer will in practice be "No."

That represents a dirty little secret. There is an amazing level of demand for tech talent. It's not exactly because these companies are raging commercial successes; most early-stage ventures by definition are not. It's because few people who are capable of making good judgements are willing to compromise their futures to work for ineffective caricatures. Many start-ups encounter difficulties in scaling their team. Or – more likely over time – they encounter high attrition rates.

While I have in the past focused for several years on the same project, lately I don't stay long in most early-stage firms, generally moving on after an organization demonstrates its nature. To paraphrase Lady Grantham from Downton Abbey, there is a point at which malice ceases to be amusing. On the one hand, that's a terrible way to leverage stock option packages. On the other hand, arguably I have pursued a portfolio career strategy. That approach has helped me build an amazing network. Long-term benefits of my network have far surpassed the potential upside of my aggregate stock options. Therein dwells an important lesson about Silicon Valley.

hadoop summit 2012: emergence of the confidence economy

moore intro

Geoffrey Moore opened his keynote at Hadoop Summit 2012 and promptly dropped the line: “You will remember this moment years from now.”

http://www.slideshare.net/Hadoop_Summit/geoffrey-moore-keynote-hadoop-summit

After a disappointing set of “sales pitch” keynotes on the first day of the conference (thanks Yahoo! — but you knew that already) many people attending seemed to roll their eyes about yet another keynote talk this morning. Surprise!

I was grateful to hear Geoffrey Moore trash Advertising as an industry at risk. If I may paraphrase: permanently caught between bleeding edge and dinosaurs, yet irreparably dependent on a broken business model. [FWIW, the last three VCs on whom I’ve used that line looked back at me like I was some kind of alien slime-mold.]

By the middle of his talk, Moore put up a slide with a half-dozen bullet points. The slide listed some of the most disruptive technologies on which businesses — Main Street, in his terms — would come to rely in the early 21st century. Those include: collab filters, behavioral targeting, predictive analytics, fraud detection, time series, etc., etc. Outside of the intelligence community and the hedge funds, the significance of these technologies is not well understood yet. Word. Up. Bitches.

Moore’s “Final Thoughts” slide really hit home. He talked about data access patterns (system of record vs. log file usage vs. real-time analytics vs. etc.) and how those access patterns create feedback loops within an organization. Moore claimed this was core DNA for Google, Amazon, etc., which all major businesses must now embrace. Or else. [That's about 95% overlap with a slide I made for (insert recent past employer) during a 2011Q1 pivot. Two pivots later, I left without any particular next gig in mind — clearly needing to get involved with a different business team. Shortly before their CEO got, um, an "opportunity" to find work elsewhere. But I digress.]

an exercise

So here’s a fun exercise for the interested reader: Pull up a 10-year chart for the S&P 500. Add to that CBS. Right.. Add to that Barnes & Noble. Bokay.. Add to that Wal-Mart. Got few bumps, some upturns.. Nothing to write home about.

Now add Google. Now add Amazon. Now add Apple. One might argue that I’m cherry-picking examples; however, one must understand those three in particular to grasp the trajectory of how Data modifies Companies.

Think about it. Imagine rolling the clock back about 13 years, just a few years before that huge financial sea change got going. Think about perceptions at the time of Apple, Amazon, Google. Most of the mainstream buzz that I heard or read in 1999 was largely disparaging about those three. They didn’t make sense to the average joe, and that was a problem. I will contend that what made sense to a handful of computer science grad students, but not to the average joe, was considered a problem for Main Street. A multi-billion dollar existential problem for some, as it turned out.

At the time, it seemed like Apple would never get past the overwhelming popularity of Dell and Microsoft. Amazon didn’t have a way to justify its enormous P/E ratio — and was probably fluff in the long run. Google was considered interesting, but a little strange, with no clear path toward revenue.

Now think about what happened to the music industry, the mobile industry, the … well, I could go on, but Apple disrupted the pants off lots of established players. Entire industries were taken down by one company. Then consider what happened to retail. One word, a verb according to Geoffrey Moore: Amazon. Think about what happened to advertising. Googled, and not in a nice way either. Amazon and Google took off in 1997Q4 and 1998Q1 respectively, with Big Data projects which became enormous cash cows: Amazon’s recommender system (plus cloud infrastructure), and Google’s search+ads (plus cloud infrastructure). Arguably, those two are the reasons we were having a "Hadoop" conference. Apple perhaps seems less in category; however Apple leveraged mountains of consumer data (plus cloud infrastructure) to drive its smartphones, App Store, etc.

Imagine what kinds of conversations which must have been occurring in the board rooms of CBS, Motorola, Barnes & Noble, Wal-Mart, etc., etc. Gone, gone, gone. Three relative underdogs became giants, tipping almost everyone else’s apple carts. (pun intended) At least three firms understood the power of leveraging their data, they understood the urgency of real-time analytics, etc. Their competitors, mostly, did not. Just look at those stock charts.

According to Moore, that was the tip of the iceberg. Most of the Global 1000 is now on notice. Over the next decade we’ll see monumental failures. Winners and losers, as always, but the magnitude of the losers may be unexpected.

central point

Moore’s central point in the keynote — since this was a Hadoop conference — was that the Hadoop tech stack and business ecosystem is maybe a year ahead of the proverbial “crossing the chasm” moment. Ergo his lead line.

Notably, enormous cultural changes of the 1990s and early 2000s have percolated through personal expectations among those coming up in the ranks. That’s happened more notably and with more impact outside the US than within it. He pointed to the “digitization of culture”, where access has become nearly universal, where broadband created emotional dimensions (Facebook, Pinterest, etc.), where mobile makes the experience ubiquitous regardless of socio-economic position.

Meanwhile, the corporate culture of how to “get stuff done” within enterprise has not kept up. There’s no Facebook for enterprise, no YouTube for enterprise, etc. [Well, actually, there are — and they are each headquartered within a bike ride of my home near the Mountain View / Palo Alto border — but you haven’t heard about them. Yet.]

Meanwhile, Facebook-esque consumer Internet companies of the world are too caught up in their own weirdly distorted realities to solve the larger business problems. Business problems where the solutions will inevitably derive from the social networks’ innovations. Oops.

In Moore’s vaulted opinion, those conditions won’t hold much longer. There will be winners. There will be losers. Big ones.

Meanwhile, for people of my ilk, Moore smiled and predicted: “This should provide at least a decade of entertainment for everyone present.” Fundamental business reasons are simple: enormous change ahead but precious few who are trained and experienced to navigate it.

key take-aways

My first key take-away is based on the observation last year that Enterprise giants bumbled into Hadoop Summit 2011 in a huge and awkward way. Oddly, the logo is an elephant, #justsayin

In contrast, this year was really smooth, completely professional, far too expensive … but almost all about data infrastructure in a world where nobody want to utter the word “Oracle”.

Mind you that neither of the two main “enterprise” keynote speakers from last year still have their same jobs. #justsayin

Also, notable Hadoop practitioners were noticeably absent. In fact, most of the cast and crew of Strata seemed to be missing. A particularly popular social network has been burning the midnight oil to make Hadoop perform backflips — they like gave a couple talks and seemed to vanish.

Let me put this in other words: several hundred million dollars have been invested by VCs (and angels) to recreate an industry in the image of Redhat and Yahoo!

Wow, did anybody think that would be a particularly good idea? No, but it’s the herd mentality in practice. Even after the 5th beer I’d still recognize that strategy as not particularly wise. Feels like when you talk with an ex-convict, and they drop a line “Yeah, I made some poor choices long ago…”

My hunch is those data infrastructure plays are mostly tax write-offs (for the “early adopter” part of Geoffrey’s famous curve) at this point.

Moore underscored how real payouts come when key verticals catch fire — with serious domain expertise leveraged. LinkedIn perhaps got close, but now it almost feels like a spamming broadcast system for HR and BD departments. We’ll see “Big Data” killer apps which mean something to lots of people. Beyond the GOOG+AMZN+AAPL tip o’ the iceberg. They will come from people who have sophisticated backgrounds in Stats + ORSA + distributed systems + functional programming + DevOps, people who can also communicate well with actual business leaders. Not those employed by some halfwit B-school grad who’s posturing as the next Steve Jobs, when in reality he drinks bad beer at a lame, faux-hipster sports bar while watching cable televison. Or something. Dude, hop on your fixed-gear bike and standstill/peddle your sleeve tattoos out of here.

Translated: the proverbial ignite moment, that spark of innovation, is not going to come from the likes of a Cloudera or a Platfora or a (banal noun)-(o|e)ra… But it’s going to come, probably not many moons away. It will be in apps.

the sound of disruption

Thirty years ago, I went into a field called “math science”, i.e. how to build predictive analytics as software apps. Stanford — the Statistics department chairman, Bradley Efron, in particular — had put together an interdisciplinary degree which combined math, statistics, operations research, programming, engineering, etc. At the time, most of my peers in the program went on to become insurance actuaries. I went instead to do graduate work in machine learning and distributed systems.

For nearly two decades, most employers could care less about any quantitative background. They wanted C++ software engineers working all day on APIs from Sun or Microsoft or Oracle. Or they wanted managers. Then, in about 2000, came the sea change.

Right about the same time as ticker symbols for Apple and Google and Amazon were strolling up to their respective launchpads, some people began to look at my resume and ask a different line of questions.

I’ll always remember the first: a microchip vendor — one which makes electronics for several products you’ve purchased — was getting squeezed by Intel and their silicon compiler vendor. Critical features were being deprecated, specifically to put this second-tier player out of business. The company was on notice. They had to find a proverbial needle in a haystack: out of tens of thousands of circuit designs, they had to identify the 1% which would no longer be licensed — then redesign those. Quickly.

An internal team at the company had tried, but given up. Too much data for their techniques, it would’ve taken years to resolve. The company hired an electronics consulting firm in Austin, and engineers went to work, but gave up as well. Too much data, not enough signal. I got called in, as a “Whatever, just see if you can get anywhere” last-ditch effort. About 20 lines of Perl and one relatively simple equation later, I dumped my results into a scatterplot.

One of the lead circuit designers picked up my plot off the laser printer and began laughing. Loudly. The whole office heard him.

His manager grew annoyed: “What?! Why are you laughing?!”

Engineer: “He found it.”

When I turned in my invoice, the manager glared. “Look,” he said in a growl, “Just go somewhere for about three weeks. Bill us the whole time. Then come back and turn that in.”

My brows furrowed, this was a high-dollar rate for 2000.

“If you don’t pad that damn invoice…” he paused, “You’ll make both us and our customer look like complete fools. Piss a lot of people off.”

That’s the sound of Disruption.

More than a decade later, the summary graf of my resume reads like bullet points from Geoffrey Moore’s slide. Collab filters, anti-fraud classifiers, predictive analytics, etc. Even in the past few years, when HR people have read that resume, several looked up with a frown, said they thought that kind of work was better suited for business analysts — yadda, yadda, yadda, keep following the herd: you put the “botch” in “beotch”.

At a time when lots of business (start-ups as well as enterprise) are starving because they cannot hire Data Scientists, I’ve been busy building teams. Teams which delivered $MM results. I’ve hired about thirty people onto Data Science teams within the past few years — at a time when many start-ups would feel lucky to hire one. #justsayin

mal*wart

I read one of the most imbecilic essays recently from Forbes/Quora: “What Would Be The Global Impact If Wal-Mart Abruptly Shut Down?” Essentially, a hagiography stating that Wal-Mart is too big to fail, that the consequences on the US economy, the global economy, would be catastrophic. Translated: may require an enormous bailout, soon.

[In case you hadn’t guessed, I just threw up a little bit in my mouth.]

What. A. Fucking. Moron. The reality is that Wal-Mart hasn’t been doing so well over the past decade. Not if you peel back enough layers of PR. Not since they tried to bamboozle the LA city council. And failed. Moreover, folks at Amazon could really care less which Senators or SecState/former-first-lady the execs in Arkansas have in pocket. Bezos has positioned to take over 150% of Wal-Mart’s business the picosecond after Bentonville implodes. Sears and Target have reinvented themselves specifically for that very instant. So long, good riddance. Remember the point about the Global 1000 on notice? About the importance of business fundamentals?

sears, a.k.a. that web site which kinda looks like amazon

A third keynote talk that day was by the Sears CTO, Phillip Shelley. I had packed up my laptop and backpack, and was getting ready to walk out of the auditorium. After his first few sentences, I put my stuff back down and started taking notes.

Dr. Shelley mentioned how Sears started as a mail-order business a century ago, though more recently got completely kicked by another “catalog” called Amazon. Now they are leveraging Hadoop + R + Linux/Xen private cloud (srsly, this is from the Sears CTO?!?) to reinvent their business with 100x more detail on regional pricing models. Literally calculating personal pricing discounts for individuals, multiple times per day, specifically for mobile.

Sears: core algorithm moved from [6000 lines of COBOL on mainframe with 3.5 hr batch window] to [50 lines of Hadoop app on Linux with 8 min batch window], while reducing TCO for enterprise IT by two orders of magnitude. So much success, that they’ve spun it out as a new business line called MetaScale.

Brilliant strategery by Sears. Some seriously high powered Data Science talent walked out of that auditorium musing how they wished their VP Engineering was half as progressive as Sears. Srsly? Um, that’s called a PR coup. [Literally at the same moment as Wal*Mart had HR droids spamming the audience with whispers and rumors of lucrative salaries. Gak.]

emergence of the confidence economy

What’s the deal? It’s about confidence. Those giants in the Global 1000 which Geoffrey Moore says are on notice? They got that way by believing that business is largely about who barks the loudest, barks the longest, and cuts the most deals under the table. The proverbial alpha male in a wolf pack.

Wal-Mart would be a prime example, in my opinion. Their business is predicated on fundamentals which simply do not hold. Misplaced confidence. Thanks to people like Hillary Clinton, Wal-Mart has gained much influence on the House and Senate floors and the halls of the State Department and the UN assembly. In other words, so long as we manage to keep fuel costs artificially low, Wal-Mart’s market valuation will keep growing. So long as we believe that bullying vendors, conducting intelligence operations against the rest of your ecosystem, etc. — that these kinds of practices are ethical and sound in the long-run, then Wal-Mart will keep growing. Bullshit. Go look at that stock chart again. Wal-Mart is about tall white guys in dark suits, acting like complete pricks, destroying and plundering anything they can get their grimy paws on. Richard Gere in Pretty Woman, before he gets Julia Roberts. And not much more than that. On notice.

Moore is pointing out, in my opinion, that the issue at hand is about uncertainty. The point of establishing a corporate charter was always to externalize risk and perpetuate wealth for shareholders. That was true four centuries ago, when the first transnational was established, and has been true ever since. The modus of that mechanism is a process called sublation. The train wreck for sublation is uncertainty. In an environment where uncertainty holds sway, having real-time analytics from petabytes of customer data wins out over having a Senator in pocket. Any day of the week. The antidote for uncertainty is confidence. While there had been a regime of an “Attention Economy” extant for the past two decades or so, we’re now entering a new regime of the “Confidence Economy”.

Here’s the deal: people like me like those of us in Moore's lecture have been the “secret sauce” fueling the rise of Amazon, Google, Apple, etc. We use techniques which are mostly not well understood outside of Langley and the hedge funds. The tools of contemporary corporate assassins. Guys in suits who act like pricks in lieu of practicing business fundamentals — those guys are our targets. The modus is Disruption. If you have an MBA or a CxO title and not much else to back it up, I put food on my family’s table by being a sniper paid to hunt you. Lots of *great* food. And some of the best wines available. I shake the tension out of my hands, correct for wind and distance, draw a bead, take a deep breath, squeeze the trigger. Kill shot.

The challenges faced by Data Scientists are daunting. On one hand, most mathematicians lack enough solid engineering to create killer apps. Conversely, most engineers lack enough math to make any headway on the business data. Most business analysts lack enough of either the math or the engineering to be worth hiring. Data Scientists provide all three areas of expertise: the engineering and the math and the business insights to contend with mountainous torrents of data, and move the needle. On the other hand, Data Scientists must also speak truth to power. In any given business, there will be winners and losers. Executives, people accustomed to their own power, taken down. Meanwhile, we Data Scientists come prancing into a business, we do our magic, and consequently we point out which executives are bullshit and must be “executed”.

The reason why I’ve built Data teams at a time when others are starving is simple: confidence. Sure, I’ve logged three decades of machine learning, statistical modeling, data management, distributed computing, etc. When I talk with a grad student about their work, I can tell them in 25 words or less what they need to do on their first day at work to become regarded as an great asset to the team. They already know the techniques, but crave confidence. Into the trenches, fresh-out, having to speak truth to power. They’ll be placed into some faltering business unit, run some detailed analysis, and point out that the VP who’s been arguing loudly was completely wrong for the last N years and his/her ego cost the company several $MM. You can bet that those execs will return fire. However, a person like me is confident that we can get a kill shot. I show new folks how to draw a bead and squeeze the trigger. Been doing it for a long while, and will be doing for a long while more.

Snipers have an eerily pragmatic sense of confidence. And, by the way, that’s a peculiarly difficult job. Praise goes out to the men and women who serve their countries in uniform — when the cause is just. [FWIW, before tackling the challenges of data+science, I wore a military uniform and carried a rifle. Sniper training has become invaluable.]

my opinions

#1: Enterprise suffers because so many people in the corporate leadership ranks (or rather, amongst those clawing and scrambling to make their way into the corporate leadership ranks) consider themselves to be a different caste — if not a different species all together — from the rest of us who do not have a salaried position with a transnational. In a “Let them eat cake” world fraught with trillion-dollar bailouts, that’s not a particularly good way to future-proof. Moreover, this is why VCs are vital… to demolish that kind of hubris via constructive Disruption. #justsayin Word. Up. Bitches.

#2: Enterprise tooling, which is now mostly dependent on JVM-based apps, suffers because it has embraced “Convention over Configuration” … CoC has its place. I can imagine that it’s an excellent idea for heart surgeons to have a standard toolset, with scalpels in the exact same positions, etc. CoC is not a particularly good way to manage complexity and uncertainty, because it simply displaces major problems into the build system. Ultimately, it fails too much and impedes spin-up. Which, I believe, represents an enormous, ticking time bomb in Enterprise. Here’s a challenge: Show me a metric for the median period it takes in your business for a newly hired engineer to push code changes into production use which is adopted by at least 80% of your customer base. Now show me a metric for the media period it takes in your business between the point where a product manager identifies a needed feature and a newly hired engineer is ready for spin-up. From those, I’ll make a prediction based on that metric for how well your business will survive the “on notice” condition which Geoffrey Moore described. Better clues for navigating complexity and uncertainty can be found in the works of Ilya Prigogine or Stephen Wolfram. To wit, functional programming is more likely to address complexity, real complexity, and also more likely to attract top talent. CoC, not so much. Perhaps your enterprise business addresses Main Street instead of Early Adopters... Recall that Google and Amazon and Apple crossed the chasm by recruiting armies from grad students — at a time when most other people erred on the side of average joes. Remember the point about real-time analytics? It counts for training your people, then retraining, and retraining, constantly — to grapple with uncertainty. Kill shot. Global 1000.

#3: MapReduce will be unrecognizable within three years. Hadoop Summit will become something quite different after Hadoop bifurcates and gets sublated into Something Else. For example, it would be not difficult to use the Simple Workflow Service from Amazon AWS to implement MapReduce using the core part of Cascading… a different kind of MapReduce, which is not constrained by JVMs… which could scale much more gracefully and robustly… which could out-perform Google infrastructure and avoid attempting to re-create the industry in the image of Yahoo! At which point, one could deploy functional programming blocks at enterprise scale, without having to rely on the morass of enterprise build tools. Hmmm… may need to get a term sheet for that one.

Geoffrey Moore, we may have a few answers for your questions.

upcoming talks

I will be speaking at three events over the next two weeks. These talks will focus on how ShareThis leverages AWS and MapReduce, based on Cascading and AsterData as vendors, plus several open source projects including Bixo and Katta.


Tue 2009-06-09, 7:00pm-9:00pm

ScaleCamp
http://scalecamp.eventbrite.com/

Santa Clara Marriott
2700 Mission College Blvd
Santa Clara, CA 95054

Will discuss "How ShareThis mashes technologies in the cloud for Big Data analysis, leveraging AsterData nCluster, Cascading, Amazon Elastic MapReduce, and other AWS services in our system architecture."

This will be run in a Bar Camp format, along with DJ Patil, Ted Dunning, and other great speakers.


Wed 2009-06-10, 8:30am-8:30pm

Hadoop Summit 2009
http://developer.yahoo.com/events/hadoopsummit09/

Santa Clara Marriott
2700 Mission College Blvd
Santa Clara, CA 95054

Two talks: "Cascading for Data Insights at ShareThis: Syntax is for humans, API's are for software", as a developer talk, 2-3pm. Also, participating on the Amazon AWS panel discussion moderated by @jinman, about running Hadoop on EC2 and Elastic MapReduce, 6:00-6:30pm.

Last year's Hadoop Summit was a must-not-miss event, and I got lucky on timing and was able to tip-off friends at AWS to check it out... Amazon sponsored the lunch, and even though I did have to miss that event, AWS sent me awesome t-shirt schwag which I treasure to this day :)


Tue 2009-06-16, 1:00pm-7:00pm

Amazon AWS Start-Up Project
http://aws.amazon.com/startupproject/

Plug & Play Tech Center
440 N Wolfe Rd
Sunnyvale, CA‎ 94085

Panel discussion, along with Netflix, SmugMug, and other AWS customers, to "help business and technical decision makers from from small start-ups to large enterprises learn how to be successful with Amazon Web Service."

Really looking forward to this. I was on the panel for AWS Start-Up Tour in Santa Monica in 2007. You may recall there were some fires... like, nearly all of LA and SD hillsides were up in flames. Even so, 250+ people braved their way through the smoke and ash to reach that night club on Santa Monica Blvd, and we had a great panel discussion. This year in Sunnyvale should be even better.

---

While I've got your attention... I've been spending a fair amount of time in Seattle and on the phone to Seattle, particularly at one reconditioned former medical center, and especially regarding the fine folks at Elastic MapReduce. While in Seattle, I got to hang out with 30+ other system architects, many incredible folks. M David, of course I'm talking about you! :) At this stage of my career, I rarely seem to get time to sit back with peers, and swap stories and ideas. Time well spent.

BTW, we've been working with EMR now for a while, and recently started using EMR for in production. One of those cases where I could not post a blog entry then, but definitely wanted to tell everybody! Now that the news has been long public, I'm actually too busy using it commercially to blog ;)

See you on twitter @pacoid

westward ho

Some friends of mine live in the Westport area of Kansas City, across the street from a marker which notes the Oregon Trail, the Santa Fe Trail, and the California Trail. For many people in North America during the 19th century, their resolve to migrate west was tested there.

View Larger Map

So it is for us, with two little girls ready to travel. Our oldest is eager to live in hotels again, because they promise TVs, room service, and jacuzzi pools. She was done with KC as soon as she got her fill of fresh snow. Our youngest has been missing the ocean, especially the seals. She's asked, "When do we go home?" just about ever since we arrived in KC.


A woman from Kansas a few days ago made a few rather hostile remarks about how Californians feel about seals -- right after my little girl got through declaring how much she loves seals and whales.

Unforgivably callous. Intolerably ignorant. Not entirely unexpected. In other words, par for the course. I'm sorry to have to feel that kind of resignation, but it is entirely realistic. I bit my tongue to avoid making an offhand retort about "fly over zone". After all, parts of the Midwest are highly recommended.

Two bright spots have made these past 10 months worthwhile...

One is my team in the Analytics department at Adknowledge -- a.k.a., "Team Mendota". Those people have been a joy and and honor to work among. In addition to several contributions to R, Hadoop, AWS, RightScale, and points inbetween, we've made big strides in methods for having statisticians and developers working side-by-side on "Big Data" projects -- with tangible results. Frankly, I would have left much soon (supra 3 yr old tugging daddy's heartstrings) if it had not been for how much I appreciate this team. Flo, Chris, Nathan, Chun, Bill, Shan, Vicky, Margi, David, Gaurav -- I wish you all very well. My regret is that I did not get to work with you long enough.

The other is our collective friends and families in the area, met through our kids -- mostly because of their school, Global Montessori Academy. My wife has served on the board there. The teachers and just everyone involved have been amazing. Not sure quite how that place emerged here inside of KC, and I'd be hard-pressed to believe that anything much like it exists between Chicago and Austin... but GMA and the people associated will be missed.

I found myself thinking, many many times while living here in KC, "How can I teach my daughters about the principles in which I believe, if I cannot live them here?" GMA actually does represent those principles. Highly recommended.

For everyone else, see you back on the Left Coast!

ceteris paribus and the basted egg heuristic

Years ago, I had a job working as a short-order cook in a reasonably nice restaurant. One thing you learn quickly in that kind of work is what you can cook simply, reliably, while handling N other orders at the same time. Perhaps this falls into the general category of load-balancing or speculative execution, but some orders work better than others.

An example of a simple order is a "basted egg". One egg, some water and some oil in a small covered pan, burner set low, for a few minutes. I happen to like eating those for breakfast. Even though they almost never appear on a breakfast menu, it's one of the simplest things that a short-order cook can be asked to make -- aside from toast.

When I'm checking out a new little cafe for a quick breakfast, oftentimes I'll ask for a basted egg. That alone provides amazing insights to the structure and vitality of the restaurant... I've found that most wait staff and short-order cooks who are any good at preparing breakfast will jump at the chance to earn 3x food cost + their 20% tip by serving the simplest thing possible. (assuming that wait staff splits tips with cooks, which they should). Generally they do quite well, I enjoy breakfast, and pay more than 20% tip. Done.

However, using this metric, I've found an heuristic: restaurants on the brink of failure -- and there are many, almost always, given the nature of the business -- tend to fumble on something even as simple as a basted egg. I've found that almost every restaurant which told me "Sorry, no basted eggs here" has closed within the following 2 months. In fact, I've got a running calculation for a binomial confidence interval on that heuristic test converging near 1.0 as a best point estimate. Which is to say, pretty damn certain. Generally, I'd attribute that kind of failure to lack of insight or communication by the restaurant management, or simply that they'd hired untrained wait staff and kitchen staff. Generally the former condition is more opaque, and thus more likely as a risk to the restaurant.

Years ago -- and not nearly as many years ago as the cooking gigs, but close -- I started working in the technology industry. My first full-time job in Silicon Valley was in 1983... though I'd been mentored about the industry by my uncle (who was deep in it) going back to the late 1960s. In other words, I've been learning about what works and what doesn't in the tech industry for approximately 40 years, with the past 25 years of it hands-on. (Yikes, I'm that old now?!)

During that time, I've developed and tuned a robust heuristic for measuring the vitality of tech organizations -- not so different from my basted egg test in restaurants. Simply put, when I walk into a meeting, I ask myself three questions about everyone in the room. I won't disclose those questions here, but they are listed on my resume, which is public. I'll leave that research as an exercise for the interested reader. Anywhoo, every participant in the meeting gets scored on a scale of 0-3 points, whether they like it or not. Here's how I evaluate scores on an individual level:

3. great to work with, nurture a long-term relationship
2. short-term only, watch carefully for cracks in the foundation
1. limit dialogue to what is required
0. migrate in opposite direction!

Okay, I'll be the first to say that may sound trite, but hey it works. Even when you're encountering people who may not be the sharpest minds in the industry, if they score highly, consistently on that scale then ceteris paribus they'll likely do well in the industry.

I recall -- vividly -- one restaurant in central Austin, where I asked for a basted egg... the manager came out, and, without asking, placed his arm around my shoulders while explaning to me, "Son, there's no such thing as a basted egg." As one might imagine, I got up and left rather suddenly, and more adroitly put, there was no such thing as that idiot's employment 4 weeks later.

On the organizational level, based on the past few decades of experience watching tech companies struggle to be viable -- struggle on par with that found in the restaurant biz -- on one hand, if a tech firm has plenty of people who score 3's consistently, they've got my attention and quite likely could get my commitment too. On the other hand, if they've got people who score 1's or 0's consistently then, just like that ill-fated breakfast nook in Austin with the smarmy manager, get up and leave quickly!