newsletter update 2016-07

Synecdoche

Been absent from newsletter feeds for a year plus. During that time, several POVs changed markedly. Home: relocated from the shadows of the new Google X building …  to a rural part of Russian River Valley. Work: swapped more than an hour commute up to a SF tech start-up … for a 20 min bike ride to O’Reilly Media. Plus a few other priorities. While the newsletter name changes, its themes carry forward. Plus a few other priorities. I’ll cover: conference summaries, open source projects, interesting news items, recommendations … data science, machine learning, advanced math, sustainable ag, art, travel, artisan foods, cider, flying cars, etc. Albeit running in slightly different circles and casting a wider net.

Changing POVs: new “backyard” has less actuators, more pollinators

Confs

Been trying to keep pace with our 20+ tech conferences each year. Lots of travel, lots of interesting people and insights into their projects. Some upcoming:

• JupyterDay Atlanta, ATL, 13 Aug: will present Oriole: a new learning medium based on Jupyter + Docker
• Artificial Intelligence, NYC, 26–27 Sep: plus a drinkup at Wassail
• Next:Economy, SF, 10–11 Oct: plus a drinkup at Upcider
• Big Data Spain, Madrid, 17–18 Nov: will deliver a keynote, hopefully some workshops too!
• Strata+HW Singapore, SG, 5–8 Dec: I’ll present Computable Content: Notebooks, Containers, and Data-Driven Organizational Learning

De Pijp in Amsterdam during Scala Days 2015

Strata EU 2016, 31 May - 03 Jun. Venue moved to the much larger ExCel London space and that afforded an expanse of incredible people, tech innovations, data insights, etc. Also, it proved to be a good base of operations for exploring two recent interests regarding London: connectography and cider. More about both in a moment :)

Strata kicked off with a “Hey, are you busy?” text from the brilliant Alistair Croll, which led to one of the best pub crawls ever. Still parsing our discussions in those London pubs. Those provided the basis for several essays linked here. Seriously. Pay close attention to the intellectual sculpting that emerges from Alistair’s endeavors.

Strata tradition: team #noscurvy w/ BLL Ladies @holdenkarau @j_houg

Anirudh Koul invited me to a session with his Microsoft colleague Saqib Shaikh, where they delivered Beyond guide dogs: How advances in deep learning can empower the blind community. Sure enough, a favorite talk at Strata. Microsoft’s been busy building cognitive services in the cloud which developers can integrate into other software in general. Following retina surgery in both eyes in the 1990s, I experienced serious vision impairment. It was way too much to write code for a while. Switched to working on VR design for a while: shapes, colors, contrasts were much easier to manipulate than source code. Years later my vision still changes abruptly. Grateful now to see (pun intended) these AI innovations by Microsoft and I look forward to many interesting applications.

The proverbial “Full English”

Arrived back to Sonoma County just in time for Foo Camp 2016. Translated: that’s an unconference while camping with 200 fascinating people from around the world. Held in the apple orchard behind our office. DJ Patil and Ed Felten opened the weekend with a lively group discussion about data ethics, where Cory Doctorow delivered the punchline: when you own a mobile device, who choses to override settings, you, the manufacturer, or the government? Lives may depend on the answer.

One decidedly tangible result of Foo: “Blockchain of Love” by Ellie Volckhausen, Edie Freedman, Ally Miller, Johnny Diggz, Tony Parisi, Brian Fan, and yours truly helped with a line or two. Tony attempted to explain the ineffable:

Geeks, guitars and a whole lotta booze. We wrote this song in 45 minutes at O’Reilly Foo Camp!!!

Got to try a Copenhagen Wheel, courtesy of Assaf Biderman @ Superpedestrian – most definitely a decade beyond the Bionx on my prior ride. A favorite session was an Ignite talk by GalaxyKate, aka Kate Compton. Check out her work on generators and related tooling at Tracery.

the old Make Mobile

Velocity CA 2016, 20–23 Jun … Noticed that sessions related to Apache Mesos and DC/OS were standing-room-only. In other words, considering the recent Mesosphere investment by HPE and Microsoft, Mesos has arrived for the mainstream. A good illustration of this? The Container orchestration wars session by Karl Isenberg.

Having been away from SV for a while … Andrew Marantz’s recent New Yorker article How “Silicon Valley” Nails Silicon Valley struck a nerve or two. That rollerblade “scene” took place right behind our former house. Recommended.

Media Theory

One evening in late 1992, between the first Cypherpunks meeting and the FringeWare launch … I climbed aboard an opaque bus, allegedly headed to a private event at Intel Santa Clara. Probably had “press credentials” from Mondo 2000, or something. Sat down, turned around, met the person sitting in the next row. Brilliant, utterly mellifluous, compelling immediate engagement from all around, someone who soon became a star in the industry and also a lifelong friend: Mark Pesce. That evening was all about wandering into different worlds. For a brief interlude, the word “environment” itself became entirely fungible. Alice, down the rabbit hole. Guided tours of the latest in VR by Eric Gullichsen, et al.

offspring in tribal garb

Some things come full circle in the next generation. Our tween daughters recently chose their first “concert” experience: a first-of-its-kind world tour by celebrity YouTubers Dan and Phil. If you haven’t been tracking what’s changed in media, take a close look.

This time, the virtual from the Interwebs gets projected live and in the flesh. Spoiler alert: millennials may have leapt the Selachimorpha.

Read more in my Medium article: Wandering into an entirely different world

Oriole

Les Guessing posed an interesting question recently:

Surprised how data “Storytelling” mostly talks about visualization, not video or other great storytelling mediums.

Well said! Oriole is our update on that point. This new medium combines a video timeline with hyperlinked rich text, code, data, results, instrumentation, and visualization. The point is about repeatable science, in a sense, applied to the matter of contextualized hands-on learning about technology. Also a great step toward personalized learning, with brilliant dev+design from our Brooklyn Team. We’re eager to see how other authors leverage this.

Oriole

For a much better introduction, try it yourself! Check out the coding challenge in Regex Golf by Peter Norvig. That’s the first Oriole we published, as an exemplar.

Read more in my Radar article: Learning alongside innovators, thought-by-thought, in context

And join us for the first public talk about Oriole, at JupyterDay Atlanta on 13 Aug.

AI

David Beyer recently published an excellent series of interviews about the Future of Machine Intelligence. Favorites include:

• Ben Recht on what ML can learn from many decades of control theory, in particular due to similarities with classic dynamical systems – two distinct ways to consider models
• Anima Anandkumar on the use of tensor decomposition and multilinear algebras for non-convex optimization – to represent higher order relationships among latent variables
• Gurjeet Singh on the use of topology: “Data has shape and shape has meaning”, i.e., mapping from one space to another

One thing has been really bothering me about recent media coverage of AI. There’s a disturbing tendency to equate ML with AI, at the expense of considering how control systems are inherent in almost any business application. For example, Uber may use lots of machine learning; however, at the heart its business a control system manages drivers, customers, offers, and other vital aspects of supply chain and contingency. Note that the field of AI emerged from control theory and early cybernetics.

Read more in my Medium article: Beyond the AI Winter

A recent conversation on video by Tim O’Reilly and Peter Norvig explores applications of AI technologies, conversational interfaces, etc. That dovetails with how Google pointed DeepMind at data center power usage, for a dramatic 40% decrease in energy needs. That prefigures many industrial uses for AI, given how “the algorithm is a general-purpose framework to understand complex dynamics” in manufacturing, transportation, energy, etc. Note that Tim and Peter will be honorary program chairs at the new/aforementioned Artificial Intelligence conference in NYC this September.

Meanwhile, other arcanum about the Greater NYC Area … note the curiously shaped post horn which marks the façade of our Brooklyn office:

O’Reilly Brooklyn office, “front door”

The Trystero are out and about, and the sign is not stable.

Neurons and Art

More about applied AI, perhaps for dessert … What have artificial neurons been doing in the art world lately? Apparently they’ve evolved far beyond assembling art boxes that recall the work of Joseph Cornell. Check out the aftermath of the DeepDream: the art of neural networks show in SF this past February by Gray Area.

For an even more intense dose of narrative closure … Terence Broad applied his dissertation, Autoencoding Video Frames to have deep learning deconstruct the film Blade Runner. Using unsupervised learning in a way that Hollywood lawyers might not comprehend for quite a while, this work prompted Warner Brothers to issue a DMCA takedown notice … Except that (surprise!) the work did not include any Blade Runner footage.

For an epic recounting of neurons and art, check out Neural Networks and the Digital Hallucination by good friends over at Lumo Interactive. Also, links highly recommended in Beyond the AI Winter:

Check out the inexplicably entertaining short film Sunspring, authored by an AI named Benjamin.

That script writing AI leverages a recurrent neural network approach called long short-term memory (LSTM ) which takes abductive reasoning to new levels.

ebumna lw’nafh

What AIs aren’t quite dreaming about (yet) are seemingly less probable matters in generated imagery: crystals in a 4.57 billion-year-old meteorite found in Chukotka, Russia showed the first naturally occurring Forbidden Symmetry. YMMV.

Reading

Recently finished Connectography by Parag Khanna. Well worth a good read on several levels. See the especially interesting insights about city-states and SEZs. Had me at the maps. Even so, I’d suggest taking this text with a sizable grain of salt. Perhaps a boulder, caveat lector. The basic premise obtains: how supply chain economics have usurped national borders and trade restrictions. To wit:

Infrastructure, markets, technologies, and supply chains are not only logistically uniting the world, but propelling us toward a more fair and sustainable future.

brilliant maps in Connectography

OTOH its trappings as a book-length hagiography of corporatist values may required antiemetics. Read more in my Goodreads review:

The analysis of geography and history is thought-provoking, albeit some flawed historical citations. Clearly he loves major cities. Plus, there’s a bunch of “US bad, China good” on the surface. Quite a bunch. Dig deeper, it’s worthwhile.

Not to change the subject, but for a truly fun read: The Peripheral by William Gibson. One of the best novels I’ve read in a long while – had much catching up to do on books by the Great Dismal. Read more in my Goodreads review:

Chapters alternate POV between a young female protagonist of near-future and our male anti-hero of farther-future. Extra points for picking up the narrative at the start of some chapters in the “wrong” tense, rear-view mirror syntactic suspense for major plot beats. Like a video rewind after an unexpected concussion.

If you’d like to compare books, reviews, etc., please friend me on Goodreads.

Orchard Brew

A few years ago I was enjoying a seasonal, organic, farm-to-table meal with a close friend at Farmhouse Evanston. Ordered a cyser to drink, notably a Puff the Magic Cyser by Vander Mill – for something local to the Midwest. One sip took me on an adventure that’s kept unfolding in marvelous ways ever since.

ICYMI, ciders are making a huge comeback. Cider used to be the most popular alcoholic beverage in the US, prior to the insanity and cultural warfare of Prohibition. Especially in Sonoma West, where we live on a property that’s partly apple orchard, where we work on a campus built amidst an apple orchard. One fun aspect is that the ciders tend to be hyperlocal. In other words, one can travel the globe and encounter novel brews which stay close to their orchards of origin.

ngram occurrence for “cider” in Google Books, 1650–2008 ce

Here in Sonoma, ciders have little to do with the nasty, too sweet, apple-juice-with-vodka mush flavors that so many folks associate with the phrase “hard cider”. Cider makers here use champagne yeasts and other techniques to produce dry brews with complex flavors. Along with that, federal legislation in the US just revamped arcane tax laws that made The new CIDER Act allows small producers to get in the game with artisan brews, without getting stomped by weird post-Prohibition wine laws. This renaissance is good for the land, good for family farms, and great for drinking IMO.

Cider Bar @ Foo

So we held a taste test at Foo Camp this year. Brought in some of the more popular and interesting ciders from orchards near Sebastopol. Almost everyone (except DJ) favored Cider with Hops & Honey: a personal favorite from our neighbors Horse & Plow run by Chris Condos and Suzanne Hagins. That’s both good and somewhat dangerous, since Chris and Suzanne just opened their new “tasting barn” across the road from O’Reilly Media.

Also … LONDON! One of the best places in the world for cider. Fine folks at Orchard Pig recommended visiting The Williams Ale & Cider House, which was outstanding. Will be hosting drinkups there whenever I’m in London. Another couple great venues for sampling UK ciders are Euston Cider Tap and The Green Man. Favorite ciders in the UK? Reveller by Orchard Pig brings the fine flavors of Somerset, available far and wide. Top delight of this trip was Scotland’s Thistly Cross: off-dry, crisp, so smooth, the closest thing I’ve experienced to a Devoto Cidre Noir outside of Northern California :) Another treasure to seek out: Wild Summer by Kentish Pip:

Hints of Elderflower alight sea foam spray in a Nordic summer. Delicate, lightly astringent aftertaste.

As we tour the world with O’Reilly conferences, I’ll host drinkups at nearby cider bars. Here’s a toast to meeting you and sharing a brew or two! Meanwhile, if you’d like to share reviews of ciders, beers, etc., please friend me on Untappd.

Kane’ohe Bay Sandbar: grateful to visit with family in Hawaii recently

Ag+Data

Loved this informed, scathing rebuttal by the brilliant Wendell Berry, Wes Jackson, et al. Money quote:

What is remarkable, but unsurprising, is that these two gentlemen managed to write an entire column about agriculture without mentioning farmland or farmers.

That, in response to a NYTimes op-ed piece earlier this year entitled We Need a New Green Revolution. Note the key word entitled in that sentence: beware just about anyone who lauds a “Green Revolution” without recognizing the terrible consequences that resulted, i.e., financialization being a polite term. In other words, vast claims of battling starvation, while the most wealthy leveraged this vector to institutionalize mass starvation as their business model. There’s far too much “science” bandied about w.r.t. agriculture, pumped into media, heavy on funding ties yet light on scientific falsification.

I believe that politicians and other corporatists who side with Big Ag are behaving immorally. No other word for it.

Read more in my Radar article: Ag+Data

OTOH, absolutely LOVE it when intellect, integrity, and endurance overcome oh so many tedious decades of thoughtless aristotelianism. Check out How a Guy From a Montana Trailer Park Overturned 150 Years of Biology:

“Toby took huge risks for many years,” says McCutcheon. “And he changed the field.”

VRTX

Apropos of the above, I’ve been testing chapter ideas and other material for a new Liber 118. In particular, I’m a huge fan of Universal Basic Income, which has intriguing historical precedents plus enormous implications for the game that comes next.

After running the numbers for what UBI might cost in the US, implications became all the more urgent. It’s within striking range of commercial alternatives, i.e., corporatist funding in lieu of government initiatives. That got me thinking … about parody and science fiction to caricature what UBI might look like if everything went horribly wrong? What if corporatist values seized the day? Key portions of that parody got trumped, so to speak, by an even more bizarre happenstance in alleged reality.

¿quién es más macho? alleged reality vs. parody

Read more in my Medium article: Guaranteed Basic Income (cynical remix)

I’ll just leave you with …

Flying cars. It’s happening. When friends at Bloomberg first gave a heads up about this two years ago, the whole spiel sounded unreal. Now 2 out of 3 of the promises foretold in Planet Unicorn have come true. The big picture is that people travel into work in metro areas via line of sight. No more traffic congestion. Or something.

I guess that’s great for people who work in large cities. Not everybody. But there are likely some other intriguing uses.

Meanwhile, in the lesser hells of drone wizardry, one regrets that the noun #WTFery does not enjoy more regular usage in the English language.

---

Thanks for your attention, and I wish you all the best in your endeavors. Sign-up for the newsletter version at liber118.com

hadoop summit 2012: emergence of the confidence economy

moore intro

Geoffrey Moore opened his keynote at Hadoop Summit 2012 and promptly dropped the line: “You will remember this moment years from now.”

http://www.slideshare.net/Hadoop_Summit/geoffrey-moore-keynote-hadoop-summit

After a disappointing set of “sales pitch” keynotes on the first day of the conference (thanks Yahoo! — but you knew that already) many people attending seemed to roll their eyes about yet another keynote talk this morning. Surprise!

I was grateful to hear Geoffrey Moore trash Advertising as an industry at risk. If I may paraphrase: permanently caught between bleeding edge and dinosaurs, yet irreparably dependent on a broken business model. [FWIW, the last three VCs on whom I’ve used that line looked back at me like I was some kind of alien slime-mold.]

By the middle of his talk, Moore put up a slide with a half-dozen bullet points. The slide listed some of the most disruptive technologies on which businesses — Main Street, in his terms — would come to rely in the early 21st century. Those include: collab filters, behavioral targeting, predictive analytics, fraud detection, time series, etc., etc. Outside of the intelligence community and the hedge funds, the significance of these technologies is not well understood yet. Word. Up. Bitches.

Moore’s “Final Thoughts” slide really hit home. He talked about data access patterns (system of record vs. log file usage vs. real-time analytics vs. etc.) and how those access patterns create feedback loops within an organization. Moore claimed this was core DNA for Google, Amazon, etc., which all major businesses must now embrace. Or else. [That's about 95% overlap with a slide I made for (insert recent past employer) during a 2011Q1 pivot. Two pivots later, I left without any particular next gig in mind — clearly needing to get involved with a different business team. Shortly before their CEO got, um, an "opportunity" to find work elsewhere. But I digress.]

an exercise

So here’s a fun exercise for the interested reader: Pull up a 10-year chart for the S&P 500. Add to that CBS. Right.. Add to that Barnes & Noble. Bokay.. Add to that Wal-Mart. Got few bumps, some upturns.. Nothing to write home about.

Now add Google. Now add Amazon. Now add Apple. One might argue that I’m cherry-picking examples; however, one must understand those three in particular to grasp the trajectory of how Data modifies Companies.

Think about it. Imagine rolling the clock back about 13 years, just a few years before that huge financial sea change got going. Think about perceptions at the time of Apple, Amazon, Google. Most of the mainstream buzz that I heard or read in 1999 was largely disparaging about those three. They didn’t make sense to the average joe, and that was a problem. I will contend that what made sense to a handful of computer science grad students, but not to the average joe, was considered a problem for Main Street. A multi-billion dollar existential problem for some, as it turned out.

At the time, it seemed like Apple would never get past the overwhelming popularity of Dell and Microsoft. Amazon didn’t have a way to justify its enormous P/E ratio — and was probably fluff in the long run. Google was considered interesting, but a little strange, with no clear path toward revenue.

Now think about what happened to the music industry, the mobile industry, the … well, I could go on, but Apple disrupted the pants off lots of established players. Entire industries were taken down by one company. Then consider what happened to retail. One word, a verb according to Geoffrey Moore: Amazon. Think about what happened to advertising. Googled, and not in a nice way either. Amazon and Google took off in 1997Q4 and 1998Q1 respectively, with Big Data projects which became enormous cash cows: Amazon’s recommender system (plus cloud infrastructure), and Google’s search+ads (plus cloud infrastructure). Arguably, those two are the reasons we were having a "Hadoop" conference. Apple perhaps seems less in category; however Apple leveraged mountains of consumer data (plus cloud infrastructure) to drive its smartphones, App Store, etc.

Imagine what kinds of conversations which must have been occurring in the board rooms of CBS, Motorola, Barnes & Noble, Wal-Mart, etc., etc. Gone, gone, gone. Three relative underdogs became giants, tipping almost everyone else’s apple carts. (pun intended) At least three firms understood the power of leveraging their data, they understood the urgency of real-time analytics, etc. Their competitors, mostly, did not. Just look at those stock charts.

According to Moore, that was the tip of the iceberg. Most of the Global 1000 is now on notice. Over the next decade we’ll see monumental failures. Winners and losers, as always, but the magnitude of the losers may be unexpected.

central point

Moore’s central point in the keynote — since this was a Hadoop conference — was that the Hadoop tech stack and business ecosystem is maybe a year ahead of the proverbial “crossing the chasm” moment. Ergo his lead line.

Notably, enormous cultural changes of the 1990s and early 2000s have percolated through personal expectations among those coming up in the ranks. That’s happened more notably and with more impact outside the US than within it. He pointed to the “digitization of culture”, where access has become nearly universal, where broadband created emotional dimensions (Facebook, Pinterest, etc.), where mobile makes the experience ubiquitous regardless of socio-economic position.

Meanwhile, the corporate culture of how to “get stuff done” within enterprise has not kept up. There’s no Facebook for enterprise, no YouTube for enterprise, etc. [Well, actually, there are — and they are each headquartered within a bike ride of my home near the Mountain View / Palo Alto border — but you haven’t heard about them. Yet.]

Meanwhile, Facebook-esque consumer Internet companies of the world are too caught up in their own weirdly distorted realities to solve the larger business problems. Business problems where the solutions will inevitably derive from the social networks’ innovations. Oops.

In Moore’s vaulted opinion, those conditions won’t hold much longer. There will be winners. There will be losers. Big ones.

Meanwhile, for people of my ilk, Moore smiled and predicted: “This should provide at least a decade of entertainment for everyone present.” Fundamental business reasons are simple: enormous change ahead but precious few who are trained and experienced to navigate it.

key take-aways

My first key take-away is based on the observation last year that Enterprise giants bumbled into Hadoop Summit 2011 in a huge and awkward way. Oddly, the logo is an elephant, #justsayin

In contrast, this year was really smooth, completely professional, far too expensive … but almost all about data infrastructure in a world where nobody want to utter the word “Oracle”.

Mind you that neither of the two main “enterprise” keynote speakers from last year still have their same jobs. #justsayin

Also, notable Hadoop practitioners were noticeably absent. In fact, most of the cast and crew of Strata seemed to be missing. A particularly popular social network has been burning the midnight oil to make Hadoop perform backflips — they like gave a couple talks and seemed to vanish.

Let me put this in other words: several hundred million dollars have been invested by VCs (and angels) to recreate an industry in the image of Redhat and Yahoo!

Wow, did anybody think that would be a particularly good idea? No, but it’s the herd mentality in practice. Even after the 5th beer I’d still recognize that strategy as not particularly wise. Feels like when you talk with an ex-convict, and they drop a line “Yeah, I made some poor choices long ago…”

My hunch is those data infrastructure plays are mostly tax write-offs (for the “early adopter” part of Geoffrey’s famous curve) at this point.

Moore underscored how real payouts come when key verticals catch fire — with serious domain expertise leveraged. LinkedIn perhaps got close, but now it almost feels like a spamming broadcast system for HR and BD departments. We’ll see “Big Data” killer apps which mean something to lots of people. Beyond the GOOG+AMZN+AAPL tip o’ the iceberg. They will come from people who have sophisticated backgrounds in Stats + ORSA + distributed systems + functional programming + DevOps, people who can also communicate well with actual business leaders. Not those employed by some halfwit B-school grad who’s posturing as the next Steve Jobs, when in reality he drinks bad beer at a lame, faux-hipster sports bar while watching cable televison. Or something. Dude, hop on your fixed-gear bike and standstill/peddle your sleeve tattoos out of here.

Translated: the proverbial ignite moment, that spark of innovation, is not going to come from the likes of a Cloudera or a Platfora or a (banal noun)-(o|e)ra… But it’s going to come, probably not many moons away. It will be in apps.

the sound of disruption

Thirty years ago, I went into a field called “math science”, i.e. how to build predictive analytics as software apps. Stanford — the Statistics department chairman, Bradley Efron, in particular — had put together an interdisciplinary degree which combined math, statistics, operations research, programming, engineering, etc. At the time, most of my peers in the program went on to become insurance actuaries. I went instead to do graduate work in machine learning and distributed systems.

For nearly two decades, most employers could care less about any quantitative background. They wanted C++ software engineers working all day on APIs from Sun or Microsoft or Oracle. Or they wanted managers. Then, in about 2000, came the sea change.

Right about the same time as ticker symbols for Apple and Google and Amazon were strolling up to their respective launchpads, some people began to look at my resume and ask a different line of questions.

I’ll always remember the first: a microchip vendor — one which makes electronics for several products you’ve purchased — was getting squeezed by Intel and their silicon compiler vendor. Critical features were being deprecated, specifically to put this second-tier player out of business. The company was on notice. They had to find a proverbial needle in a haystack: out of tens of thousands of circuit designs, they had to identify the 1% which would no longer be licensed — then redesign those. Quickly.

An internal team at the company had tried, but given up. Too much data for their techniques, it would’ve taken years to resolve. The company hired an electronics consulting firm in Austin, and engineers went to work, but gave up as well. Too much data, not enough signal. I got called in, as a “Whatever, just see if you can get anywhere” last-ditch effort. About 20 lines of Perl and one relatively simple equation later, I dumped my results into a scatterplot.

One of the lead circuit designers picked up my plot off the laser printer and began laughing. Loudly. The whole office heard him.

His manager grew annoyed: “What?! Why are you laughing?!”

Engineer: “He found it.”

When I turned in my invoice, the manager glared. “Look,” he said in a growl, “Just go somewhere for about three weeks. Bill us the whole time. Then come back and turn that in.”

My brows furrowed, this was a high-dollar rate for 2000.

“If you don’t pad that damn invoice…” he paused, “You’ll make both us and our customer look like complete fools. Piss a lot of people off.”

That’s the sound of Disruption.

More than a decade later, the summary graf of my resume reads like bullet points from Geoffrey Moore’s slide. Collab filters, anti-fraud classifiers, predictive analytics, etc. Even in the past few years, when HR people have read that resume, several looked up with a frown, said they thought that kind of work was better suited for business analysts — yadda, yadda, yadda, keep following the herd: you put the “botch” in “beotch”.

At a time when lots of business (start-ups as well as enterprise) are starving because they cannot hire Data Scientists, I’ve been busy building teams. Teams which delivered $MM results. I’ve hired about thirty people onto Data Science teams within the past few years — at a time when many start-ups would feel lucky to hire one. #justsayin

mal*wart

I read one of the most imbecilic essays recently from Forbes/Quora: “What Would Be The Global Impact If Wal-Mart Abruptly Shut Down?” Essentially, a hagiography stating that Wal-Mart is too big to fail, that the consequences on the US economy, the global economy, would be catastrophic. Translated: may require an enormous bailout, soon.

[In case you hadn’t guessed, I just threw up a little bit in my mouth.]

What. A. Fucking. Moron. The reality is that Wal-Mart hasn’t been doing so well over the past decade. Not if you peel back enough layers of PR. Not since they tried to bamboozle the LA city council. And failed. Moreover, folks at Amazon could really care less which Senators or SecState/former-first-lady the execs in Arkansas have in pocket. Bezos has positioned to take over 150% of Wal-Mart’s business the picosecond after Bentonville implodes. Sears and Target have reinvented themselves specifically for that very instant. So long, good riddance. Remember the point about the Global 1000 on notice? About the importance of business fundamentals?

sears, a.k.a. that web site which kinda looks like amazon

A third keynote talk that day was by the Sears CTO, Phillip Shelley. I had packed up my laptop and backpack, and was getting ready to walk out of the auditorium. After his first few sentences, I put my stuff back down and started taking notes.

Dr. Shelley mentioned how Sears started as a mail-order business a century ago, though more recently got completely kicked by another “catalog” called Amazon. Now they are leveraging Hadoop + R + Linux/Xen private cloud (srsly, this is from the Sears CTO?!?) to reinvent their business with 100x more detail on regional pricing models. Literally calculating personal pricing discounts for individuals, multiple times per day, specifically for mobile.

Sears: core algorithm moved from [6000 lines of COBOL on mainframe with 3.5 hr batch window] to [50 lines of Hadoop app on Linux with 8 min batch window], while reducing TCO for enterprise IT by two orders of magnitude. So much success, that they’ve spun it out as a new business line called MetaScale.

Brilliant strategery by Sears. Some seriously high powered Data Science talent walked out of that auditorium musing how they wished their VP Engineering was half as progressive as Sears. Srsly? Um, that’s called a PR coup. [Literally at the same moment as Wal*Mart had HR droids spamming the audience with whispers and rumors of lucrative salaries. Gak.]

emergence of the confidence economy

What’s the deal? It’s about confidence. Those giants in the Global 1000 which Geoffrey Moore says are on notice? They got that way by believing that business is largely about who barks the loudest, barks the longest, and cuts the most deals under the table. The proverbial alpha male in a wolf pack.

Wal-Mart would be a prime example, in my opinion. Their business is predicated on fundamentals which simply do not hold. Misplaced confidence. Thanks to people like Hillary Clinton, Wal-Mart has gained much influence on the House and Senate floors and the halls of the State Department and the UN assembly. In other words, so long as we manage to keep fuel costs artificially low, Wal-Mart’s market valuation will keep growing. So long as we believe that bullying vendors, conducting intelligence operations against the rest of your ecosystem, etc. — that these kinds of practices are ethical and sound in the long-run, then Wal-Mart will keep growing. Bullshit. Go look at that stock chart again. Wal-Mart is about tall white guys in dark suits, acting like complete pricks, destroying and plundering anything they can get their grimy paws on. Richard Gere in Pretty Woman, before he gets Julia Roberts. And not much more than that. On notice.

Moore is pointing out, in my opinion, that the issue at hand is about uncertainty. The point of establishing a corporate charter was always to externalize risk and perpetuate wealth for shareholders. That was true four centuries ago, when the first transnational was established, and has been true ever since. The modus of that mechanism is a process called sublation. The train wreck for sublation is uncertainty. In an environment where uncertainty holds sway, having real-time analytics from petabytes of customer data wins out over having a Senator in pocket. Any day of the week. The antidote for uncertainty is confidence. While there had been a regime of an “Attention Economy” extant for the past two decades or so, we’re now entering a new regime of the “Confidence Economy”.

Here’s the deal: people like me like those of us in Moore's lecture have been the “secret sauce” fueling the rise of Amazon, Google, Apple, etc. We use techniques which are mostly not well understood outside of Langley and the hedge funds. The tools of contemporary corporate assassins. Guys in suits who act like pricks in lieu of practicing business fundamentals — those guys are our targets. The modus is Disruption. If you have an MBA or a CxO title and not much else to back it up, I put food on my family’s table by being a sniper paid to hunt you. Lots of *great* food. And some of the best wines available. I shake the tension out of my hands, correct for wind and distance, draw a bead, take a deep breath, squeeze the trigger. Kill shot.

The challenges faced by Data Scientists are daunting. On one hand, most mathematicians lack enough solid engineering to create killer apps. Conversely, most engineers lack enough math to make any headway on the business data. Most business analysts lack enough of either the math or the engineering to be worth hiring. Data Scientists provide all three areas of expertise: the engineering and the math and the business insights to contend with mountainous torrents of data, and move the needle. On the other hand, Data Scientists must also speak truth to power. In any given business, there will be winners and losers. Executives, people accustomed to their own power, taken down. Meanwhile, we Data Scientists come prancing into a business, we do our magic, and consequently we point out which executives are bullshit and must be “executed”.

The reason why I’ve built Data teams at a time when others are starving is simple: confidence. Sure, I’ve logged three decades of machine learning, statistical modeling, data management, distributed computing, etc. When I talk with a grad student about their work, I can tell them in 25 words or less what they need to do on their first day at work to become regarded as an great asset to the team. They already know the techniques, but crave confidence. Into the trenches, fresh-out, having to speak truth to power. They’ll be placed into some faltering business unit, run some detailed analysis, and point out that the VP who’s been arguing loudly was completely wrong for the last N years and his/her ego cost the company several $MM. You can bet that those execs will return fire. However, a person like me is confident that we can get a kill shot. I show new folks how to draw a bead and squeeze the trigger. Been doing it for a long while, and will be doing for a long while more.

Snipers have an eerily pragmatic sense of confidence. And, by the way, that’s a peculiarly difficult job. Praise goes out to the men and women who serve their countries in uniform — when the cause is just. [FWIW, before tackling the challenges of data+science, I wore a military uniform and carried a rifle. Sniper training has become invaluable.]

my opinions

#1: Enterprise suffers because so many people in the corporate leadership ranks (or rather, amongst those clawing and scrambling to make their way into the corporate leadership ranks) consider themselves to be a different caste — if not a different species all together — from the rest of us who do not have a salaried position with a transnational. In a “Let them eat cake” world fraught with trillion-dollar bailouts, that’s not a particularly good way to future-proof. Moreover, this is why VCs are vital… to demolish that kind of hubris via constructive Disruption. #justsayin Word. Up. Bitches.

#2: Enterprise tooling, which is now mostly dependent on JVM-based apps, suffers because it has embraced “Convention over Configuration” … CoC has its place. I can imagine that it’s an excellent idea for heart surgeons to have a standard toolset, with scalpels in the exact same positions, etc. CoC is not a particularly good way to manage complexity and uncertainty, because it simply displaces major problems into the build system. Ultimately, it fails too much and impedes spin-up. Which, I believe, represents an enormous, ticking time bomb in Enterprise. Here’s a challenge: Show me a metric for the median period it takes in your business for a newly hired engineer to push code changes into production use which is adopted by at least 80% of your customer base. Now show me a metric for the media period it takes in your business between the point where a product manager identifies a needed feature and a newly hired engineer is ready for spin-up. From those, I’ll make a prediction based on that metric for how well your business will survive the “on notice” condition which Geoffrey Moore described. Better clues for navigating complexity and uncertainty can be found in the works of Ilya Prigogine or Stephen Wolfram. To wit, functional programming is more likely to address complexity, real complexity, and also more likely to attract top talent. CoC, not so much. Perhaps your enterprise business addresses Main Street instead of Early Adopters... Recall that Google and Amazon and Apple crossed the chasm by recruiting armies from grad students — at a time when most other people erred on the side of average joes. Remember the point about real-time analytics? It counts for training your people, then retraining, and retraining, constantly — to grapple with uncertainty. Kill shot. Global 1000.

#3: MapReduce will be unrecognizable within three years. Hadoop Summit will become something quite different after Hadoop bifurcates and gets sublated into Something Else. For example, it would be not difficult to use the Simple Workflow Service from Amazon AWS to implement MapReduce using the core part of Cascading… a different kind of MapReduce, which is not constrained by JVMs… which could scale much more gracefully and robustly… which could out-perform Google infrastructure and avoid attempting to re-create the industry in the image of Yahoo! At which point, one could deploy functional programming blocks at enterprise scale, without having to rely on the morass of enterprise build tools. Hmmm… may need to get a term sheet for that one.

Geoffrey Moore, we may have a few answers for your questions.

election visualizations

I use the following tools to watch the 2008 US presidential election take shape. These tools show great applications of geographic analytics, prediction markets, data visualization, etc.

• CNN Electoral Map Calculator


• FiveThirtyEight Electoral Projections


• RealClearPolitics Electoral College


• CS Monitor Patchwork Nation


• Election Night Timetable


• Betfair Next US President Odds

My one wish would be to see an aggregate view of them as a cartogram.

ceteris paribus and the basted egg heuristic

Years ago, I had a job working as a short-order cook in a reasonably nice restaurant. One thing you learn quickly in that kind of work is what you can cook simply, reliably, while handling N other orders at the same time. Perhaps this falls into the general category of load-balancing or speculative execution, but some orders work better than others.

An example of a simple order is a "basted egg". One egg, some water and some oil in a small covered pan, burner set low, for a few minutes. I happen to like eating those for breakfast. Even though they almost never appear on a breakfast menu, it's one of the simplest things that a short-order cook can be asked to make -- aside from toast.

When I'm checking out a new little cafe for a quick breakfast, oftentimes I'll ask for a basted egg. That alone provides amazing insights to the structure and vitality of the restaurant... I've found that most wait staff and short-order cooks who are any good at preparing breakfast will jump at the chance to earn 3x food cost + their 20% tip by serving the simplest thing possible. (assuming that wait staff splits tips with cooks, which they should). Generally they do quite well, I enjoy breakfast, and pay more than 20% tip. Done.

However, using this metric, I've found an heuristic: restaurants on the brink of failure -- and there are many, almost always, given the nature of the business -- tend to fumble on something even as simple as a basted egg. I've found that almost every restaurant which told me "Sorry, no basted eggs here" has closed within the following 2 months. In fact, I've got a running calculation for a binomial confidence interval on that heuristic test converging near 1.0 as a best point estimate. Which is to say, pretty damn certain. Generally, I'd attribute that kind of failure to lack of insight or communication by the restaurant management, or simply that they'd hired untrained wait staff and kitchen staff. Generally the former condition is more opaque, and thus more likely as a risk to the restaurant.

Years ago -- and not nearly as many years ago as the cooking gigs, but close -- I started working in the technology industry. My first full-time job in Silicon Valley was in 1983... though I'd been mentored about the industry by my uncle (who was deep in it) going back to the late 1960s. In other words, I've been learning about what works and what doesn't in the tech industry for approximately 40 years, with the past 25 years of it hands-on. (Yikes, I'm that old now?!)

During that time, I've developed and tuned a robust heuristic for measuring the vitality of tech organizations -- not so different from my basted egg test in restaurants. Simply put, when I walk into a meeting, I ask myself three questions about everyone in the room. I won't disclose those questions here, but they are listed on my resume, which is public. I'll leave that research as an exercise for the interested reader. Anywhoo, every participant in the meeting gets scored on a scale of 0-3 points, whether they like it or not. Here's how I evaluate scores on an individual level:

3. great to work with, nurture a long-term relationship
2. short-term only, watch carefully for cracks in the foundation
1. limit dialogue to what is required
0. migrate in opposite direction!

Okay, I'll be the first to say that may sound trite, but hey it works. Even when you're encountering people who may not be the sharpest minds in the industry, if they score highly, consistently on that scale then ceteris paribus they'll likely do well in the industry.

I recall -- vividly -- one restaurant in central Austin, where I asked for a basted egg... the manager came out, and, without asking, placed his arm around my shoulders while explaning to me, "Son, there's no such thing as a basted egg." As one might imagine, I got up and left rather suddenly, and more adroitly put, there was no such thing as that idiot's employment 4 weeks later.

On the organizational level, based on the past few decades of experience watching tech companies struggle to be viable -- struggle on par with that found in the restaurant biz -- on one hand, if a tech firm has plenty of people who score 3's consistently, they've got my attention and quite likely could get my commitment too. On the other hand, if they've got people who score 1's or 0's consistently then, just like that ill-fated breakfast nook in Austin with the smarmy manager, get up and leave quickly!

schwa meets darpa

Engadget ran a story today entitled "US Army to arm UAVs with xenon-based paralysis inducer". The article describes that technology as a "non-lethal crowd-control device".

Wikipedia's entry for Schwa describes it as the "underground conceptual artwork" of Bill Barker. A true original among the popular alien image fetish / conspiracy theory "parody" themes of the 1990s. One of Bill's original serigraphs for Schwa hangs on my wall. We had lots of fun working with Bill at FringeWare, especially for the cover of FWR #5. That issue was our first to sell through completely, though I've stashed away a box or two for posterity.

Bill's first pitch to me about Schwa was that it was a government conspiracy for alien UFOs to use Xenon-based technology for non-lethal crowd control. In other words mass media brainwashing. Holy shit! Apparently somebody at DARPA was actually listening to us back in the 1990s. Either that, or Bill knew more back then than he could ever say in words. I'd bank on the latter.

BTW, Wikipedia states that Bill has been "MIA" since 2001. I've talked with him since then. More like doing fine while avoiding the paparazzi.

spinning iran

If you read just one article today, do just one thing to understand the world a little better, I recommend that it be to read Spinning Iran on Rafe Colburn's blog.

Bush's rhetoric about "War with Iran" is more neocon (read: pro-Israeli lobby) nonsense. The Newsweek "photo gallery" is like a page out of 1984. US diplomats are negotiating with Iranian diplomats now, brokered via Russia and Qatar. The results of those negotiations will be normalized relations, which are slated to become Bush's legacy (what's left of it).

The age of the "Great Superpower" is over, the Kissinger Doctrine failed, and with it goes the predominance of a whole other mess: Saudis, Tel-Aviv, etc. Welcome to a world where Asia becomes much, much more quintessential.

wal-mart's peanut butter manifesto and "Slavery 2.0"

Is the world's largest multinational corporation, Wal-Mart, on the decline? Or is it simply riding the political ebb and flow as it tracks a neoconservative agenda to undermine people of color and drive a wedge through the Democratic Party?

Interesting news comes today that a judge in Chelan, WA has declared the building permit illegal for a new Wal-Mart "supercenter" being constructed there.

The mix of reactions from locals in Chelan seems characteristic of other areas in which Wal-Mart has faced opposition. Older or poorer residents look at consumer prices and tend to want access to cheap goods, which Wal-Mart provides. Local retail businesses generally know that they'll be forced into a situation in which they can no longer stay open, resulting from changes in traffic and spending patterns, changes in zoning, etc. Real estate organizations lobby ardently on behalf of Wal-Mart, often in direct opposition to homeowners. Local political officials get paid off and cut corners legally to accommodate corporate plans. Homeowners know that as a supercenter comes into an area, the crime rate tends to rise, the traffic becomes clogged, and the home values tend to fall.

Where we own a home in Austin, news came last month that Wal-Mart had obtain a building permit to open a 220,000 square foot, 24/7 "supercenter" in our residential neighborhood. To place this into perspective, that would become one of the largest "big box" stores in the state of Texas - in our residential neighborhood.

In brief, there is an old shopping center called "Northcross Mall" which was built decades ago when this area was on the very outskirts of Austin. When my wife was a kid here, Northcross was the shopping center of the region, but for several years it has been largely vacant - mostly due to rather harsh policies by its absentee landlord. The city has grown to a point where this neighborhood is now considered "central" and is mostly residential. When I bought a home in this neighborhood 12 years ago, nearby retail properties were mostly in decline; however, several innovative small businesses have worked hard to renovate the area, so that we have a place were neighbors can walk to local businesses. WMT came in, arranged with Austin's city manager to obtain a building permit without a public hearing, the city manger recused herself from further comment on the issue, and now we're going to have semi-trucks driving through our neighborhood to supply a 24/7 superstore. On one side there is the Real Estate Council of Austin, pushing hard to promote the WMT/Clinton agenda. On the other side, there are five neighborhood associations - in other words, the political power of approximately 2500 soccer moms plus their best friends, family, and neighbors, which multiples into about 10% of the city population - looking ahead toward rising crime rates, lower home property values, degraded public schools, etc.

It is interesting to look at the political activity of Wal-Mart. Notably, they spend most of their political funding - more than 80% - on conservative, "Religious Right" Republican candidates. At the executive levels, WMT contributions to RNC exceed their contributions to DNC by approximately a 30:1 ratio. Those candidates tend to represent the bulk of WMT customers, two-thirds of whom come from "Red" states: poor, under-educated, working-class people in rural America. However, there is also a healthy stream of WMT political funding heading toward Democratic leadership, particularly in the Senate, such as Clinton, Ford, Bayh. Clinton in particular was on WMT's board of directors for several years during its major growth period - during its main period of destroying "ma & pa" retail businesses throughout the US, during its main period of undermining laws for employee benefits.

What is intriguing and arguably appalling is how much emphasis WMT has placed on African-American political lobbies. One blantant example is found the National Coalition on Black Civic Participation, where WMT with its Democratic party candidates in tow has been systematically acting as a wolf in sheep's clothing to tear apart support for Black voting rights, labor protections, etc. Lee Walker of Heartland Institute argues a racial case for "sell on the downside" case on behalf of WMT:

When a city government steps in and makes it illegal for some businesses to pay less than $10.00 an hour and an additional $3.00 an hour in benefits, it effectively makes it illegal for those businesses to hire many black applicants. Crucial entry-level jobs that would enable young blacks and ex-offenders to start climbing the career ladder to better-paying and more fulfilling jobs will be closed to them.

IMHO, if Walker, et al., do not represent a neo-conservative push toward "Slavery 2.0", I don't know what else would be reasonable to call them.

Looking at the arguments about Wal-Mart, the issues are not about "access to reasonably priced consumer goods" or "protection of a free market". WMT is just about as oppositional to "free market" economics as any entity in the world, considering how their growth in recent years has been built by literally threatening local elected officials into changing laws which support WMT at the cost of destroying existing businesses and communities, not to mention their lobbying against labor protection laws, minimum wage, health care rights, etc. The underlying issue is about how to compel poor people - mostly people of color - to work for less than an affordable living, as the conflict in Chicago exposed so directly.

Admittedly, I am a Green, and in some ways the WMT corporate history of predatory acts have helped to illustrate what I believe are the fundamental and unresolvable flaws of the Democratic Party - racial prejudice being among the foremost. Even so, I am encouraged to find that currently DNC front-runners such as John Edwards and Barak Obama have broken ranks with Clinton, et al., with respect to WMT. The issue there appears to be foreign policy: Clinton arguably has no foreign policy beyond "Let's increase the trade deficit with China" or "Maybe we should invade Iran". Thankfully, other DNC candidates appear to have a more global (read: less self-serving) perspective for the long-run.

If one were to take a counterpoint position, what would be important to consider vis-a-vis WMT in terms of conservative American interests? Notably, WMT is one of the largest employers in the US, and it accounts for a substantive chunk of our foreign trade. Given those measures of strategic importance, it is interesting to review the security risks which WMT faces. For example, while they appear to have built and continue to nurture a solid foundation for political support and a huge base of consumers, they have a systematic weakness in their vendor relations. Quite arguably, a culture of fear has emerged where WMT actively pits one vendor against its competitors, to leverage that fear for lower wholesale pricing. While in the long run their pattern of sublation will most likely expose WMT to antitrust actions, in the short term their vendor policies expose a significant vulnerability to so-called "terrorist" attacks. If a non-state actor were to arrange for systematic delays of the supply chain of vendor deliveries - even on the order of trucks being delayed on the highways by mere minutes - then some of the most essential WMT policies would be triggered and come into play. Effectively the corporation would begin to attack its own internal operations.

Not that I am encouraging such acts, but as a thought experiment, does that risk worry anybody? If some non-state actor were ambitious enough to stage that kind of attack, the most likely outcomes imply that it's time to "stop eating peanut butter" in Bentonville.