newsletter update 2016-07

Synecdoche

Been absent from newsletter feeds for a year plus. During that time, several POVs changed markedly. Home: relocated from the shadows of the new Google X building …  to a rural part of Russian River Valley. Work: swapped more than an hour commute up to a SF tech start-up … for a 20 min bike ride to O’Reilly Media. Plus a few other priorities. While the newsletter name changes, its themes carry forward. Plus a few other priorities. I’ll cover: conference summaries, open source projects, interesting news items, recommendations … data science, machine learning, advanced math, sustainable ag, art, travel, artisan foods, cider, flying cars, etc. Albeit running in slightly different circles and casting a wider net.

Changing POVs: new “backyard” has less actuators, more pollinators

Confs

Been trying to keep pace with our 20+ tech conferences each year. Lots of travel, lots of interesting people and insights into their projects. Some upcoming:

• JupyterDay Atlanta, ATL, 13 Aug: will present Oriole: a new learning medium based on Jupyter + Docker
• Artificial Intelligence, NYC, 26–27 Sep: plus a drinkup at Wassail
• Next:Economy, SF, 10–11 Oct: plus a drinkup at Upcider
• Big Data Spain, Madrid, 17–18 Nov: will deliver a keynote, hopefully some workshops too!
• Strata+HW Singapore, SG, 5–8 Dec: I’ll present Computable Content: Notebooks, Containers, and Data-Driven Organizational Learning

De Pijp in Amsterdam during Scala Days 2015

Strata EU 2016, 31 May - 03 Jun. Venue moved to the much larger ExCel London space and that afforded an expanse of incredible people, tech innovations, data insights, etc. Also, it proved to be a good base of operations for exploring two recent interests regarding London: connectography and cider. More about both in a moment :)

Strata kicked off with a “Hey, are you busy?” text from the brilliant Alistair Croll, which led to one of the best pub crawls ever. Still parsing our discussions in those London pubs. Those provided the basis for several essays linked here. Seriously. Pay close attention to the intellectual sculpting that emerges from Alistair’s endeavors.

Strata tradition: team #noscurvy w/ BLL Ladies @holdenkarau @j_houg

Anirudh Koul invited me to a session with his Microsoft colleague Saqib Shaikh, where they delivered Beyond guide dogs: How advances in deep learning can empower the blind community. Sure enough, a favorite talk at Strata. Microsoft’s been busy building cognitive services in the cloud which developers can integrate into other software in general. Following retina surgery in both eyes in the 1990s, I experienced serious vision impairment. It was way too much to write code for a while. Switched to working on VR design for a while: shapes, colors, contrasts were much easier to manipulate than source code. Years later my vision still changes abruptly. Grateful now to see (pun intended) these AI innovations by Microsoft and I look forward to many interesting applications.

The proverbial “Full English”

Arrived back to Sonoma County just in time for Foo Camp 2016. Translated: that’s an unconference while camping with 200 fascinating people from around the world. Held in the apple orchard behind our office. DJ Patil and Ed Felten opened the weekend with a lively group discussion about data ethics, where Cory Doctorow delivered the punchline: when you own a mobile device, who choses to override settings, you, the manufacturer, or the government? Lives may depend on the answer.

One decidedly tangible result of Foo: “Blockchain of Love” by Ellie Volckhausen, Edie Freedman, Ally Miller, Johnny Diggz, Tony Parisi, Brian Fan, and yours truly helped with a line or two. Tony attempted to explain the ineffable:

Geeks, guitars and a whole lotta booze. We wrote this song in 45 minutes at O’Reilly Foo Camp!!!

Got to try a Copenhagen Wheel, courtesy of Assaf Biderman @ Superpedestrian – most definitely a decade beyond the Bionx on my prior ride. A favorite session was an Ignite talk by GalaxyKate, aka Kate Compton. Check out her work on generators and related tooling at Tracery.

the old Make Mobile

Velocity CA 2016, 20–23 Jun … Noticed that sessions related to Apache Mesos and DC/OS were standing-room-only. In other words, considering the recent Mesosphere investment by HPE and Microsoft, Mesos has arrived for the mainstream. A good illustration of this? The Container orchestration wars session by Karl Isenberg.

Having been away from SV for a while … Andrew Marantz’s recent New Yorker article How “Silicon Valley” Nails Silicon Valley struck a nerve or two. That rollerblade “scene” took place right behind our former house. Recommended.

Media Theory

One evening in late 1992, between the first Cypherpunks meeting and the FringeWare launch … I climbed aboard an opaque bus, allegedly headed to a private event at Intel Santa Clara. Probably had “press credentials” from Mondo 2000, or something. Sat down, turned around, met the person sitting in the next row. Brilliant, utterly mellifluous, compelling immediate engagement from all around, someone who soon became a star in the industry and also a lifelong friend: Mark Pesce. That evening was all about wandering into different worlds. For a brief interlude, the word “environment” itself became entirely fungible. Alice, down the rabbit hole. Guided tours of the latest in VR by Eric Gullichsen, et al.

offspring in tribal garb

Some things come full circle in the next generation. Our tween daughters recently chose their first “concert” experience: a first-of-its-kind world tour by celebrity YouTubers Dan and Phil. If you haven’t been tracking what’s changed in media, take a close look.

This time, the virtual from the Interwebs gets projected live and in the flesh. Spoiler alert: millennials may have leapt the Selachimorpha.

Read more in my Medium article: Wandering into an entirely different world

Oriole

Les Guessing posed an interesting question recently:

Surprised how data “Storytelling” mostly talks about visualization, not video or other great storytelling mediums.

Well said! Oriole is our update on that point. This new medium combines a video timeline with hyperlinked rich text, code, data, results, instrumentation, and visualization. The point is about repeatable science, in a sense, applied to the matter of contextualized hands-on learning about technology. Also a great step toward personalized learning, with brilliant dev+design from our Brooklyn Team. We’re eager to see how other authors leverage this.

Oriole

For a much better introduction, try it yourself! Check out the coding challenge in Regex Golf by Peter Norvig. That’s the first Oriole we published, as an exemplar.

Read more in my Radar article: Learning alongside innovators, thought-by-thought, in context

And join us for the first public talk about Oriole, at JupyterDay Atlanta on 13 Aug.

AI

David Beyer recently published an excellent series of interviews about the Future of Machine Intelligence. Favorites include:

• Ben Recht on what ML can learn from many decades of control theory, in particular due to similarities with classic dynamical systems – two distinct ways to consider models
• Anima Anandkumar on the use of tensor decomposition and multilinear algebras for non-convex optimization – to represent higher order relationships among latent variables
• Gurjeet Singh on the use of topology: “Data has shape and shape has meaning”, i.e., mapping from one space to another

One thing has been really bothering me about recent media coverage of AI. There’s a disturbing tendency to equate ML with AI, at the expense of considering how control systems are inherent in almost any business application. For example, Uber may use lots of machine learning; however, at the heart its business a control system manages drivers, customers, offers, and other vital aspects of supply chain and contingency. Note that the field of AI emerged from control theory and early cybernetics.

Read more in my Medium article: Beyond the AI Winter

A recent conversation on video by Tim O’Reilly and Peter Norvig explores applications of AI technologies, conversational interfaces, etc. That dovetails with how Google pointed DeepMind at data center power usage, for a dramatic 40% decrease in energy needs. That prefigures many industrial uses for AI, given how “the algorithm is a general-purpose framework to understand complex dynamics” in manufacturing, transportation, energy, etc. Note that Tim and Peter will be honorary program chairs at the new/aforementioned Artificial Intelligence conference in NYC this September.

Meanwhile, other arcanum about the Greater NYC Area … note the curiously shaped post horn which marks the façade of our Brooklyn office:

O’Reilly Brooklyn office, “front door”

The Trystero are out and about, and the sign is not stable.

Neurons and Art

More about applied AI, perhaps for dessert … What have artificial neurons been doing in the art world lately? Apparently they’ve evolved far beyond assembling art boxes that recall the work of Joseph Cornell. Check out the aftermath of the DeepDream: the art of neural networks show in SF this past February by Gray Area.

For an even more intense dose of narrative closure … Terence Broad applied his dissertation, Autoencoding Video Frames to have deep learning deconstruct the film Blade Runner. Using unsupervised learning in a way that Hollywood lawyers might not comprehend for quite a while, this work prompted Warner Brothers to issue a DMCA takedown notice … Except that (surprise!) the work did not include any Blade Runner footage.

For an epic recounting of neurons and art, check out Neural Networks and the Digital Hallucination by good friends over at Lumo Interactive. Also, links highly recommended in Beyond the AI Winter:

Check out the inexplicably entertaining short film Sunspring, authored by an AI named Benjamin.

That script writing AI leverages a recurrent neural network approach called long short-term memory (LSTM ) which takes abductive reasoning to new levels.

ebumna lw’nafh

What AIs aren’t quite dreaming about (yet) are seemingly less probable matters in generated imagery: crystals in a 4.57 billion-year-old meteorite found in Chukotka, Russia showed the first naturally occurring Forbidden Symmetry. YMMV.

Reading

Recently finished Connectography by Parag Khanna. Well worth a good read on several levels. See the especially interesting insights about city-states and SEZs. Had me at the maps. Even so, I’d suggest taking this text with a sizable grain of salt. Perhaps a boulder, caveat lector. The basic premise obtains: how supply chain economics have usurped national borders and trade restrictions. To wit:

Infrastructure, markets, technologies, and supply chains are not only logistically uniting the world, but propelling us toward a more fair and sustainable future.

brilliant maps in Connectography

OTOH its trappings as a book-length hagiography of corporatist values may required antiemetics. Read more in my Goodreads review:

The analysis of geography and history is thought-provoking, albeit some flawed historical citations. Clearly he loves major cities. Plus, there’s a bunch of “US bad, China good” on the surface. Quite a bunch. Dig deeper, it’s worthwhile.

Not to change the subject, but for a truly fun read: The Peripheral by William Gibson. One of the best novels I’ve read in a long while – had much catching up to do on books by the Great Dismal. Read more in my Goodreads review:

Chapters alternate POV between a young female protagonist of near-future and our male anti-hero of farther-future. Extra points for picking up the narrative at the start of some chapters in the “wrong” tense, rear-view mirror syntactic suspense for major plot beats. Like a video rewind after an unexpected concussion.

If you’d like to compare books, reviews, etc., please friend me on Goodreads.

Orchard Brew

A few years ago I was enjoying a seasonal, organic, farm-to-table meal with a close friend at Farmhouse Evanston. Ordered a cyser to drink, notably a Puff the Magic Cyser by Vander Mill – for something local to the Midwest. One sip took me on an adventure that’s kept unfolding in marvelous ways ever since.

ICYMI, ciders are making a huge comeback. Cider used to be the most popular alcoholic beverage in the US, prior to the insanity and cultural warfare of Prohibition. Especially in Sonoma West, where we live on a property that’s partly apple orchard, where we work on a campus built amidst an apple orchard. One fun aspect is that the ciders tend to be hyperlocal. In other words, one can travel the globe and encounter novel brews which stay close to their orchards of origin.

ngram occurrence for “cider” in Google Books, 1650–2008 ce

Here in Sonoma, ciders have little to do with the nasty, too sweet, apple-juice-with-vodka mush flavors that so many folks associate with the phrase “hard cider”. Cider makers here use champagne yeasts and other techniques to produce dry brews with complex flavors. Along with that, federal legislation in the US just revamped arcane tax laws that made The new CIDER Act allows small producers to get in the game with artisan brews, without getting stomped by weird post-Prohibition wine laws. This renaissance is good for the land, good for family farms, and great for drinking IMO.

Cider Bar @ Foo

So we held a taste test at Foo Camp this year. Brought in some of the more popular and interesting ciders from orchards near Sebastopol. Almost everyone (except DJ) favored Cider with Hops & Honey: a personal favorite from our neighbors Horse & Plow run by Chris Condos and Suzanne Hagins. That’s both good and somewhat dangerous, since Chris and Suzanne just opened their new “tasting barn” across the road from O’Reilly Media.

Also … LONDON! One of the best places in the world for cider. Fine folks at Orchard Pig recommended visiting The Williams Ale & Cider House, which was outstanding. Will be hosting drinkups there whenever I’m in London. Another couple great venues for sampling UK ciders are Euston Cider Tap and The Green Man. Favorite ciders in the UK? Reveller by Orchard Pig brings the fine flavors of Somerset, available far and wide. Top delight of this trip was Scotland’s Thistly Cross: off-dry, crisp, so smooth, the closest thing I’ve experienced to a Devoto Cidre Noir outside of Northern California :) Another treasure to seek out: Wild Summer by Kentish Pip:

Hints of Elderflower alight sea foam spray in a Nordic summer. Delicate, lightly astringent aftertaste.

As we tour the world with O’Reilly conferences, I’ll host drinkups at nearby cider bars. Here’s a toast to meeting you and sharing a brew or two! Meanwhile, if you’d like to share reviews of ciders, beers, etc., please friend me on Untappd.

Kane’ohe Bay Sandbar: grateful to visit with family in Hawaii recently

Ag+Data

Loved this informed, scathing rebuttal by the brilliant Wendell Berry, Wes Jackson, et al. Money quote:

What is remarkable, but unsurprising, is that these two gentlemen managed to write an entire column about agriculture without mentioning farmland or farmers.

That, in response to a NYTimes op-ed piece earlier this year entitled We Need a New Green Revolution. Note the key word entitled in that sentence: beware just about anyone who lauds a “Green Revolution” without recognizing the terrible consequences that resulted, i.e., financialization being a polite term. In other words, vast claims of battling starvation, while the most wealthy leveraged this vector to institutionalize mass starvation as their business model. There’s far too much “science” bandied about w.r.t. agriculture, pumped into media, heavy on funding ties yet light on scientific falsification.

I believe that politicians and other corporatists who side with Big Ag are behaving immorally. No other word for it.

Read more in my Radar article: Ag+Data

OTOH, absolutely LOVE it when intellect, integrity, and endurance overcome oh so many tedious decades of thoughtless aristotelianism. Check out How a Guy From a Montana Trailer Park Overturned 150 Years of Biology:

“Toby took huge risks for many years,” says McCutcheon. “And he changed the field.”

VRTX

Apropos of the above, I’ve been testing chapter ideas and other material for a new Liber 118. In particular, I’m a huge fan of Universal Basic Income, which has intriguing historical precedents plus enormous implications for the game that comes next.

After running the numbers for what UBI might cost in the US, implications became all the more urgent. It’s within striking range of commercial alternatives, i.e., corporatist funding in lieu of government initiatives. That got me thinking … about parody and science fiction to caricature what UBI might look like if everything went horribly wrong? What if corporatist values seized the day? Key portions of that parody got trumped, so to speak, by an even more bizarre happenstance in alleged reality.

¿quién es más macho? alleged reality vs. parody

Read more in my Medium article: Guaranteed Basic Income (cynical remix)

I’ll just leave you with …

Flying cars. It’s happening. When friends at Bloomberg first gave a heads up about this two years ago, the whole spiel sounded unreal. Now 2 out of 3 of the promises foretold in Planet Unicorn have come true. The big picture is that people travel into work in metro areas via line of sight. No more traffic congestion. Or something.

I guess that’s great for people who work in large cities. Not everybody. But there are likely some other intriguing uses.

Meanwhile, in the lesser hells of drone wizardry, one regrets that the noun #WTFery does not enjoy more regular usage in the English language.

---

Thanks for your attention, and I wish you all the best in your endeavors. Sign-up for the newsletter version at liber118.com

Newsletter Updates for July 2014

Two aspects about leveraging machine learning are largely under-represented in the lit, especially when it comes to production use cases: feature engineering and the comparative evaluation of multiple modeling approaches. To that point, check out “Streamlining feature engineering: Researchers and startups are building tools that enable feature discovery” by Ben Lorica. The article mentions Spark Beyond, which “finds deep patterns in your data.” I was lucky to get a demo of Spark Beyond earlier this year and talk with the principals – and highly recommend taking a good look at their wares. Between the ongoing advances in deep learning and symbolic regression, a direction seems to be emerging … that perhaps one of the more difficult parts of machine learning workflows, namely the feature engineering aspects, could become more automated.

For another great article, check out Including Men in the Conversation About Women by Scarlett Sieber. Among my biggest peeves about Silicon Valley are the “brogrammer” lopsided demographics, and the gender bias which is quite real and nearly epidemic. Our data science teams have generally been quite mixed, why can’t engineering teams in general leave the 19th century behind, let alone stop being so hostile? Not naming names, but two of the SV firms in which I’ve worked in the past five years are both well known and well poised for harassment lawsuits. Taking a stand against that nonsense as an engineering manager is a great way to catch hell, which I’ve gladly engaged before. Another related pet peeve is where one of the same firms was actively pressuring their engineering interns to quit university degree programs. As a behavior for an engineering manager, I find that highly unethical. Some of those who are engaged in these practices know quite well who I’m talking about.

Spark Summit

The big, BIG news last month was … (wait for it) … Spark Summit. All of the speaker videos have been posted – those are probably the single-best resource for learning about Apache Spark. Of course, the big surprise at the conf was the announcement of Databricks Cloud. If you missed the conf, you can watch Ali Ghodsi’s spectacular demo which kicks in at about the 14:40 time marker.

Spark Summit keynote practice, T-15 hours

One surprise learning from the conf was that one product line from SAP generates more annual revenue than all of the other Big Data vendors (HW, Cloudera, etc.) combined. Other pleasant surprises included: Flambo, a Clojure DSL for Spark; and Thunder, for large-scale neural data analysis, which shows some excellent integration of PySpark, SciPy, scikit-learn, etc.

Our training sessions at Spark Summit set some kind of new records. In particular, check out the advanced material for great lectures there. Those who attended the conf received a free ebook preview for the upcoming Learning Spark: Lightning-Fast Big Data Analytics by Holden Karau, Andy Konwinski, Patrick Wendell, Matei Zaharia; O’Reilly Media (2014).

Also, I got to host the Research track of session talks at Spark Summit, which was a real treat. We had a special #geo break-out session following the Geotrellis talk by Rob Emanuele. We will hopefully be expanding that focus in future confs. There were so many other great talks that it’s hard to pick favorites. Even so, I’ll be studying up about two in particular: Quadratic Programing Solver for Non-negative Matrix Factorization with Spark by Debasish Das, Santanu Das; and Distributed Reinforcement Learning for Electricity Market Bidding with Spark by Vijay Srinivas Agneeswaran, Vishnuteja Nanduri. The latter seems almost ideal for integration with recent work on genetic programming.

Stay tuned for the next Spark Summit, which will be held on NYC in early 2015.

OSCON

I’ve just returned from OSCON 2014. What an excellent conference! Check out the content recently posted online: keynotes, photos, speaker slides.

Of course, this event was carefully timed to overlap with the Oregon Brewer’s Festival. Top two picks: Double Latte. by Sierra Nevada Brewing Co.; and Lorenzini Blood Orange Double IPA by Maui Brewing Company. Many thanks to Erin Rasmussen for suggesting about OBF!

Blood Orange IPA

Back at OSCON… one of my favorite Ignite talks was What Science Fiction Can Teach Us About Building Communities by Dawn Foster. Another favorite, speaking of #geo, was a preso/proposal for Open Aerial Map by Kate Chapman.

During the conf, Andy Orem did a video interview where we discussed perspectives and current projects: Ag+Data, Industrial Internet, sketch algorithms, Apache Spark, etc. Andy was the very first editor I worked with at O’Reilly Media, ten years ago. He’s a much better interviewer than I am an interviewee, so I enjoyed learning much through our work together. Also fun to work again with the amazing video team.

"With great power comes some data, plus wrinkled shirts"

The tutorial for Just Enough Math had 50+ people attending, and we got to evaluate an intermediate stage of a new tutorial software platform. For that, I needed to get a bunch of USB drives from Amazon, but the order/delivery #failed. At the last minute our 10 y.o. daughter and I made an emergency run to Fry’s Electronics (she was eager to observe ground zero for nerdliness) … but the only 4Gb flash drives that they had left in stock were Marvel Universe comix characters. Arriving back home, our 9 y.o. daughter was aghast that adults would be receiving comix figures in a lecture :)

The Data Workflows for Machine Learning talk received lots of great responses – as did earlier versions during meetups in Seattle and SF. It become of the “top-shared” slide decks featured on the SlideShare home page. Perhaps that needs to be turned into a mini-book?

new book kiosk

As my last-o’-the-day book signing was winding down, after almost everyone had left the convention center for “nearby locations of beer taps”, a friend mentioned “Hey, look there’s another pile of books – these look different.” So a few lucky latecomers got signed copies of the galley drafts for our new book Just Enough Math, which probably still won’t be released for months – this rev is quite rough :) Oddly enough, the first person to read it looked up and said, “Where are the other O’Reilly books about math?” Indeed.

Sketchy Things

Speaking of Just Enough Math, we’ve put up a companion site for the video+book+tutorial at http://justenoughmath.com/ to provide additional resources and related links:

• set up a Python programming envon your laptop
• code+data files for examples in the video+book
• “gists” that show expected results for the examples
• links to external resources that get referenced
• recommended books and videos for further study
• monthly newsletter sign-up

The tutorial at OSCON previewed a new chapter recently added about sketch algorithms, following from notes at an excellent Foo Camp session led by Avi Bryant. I will be focussing on Spark Streaming use cases for Strata EU in Barcelona this fall, particularly where approximation techniques (think: examples of monoids in action) can leverage both Spark and Cassandra. If you have examples to share of Spark Streaming production use cases in general, I’m eager to build case studies to publish in Radar. Meanwhile, for a great resource about sketch algorithms, check out the archives of the AK Data Science Summit – Streaming and Sketching from last summer.

Card-Carrying Green

A friend recently brought up the topic of navigating questions about extinction and climate change for preschoolers… I’m getting those too; however, in my experience the questions become much better formulated after an additional 5–6 years or so. As a parent, as a human, it kills me to see all the ginormous FUD spewing from the political lobbies for the coal industry, fracking, Monsanto, GM, etc. How about giving ample air time and consideration for some points from the other side?

First off, I’ve mentioned it before but it bears repeating: The Land Institute is a phenomenally excellent resource for understanding some of the insanity and pure tragedy of contemporary agricultural practices, particularly when it comes to monocultures, annuals, hybrids, let alone unnecessary tillage. To paraphrase Wes Jackson, “The plow share has destroyed more options for future generations than the sword.” On a related note, I’ll also point to an excellent article by Michael Pollan, as a forward to Grass, Soil, Hope: A Journey through Carbon Country by Courtney White. Moreover, check out The Solutions Project. That latter site has more substance than perhaps its web-design polish indicates: it’s about the work by Mark Jacobson, et al., on how to power the planet via renewables now while mitigating hurricane damages, etc. One would think that the reinsurance revenues alone would justify a significant investment. In any case, these three links point to the fact that any emerging “dialog of despair” about global warming, etc., is purely FUD. Much can and will be done.

Phylo, the trading card game

I’m particularly grateful to be associated with O’Reilly Media, which provided OSCON attendees with a nice treat in their schwag bags: Phylo, a trading card game. Its gameplay emphasizes endangered species, climate change, food chains, and other environmental pressures. “Phylo is a project that began as a reaction to the following nugget of information: Kids know more about Pokemon creatures than they do about real creatures. We think there’s something wrong with that. Apparently, so do many others.”

In a related development, check out Nerds Without Borders: “We are looking for all sorts of people to help: Engineers, Scientists, Writers, Artists, Dreamers, Activists, Organizers, Fundraisers, Financiers, etc…” Starting with use of IoT sensors and cell phone networks to protect sea turtle hatchlings. Good stuff.

Looking Ahead

Another fun follow-up from Foo Camp and OSCON: getting to talk with Scott Jenson about his work on The Physical Web at Google. Check out his preso, Why Mobile Apps Must Die. The big idea is a kind of “micro-DNS” for low-cost digital tagging of physical items that can be accessed by mobile devices. No app installs required.

In other news, Trafodion was recently released as open source by HP. The name is based on the Welsh word for “transaction”. If you recall about Tandem Computers and NonStop, this product line has a long history of tech innovations – for highly reliable, highly optimized real-time SQL at scale. My uncle retired from Tandem, and lately I’ve spent time with the Trafodion team and am quite impressed. This release brings an interesting new level of Enterprise robustness to real-time transactions+analysis atop Linux+Hadoop. One to watch.

Another to watch closely is The Distributed Developer Stack Field Guide by Andrew Odewahn, Courtney Nash, Mike Loukides, et al. This is a GitHub-based book from O’Reilly. If you see any points in there that need editing, embellishing, etc., then two words: pull request, for the win.

In terms of upcoming events, registration is now open for Data Day Texas 2015, and I’m really looking forward to that. Will be teaching Spark at Scala by the Bay in SF on Aug 8–9, speaking at #MesosCon in Chicago on Aug 21, followed by another Spark course in Chicago on Aug 25.

Flashbacks

I’ll close with a look back to a 1990 Documentary about Cyberpunk. That provides a good summary of what we up to in the early 1990s with Mondo 2000, bOING-bOING, FringeWare, WiReD, The WELL, Turkey City, etc. Tim’s monologue around 15:30-ff is hilarious – both because of his ever-optimistic “There will be mass democracy in the streets” miss, and how much it contrasts with just about every other major point coming true within 25 years. Warning: gratuitous F242 clips, throughout. Time marker 27:11 shows what I was doing as a vendor at many, many raves… Meanwhile, check out a recent bOING-bOING article Alien Autopsy: William Barker on Schwa, two decades later for some of the more astute counterpoint about what was really going on, then and now.

That's the update for now. See you in Chicago with San Diego on the event horizon!

hadoop summit 2012: emergence of the confidence economy

moore intro

Geoffrey Moore opened his keynote at Hadoop Summit 2012 and promptly dropped the line: “You will remember this moment years from now.”

http://www.slideshare.net/Hadoop_Summit/geoffrey-moore-keynote-hadoop-summit

After a disappointing set of “sales pitch” keynotes on the first day of the conference (thanks Yahoo! — but you knew that already) many people attending seemed to roll their eyes about yet another keynote talk this morning. Surprise!

I was grateful to hear Geoffrey Moore trash Advertising as an industry at risk. If I may paraphrase: permanently caught between bleeding edge and dinosaurs, yet irreparably dependent on a broken business model. [FWIW, the last three VCs on whom I’ve used that line looked back at me like I was some kind of alien slime-mold.]

By the middle of his talk, Moore put up a slide with a half-dozen bullet points. The slide listed some of the most disruptive technologies on which businesses — Main Street, in his terms — would come to rely in the early 21st century. Those include: collab filters, behavioral targeting, predictive analytics, fraud detection, time series, etc., etc. Outside of the intelligence community and the hedge funds, the significance of these technologies is not well understood yet. Word. Up. Bitches.

Moore’s “Final Thoughts” slide really hit home. He talked about data access patterns (system of record vs. log file usage vs. real-time analytics vs. etc.) and how those access patterns create feedback loops within an organization. Moore claimed this was core DNA for Google, Amazon, etc., which all major businesses must now embrace. Or else. [That's about 95% overlap with a slide I made for (insert recent past employer) during a 2011Q1 pivot. Two pivots later, I left without any particular next gig in mind — clearly needing to get involved with a different business team. Shortly before their CEO got, um, an "opportunity" to find work elsewhere. But I digress.]

an exercise

So here’s a fun exercise for the interested reader: Pull up a 10-year chart for the S&P 500. Add to that CBS. Right.. Add to that Barnes & Noble. Bokay.. Add to that Wal-Mart. Got few bumps, some upturns.. Nothing to write home about.

Now add Google. Now add Amazon. Now add Apple. One might argue that I’m cherry-picking examples; however, one must understand those three in particular to grasp the trajectory of how Data modifies Companies.

Think about it. Imagine rolling the clock back about 13 years, just a few years before that huge financial sea change got going. Think about perceptions at the time of Apple, Amazon, Google. Most of the mainstream buzz that I heard or read in 1999 was largely disparaging about those three. They didn’t make sense to the average joe, and that was a problem. I will contend that what made sense to a handful of computer science grad students, but not to the average joe, was considered a problem for Main Street. A multi-billion dollar existential problem for some, as it turned out.

At the time, it seemed like Apple would never get past the overwhelming popularity of Dell and Microsoft. Amazon didn’t have a way to justify its enormous P/E ratio — and was probably fluff in the long run. Google was considered interesting, but a little strange, with no clear path toward revenue.

Now think about what happened to the music industry, the mobile industry, the … well, I could go on, but Apple disrupted the pants off lots of established players. Entire industries were taken down by one company. Then consider what happened to retail. One word, a verb according to Geoffrey Moore: Amazon. Think about what happened to advertising. Googled, and not in a nice way either. Amazon and Google took off in 1997Q4 and 1998Q1 respectively, with Big Data projects which became enormous cash cows: Amazon’s recommender system (plus cloud infrastructure), and Google’s search+ads (plus cloud infrastructure). Arguably, those two are the reasons we were having a "Hadoop" conference. Apple perhaps seems less in category; however Apple leveraged mountains of consumer data (plus cloud infrastructure) to drive its smartphones, App Store, etc.

Imagine what kinds of conversations which must have been occurring in the board rooms of CBS, Motorola, Barnes & Noble, Wal-Mart, etc., etc. Gone, gone, gone. Three relative underdogs became giants, tipping almost everyone else’s apple carts. (pun intended) At least three firms understood the power of leveraging their data, they understood the urgency of real-time analytics, etc. Their competitors, mostly, did not. Just look at those stock charts.

According to Moore, that was the tip of the iceberg. Most of the Global 1000 is now on notice. Over the next decade we’ll see monumental failures. Winners and losers, as always, but the magnitude of the losers may be unexpected.

central point

Moore’s central point in the keynote — since this was a Hadoop conference — was that the Hadoop tech stack and business ecosystem is maybe a year ahead of the proverbial “crossing the chasm” moment. Ergo his lead line.

Notably, enormous cultural changes of the 1990s and early 2000s have percolated through personal expectations among those coming up in the ranks. That’s happened more notably and with more impact outside the US than within it. He pointed to the “digitization of culture”, where access has become nearly universal, where broadband created emotional dimensions (Facebook, Pinterest, etc.), where mobile makes the experience ubiquitous regardless of socio-economic position.

Meanwhile, the corporate culture of how to “get stuff done” within enterprise has not kept up. There’s no Facebook for enterprise, no YouTube for enterprise, etc. [Well, actually, there are — and they are each headquartered within a bike ride of my home near the Mountain View / Palo Alto border — but you haven’t heard about them. Yet.]

Meanwhile, Facebook-esque consumer Internet companies of the world are too caught up in their own weirdly distorted realities to solve the larger business problems. Business problems where the solutions will inevitably derive from the social networks’ innovations. Oops.

In Moore’s vaulted opinion, those conditions won’t hold much longer. There will be winners. There will be losers. Big ones.

Meanwhile, for people of my ilk, Moore smiled and predicted: “This should provide at least a decade of entertainment for everyone present.” Fundamental business reasons are simple: enormous change ahead but precious few who are trained and experienced to navigate it.

key take-aways

My first key take-away is based on the observation last year that Enterprise giants bumbled into Hadoop Summit 2011 in a huge and awkward way. Oddly, the logo is an elephant, #justsayin

In contrast, this year was really smooth, completely professional, far too expensive … but almost all about data infrastructure in a world where nobody want to utter the word “Oracle”.

Mind you that neither of the two main “enterprise” keynote speakers from last year still have their same jobs. #justsayin

Also, notable Hadoop practitioners were noticeably absent. In fact, most of the cast and crew of Strata seemed to be missing. A particularly popular social network has been burning the midnight oil to make Hadoop perform backflips — they like gave a couple talks and seemed to vanish.

Let me put this in other words: several hundred million dollars have been invested by VCs (and angels) to recreate an industry in the image of Redhat and Yahoo!

Wow, did anybody think that would be a particularly good idea? No, but it’s the herd mentality in practice. Even after the 5th beer I’d still recognize that strategy as not particularly wise. Feels like when you talk with an ex-convict, and they drop a line “Yeah, I made some poor choices long ago…”

My hunch is those data infrastructure plays are mostly tax write-offs (for the “early adopter” part of Geoffrey’s famous curve) at this point.

Moore underscored how real payouts come when key verticals catch fire — with serious domain expertise leveraged. LinkedIn perhaps got close, but now it almost feels like a spamming broadcast system for HR and BD departments. We’ll see “Big Data” killer apps which mean something to lots of people. Beyond the GOOG+AMZN+AAPL tip o’ the iceberg. They will come from people who have sophisticated backgrounds in Stats + ORSA + distributed systems + functional programming + DevOps, people who can also communicate well with actual business leaders. Not those employed by some halfwit B-school grad who’s posturing as the next Steve Jobs, when in reality he drinks bad beer at a lame, faux-hipster sports bar while watching cable televison. Or something. Dude, hop on your fixed-gear bike and standstill/peddle your sleeve tattoos out of here.

Translated: the proverbial ignite moment, that spark of innovation, is not going to come from the likes of a Cloudera or a Platfora or a (banal noun)-(o|e)ra… But it’s going to come, probably not many moons away. It will be in apps.

the sound of disruption

Thirty years ago, I went into a field called “math science”, i.e. how to build predictive analytics as software apps. Stanford — the Statistics department chairman, Bradley Efron, in particular — had put together an interdisciplinary degree which combined math, statistics, operations research, programming, engineering, etc. At the time, most of my peers in the program went on to become insurance actuaries. I went instead to do graduate work in machine learning and distributed systems.

For nearly two decades, most employers could care less about any quantitative background. They wanted C++ software engineers working all day on APIs from Sun or Microsoft or Oracle. Or they wanted managers. Then, in about 2000, came the sea change.

Right about the same time as ticker symbols for Apple and Google and Amazon were strolling up to their respective launchpads, some people began to look at my resume and ask a different line of questions.

I’ll always remember the first: a microchip vendor — one which makes electronics for several products you’ve purchased — was getting squeezed by Intel and their silicon compiler vendor. Critical features were being deprecated, specifically to put this second-tier player out of business. The company was on notice. They had to find a proverbial needle in a haystack: out of tens of thousands of circuit designs, they had to identify the 1% which would no longer be licensed — then redesign those. Quickly.

An internal team at the company had tried, but given up. Too much data for their techniques, it would’ve taken years to resolve. The company hired an electronics consulting firm in Austin, and engineers went to work, but gave up as well. Too much data, not enough signal. I got called in, as a “Whatever, just see if you can get anywhere” last-ditch effort. About 20 lines of Perl and one relatively simple equation later, I dumped my results into a scatterplot.

One of the lead circuit designers picked up my plot off the laser printer and began laughing. Loudly. The whole office heard him.

His manager grew annoyed: “What?! Why are you laughing?!”

Engineer: “He found it.”

When I turned in my invoice, the manager glared. “Look,” he said in a growl, “Just go somewhere for about three weeks. Bill us the whole time. Then come back and turn that in.”

My brows furrowed, this was a high-dollar rate for 2000.

“If you don’t pad that damn invoice…” he paused, “You’ll make both us and our customer look like complete fools. Piss a lot of people off.”

That’s the sound of Disruption.

More than a decade later, the summary graf of my resume reads like bullet points from Geoffrey Moore’s slide. Collab filters, anti-fraud classifiers, predictive analytics, etc. Even in the past few years, when HR people have read that resume, several looked up with a frown, said they thought that kind of work was better suited for business analysts — yadda, yadda, yadda, keep following the herd: you put the “botch” in “beotch”.

At a time when lots of business (start-ups as well as enterprise) are starving because they cannot hire Data Scientists, I’ve been busy building teams. Teams which delivered $MM results. I’ve hired about thirty people onto Data Science teams within the past few years — at a time when many start-ups would feel lucky to hire one. #justsayin

mal*wart

I read one of the most imbecilic essays recently from Forbes/Quora: “What Would Be The Global Impact If Wal-Mart Abruptly Shut Down?” Essentially, a hagiography stating that Wal-Mart is too big to fail, that the consequences on the US economy, the global economy, would be catastrophic. Translated: may require an enormous bailout, soon.

[In case you hadn’t guessed, I just threw up a little bit in my mouth.]

What. A. Fucking. Moron. The reality is that Wal-Mart hasn’t been doing so well over the past decade. Not if you peel back enough layers of PR. Not since they tried to bamboozle the LA city council. And failed. Moreover, folks at Amazon could really care less which Senators or SecState/former-first-lady the execs in Arkansas have in pocket. Bezos has positioned to take over 150% of Wal-Mart’s business the picosecond after Bentonville implodes. Sears and Target have reinvented themselves specifically for that very instant. So long, good riddance. Remember the point about the Global 1000 on notice? About the importance of business fundamentals?

sears, a.k.a. that web site which kinda looks like amazon

A third keynote talk that day was by the Sears CTO, Phillip Shelley. I had packed up my laptop and backpack, and was getting ready to walk out of the auditorium. After his first few sentences, I put my stuff back down and started taking notes.

Dr. Shelley mentioned how Sears started as a mail-order business a century ago, though more recently got completely kicked by another “catalog” called Amazon. Now they are leveraging Hadoop + R + Linux/Xen private cloud (srsly, this is from the Sears CTO?!?) to reinvent their business with 100x more detail on regional pricing models. Literally calculating personal pricing discounts for individuals, multiple times per day, specifically for mobile.

Sears: core algorithm moved from [6000 lines of COBOL on mainframe with 3.5 hr batch window] to [50 lines of Hadoop app on Linux with 8 min batch window], while reducing TCO for enterprise IT by two orders of magnitude. So much success, that they’ve spun it out as a new business line called MetaScale.

Brilliant strategery by Sears. Some seriously high powered Data Science talent walked out of that auditorium musing how they wished their VP Engineering was half as progressive as Sears. Srsly? Um, that’s called a PR coup. [Literally at the same moment as Wal*Mart had HR droids spamming the audience with whispers and rumors of lucrative salaries. Gak.]

emergence of the confidence economy

What’s the deal? It’s about confidence. Those giants in the Global 1000 which Geoffrey Moore says are on notice? They got that way by believing that business is largely about who barks the loudest, barks the longest, and cuts the most deals under the table. The proverbial alpha male in a wolf pack.

Wal-Mart would be a prime example, in my opinion. Their business is predicated on fundamentals which simply do not hold. Misplaced confidence. Thanks to people like Hillary Clinton, Wal-Mart has gained much influence on the House and Senate floors and the halls of the State Department and the UN assembly. In other words, so long as we manage to keep fuel costs artificially low, Wal-Mart’s market valuation will keep growing. So long as we believe that bullying vendors, conducting intelligence operations against the rest of your ecosystem, etc. — that these kinds of practices are ethical and sound in the long-run, then Wal-Mart will keep growing. Bullshit. Go look at that stock chart again. Wal-Mart is about tall white guys in dark suits, acting like complete pricks, destroying and plundering anything they can get their grimy paws on. Richard Gere in Pretty Woman, before he gets Julia Roberts. And not much more than that. On notice.

Moore is pointing out, in my opinion, that the issue at hand is about uncertainty. The point of establishing a corporate charter was always to externalize risk and perpetuate wealth for shareholders. That was true four centuries ago, when the first transnational was established, and has been true ever since. The modus of that mechanism is a process called sublation. The train wreck for sublation is uncertainty. In an environment where uncertainty holds sway, having real-time analytics from petabytes of customer data wins out over having a Senator in pocket. Any day of the week. The antidote for uncertainty is confidence. While there had been a regime of an “Attention Economy” extant for the past two decades or so, we’re now entering a new regime of the “Confidence Economy”.

Here’s the deal: people like me like those of us in Moore's lecture have been the “secret sauce” fueling the rise of Amazon, Google, Apple, etc. We use techniques which are mostly not well understood outside of Langley and the hedge funds. The tools of contemporary corporate assassins. Guys in suits who act like pricks in lieu of practicing business fundamentals — those guys are our targets. The modus is Disruption. If you have an MBA or a CxO title and not much else to back it up, I put food on my family’s table by being a sniper paid to hunt you. Lots of *great* food. And some of the best wines available. I shake the tension out of my hands, correct for wind and distance, draw a bead, take a deep breath, squeeze the trigger. Kill shot.

The challenges faced by Data Scientists are daunting. On one hand, most mathematicians lack enough solid engineering to create killer apps. Conversely, most engineers lack enough math to make any headway on the business data. Most business analysts lack enough of either the math or the engineering to be worth hiring. Data Scientists provide all three areas of expertise: the engineering and the math and the business insights to contend with mountainous torrents of data, and move the needle. On the other hand, Data Scientists must also speak truth to power. In any given business, there will be winners and losers. Executives, people accustomed to their own power, taken down. Meanwhile, we Data Scientists come prancing into a business, we do our magic, and consequently we point out which executives are bullshit and must be “executed”.

The reason why I’ve built Data teams at a time when others are starving is simple: confidence. Sure, I’ve logged three decades of machine learning, statistical modeling, data management, distributed computing, etc. When I talk with a grad student about their work, I can tell them in 25 words or less what they need to do on their first day at work to become regarded as an great asset to the team. They already know the techniques, but crave confidence. Into the trenches, fresh-out, having to speak truth to power. They’ll be placed into some faltering business unit, run some detailed analysis, and point out that the VP who’s been arguing loudly was completely wrong for the last N years and his/her ego cost the company several $MM. You can bet that those execs will return fire. However, a person like me is confident that we can get a kill shot. I show new folks how to draw a bead and squeeze the trigger. Been doing it for a long while, and will be doing for a long while more.

Snipers have an eerily pragmatic sense of confidence. And, by the way, that’s a peculiarly difficult job. Praise goes out to the men and women who serve their countries in uniform — when the cause is just. [FWIW, before tackling the challenges of data+science, I wore a military uniform and carried a rifle. Sniper training has become invaluable.]

my opinions

#1: Enterprise suffers because so many people in the corporate leadership ranks (or rather, amongst those clawing and scrambling to make their way into the corporate leadership ranks) consider themselves to be a different caste — if not a different species all together — from the rest of us who do not have a salaried position with a transnational. In a “Let them eat cake” world fraught with trillion-dollar bailouts, that’s not a particularly good way to future-proof. Moreover, this is why VCs are vital… to demolish that kind of hubris via constructive Disruption. #justsayin Word. Up. Bitches.

#2: Enterprise tooling, which is now mostly dependent on JVM-based apps, suffers because it has embraced “Convention over Configuration” … CoC has its place. I can imagine that it’s an excellent idea for heart surgeons to have a standard toolset, with scalpels in the exact same positions, etc. CoC is not a particularly good way to manage complexity and uncertainty, because it simply displaces major problems into the build system. Ultimately, it fails too much and impedes spin-up. Which, I believe, represents an enormous, ticking time bomb in Enterprise. Here’s a challenge: Show me a metric for the median period it takes in your business for a newly hired engineer to push code changes into production use which is adopted by at least 80% of your customer base. Now show me a metric for the media period it takes in your business between the point where a product manager identifies a needed feature and a newly hired engineer is ready for spin-up. From those, I’ll make a prediction based on that metric for how well your business will survive the “on notice” condition which Geoffrey Moore described. Better clues for navigating complexity and uncertainty can be found in the works of Ilya Prigogine or Stephen Wolfram. To wit, functional programming is more likely to address complexity, real complexity, and also more likely to attract top talent. CoC, not so much. Perhaps your enterprise business addresses Main Street instead of Early Adopters... Recall that Google and Amazon and Apple crossed the chasm by recruiting armies from grad students — at a time when most other people erred on the side of average joes. Remember the point about real-time analytics? It counts for training your people, then retraining, and retraining, constantly — to grapple with uncertainty. Kill shot. Global 1000.

#3: MapReduce will be unrecognizable within three years. Hadoop Summit will become something quite different after Hadoop bifurcates and gets sublated into Something Else. For example, it would be not difficult to use the Simple Workflow Service from Amazon AWS to implement MapReduce using the core part of Cascading… a different kind of MapReduce, which is not constrained by JVMs… which could scale much more gracefully and robustly… which could out-perform Google infrastructure and avoid attempting to re-create the industry in the image of Yahoo! At which point, one could deploy functional programming blocks at enterprise scale, without having to rely on the morass of enterprise build tools. Hmmm… may need to get a term sheet for that one.

Geoffrey Moore, we may have a few answers for your questions.