newsletter update 2016-07

Synecdoche

Been absent from newsletter feeds for a year plus. During that time, several POVs changed markedly. Home: relocated from the shadows of the new Google X building …  to a rural part of Russian River Valley. Work: swapped more than an hour commute up to a SF tech start-up … for a 20 min bike ride to O’Reilly Media. Plus a few other priorities. While the newsletter name changes, its themes carry forward. Plus a few other priorities. I’ll cover: conference summaries, open source projects, interesting news items, recommendations … data science, machine learning, advanced math, sustainable ag, art, travel, artisan foods, cider, flying cars, etc. Albeit running in slightly different circles and casting a wider net.

Changing POVs: new “backyard” has less actuators, more pollinators

Confs

Been trying to keep pace with our 20+ tech conferences each year. Lots of travel, lots of interesting people and insights into their projects. Some upcoming:

• JupyterDay Atlanta, ATL, 13 Aug: will present Oriole: a new learning medium based on Jupyter + Docker
• Artificial Intelligence, NYC, 26–27 Sep: plus a drinkup at Wassail
• Next:Economy, SF, 10–11 Oct: plus a drinkup at Upcider
• Big Data Spain, Madrid, 17–18 Nov: will deliver a keynote, hopefully some workshops too!
• Strata+HW Singapore, SG, 5–8 Dec: I’ll present Computable Content: Notebooks, Containers, and Data-Driven Organizational Learning

De Pijp in Amsterdam during Scala Days 2015

Strata EU 2016, 31 May - 03 Jun. Venue moved to the much larger ExCel London space and that afforded an expanse of incredible people, tech innovations, data insights, etc. Also, it proved to be a good base of operations for exploring two recent interests regarding London: connectography and cider. More about both in a moment :)

Strata kicked off with a “Hey, are you busy?” text from the brilliant Alistair Croll, which led to one of the best pub crawls ever. Still parsing our discussions in those London pubs. Those provided the basis for several essays linked here. Seriously. Pay close attention to the intellectual sculpting that emerges from Alistair’s endeavors.

Strata tradition: team #noscurvy w/ BLL Ladies @holdenkarau @j_houg

Anirudh Koul invited me to a session with his Microsoft colleague Saqib Shaikh, where they delivered Beyond guide dogs: How advances in deep learning can empower the blind community. Sure enough, a favorite talk at Strata. Microsoft’s been busy building cognitive services in the cloud which developers can integrate into other software in general. Following retina surgery in both eyes in the 1990s, I experienced serious vision impairment. It was way too much to write code for a while. Switched to working on VR design for a while: shapes, colors, contrasts were much easier to manipulate than source code. Years later my vision still changes abruptly. Grateful now to see (pun intended) these AI innovations by Microsoft and I look forward to many interesting applications.

The proverbial “Full English”

Arrived back to Sonoma County just in time for Foo Camp 2016. Translated: that’s an unconference while camping with 200 fascinating people from around the world. Held in the apple orchard behind our office. DJ Patil and Ed Felten opened the weekend with a lively group discussion about data ethics, where Cory Doctorow delivered the punchline: when you own a mobile device, who choses to override settings, you, the manufacturer, or the government? Lives may depend on the answer.

One decidedly tangible result of Foo: “Blockchain of Love” by Ellie Volckhausen, Edie Freedman, Ally Miller, Johnny Diggz, Tony Parisi, Brian Fan, and yours truly helped with a line or two. Tony attempted to explain the ineffable:

Geeks, guitars and a whole lotta booze. We wrote this song in 45 minutes at O’Reilly Foo Camp!!!

Got to try a Copenhagen Wheel, courtesy of Assaf Biderman @ Superpedestrian – most definitely a decade beyond the Bionx on my prior ride. A favorite session was an Ignite talk by GalaxyKate, aka Kate Compton. Check out her work on generators and related tooling at Tracery.

the old Make Mobile

Velocity CA 2016, 20–23 Jun … Noticed that sessions related to Apache Mesos and DC/OS were standing-room-only. In other words, considering the recent Mesosphere investment by HPE and Microsoft, Mesos has arrived for the mainstream. A good illustration of this? The Container orchestration wars session by Karl Isenberg.

Having been away from SV for a while … Andrew Marantz’s recent New Yorker article How “Silicon Valley” Nails Silicon Valley struck a nerve or two. That rollerblade “scene” took place right behind our former house. Recommended.

Media Theory

One evening in late 1992, between the first Cypherpunks meeting and the FringeWare launch … I climbed aboard an opaque bus, allegedly headed to a private event at Intel Santa Clara. Probably had “press credentials” from Mondo 2000, or something. Sat down, turned around, met the person sitting in the next row. Brilliant, utterly mellifluous, compelling immediate engagement from all around, someone who soon became a star in the industry and also a lifelong friend: Mark Pesce. That evening was all about wandering into different worlds. For a brief interlude, the word “environment” itself became entirely fungible. Alice, down the rabbit hole. Guided tours of the latest in VR by Eric Gullichsen, et al.

offspring in tribal garb

Some things come full circle in the next generation. Our tween daughters recently chose their first “concert” experience: a first-of-its-kind world tour by celebrity YouTubers Dan and Phil. If you haven’t been tracking what’s changed in media, take a close look.

This time, the virtual from the Interwebs gets projected live and in the flesh. Spoiler alert: millennials may have leapt the Selachimorpha.

Read more in my Medium article: Wandering into an entirely different world

Oriole

Les Guessing posed an interesting question recently:

Surprised how data “Storytelling” mostly talks about visualization, not video or other great storytelling mediums.

Well said! Oriole is our update on that point. This new medium combines a video timeline with hyperlinked rich text, code, data, results, instrumentation, and visualization. The point is about repeatable science, in a sense, applied to the matter of contextualized hands-on learning about technology. Also a great step toward personalized learning, with brilliant dev+design from our Brooklyn Team. We’re eager to see how other authors leverage this.

Oriole

For a much better introduction, try it yourself! Check out the coding challenge in Regex Golf by Peter Norvig. That’s the first Oriole we published, as an exemplar.

Read more in my Radar article: Learning alongside innovators, thought-by-thought, in context

And join us for the first public talk about Oriole, at JupyterDay Atlanta on 13 Aug.

AI

David Beyer recently published an excellent series of interviews about the Future of Machine Intelligence. Favorites include:

• Ben Recht on what ML can learn from many decades of control theory, in particular due to similarities with classic dynamical systems – two distinct ways to consider models
• Anima Anandkumar on the use of tensor decomposition and multilinear algebras for non-convex optimization – to represent higher order relationships among latent variables
• Gurjeet Singh on the use of topology: “Data has shape and shape has meaning”, i.e., mapping from one space to another

One thing has been really bothering me about recent media coverage of AI. There’s a disturbing tendency to equate ML with AI, at the expense of considering how control systems are inherent in almost any business application. For example, Uber may use lots of machine learning; however, at the heart its business a control system manages drivers, customers, offers, and other vital aspects of supply chain and contingency. Note that the field of AI emerged from control theory and early cybernetics.

Read more in my Medium article: Beyond the AI Winter

A recent conversation on video by Tim O’Reilly and Peter Norvig explores applications of AI technologies, conversational interfaces, etc. That dovetails with how Google pointed DeepMind at data center power usage, for a dramatic 40% decrease in energy needs. That prefigures many industrial uses for AI, given how “the algorithm is a general-purpose framework to understand complex dynamics” in manufacturing, transportation, energy, etc. Note that Tim and Peter will be honorary program chairs at the new/aforementioned Artificial Intelligence conference in NYC this September.

Meanwhile, other arcanum about the Greater NYC Area … note the curiously shaped post horn which marks the façade of our Brooklyn office:

O’Reilly Brooklyn office, “front door”

The Trystero are out and about, and the sign is not stable.

Neurons and Art

More about applied AI, perhaps for dessert … What have artificial neurons been doing in the art world lately? Apparently they’ve evolved far beyond assembling art boxes that recall the work of Joseph Cornell. Check out the aftermath of the DeepDream: the art of neural networks show in SF this past February by Gray Area.

For an even more intense dose of narrative closure … Terence Broad applied his dissertation, Autoencoding Video Frames to have deep learning deconstruct the film Blade Runner. Using unsupervised learning in a way that Hollywood lawyers might not comprehend for quite a while, this work prompted Warner Brothers to issue a DMCA takedown notice … Except that (surprise!) the work did not include any Blade Runner footage.

For an epic recounting of neurons and art, check out Neural Networks and the Digital Hallucination by good friends over at Lumo Interactive. Also, links highly recommended in Beyond the AI Winter:

Check out the inexplicably entertaining short film Sunspring, authored by an AI named Benjamin.

That script writing AI leverages a recurrent neural network approach called long short-term memory (LSTM ) which takes abductive reasoning to new levels.

ebumna lw’nafh

What AIs aren’t quite dreaming about (yet) are seemingly less probable matters in generated imagery: crystals in a 4.57 billion-year-old meteorite found in Chukotka, Russia showed the first naturally occurring Forbidden Symmetry. YMMV.

Reading

Recently finished Connectography by Parag Khanna. Well worth a good read on several levels. See the especially interesting insights about city-states and SEZs. Had me at the maps. Even so, I’d suggest taking this text with a sizable grain of salt. Perhaps a boulder, caveat lector. The basic premise obtains: how supply chain economics have usurped national borders and trade restrictions. To wit:

Infrastructure, markets, technologies, and supply chains are not only logistically uniting the world, but propelling us toward a more fair and sustainable future.

brilliant maps in Connectography

OTOH its trappings as a book-length hagiography of corporatist values may required antiemetics. Read more in my Goodreads review:

The analysis of geography and history is thought-provoking, albeit some flawed historical citations. Clearly he loves major cities. Plus, there’s a bunch of “US bad, China good” on the surface. Quite a bunch. Dig deeper, it’s worthwhile.

Not to change the subject, but for a truly fun read: The Peripheral by William Gibson. One of the best novels I’ve read in a long while – had much catching up to do on books by the Great Dismal. Read more in my Goodreads review:

Chapters alternate POV between a young female protagonist of near-future and our male anti-hero of farther-future. Extra points for picking up the narrative at the start of some chapters in the “wrong” tense, rear-view mirror syntactic suspense for major plot beats. Like a video rewind after an unexpected concussion.

If you’d like to compare books, reviews, etc., please friend me on Goodreads.

Orchard Brew

A few years ago I was enjoying a seasonal, organic, farm-to-table meal with a close friend at Farmhouse Evanston. Ordered a cyser to drink, notably a Puff the Magic Cyser by Vander Mill – for something local to the Midwest. One sip took me on an adventure that’s kept unfolding in marvelous ways ever since.

ICYMI, ciders are making a huge comeback. Cider used to be the most popular alcoholic beverage in the US, prior to the insanity and cultural warfare of Prohibition. Especially in Sonoma West, where we live on a property that’s partly apple orchard, where we work on a campus built amidst an apple orchard. One fun aspect is that the ciders tend to be hyperlocal. In other words, one can travel the globe and encounter novel brews which stay close to their orchards of origin.

ngram occurrence for “cider” in Google Books, 1650–2008 ce

Here in Sonoma, ciders have little to do with the nasty, too sweet, apple-juice-with-vodka mush flavors that so many folks associate with the phrase “hard cider”. Cider makers here use champagne yeasts and other techniques to produce dry brews with complex flavors. Along with that, federal legislation in the US just revamped arcane tax laws that made The new CIDER Act allows small producers to get in the game with artisan brews, without getting stomped by weird post-Prohibition wine laws. This renaissance is good for the land, good for family farms, and great for drinking IMO.

Cider Bar @ Foo

So we held a taste test at Foo Camp this year. Brought in some of the more popular and interesting ciders from orchards near Sebastopol. Almost everyone (except DJ) favored Cider with Hops & Honey: a personal favorite from our neighbors Horse & Plow run by Chris Condos and Suzanne Hagins. That’s both good and somewhat dangerous, since Chris and Suzanne just opened their new “tasting barn” across the road from O’Reilly Media.

Also … LONDON! One of the best places in the world for cider. Fine folks at Orchard Pig recommended visiting The Williams Ale & Cider House, which was outstanding. Will be hosting drinkups there whenever I’m in London. Another couple great venues for sampling UK ciders are Euston Cider Tap and The Green Man. Favorite ciders in the UK? Reveller by Orchard Pig brings the fine flavors of Somerset, available far and wide. Top delight of this trip was Scotland’s Thistly Cross: off-dry, crisp, so smooth, the closest thing I’ve experienced to a Devoto Cidre Noir outside of Northern California :) Another treasure to seek out: Wild Summer by Kentish Pip:

Hints of Elderflower alight sea foam spray in a Nordic summer. Delicate, lightly astringent aftertaste.

As we tour the world with O’Reilly conferences, I’ll host drinkups at nearby cider bars. Here’s a toast to meeting you and sharing a brew or two! Meanwhile, if you’d like to share reviews of ciders, beers, etc., please friend me on Untappd.

Kane’ohe Bay Sandbar: grateful to visit with family in Hawaii recently

Ag+Data

Loved this informed, scathing rebuttal by the brilliant Wendell Berry, Wes Jackson, et al. Money quote:

What is remarkable, but unsurprising, is that these two gentlemen managed to write an entire column about agriculture without mentioning farmland or farmers.

That, in response to a NYTimes op-ed piece earlier this year entitled We Need a New Green Revolution. Note the key word entitled in that sentence: beware just about anyone who lauds a “Green Revolution” without recognizing the terrible consequences that resulted, i.e., financialization being a polite term. In other words, vast claims of battling starvation, while the most wealthy leveraged this vector to institutionalize mass starvation as their business model. There’s far too much “science” bandied about w.r.t. agriculture, pumped into media, heavy on funding ties yet light on scientific falsification.

I believe that politicians and other corporatists who side with Big Ag are behaving immorally. No other word for it.

Read more in my Radar article: Ag+Data

OTOH, absolutely LOVE it when intellect, integrity, and endurance overcome oh so many tedious decades of thoughtless aristotelianism. Check out How a Guy From a Montana Trailer Park Overturned 150 Years of Biology:

“Toby took huge risks for many years,” says McCutcheon. “And he changed the field.”

VRTX

Apropos of the above, I’ve been testing chapter ideas and other material for a new Liber 118. In particular, I’m a huge fan of Universal Basic Income, which has intriguing historical precedents plus enormous implications for the game that comes next.

After running the numbers for what UBI might cost in the US, implications became all the more urgent. It’s within striking range of commercial alternatives, i.e., corporatist funding in lieu of government initiatives. That got me thinking … about parody and science fiction to caricature what UBI might look like if everything went horribly wrong? What if corporatist values seized the day? Key portions of that parody got trumped, so to speak, by an even more bizarre happenstance in alleged reality.

¿quién es más macho? alleged reality vs. parody

Read more in my Medium article: Guaranteed Basic Income (cynical remix)

I’ll just leave you with …

Flying cars. It’s happening. When friends at Bloomberg first gave a heads up about this two years ago, the whole spiel sounded unreal. Now 2 out of 3 of the promises foretold in Planet Unicorn have come true. The big picture is that people travel into work in metro areas via line of sight. No more traffic congestion. Or something.

I guess that’s great for people who work in large cities. Not everybody. But there are likely some other intriguing uses.

Meanwhile, in the lesser hells of drone wizardry, one regrets that the noun #WTFery does not enjoy more regular usage in the English language.

---

Thanks for your attention, and I wish you all the best in your endeavors. Sign-up for the newsletter version at liber118.com

Newsletter Updates for December 2014

Chicago, Boulder, NYC, DC, SF, Stanford, London, Stockholm, Madrid, Barcelona, Amsterdam, Dulles, Baltimore, LA. The range of speaking events and business travel over the past quarter almost bewilders, but I’m grateful to get to meet many interesting people and learn about new projects. 

Also feeling grateful to enjoy some quiet time at home with family over the holidays, and I wish very happy holidays to you and yours.

Conference Summaries

Strata NY set a new record with about 450 people attending Spark Camp. There was a spare room, plus an hour break in the fray, so we held an impromptu “Ask Us Anything” about Spark – that has turned into a new kind of open source ritual at Strata confs, especially for handling the more advanced audience questions. Also, Bloomberg kindly hosted a large Spark Committer Night meetup event, their largest to-date.

Manhattan, from NY Water Taxi at Port Imperial

Throughout many conferences and meetup events over the past few months, one demo in particular stood out. David Jonker and Rob Harper from Oculus Info in Toronto gave a talk about Aperture Tiles at Strata NY. Last talk of the show, and quite arguably the best. This open source framework, partly built atop Spark, provides interactive data exploration with continuous zooming on large scale datasets. Highly recommended.

The week after Strata conf in NYC, some of our team found our way slightly south to the University of Maryland, where we got to teach alongside the renowned Jimmy Lin. The week included a Spark Tutorial on campus, plus the initial meeting of the Apache Spark Maryland meetup. Much fun, and we look forward to returning to UMD again soon.

Arriving back to the Bay Area just in time, I caught the launch of the new GalvanizeU program in downtown SF. One challenge that particular evening was getting scheduled to speak head-to-head with the final game in the World Series. That keynote, Data Science in Future Tense, examined some of the near-past and near-future of the field – hopefully indicating some non-intuitive directions. 

GalvanizeU is located next to the Transbay Center, just a few blocks away from the new Databricks office. They provide a hands-on graduate program in Data Science, in an urban setting and working closely with industry partners. Galvanize started in Boulder and is also expanding soon into Seattle. We’re thrilled about our new neighbors.

Home just long enough to take the kiddos trick-or-treating and attend GCPLive, then on to Europe… During a brief visit in the UK, I got to present about the latest in Spark Streaming at the London Spark Meetup: Tiny Batches in the wine (a callback to Don Ho, for those who were born more recently – ideal for getting your luau on). Then on to Stockholm with gracious hosting by Spotify, Ericsson, and SICS.

Good times @ Big Data Spain, Madrid

Madrid came next, for the annual Big Data Spain conf. Noticing a joke painted on the side of a jet at the airport, I had a hunch immediately that Madrid would be lots of fun. I was not disappointed. Our hosts at Paradigma Tecnólogico and Stratio presented an amazing conference, one of my favorites in a long, long time. I was fortunate to give a keynote talk, alongside many other excellent talks, such as from friends at Cloudera and Google BigQuery. I highly recommend Big Data Spain. More about Stratio in a bit…

The beach at El Poblenou, Barcelona

Taking a train from Madrid to Barcelona, admittedly I was missing the former, but Barcelona is a wonderful place. Imagine yourself in Santa Barbara, except that the city is 50 times larger, thousands of years older, and packed full o’ amazing culture. Strata EU was located at a conference center right next to the beach. We held the first official Spark developer certificate exam, plus a large Spark Camp event (25% of the conference attended), a meetup at UPC, and a second iteration of our “Ask Us Anything” about Spark.

Locavore feasting in Catalunya

Business travel Spark-style does not allow much downtime. Effectively one day off during two full weeks in EU. Fortunately that just happened to be during a weekend in Barcelona, the day after Strata concluded. I rented an Airbnb condo near the beach in El Poblenou, then wandered busy Rambla markets, through the crowd surrounding a busker string trio, gathering items to make a small feast. Only in Catalunya.

Amstel River in Amsterdam

A quick stop in Amsterdam, with a very fun talk hosted at eBay with hours of Q&A, then back home. Long enough for a family Thanksgiving feast, then off to DC, Baltimore, and LA. Excellent events and good friends met along the way, particularly the Los Angeles Apache Spark meetup hosted by Rubicon Project. Much appreciated.

Spark

The curiously named Likelihood T. Prior noted on Twitter: Spark spark spark spark, spark spark spark spark. #Strataconf synopsis complete. Some went as far as to begin calling “Strata + Hadoop World” by a new name, “Strata + Spark World”. I like the sound of that.

To help keep track of this rocket ride, I’ve begun curating an ongoing list http://goo.gl/2YqJZK of the talks, workshops, etc., related to Spark worldwide. Please let me know if you have events to add.

Speaking of events, recently we began to increase the cadence for Bay Area Spark meetup events. These talks get live-streamed, with the archives published on the Apache Spark channel on YouTube. Databricks also recently announced Spark Packages a community index of packages. The site had to be moved shortly after its launch, due to overwhelming popularity. Good stuff on both the video channel and package repo.

So much news about Spark has happened in the past few months. I’d like to summarize with a few gems collected along the way…

• Stratio has distinguished as arguably one of the most sophisticated use cases for Spark Streaming, with a CEP engine layered atop plus substantial other integrations
• Virdata published the highly recommended Tuning Spark Streaming for Throughput, along with a shared blog post about their Spark use cases
• Large use cases for Spark Streaming became public, notably Netflix and Pearson
• DataStax published an excellent tutorial, Interactive Advanced Analytics with DSE and Spark MLlib based on use of the Spark Cassandra Connector
• Elasticsearch provides Spark SQL integration since 2.1 – check their developer guide for a great tutorial
• Michael Noll published a detailed tutorial about Integrating Kafka and Spark Streaming: Code Examples and State of the Game

Not least of these items, the Databricks team broke YHOO’s previous world record for the Daytona GraySort contest. That tied for the 100 PB sort on AWS, using 1/10 the number of servers and running 3x faster than YHOO Hadoop clusters. #justsayin

MOOCs

Part of my job involves the curriculum for Spark instruction. Our big news recently is that edX and the University of California will be offering two new MOOCs about Spark, sponsored by Databricks.

The first is Introduction to Big Data with Apache Spark by Prof. Anthony Joseph at UC Berkeley. This comprehensive introduction to Spark, as well as Big Data, is based entirely on Python programming and aimed at developing Data Science skills. This course begins on 2015–02–23.

The second is Scalable Machine Learning by Prof. Ameet Talwalkar at UCLA. This hands-on course focuses on distributed machine learning at scale, based on examples using open data, also in Python. This course begins on 2015–04–14.

Note that some taking Spark MOOCs will have the option to use Databricks Cloud free student accounts. Similarly, we will be integrating use of DBC free accounts into our other Spark training events.

Workplace

Several years ago, I was fortunate to work for a CEO who understood how to leverage a distributed workplace. I studied the management practices involved, and in particular have grown to appreciate ROWE greatly. These practices seem all too rare among early-stage tech start-ups Silicon Valley. However, a few tech firms (DataStax and Typesafe come to mind) have embraced distributed workplace models. Frankly, correlations between effective approaches to gender equality and practices such as ROWE should be on every VC’s radar.

With respect to workplace practices – effective or otherwise – two recent articles caught my attention:

• Stop Wasting Everyone’s Time: Meetings and Emails Kill Hours, but You Can Identify the Worst Offenders on WSJ
• Killing the Crunch Mode Antipattern by Chad Fowler

Great words of wisdom about two of the worst anti-patterns for successful tech organizations. The most telling part is the “canary in a coal mine” effect: to watch and see who becomes the most offended by these points. Egregious (sometimes outright hostile) use of email, chat, meetings, etc., and the fallacy of “crunch mode” stand as two of my top determinants for evaluating a company. Right alongside we provide free snacks and meals vs. we offer reasonable health care plans – which somehow turn out to be at odds in far too many start-ups.

BTW, really looking forward to catching Chad speak at GOTO Chicago next May.

Just Enough Math

The Just Enough Math material continues to evolve… Allen and I gave a tutorial at Strata NY, working closely with O’Reilly Media to export content to IPython Notebook within a Docker container for participants to run in the cloud. Rackspace provided the hosting, which in turn was an alpha test for their Nature magazine IPython interactive demo. Welcome to the future of publishing.

Andrew Odewahn and I entered a version of this for the Boston instance of Docker Global Hack Day #2 – frankly, Andrew did like 99.9999% of the work on that one :) Meanwhile, speaking of the future of publishing, JEM provides an example in the new Publishing Workflows for Jupyter by Andrew Odewahn, Kyle Kelley, Rune Madsen.

Beyond publishing, we do have some math to suggest… Two papers caught my attention recently:

• Getting the Most Out of Ensemble Selection by Rich Caruana, Art Munson, Alexandru Niculescu-Mizil @Cornell – turns out that RMS is a pretty good proxy metric…
• The Thermodynamics of High Frequency Markets by Kevin Thomas Webster @Princeton – lots of unexpected in that gem

Oh, and riffing off the “Quantum Algorithms on the Moon” meme from JEM, note that NASA, Google and USRA establish Quantum Computing Research Collaboration such that 20% of computing time will be provided to the university community. In case you have some large data set that’s just screaming to get crunched on a D-Wave. Like you do.

Mesos

BenH wrote in O’Reilly Radar recently, Why the data center needs an operating system: It’s time for applications — not servers — to rule the data center.

Other big news was the Google Cloud Platform Live conference in SF on Nov 1. The message from #GCPLive was largely about containers… in short, the notion of The datacenter IS the computer going mainstream. To paraphrase one comment during the conf: “Customers get locked into host-based patterns, so they struggle with intertwined systems.” Well said. Definitely looking forward to the new GKE service based on Kubernetes.

Other big news was awaiting in London. Namely, the team behind Weave. Recall that the JEM tutorial had been an alpha test for the IPython + Docker + Rackspace + Nature magazine thing? We learned a truism the hard way, with minutes to go before the event started: Docker does little to resolve crucial issues outside of the containers. Enter Weave, handling difficult matters outside the container, such as networking and crypto. Check their blog for tasty insights, e.g., Automated provisioning of multi-cloud weave network with Terraform. Highly recommended.

Speaking of Docker, I really enjoyed this talk by Adrian Cockcroft @DockerCon: State of the Art in Microservices. Especially slides #8–19, product development process.

Speaking of Microservices, here’s a good overview: The Strengths and Weaknesses of Microservices by Abel Avram on InfoQ.

Ag+Data

Continuing on the Ag+Data front, check out the excellent article GeoTrellis Adapts to Climate Change and Spark about how Climate Change analytics drove Spark adoption at Azavea. They integrated Spark and Accumulo to support fast computation of climate impact metrics for DoE, which should be included in the 0.10 release of GeoTrellis.

NYT ran an interactive analysis/visualization, Flooding Risk From Climate Change, Country by Country, which perhaps helps explain Silicon Valley rumors about Google building ferry ports at corporate campuses along SF Bay.

I’m a big fan of Danielle Nierenberg @FoodTank in Chicago. A recent article, How Vegetables Can Save the World, is brief, accessible, and quite to the point. More of that on FoodTank.

Meanwhile, considering the many challenges ahead in Ag worldwide, I’m curious whether some programmable matter could become useful on farms to leverage data? Sort of an asymptote for IoT.

Upcoming Events

Many interesting conferences and other events are planned for the months ahead. Please do check the http://goo.gl/2YqJZK listings. In particular, mark your calendars for:

• Austin, Jan 10: Data Day Texas
• San Jose, Feb 18–20: Strata CA
• NYC, Mar 18–19: Spark Summit East
• SF, Jun 15–17: Spark Summit 2015

O'Reilly studio in Sebastopol, for new "Intro Spark" video

Misc.

I’ll leave you with something fun and something epic.

First, the fun – though it’s quite epic in a way: LumiGeek. We make Arduino shields for LEDs, audio-reactive drivers, and custom solutions for architectural and artistic endeavors. Check their installation at the new Galvanize Cafe in SF, and look about carefully for a subtle case of anamorphosis.

Second, the epic – if you haven’t seen it yet, it’s well worth four gorgeous minutes of video: Wanderers by Erik Wernquist, narrated by Carl Sagan. Money quote @1:45: “Herman Melville in Moby Dick spoke for wanderers in all epochs and meridians…”

That's the update for now. See you in Austin, San Jose, and NYC on the event horizon!

ceteris paribus and the basted egg heuristic

Years ago, I had a job working as a short-order cook in a reasonably nice restaurant. One thing you learn quickly in that kind of work is what you can cook simply, reliably, while handling N other orders at the same time. Perhaps this falls into the general category of load-balancing or speculative execution, but some orders work better than others.

An example of a simple order is a "basted egg". One egg, some water and some oil in a small covered pan, burner set low, for a few minutes. I happen to like eating those for breakfast. Even though they almost never appear on a breakfast menu, it's one of the simplest things that a short-order cook can be asked to make -- aside from toast.

When I'm checking out a new little cafe for a quick breakfast, oftentimes I'll ask for a basted egg. That alone provides amazing insights to the structure and vitality of the restaurant... I've found that most wait staff and short-order cooks who are any good at preparing breakfast will jump at the chance to earn 3x food cost + their 20% tip by serving the simplest thing possible. (assuming that wait staff splits tips with cooks, which they should). Generally they do quite well, I enjoy breakfast, and pay more than 20% tip. Done.

However, using this metric, I've found an heuristic: restaurants on the brink of failure -- and there are many, almost always, given the nature of the business -- tend to fumble on something even as simple as a basted egg. I've found that almost every restaurant which told me "Sorry, no basted eggs here" has closed within the following 2 months. In fact, I've got a running calculation for a binomial confidence interval on that heuristic test converging near 1.0 as a best point estimate. Which is to say, pretty damn certain. Generally, I'd attribute that kind of failure to lack of insight or communication by the restaurant management, or simply that they'd hired untrained wait staff and kitchen staff. Generally the former condition is more opaque, and thus more likely as a risk to the restaurant.

Years ago -- and not nearly as many years ago as the cooking gigs, but close -- I started working in the technology industry. My first full-time job in Silicon Valley was in 1983... though I'd been mentored about the industry by my uncle (who was deep in it) going back to the late 1960s. In other words, I've been learning about what works and what doesn't in the tech industry for approximately 40 years, with the past 25 years of it hands-on. (Yikes, I'm that old now?!)

During that time, I've developed and tuned a robust heuristic for measuring the vitality of tech organizations -- not so different from my basted egg test in restaurants. Simply put, when I walk into a meeting, I ask myself three questions about everyone in the room. I won't disclose those questions here, but they are listed on my resume, which is public. I'll leave that research as an exercise for the interested reader. Anywhoo, every participant in the meeting gets scored on a scale of 0-3 points, whether they like it or not. Here's how I evaluate scores on an individual level:

3. great to work with, nurture a long-term relationship
2. short-term only, watch carefully for cracks in the foundation
1. limit dialogue to what is required
0. migrate in opposite direction!

Okay, I'll be the first to say that may sound trite, but hey it works. Even when you're encountering people who may not be the sharpest minds in the industry, if they score highly, consistently on that scale then ceteris paribus they'll likely do well in the industry.

I recall -- vividly -- one restaurant in central Austin, where I asked for a basted egg... the manager came out, and, without asking, placed his arm around my shoulders while explaning to me, "Son, there's no such thing as a basted egg." As one might imagine, I got up and left rather suddenly, and more adroitly put, there was no such thing as that idiot's employment 4 weeks later.

On the organizational level, based on the past few decades of experience watching tech companies struggle to be viable -- struggle on par with that found in the restaurant biz -- on one hand, if a tech firm has plenty of people who score 3's consistently, they've got my attention and quite likely could get my commitment too. On the other hand, if they've got people who score 1's or 0's consistently then, just like that ill-fated breakfast nook in Austin with the smarmy manager, get up and leave quickly!

attack at cia headquarters

I can neither confirm nor deny that members of our party were in any way responsible for several unfortunate events which occurred last Friday evening at the Culinary Institute of America (California Branch) in St Helena. Because I plead the Fifth, my comments may not be construed to support any of the alleged acts, including but not limited to: (a) the disappearance of a wrought-iron rooster from the wall ("Sir, we know that you have the rooster"); (b) contributing to the delinquency of a wait-server by suggesting to bribe said CIA employee in the face of mounting distress on the part of the maître de ("Only if it comes from Humbolt"); or the willful inciting of a "cougar" attack ("Gee, you smell great!") on our way out the door. I can confirm however that two of my friends, the aforementioned rooster rustlers and designated "cougar" bait - to whom we shall refer by the utterly fictitious pseudonyms "Karsten" and "Charlie" - did in fact score an autographed chef's hat for somewhat less than the winery price of a Stag's Leap 2002 Estate Grown Petite Syrah.

A member of our board has a bed and breakfast in the family, located on a beautiful stretch of Silverado just a few clicks up-valley from St Helena. They graciously hosted our firm's first "off site" week. We initiated festivities with a visit to Dean & DeLuca which involved intense negotiations en français followed by dropping a few bills (non-expensable) and holding up the checkout line for nearly an hour. Our ensuing ratatouille feast and successive raids on said family's mountain cellar provided fine food, wine, and conversations all week long, fueling some amazing sorties into the metaverse, which I hope we can begin to share with the world soon.

After months of crunch time, preparing for a wide set of technology demos, we deserved a little pampering - so we availed ourselves of the local terroir. In a private room at Tra Vigne we learned the intimate details of Pighed's recent forcible detainment by LAPD for running a red light after midnight on a girl's bike. For the up-scale down-home version of local fare, various attendees made excursions to Market throughout the week.

Lobster Roll at the latter was so amazing that Erin and I missed our flight home from Oakland. I've reviewed it for a first contribution at Lindsey Simon's new Dishola site. And as I type, our girlies are both nearly unrecognizable after munching on their chocolate duck bling-bling from Woodhouse, right near Market. Be certain to check out The Olivier next door.

Ah, Napa! Not quite as dear as my home terrior in Edna Valley, but the best thing this side of Tuscany. Moreover I can confirm that thanks to the oneological genius of Opus One, I got more smashed than I've ever been outside of Russia or Japan, and the Scallop Gnocci at CIA alone is well worth an immediate trek up to Napa.

bay-nays foster

My daughters call bananas "bay-nays". This recipe, with which I've been experimenting since 1996, is approximately as old as my first Java code. It's also the only dish that I've ever served to (otherwise seemingly civil) adults which caused them to fight and steal at a dinner party ;) Be sure to use dark chocolate and light, fresh flour ... and in the name of gopod, prepare enough to avoid the fighting.

• Serves: 4
• Prep Time: 60 minutes


• Ravioli Pasta:
  • 1 cup Banana pulp
  • 1/3 cup Sugar
  • 1 tsp Vanilla extract
  • 1 Egg
  • 2 cup Flour (Semolina)
• Filling:
  • 1 Chocolate bar
  • 1 Egg yolk
  • 2 Tbs Water
• Sauce:
  • 3 Tbs Butter
  • 3 Tbs Sugar
  • 1 Lemon, small
  • 1 shot Liqueur
  • dash Cinnamon, Nutmeg
  • 1 Banana
  • 1 tsp Powdered Sugar
  • 1/2 cup Chocolate Sauce

Blend banana pulp, sugar, vanilla, egg. Blend in flour. Process to make pasta sheets. Pound chocolate bar into powdery small chunks, then spoon in to ravioli. Coat seams with egg/water mix and cut. Boil until ready.

Saute butter, sugar, spices on low heat until caramelized. Add sliced banana and lemon juice, then stir. Carefully add ravioli and saute until browned. Pour in liqueur (a mix of rum, Everclear and banana liqueur works well) and let heat for 30 sec, then light with a match. Spoon flaming ravioli into dishes, and lightly sprinkle with rum caramel from the pan, chocolate sauce and powdered sugar.

sufi coffee shop

On a recent trip to Palo Alto, I had missed nearly a full day and a half without an actual meal. There were too many hassles enroute and then lots of exciting work to discuss, once I got on the ground.

By the time I had a chance to call home and then get a little food, not much around my hotel was open (or looked any good). I wandered, frustrated and hungry. Amazingly, here in the heart of Silicon Valley, I stumbled upon a jewel called Sufi coffee shop. After I ordered an enormous amount of food, the proprietor, Parviz Rasti, looked at me carefully and asked, "Have you not eaten in days?" "Actually," I replied, "not in about 1.5 days!"

Okay, this place has wonderful food - huge portions, great quality, reasonably low price. Even more wonderful is the ambience and the spirit of the place.

I'm hungry now, I'm heading back to Silicon Valley soon, and I'm looking forward to visiting Sufi.

in praise of seitan

Recently my wife asked me how to cook eggplant. First, we had received some nice egg-shaped eggplant with our Greenling delivery last week. Secondly, there's a popular eggplant dish in Persian cuisine which I like to fix for the family, and Erin was perhaps hinting :)

The Persian word for eggplant is "bademjan". I remember a similar word used to describe a popular eggplant salad in Russia - which was delicious, but perhaps not quite as savory as the Persian variety.

However, this time we changed up the recipes a bit. We had a craving for "setian" - also called wheat roast or fu. So here goes one for our recipe book, an adaptation of a Persian lamb stew called "khoresh" with our substitutes for the meat - in prase of Seitan:

Slice eggplant - optionally peel them if the peels are too think and tough.

Soak eggplant in brine overnight, to leach out the bitterness - not required for Japanese style eggplant.

Drain eggplant, removing the bitter brine.

In a large iron skillet, heat a centimeter of grapeseed oil, adding freshly grated nutmeg, black pepper, and Hungarian paprika.

Once that begins to snap, add eggplant slices.

Cover and cook, stirring occaisionally so that all the eggplant becomes cooked evenly.

When the eggplant flesh becomes soft and somewhat brown/purple, add equal amounts of seitan (roasted kind).

Stir together, adding enough tomato paste to coat all the "meat".

I like to garnish with a chiffonade of cilatro and garlic chives, which just happen to be locally in season.

I also like to serve this over rice. A nice basmati should work well enough... Persian dishes tend to use a technique called "chelou bah tadig" where the rice is slightly under cooked, then placed into a tall pot which has hot oil and saffron in the bottom. Make "holes" with a wooden spoon all the way from top to bottom of the rice, and wrap the pot's lid with a towel.

The result is a golden crisp at the bottom, where the the rice above becomes a little fluffy and gains an almost "nutlike" or "popcorn" flavor. Place the tomato + seitan + bademjan mix atop that and you'll enjoy a tasty entree.

In fact, I have been tempted to name this dish The Great Seitan.

greenling

I prefer my food slow, local, and organic.

Eating at restaurants too often - in other words, more than a few times per week - just does not feel wholesome to me. Eating fast food often at all just feels wrong.

For one thing, I cannot stand the size of portions which are typical at American restaurants. It's absurd. A friend of mine in Japan remarked that it's great when he visits because he can just order one meal and then carry home another entire meal!

The amount of fat, the horrid condition of ingredients, the excesses in packaging - all of these factors make restaurant food unlikely for me, except for really good meals, which are probably 4x food cost anyway and not what I'd care to buy other than for a special treat or a social gathering outside a home.

I prefer slow food. I love to cook at home - and spent a whole bunch on building a nice home kitchen. After those early years where I earned my keep as a cook at times, and after so many fun evenings hosting dinner parties - or "FringeWare Fridays" back when we had the bookstore in the late 1990s - a meal cooked by a stranger just doesn't necessarily sound fun.

Ah, but where do all those lovely ingredients for fine home-cooked meals come from?

There has been a notion of CSA and excellent farmers markets around Austin for a long while - quite appealing. Several years ago I met the couple who run Tecolote Farm... imagine getting a basket of fresh, locally, grown organic vegetables delivered to your doorstep weekly. Then imagine having to wait on a list for a year or two! Alternatively, the Austin Farmer's Market is a treat - I enjoy taking my daughters there on the weekend. For that matter, Austin is the headquarters for both Central Market (which I adore, admittedly corporate but supportive of locals and independents) and Whole Foods (which I detest, ultra corporate green-posers aka "Hoalf Udz"), and the area features some wonderful natural food co-ops. We even have an organic farm inside the city - Boggy Creek Farm, which has sustained since the years when Austin was the source for most fresh spinach in the region.

As a home cook, I've been a fan of growing veggies, herbs, raising chickens, etc., since I was a kid... Growing up on the Central California coast, I had not questioned the idea of "slow food"... Local seafood was abundant - each week we ate the abalone which my uncle couldn't sell from his catch. Many of the neighbors were from Portuguese families which had immigrated there, largely for the fishing - and they showed up everbody when it came to fixing seafood well! Local farms featured fresh berries, apple orchards, all kinds of vegetables - especially artichokes - and fresh dairy. We used to buy a side of beef for the freezer, selecting a steer out on the ranch... ranches tended by families who had gained the land 200 years earlier as Spanish land grants, or often some Swiss-Italian immigrants who brought Tyrolian styles. And of course, what wasn't ranch or farm was soon to become wall-to-wall vineyards, with soil and microclimates approximating a range from Burgundy to Tuscany.

Moving away from SLO, I found the rest of the US held no such values. Especially in Texas, where a "meal" seems synonymous with a drive around the block to Whataburger.

There have been exceptions. One chef transplanted from France to Austin - Jean Luc Salles - teamed up with Boggy Creek Farm to feature local, organic vegetables in season at his bistro. I was lucky enough to catch Salles and author Carol Ann Sayle from Boggy Creek give a cooking course at Central Market, and I highly recommend Sayle's cookbook Eating in Season.

Of course, all of this requires extra time and effort, and probably would seem obscure to people who don't share my values (or half-French blood?)


Which is why I've been stunned to find the wonderful value of a new service called Greenling. They take the notion of "CSA" plus home delivery and cooking foods in-season - and bundle it into a family-oriented package. Wonderful. It convinced my wife, and is convincing my neighbors - about as fast as they hear about it. Greenling delivers an ice chest of fresh vegetables, plus extras - each 1-2 weeks. For our family, we've chosen 2 week intervals at first, but we're switching soon to weekly. We get a surprise selection of in-season produce from local organic farms. We manage our delivery account on the Greenling website, and can take on extras like organic dairy, honey, etc.

We're forced now to cook at home, using whatever is seasonal and local. It's delicious. I'm actually learning to love okra again. We've reduced our trips to the grocery store (I have mixed feelings, because I love to shop at Central Market, but I appreciate burning less fuel to drive there.) We've been sharing with neighbors and friends.

The idea of a CSA is great, but the close pairing of one farm to a set of subscriber families - it comes out a bit awkward. With a service like Greenling, there can be some economy of scale, some buffering and variety, by incorporating producer/consumer relations between many farms and many residences. I could imagine this kind of service hanging around the farmers market just about when it closes - to bid on what's left over.

More importantly, one could imagine this kind of service going national. Soon.