newsletter update 2016-07

Synecdoche

Been absent from newsletter feeds for a year plus. During that time, several POVs changed markedly. Home: relocated from the shadows of the new Google X building …  to a rural part of Russian River Valley. Work: swapped more than an hour commute up to a SF tech start-up … for a 20 min bike ride to O’Reilly Media. Plus a few other priorities. While the newsletter name changes, its themes carry forward. Plus a few other priorities. I’ll cover: conference summaries, open source projects, interesting news items, recommendations … data science, machine learning, advanced math, sustainable ag, art, travel, artisan foods, cider, flying cars, etc. Albeit running in slightly different circles and casting a wider net.

Changing POVs: new “backyard” has less actuators, more pollinators

Confs

Been trying to keep pace with our 20+ tech conferences each year. Lots of travel, lots of interesting people and insights into their projects. Some upcoming:

• JupyterDay Atlanta, ATL, 13 Aug: will present Oriole: a new learning medium based on Jupyter + Docker
• Artificial Intelligence, NYC, 26–27 Sep: plus a drinkup at Wassail
• Next:Economy, SF, 10–11 Oct: plus a drinkup at Upcider
• Big Data Spain, Madrid, 17–18 Nov: will deliver a keynote, hopefully some workshops too!
• Strata+HW Singapore, SG, 5–8 Dec: I’ll present Computable Content: Notebooks, Containers, and Data-Driven Organizational Learning

De Pijp in Amsterdam during Scala Days 2015

Strata EU 2016, 31 May - 03 Jun. Venue moved to the much larger ExCel London space and that afforded an expanse of incredible people, tech innovations, data insights, etc. Also, it proved to be a good base of operations for exploring two recent interests regarding London: connectography and cider. More about both in a moment :)

Strata kicked off with a “Hey, are you busy?” text from the brilliant Alistair Croll, which led to one of the best pub crawls ever. Still parsing our discussions in those London pubs. Those provided the basis for several essays linked here. Seriously. Pay close attention to the intellectual sculpting that emerges from Alistair’s endeavors.

Strata tradition: team #noscurvy w/ BLL Ladies @holdenkarau @j_houg

Anirudh Koul invited me to a session with his Microsoft colleague Saqib Shaikh, where they delivered Beyond guide dogs: How advances in deep learning can empower the blind community. Sure enough, a favorite talk at Strata. Microsoft’s been busy building cognitive services in the cloud which developers can integrate into other software in general. Following retina surgery in both eyes in the 1990s, I experienced serious vision impairment. It was way too much to write code for a while. Switched to working on VR design for a while: shapes, colors, contrasts were much easier to manipulate than source code. Years later my vision still changes abruptly. Grateful now to see (pun intended) these AI innovations by Microsoft and I look forward to many interesting applications.

The proverbial “Full English”

Arrived back to Sonoma County just in time for Foo Camp 2016. Translated: that’s an unconference while camping with 200 fascinating people from around the world. Held in the apple orchard behind our office. DJ Patil and Ed Felten opened the weekend with a lively group discussion about data ethics, where Cory Doctorow delivered the punchline: when you own a mobile device, who choses to override settings, you, the manufacturer, or the government? Lives may depend on the answer.

One decidedly tangible result of Foo: “Blockchain of Love” by Ellie Volckhausen, Edie Freedman, Ally Miller, Johnny Diggz, Tony Parisi, Brian Fan, and yours truly helped with a line or two. Tony attempted to explain the ineffable:

Geeks, guitars and a whole lotta booze. We wrote this song in 45 minutes at O’Reilly Foo Camp!!!

Got to try a Copenhagen Wheel, courtesy of Assaf Biderman @ Superpedestrian – most definitely a decade beyond the Bionx on my prior ride. A favorite session was an Ignite talk by GalaxyKate, aka Kate Compton. Check out her work on generators and related tooling at Tracery.

the old Make Mobile

Velocity CA 2016, 20–23 Jun … Noticed that sessions related to Apache Mesos and DC/OS were standing-room-only. In other words, considering the recent Mesosphere investment by HPE and Microsoft, Mesos has arrived for the mainstream. A good illustration of this? The Container orchestration wars session by Karl Isenberg.

Having been away from SV for a while … Andrew Marantz’s recent New Yorker article How “Silicon Valley” Nails Silicon Valley struck a nerve or two. That rollerblade “scene” took place right behind our former house. Recommended.

Media Theory

One evening in late 1992, between the first Cypherpunks meeting and the FringeWare launch … I climbed aboard an opaque bus, allegedly headed to a private event at Intel Santa Clara. Probably had “press credentials” from Mondo 2000, or something. Sat down, turned around, met the person sitting in the next row. Brilliant, utterly mellifluous, compelling immediate engagement from all around, someone who soon became a star in the industry and also a lifelong friend: Mark Pesce. That evening was all about wandering into different worlds. For a brief interlude, the word “environment” itself became entirely fungible. Alice, down the rabbit hole. Guided tours of the latest in VR by Eric Gullichsen, et al.

offspring in tribal garb

Some things come full circle in the next generation. Our tween daughters recently chose their first “concert” experience: a first-of-its-kind world tour by celebrity YouTubers Dan and Phil. If you haven’t been tracking what’s changed in media, take a close look.

This time, the virtual from the Interwebs gets projected live and in the flesh. Spoiler alert: millennials may have leapt the Selachimorpha.

Read more in my Medium article: Wandering into an entirely different world

Oriole

Les Guessing posed an interesting question recently:

Surprised how data “Storytelling” mostly talks about visualization, not video or other great storytelling mediums.

Well said! Oriole is our update on that point. This new medium combines a video timeline with hyperlinked rich text, code, data, results, instrumentation, and visualization. The point is about repeatable science, in a sense, applied to the matter of contextualized hands-on learning about technology. Also a great step toward personalized learning, with brilliant dev+design from our Brooklyn Team. We’re eager to see how other authors leverage this.

Oriole

For a much better introduction, try it yourself! Check out the coding challenge in Regex Golf by Peter Norvig. That’s the first Oriole we published, as an exemplar.

Read more in my Radar article: Learning alongside innovators, thought-by-thought, in context

And join us for the first public talk about Oriole, at JupyterDay Atlanta on 13 Aug.

AI

David Beyer recently published an excellent series of interviews about the Future of Machine Intelligence. Favorites include:

• Ben Recht on what ML can learn from many decades of control theory, in particular due to similarities with classic dynamical systems – two distinct ways to consider models
• Anima Anandkumar on the use of tensor decomposition and multilinear algebras for non-convex optimization – to represent higher order relationships among latent variables
• Gurjeet Singh on the use of topology: “Data has shape and shape has meaning”, i.e., mapping from one space to another

One thing has been really bothering me about recent media coverage of AI. There’s a disturbing tendency to equate ML with AI, at the expense of considering how control systems are inherent in almost any business application. For example, Uber may use lots of machine learning; however, at the heart its business a control system manages drivers, customers, offers, and other vital aspects of supply chain and contingency. Note that the field of AI emerged from control theory and early cybernetics.

Read more in my Medium article: Beyond the AI Winter

A recent conversation on video by Tim O’Reilly and Peter Norvig explores applications of AI technologies, conversational interfaces, etc. That dovetails with how Google pointed DeepMind at data center power usage, for a dramatic 40% decrease in energy needs. That prefigures many industrial uses for AI, given how “the algorithm is a general-purpose framework to understand complex dynamics” in manufacturing, transportation, energy, etc. Note that Tim and Peter will be honorary program chairs at the new/aforementioned Artificial Intelligence conference in NYC this September.

Meanwhile, other arcanum about the Greater NYC Area … note the curiously shaped post horn which marks the façade of our Brooklyn office:

O’Reilly Brooklyn office, “front door”

The Trystero are out and about, and the sign is not stable.

Neurons and Art

More about applied AI, perhaps for dessert … What have artificial neurons been doing in the art world lately? Apparently they’ve evolved far beyond assembling art boxes that recall the work of Joseph Cornell. Check out the aftermath of the DeepDream: the art of neural networks show in SF this past February by Gray Area.

For an even more intense dose of narrative closure … Terence Broad applied his dissertation, Autoencoding Video Frames to have deep learning deconstruct the film Blade Runner. Using unsupervised learning in a way that Hollywood lawyers might not comprehend for quite a while, this work prompted Warner Brothers to issue a DMCA takedown notice … Except that (surprise!) the work did not include any Blade Runner footage.

For an epic recounting of neurons and art, check out Neural Networks and the Digital Hallucination by good friends over at Lumo Interactive. Also, links highly recommended in Beyond the AI Winter:

Check out the inexplicably entertaining short film Sunspring, authored by an AI named Benjamin.

That script writing AI leverages a recurrent neural network approach called long short-term memory (LSTM ) which takes abductive reasoning to new levels.

ebumna lw’nafh

What AIs aren’t quite dreaming about (yet) are seemingly less probable matters in generated imagery: crystals in a 4.57 billion-year-old meteorite found in Chukotka, Russia showed the first naturally occurring Forbidden Symmetry. YMMV.

Reading

Recently finished Connectography by Parag Khanna. Well worth a good read on several levels. See the especially interesting insights about city-states and SEZs. Had me at the maps. Even so, I’d suggest taking this text with a sizable grain of salt. Perhaps a boulder, caveat lector. The basic premise obtains: how supply chain economics have usurped national borders and trade restrictions. To wit:

Infrastructure, markets, technologies, and supply chains are not only logistically uniting the world, but propelling us toward a more fair and sustainable future.

brilliant maps in Connectography

OTOH its trappings as a book-length hagiography of corporatist values may required antiemetics. Read more in my Goodreads review:

The analysis of geography and history is thought-provoking, albeit some flawed historical citations. Clearly he loves major cities. Plus, there’s a bunch of “US bad, China good” on the surface. Quite a bunch. Dig deeper, it’s worthwhile.

Not to change the subject, but for a truly fun read: The Peripheral by William Gibson. One of the best novels I’ve read in a long while – had much catching up to do on books by the Great Dismal. Read more in my Goodreads review:

Chapters alternate POV between a young female protagonist of near-future and our male anti-hero of farther-future. Extra points for picking up the narrative at the start of some chapters in the “wrong” tense, rear-view mirror syntactic suspense for major plot beats. Like a video rewind after an unexpected concussion.

If you’d like to compare books, reviews, etc., please friend me on Goodreads.

Orchard Brew

A few years ago I was enjoying a seasonal, organic, farm-to-table meal with a close friend at Farmhouse Evanston. Ordered a cyser to drink, notably a Puff the Magic Cyser by Vander Mill – for something local to the Midwest. One sip took me on an adventure that’s kept unfolding in marvelous ways ever since.

ICYMI, ciders are making a huge comeback. Cider used to be the most popular alcoholic beverage in the US, prior to the insanity and cultural warfare of Prohibition. Especially in Sonoma West, where we live on a property that’s partly apple orchard, where we work on a campus built amidst an apple orchard. One fun aspect is that the ciders tend to be hyperlocal. In other words, one can travel the globe and encounter novel brews which stay close to their orchards of origin.

ngram occurrence for “cider” in Google Books, 1650–2008 ce

Here in Sonoma, ciders have little to do with the nasty, too sweet, apple-juice-with-vodka mush flavors that so many folks associate with the phrase “hard cider”. Cider makers here use champagne yeasts and other techniques to produce dry brews with complex flavors. Along with that, federal legislation in the US just revamped arcane tax laws that made The new CIDER Act allows small producers to get in the game with artisan brews, without getting stomped by weird post-Prohibition wine laws. This renaissance is good for the land, good for family farms, and great for drinking IMO.

Cider Bar @ Foo

So we held a taste test at Foo Camp this year. Brought in some of the more popular and interesting ciders from orchards near Sebastopol. Almost everyone (except DJ) favored Cider with Hops & Honey: a personal favorite from our neighbors Horse & Plow run by Chris Condos and Suzanne Hagins. That’s both good and somewhat dangerous, since Chris and Suzanne just opened their new “tasting barn” across the road from O’Reilly Media.

Also … LONDON! One of the best places in the world for cider. Fine folks at Orchard Pig recommended visiting The Williams Ale & Cider House, which was outstanding. Will be hosting drinkups there whenever I’m in London. Another couple great venues for sampling UK ciders are Euston Cider Tap and The Green Man. Favorite ciders in the UK? Reveller by Orchard Pig brings the fine flavors of Somerset, available far and wide. Top delight of this trip was Scotland’s Thistly Cross: off-dry, crisp, so smooth, the closest thing I’ve experienced to a Devoto Cidre Noir outside of Northern California :) Another treasure to seek out: Wild Summer by Kentish Pip:

Hints of Elderflower alight sea foam spray in a Nordic summer. Delicate, lightly astringent aftertaste.

As we tour the world with O’Reilly conferences, I’ll host drinkups at nearby cider bars. Here’s a toast to meeting you and sharing a brew or two! Meanwhile, if you’d like to share reviews of ciders, beers, etc., please friend me on Untappd.

Kane’ohe Bay Sandbar: grateful to visit with family in Hawaii recently

Ag+Data

Loved this informed, scathing rebuttal by the brilliant Wendell Berry, Wes Jackson, et al. Money quote:

What is remarkable, but unsurprising, is that these two gentlemen managed to write an entire column about agriculture without mentioning farmland or farmers.

That, in response to a NYTimes op-ed piece earlier this year entitled We Need a New Green Revolution. Note the key word entitled in that sentence: beware just about anyone who lauds a “Green Revolution” without recognizing the terrible consequences that resulted, i.e., financialization being a polite term. In other words, vast claims of battling starvation, while the most wealthy leveraged this vector to institutionalize mass starvation as their business model. There’s far too much “science” bandied about w.r.t. agriculture, pumped into media, heavy on funding ties yet light on scientific falsification.

I believe that politicians and other corporatists who side with Big Ag are behaving immorally. No other word for it.

Read more in my Radar article: Ag+Data

OTOH, absolutely LOVE it when intellect, integrity, and endurance overcome oh so many tedious decades of thoughtless aristotelianism. Check out How a Guy From a Montana Trailer Park Overturned 150 Years of Biology:

“Toby took huge risks for many years,” says McCutcheon. “And he changed the field.”

VRTX

Apropos of the above, I’ve been testing chapter ideas and other material for a new Liber 118. In particular, I’m a huge fan of Universal Basic Income, which has intriguing historical precedents plus enormous implications for the game that comes next.

After running the numbers for what UBI might cost in the US, implications became all the more urgent. It’s within striking range of commercial alternatives, i.e., corporatist funding in lieu of government initiatives. That got me thinking … about parody and science fiction to caricature what UBI might look like if everything went horribly wrong? What if corporatist values seized the day? Key portions of that parody got trumped, so to speak, by an even more bizarre happenstance in alleged reality.

¿quién es más macho? alleged reality vs. parody

Read more in my Medium article: Guaranteed Basic Income (cynical remix)

I’ll just leave you with …

Flying cars. It’s happening. When friends at Bloomberg first gave a heads up about this two years ago, the whole spiel sounded unreal. Now 2 out of 3 of the promises foretold in Planet Unicorn have come true. The big picture is that people travel into work in metro areas via line of sight. No more traffic congestion. Or something.

I guess that’s great for people who work in large cities. Not everybody. But there are likely some other intriguing uses.

Meanwhile, in the lesser hells of drone wizardry, one regrets that the noun #WTFery does not enjoy more regular usage in the English language.

---

Thanks for your attention, and I wish you all the best in your endeavors. Sign-up for the newsletter version at liber118.com

Four Quick Links for July

For those who live in the Pacific Northwest – or may be heading in that direction soon – I'll be doing a few talks in Portland and Seattle next week:

Portland, Tue Jul 21, 13:30–17:00

The Apache Spark tutorial at OSCON presents a hands-on introduction to Spark, with deep-dives into important components, SparkR and Data Sources API. The half-day tutorial considers examples from several Huawei case studies – production Spark deployments at scale for Telco use cases. I’ll be teaching this along with Haichuan Wang, Jacky Li, and Vimal Das Kammath V from Huawei.

Also: my newly updated video, Introduction to Apache Spark, will be featured as the Video of the Week during OSCON. This features new updates for DataFrames.

Portland, Thu Jul 23, 10:40–11:20

Microservices, containers, and machine learning provides a deep-dive into a project called Exsto that we’re using to explore the structure and dynamics of open source developer communities. It incorporates natural language processing, graph algorithms, etc., and leverages DataFrames and GraphX in Spark. We’ll explore the Apache Spark developer community as a case study.

Seattle, Fri Jul 24, 18:30–21:30

Eleven Almost-Truisms About Data will be a keynote at a launch party for the new GalvanizeU program in Seattle. Almost a dozen almost-truisms about Data that almost everyone should consider carefully as they embark on a journey into Data Science. There are a number preconceptions about working with data at scale where the realities beg to differ. This talk estimates that number to be at least eleven, through probably much larger. Let’s consider some of the less-intuitive directions in which this field is heading, along with likely consequences and corollaries – especially for those who are just now beginning to study about the technologies, the processes, and the people involved.

Seattle, Sun Jul 26, 14:15–14:55

PyData Seattle: NLP and text analytics at scale with PySpark and notebooks at PyData Seattle will go into more detail about the PySpark components of the Exsto pipeline. I’m also super excited about the keynote by Lorena Barba. We’re leveraging Project Jupyter for O’Reilly Learning and I’m really looking forward to talking with lots of people who are working on Jupyter for Education.

See you in Portland and Seattle!

Newsletter Updates for May 2015

If you haven’t already been listening in to the O’Reilly Data Show Podcast hosted by Ben Lorica, then by all means do not walk, run to check it out! Episode linked above is with Anima Anandkumar @UC Irvine, recommended previously here, re: tensor analysis. Similarly, recent collaboration among David Gleich @Purdue, Austin Benson @Stanford, and Lek-Heng Lim @ UChicago, using tensor analysis to resolve hard problems in higher dimensional Markov chains, resulted in “Spacey Random Walks”. Punchline on slide #17. I detect a trend…

Confs

Three months and so much travel since my previous post: to paraphrase Ricardo Alberto Fernando Ricardo y de Acha, I’ve got some ’splaining to do :)

Highlights include:

Spark Summit East 2015, NYC

Really enjoyed the track chair gig for Data Science. My two favorite talks, both highly recommended:

• Un-collaborative filtering: Giving the right recommendations when your users aren’t helping you by Leah McGuire @Salesforce
• Next-Generation Genomics Analysis Using Spark and ADAM by Timothy Danford @AMPLab

That was a busy conf, indeed! You can tell since no time was carved off for sacred pilgrimages to Wassail NYC cider bar, Mast Brothers cold brew chocolate, or Blue Hill Farm. Instead my flight left the following day for…

QCon São Paulo

My first time to Brazil. I’m hugely impressed by the developer community there in SP. Most inspirational quote from the conf: “A team is like a symphony, not a factory.” by Randy Shoup.

Great sessions on Spark, Docker, and much much more. Fortunately, we did have time to sample the local cuisine… e.g., Italian food made with tropical ingredients, or my favorite meal of the week, Pirarucu steamed in banana leaves, with a local favorite recipe for pumpkin soup. Then back to the US, before one could even say “alfajores” – assisted on a course at Stanford, then back to NYC…

Pirarucu – Brasil a gosto, São Paulo

CodeNeuro East 2015

Organized by Jeremy Freeman and crew. Truly excellent: top neuroscience researchers in the world gather for a hackathon (i.e., coding together). Perhaps a few savvy Finance people lean in too, eagerly drooling over results that apply for their high-dimensional, time-series, non-linear correlations work as well… in what VCs insist on calling an “ecosystem”. Or something. See these excellent notes. My favorites:

• Michael Dewar : streamtools real-time analytics from NY Times
• Olga Botvinnik : flotilla - Py package for iterative machine learning analyses
• Eiman Azim on motor circuit function … with actual electronic circuits to emulate what Columbia discovered through ablation studies about cerebellum connections
• Brendan Lake teaching computers to scribble characters like humans, if you want some really interesting use cases for Deep Learning

We worked together the following day, between tutorials, to build an online platform for submitting algorithms to run against standard neuroscience data sets. This hackathon literally was research. If you want to understand more about Big Data being used expertly in life sciences, attend CodeNeuro!

Lower East Side, Manhattan – from New Museum roof

BBBT.us

Many thanks to Marilyn Waldman, Claudia Imhoff, and crew for a fantastic Spark tutorial at CU Leeds Business School! Followed by excellent convo via webcast with several hundred of the top BI analysts in the world. Then on to Boston for…

Big Data TechCon, Boston and Next.ML, MIT

Matei and I were multiplexing between these two conferences so much, UberX in oscillation overthruster mode, that we didn’t even see each other. Even so, lots of great Spark talks in Boston that week! Meanwhile, kind hotel staff redirected me toward The Barking Crab for dinner, and a good friend introduced L.A. Burdick. Also, there was lots of excellent hard cider in the area. Along with all that Big Data conf talk stuff. Then a red-eye flight took off for Europe…

Boston Seaport, by water taxi

A Coruña, Spain

Many thanks to all the work by Amparo Alonso-Betanzos, David Martínez-Rego, and colleagues for organizing the Spark tutorial at A Coruña. I’d never visited Galicia before, a place with lots of rain and people with red hair playing bagpipes amidst rolling green hills (no en UK – pero en España) … a place where there are software companies next to world championship surfing competitions and albariño vineyards (no en Santa Cruz, California – pero en España) … a place where they speak a language close to Portuguese – not so unfamiliar, right after São Paulo! Excellent other talks, along with a Spark tutorial by Juantomás García. What incredible people, Computer Science excellence at the university, and oh such good sea food. My keynote was broadcast on Spanish television – that’s a first! Use of runways in this corner of Spain is quite abbreviated, so we catapulted next for…

Frente a la Torre de Hercules

Strata + Hadoop World EU 2015, London

See a good summary of the conf online. Spark Camp this time had 12% of the conference attending, whereas before we’d been trending steady at just over 8%. Many thanks to all who participated. Great to meet so many people enthusiastic about using Apache Spark! Also got to host the Hadoop and Beyond track. Which, oddly enough, was mostly about Spark. My two favorite talks:

• Taming the firehose: Build analytics over 45 billion tweets using Elasticsearch and Spark by Anirudh Koul and Shashank Singh @Microsoft … I’d never seen a two-person session talk handled with such amazing orchestration, almost like an old-time radio play. Brilliant.
• Say goodbye to batch by Tyler Akidau @Google … an expert but highly accessible overview of streaming analytics, from one of the leads on MillWheel, along with a deep-dive into one of the most sophisticated streaming platforms

Plenty of hard ciders sampled while in London. Whenever visiting near West London, I make a point to drop by Princess Victoria. Also, got to see some friends at UCL and the Barclays incubator program. Then back to the US via customs in my former part-time home city, Vancouver…

Chicagoland

GOTO, Chicago

Dean Wampler, et al., have been hosting great Spark events in the Windy City. Always a treat to visit. We had a good turnout for the Spark tutorial at GOTO, one of my favorite software conferences. Walking alongside the Tribune Tower after my tutorial, I noticed that its walls contain rocks from other famous buildings all over the world. With labels, like an inverted museum. Check it out when you visit. There are also rumors of a cider bar being built, ahem, soon. Then a flight back to the Bay Area…

Future Learning 2020 Summit, Stanford

Serendipity. Got invited by a dear friend, Donna Kidwell @Webstudent to join this conf about the future of education, a collaboration among Future Learning Lab, H-STAR, EdCast, etc. Many thanks to Oddgeir Tveiten and crew.

Most discussions focused on experiences with MOOCs – from highly successful examples, e.g., Intro to Robotics by Peter Corke @QUT, or Trust Academy @Salesforce by Masha Sedova. However, the overall themes transcended MOOCs, asking the question of what comes “After Gutenberg”, how peer evaluation is transforming education at scale, and … Peter Norvig’s point that when so much learning material is available (e.g., via Google) the problem becomes a matter of how do you get people to want to interact with it? Generally, the social context of learning becomes key.

Along similar lines, Michael Shanks stressed that – in contrast to traditional academe that tends to decontextualize learning – more contemporary advances are focusing on how to contextualize, locally. That’s part of the essence of interdisciplinary work, e.g., Data Science. Also, delighted to meet Keith Devlin (our new neighbor) with amazing work in math education using Minecraft, etc. (sound familiar?) And was very fortunate to meet teaching superhero David Conover, who uses game design to teach topics like IoT in an at-risk high school in Austin. Brilliant.

Of course, we’ll be inviting the whole lot to propose talks for Strata! Speaking of advances in learning platforms, check out the recent beta site and related article Embracing Jupyter Notebooks at O’Reilly by Andrew Odewahn. Par example, Data visualization with Seaborn … that is integrating use of IPython/Jupyter, Docker, Thebe, etc. Brilliant++.

Data Science

Another recommend: a new series of ML-related interviews by David Beyer, beginning with my friend and colleague Reza Zadeh @Stanford – on the evolution of ML, deep learning, Stanford ICME, and Apache Spark.

If you haven’t seen the news, Nature banned use of p-values … #finally  Note that Fisher did not intend for p-values to be (ab)used that way. So I consider these tests to be truly excellent, for identifying intellectual limits. Related: The Nine Circles of Scientific Hell.

On the subject of pseudoscience – appalling to see recent and ongoing unscientific gaffes by people who should know better, e.g., Neil deGrasse Tyson about GMOs. In great contrast, antidotally, I’d point toward this gem – and please read it at least twice: Die, selfish gene, die by David Dobbs. So glad to see Dawkins getting served … #finally

Neil, Richard: just because you’re published, doesn’t mean you’ve become superior thinkers. Please keep your day jobs, respectively #thankyouverymuch

Speaking of ’splaining to do… another gem is Visualization Explanations @Setosa. Extra points if you grok the callback in their name.

If some of these items are what you’ve been talking about recently, you may just be a Data Science instructor … or interested in becoming one? Check out Become a Data Science instructor @ Galvanize (Seattle, SF, Boulder)

Spark

También se recomienda: una excelente introducción en español a la Scala y Apache Spark por Isra Gaytan. The Latin World has been busting a move lately on Spark, #justsayin

A Coruña – from Playa de Oza

Adatao published an excellent article about how they anticipated the inflection point for Spark adoption. Check out the cost curves. Brilliant.

Big news from AMPLab: Keystone.ML is released as open source, to make the process of constructing complicated machine learning pipelines easier. Good stuff from the eponymous Evan Sparks, et al.

I’ve got some graph analytics talks coming up… and was excited to see a streaming/incremental SSSP impl for Spark.

Also, speaking of the Apache Spark Developer Certificate, we’ve got another new neighbor: ORM + DataStax partner on C* cert. Collect all three!

Meanwhile, the big BIG news is Spark Summit 2015 coming up next month in SF. Use the discount code SparkSummitPC25 for 25% off registration. Not retroactive, but nice try :) Followed by Spark Summit EU in Amsterdam, this autumn. Spark it up!

IoT

Solid Conference is coming up again soon! Highly recommended. As an appetizer, check out this excellent article by Cameron Turner @The Data Guild: Caltrain Quantified - An Exploration in IoT which we could hear in our previous backyard every morning starting at about zero-dark-thirty. Now that former backyard has become the new GoogleX building, and SciFi tech experiments compete with the trains for attention.

For IoT in practice, I’m totally stoked to see: Surfers on acid… What an excellent application. And, culturally not far off that mark, here’s an interesting take on marine plastic: Net+Positiva.

Ag + Data

I’ve really been enjoying Biocoder News in quarterly installments, some of my favorite new articles in the world. Period.

On that note, I’m thoroughly ecstatic to announce that I’m moving to O’Reilly Media full-time. Even so, I'll stay involved with Spark and Databricks, assisting on Spark Summit, etc. We’re moving the family to a tiny farm, an old apple orchard that really needs some tending. Perfect as a research station for Ag+Data.

The Tiny Farm – redwoods 30m tall, planted 65 yrs ago by previous owner

In highly related news, check out How to Grow a Forest Really, Really Fast, about fantastic work by Shubhendu Sharma. I’m eager to try this out.

One of the top intellects of the early 21st century, Paul Stamets, had some excellent coverage: He Holds The Patent That Could DESTROY Monsanto And Change The World!  See also: BioMason  and Ecovative Fungi, FTW – and mycorrhiza in particular, as Mohamed Hijri explains quite succinctly.

Ag-related tech approaches in SV have become largely derailed by asinine priorities dictated by Monsanto – more about taking over hedge funds on commodity trading globally, than about feeding anyone. Perhaps the best analysis that I’ve read recently – and certainly one of the best books that I’ve ready recently – is the highly recommended The Third Plate by Dan Barber. I learned about that via Gastropod – where Cynthia Graber and Nicola Twilley consider food through the lens of science and history. Brilliant.

There’s been a terrible drought / water crisis in Brazil – largely exacerbated by transnational corporate interests. This was weighing on my mind as our flight landed in São Paulo. I got to speak with friends there who are working on Ag+Data analysis, very good to see.

As predicted: Finance is driving California water into the dust… take a moment to consider the jump in almond production versus the temporally co-located jump in variance for snow pack levels. That’s the tip of the iceberg for the near-term shape of major political battles brewing in California. To wit, some of our local mafia have become known under the more apt monicker of Oligarch Valley. While Fox News, et al., promotes the Israeli approach of desalinization at scale, many people who can actually think for themselves question the impact of that approach, and recognize what an utter environmental disaster it could produce. This is not an area of judgment where one gets to call #oops as an excuse, regardless of which side the local mafia may be taking.

Industry Insights

“Software eats the world” is a catchphrase used by A16z. While I slightly agree with the title from this Datanami article, How Machine Learning Is Eating the Software World, its conclusions are pretty much the opposite of what we’ve observed with Apache Spark use cases in the field. Don't get me wrong – Reynold is a good friend, and IMO one of the most talented people working in distributed systems today. However, I have a hunch that the reporter munged the line.

Two key reasons why organizations adopt cloud-based notebooks are (1) to reduce their need for DevOps people to run clusters; and (2) to reduce the need for programmers to assist business people with queries for insights Big Data. Done and done. In other words, domain experts trump all in Data Science applications, while application developers (in relatively large supply, but relatively expensive) and expert systems engineers (in relatively short supply, extremely expensive) both become less of an existential bottleneck for new ventures. I’ll let you do the math on that one.

Some of the themes that I’ve been researching and illustrating over recent years include: Functional Programming for Big Data, Approximation Algorithms, Tensor Factorization, etc. Recognize that each of these point toward less emphasis on developers leveraging APIs, and meanwhile more emphasis on domain experts leveraging simple-to-use frameworks. That’s the bottom line of Apache Spark. Meanwhile, I have no doubt that A16z will continue to rake in loads of money – some of their partners are well-connected billionaires – just perhaps not as a consequence of their thesis. That ship is already sailing. Off, perhaps, toward the oh-not-so Great Pacific Garbage Patch.

Diversity

Speaking of VCs in SV… TechCrunch analysis recently found that female founders nearly doubled in 5 years. Par example, check out the recent Women in Data: Their Work and Achievements.

Meanwhile, I thoroughly enjoyed this gem by Karin Rubin: How women are conquering SP500… My feelings about the overall ethics of algorithmic trading are arguably mixed. However, if it’s going to happen, why not guide it based on diversity, since that demonstrates a #winning strategy?

Fun Stuff, friends in the news…

Check out Lumo Interactive Projector by Meghan Athavale and crew. It’s an interactive floor projector, transforming a floor into games that kiddos can design themselves.

Also, this bit Our Coming Robot Overlords about David, Amanda, and Zeno Hanson – friends back in Texas.

And, what William Barker called one of his most honest interviews, ever.

Upcoming Events

• 2015–06–04: Graph Analytics in Spark @BloomReach, Mountain View
• 2015–06–07: Apache Spark: Intro Workshop for Developers with Andy Petrella, @Scala Days, Amsterdam
• 2015–06–08: Data Science at scale and graph analysis with Dean Wampler, Amsterdam Spark Meetup
• 2015–06–09: GraphX: Graph analytics for insights about developer communities, @Scala Days, Amsterdam
• 2015–06–15: Spark Summit 2015, SF
• 2015–06–15: Apache Spark Community Event, @GalvanizeU SF
• 2015–07–23: Microservices, containers, and machine learning, @OSCON, PDX
• 2015–08–03: ICT Labs Summer School, @SICS Stockholm
• 2015–08–14: Scala By The Bay, SF
• 2015–09–30: Strata + Hadoop World NY, NYC

Will just leave you with…

This article. Wonderful, on so many levels.