Newsletter Updates for May 2015
If you haven’t already been listening in to the O’Reilly Data Show Podcast hosted by Ben Lorica, then by all means do not walk, run to check it out! Episode linked above is with Anima Anandkumar @UC Irvine, recommended previously here, re: tensor analysis. Similarly, recent collaboration among David Gleich @Purdue, Austin Benson @Stanford, and Lek-Heng Lim @ UChicago, using tensor analysis to resolve hard problems in higher dimensional Markov chains, resulted in “Spacey Random Walks”. Punchline on slide #17. I detect a trend…
Confs
Three months and so much travel since my previous post: to paraphrase Ricardo Alberto Fernando Ricardo y de Acha, I’ve got some ’splaining to do :)
Highlights include:
Really enjoyed the track chair gig for Data Science. My two favorite talks, both highly recommended:
- Un-collaborative filtering: Giving the right recommendations when your users aren’t helping you by Leah McGuire @Salesforce
- Next-Generation Genomics Analysis Using Spark and ADAM by Timothy Danford @AMPLab
That was a busy conf, indeed! You can tell since no time was carved off for sacred pilgrimages to Wassail NYC cider bar, Mast Brothers cold brew chocolate, or Blue Hill Farm. Instead my flight left the following day for…
My first time to Brazil. I’m hugely impressed by the developer community there in SP. Most inspirational quote from the conf: “A team is like a symphony, not a factory.” by Randy Shoup.
Great sessions on Spark, Docker, and much much more. Fortunately, we did have time to sample the local cuisine… e.g., Italian food made with tropical ingredients, or my favorite meal of the week, Pirarucu steamed in banana leaves, with a local favorite recipe for pumpkin soup. Then back to the US, before one could even say “alfajores” – assisted on a course at Stanford, then back to NYC…
 |
| Pirarucu – Brasil a gosto, São Paulo |
Organized by Jeremy Freeman and crew. Truly excellent: top neuroscience researchers in the world gather for a hackathon (i.e., coding together). Perhaps a few savvy Finance people lean in too, eagerly drooling over results that apply for their high-dimensional, time-series, non-linear correlations work as well… in what VCs insist on calling an “ecosystem”. Or something. See these excellent notes. My favorites:
- Michael Dewar : streamtools real-time analytics from NY Times
- Olga Botvinnik : flotilla - Py package for iterative machine learning analyses
- Eiman Azim on motor circuit function … with actual electronic circuits to emulate what Columbia discovered through ablation studies about cerebellum connections
- Brendan Lake teaching computers to scribble characters like humans, if you want some really interesting use cases for Deep Learning
We worked together the following day, between tutorials, to build an online platform for submitting algorithms to run against standard neuroscience data sets. This hackathon literally was research. If you want to understand more about Big Data being used expertly in life sciences, attend CodeNeuro!
 |
| Lower East Side, Manhattan – from New Museum roof |
Many thanks to Marilyn Waldman, Claudia Imhoff, and crew for a fantastic Spark tutorial at CU Leeds Business School! Followed by excellent convo via webcast with several hundred of the top BI analysts in the world. Then on to Boston for…
Matei and I were multiplexing between these two conferences so much, UberX in oscillation overthruster mode, that we didn’t even see each other. Even so, lots of great Spark talks in Boston that week! Meanwhile, kind hotel staff redirected me toward The Barking Crab for dinner, and a good friend introduced L.A. Burdick. Also, there was lots of excellent hard cider in the area. Along with all that Big Data conf talk stuff. Then a red-eye flight took off for Europe…
 |
| Boston Seaport, by water taxi |
Many thanks to all the work by Amparo Alonso-Betanzos, David Martínez-Rego, and colleagues for organizing the Spark tutorial at A Coruña. I’d never visited Galicia before, a place with lots of rain and people with red hair playing bagpipes amidst rolling green hills (no en UK – pero en España) … a place where there are software companies next to world championship surfing competitions and albariño vineyards (no en Santa Cruz, California – pero en España) … a place where they speak a language close to Portuguese – not so unfamiliar, right after São Paulo! Excellent other talks, along with a Spark tutorial by Juantomás García. What incredible people, Computer Science excellence at the university, and oh such good sea food. My keynote was broadcast on Spanish television – that’s a first! Use of runways in this corner of Spain is quite abbreviated, so we catapulted next for…
 |
| Frente a la Torre de Hercules |
See a good summary of the conf online. Spark Camp this time had 12% of the conference attending, whereas before we’d been trending steady at just over 8%. Many thanks to all who participated. Great to meet so many people enthusiastic about using Apache Spark! Also got to host the Hadoop and Beyond track. Which, oddly enough, was mostly about Spark. My two favorite talks:
- Taming the firehose: Build analytics over 45 billion tweets using Elasticsearch and Spark by Anirudh Koul and Shashank Singh @Microsoft … I’d never seen a two-person session talk handled with such amazing orchestration, almost like an old-time radio play. Brilliant.
- Say goodbye to batch by Tyler Akidau @Google … an expert but highly accessible overview of streaming analytics, from one of the leads on MillWheel, along with a deep-dive into one of the most sophisticated streaming platforms
Plenty of hard ciders sampled while in London. Whenever visiting near West London, I make a point to drop by Princess Victoria. Also, got to see some friends at UCL and the Barclays incubator program. Then back to the US via customs in my former part-time home city, Vancouver…
 |
| Chicagoland |
Dean Wampler, et al., have been hosting great Spark events in the Windy City. Always a treat to visit. We had a good turnout for the Spark tutorial at GOTO, one of my favorite software conferences. Walking alongside the Tribune Tower after my tutorial, I noticed that its walls contain rocks from other famous buildings all over the world. With labels, like an inverted museum. Check it out when you visit. There are also rumors of a cider bar being built, ahem, soon. Then a flight back to the Bay Area…
Serendipity. Got invited by a dear friend, Donna Kidwell @Webstudent to join this conf about the future of education, a collaboration among Future Learning Lab, H-STAR, EdCast, etc. Many thanks to Oddgeir Tveiten and crew.
Most discussions focused on experiences with MOOCs – from highly successful examples, e.g., Intro to Robotics by Peter Corke @QUT, or Trust Academy @Salesforce by Masha Sedova. However, the overall themes transcended MOOCs, asking the question of what comes “After Gutenberg”, how peer evaluation is transforming education at scale, and … Peter Norvig’s point that when so much learning material is available (e.g., via Google) the problem becomes a matter of how do you get people to want to interact with it? Generally, the social context of learning becomes key.
Along similar lines, Michael Shanks stressed that – in contrast to traditional academe that tends to decontextualize learning – more contemporary advances are focusing on how to contextualize, locally. That’s part of the essence of interdisciplinary work, e.g., Data Science. Also, delighted to meet Keith Devlin (our new neighbor) with amazing work in math education using Minecraft, etc. (sound familiar?) And was very fortunate to meet teaching superhero David Conover, who uses game design to teach topics like IoT in an at-risk high school in Austin. Brilliant.
Of course, we’ll be inviting the whole lot to propose talks for Strata! Speaking of advances in learning platforms, check out the recent beta site and related article Embracing Jupyter Notebooks at O’Reilly by Andrew Odewahn. Par example, Data visualization with Seaborn … that is integrating use of IPython/Jupyter, Docker, Thebe, etc. Brilliant++.
Data Science
Another recommend: a new series of ML-related interviews by David Beyer, beginning with my friend and colleague Reza Zadeh @Stanford – on the evolution of ML, deep learning, Stanford ICME, and Apache Spark.
If you haven’t seen the news, Nature banned use of p-values … #finally Note that Fisher did not intend for p-values to be (ab)used that way. So I consider these tests to be truly excellent, for identifying intellectual limits. Related: The Nine Circles of Scientific Hell.
On the subject of pseudoscience – appalling to see recent and ongoing unscientific gaffes by people who should know better, e.g., Neil deGrasse Tyson about GMOs. In great contrast, antidotally, I’d point toward this gem – and please read it at least twice: Die, selfish gene, die by David Dobbs. So glad to see Dawkins getting served … #finally
Neil, Richard: just because you’re published, doesn’t mean you’ve become superior thinkers. Please keep your day jobs, respectively #thankyouverymuch
Speaking of ’splaining to do… another gem is Visualization Explanations @Setosa. Extra points if you grok the callback in their name.
If some of these items are what you’ve been talking about recently, you may just be a Data Science instructor … or interested in becoming one? Check out Become a Data Science instructor @ Galvanize (Seattle, SF, Boulder)
Spark
También se recomienda: una excelente introducción en español a la Scala y Apache Spark por Isra Gaytan. The Latin World has been busting a move lately on Spark, #justsayin
 |
| A Coruña – from Playa de Oza |
Adatao published an excellent article about how they anticipated the inflection point for Spark adoption. Check out the cost curves. Brilliant.
Big news from AMPLab: Keystone.ML is released as open source, to make the process of constructing complicated machine learning pipelines easier. Good stuff from the eponymous Evan Sparks, et al.
I’ve got some graph analytics talks coming up… and was excited to see a streaming/incremental SSSP impl for Spark.
Also, speaking of the Apache Spark Developer Certificate, we’ve got another new neighbor: ORM + DataStax partner on C* cert. Collect all three!
Meanwhile, the big BIG news is Spark Summit 2015 coming up next month in SF. Use the discount code SparkSummitPC25 for 25% off registration. Not retroactive, but nice try :) Followed by Spark Summit EU in Amsterdam, this autumn. Spark it up!
IoT
Solid Conference is coming up again soon! Highly recommended. As an appetizer, check out this excellent article by Cameron Turner @The Data Guild: Caltrain Quantified - An Exploration in IoT which we could hear in our previous backyard every morning starting at about zero-dark-thirty. Now that former backyard has become the new GoogleX building, and SciFi tech experiments compete with the trains for attention.
For IoT in practice, I’m totally stoked to see: Surfers on acid… What an excellent application. And, culturally not far off that mark, here’s an interesting take on marine plastic: Net+Positiva.
Ag + Data
I’ve really been enjoying Biocoder News in quarterly installments, some of my favorite new articles in the world. Period.
On that note, I’m thoroughly ecstatic to announce that I’m moving to O’Reilly Media full-time. Even so, I'll stay involved with Spark and Databricks, assisting on Spark Summit, etc. We’re moving the family to a tiny farm, an old apple orchard that really needs some tending. Perfect as a research station for Ag+Data.
 |
| The Tiny Farm – redwoods 30m tall, planted 65 yrs ago by previous owner |
In highly related news, check out How to Grow a Forest Really, Really Fast, about fantastic work by Shubhendu Sharma. I’m eager to try this out.
One of the top intellects of the early 21st century, Paul Stamets, had some excellent coverage: He Holds The Patent That Could DESTROY Monsanto And Change The World! See also: BioMason and Ecovative Fungi, FTW – and mycorrhiza in particular, as Mohamed Hijri explains quite succinctly.
Ag-related tech approaches in SV have become largely derailed by asinine priorities dictated by Monsanto – more about taking over hedge funds on commodity trading globally, than about feeding anyone. Perhaps the best analysis that I’ve read recently – and certainly one of the best books that I’ve ready recently – is the highly recommended The Third Plate by Dan Barber. I learned about that via Gastropod – where Cynthia Graber and Nicola Twilley consider food through the lens of science and history. Brilliant.
There’s been a terrible drought / water crisis in Brazil – largely exacerbated by transnational corporate interests. This was weighing on my mind as our flight landed in São Paulo. I got to speak with friends there who are working on Ag+Data analysis, very good to see.
As predicted: Finance is driving California water into the dust… take a moment to consider the jump in almond production versus the temporally co-located jump in variance for snow pack levels. That’s the tip of the iceberg for the near-term shape of major political battles brewing in California. To wit, some of our local mafia have become known under the more apt monicker of Oligarch Valley. While Fox News, et al., promotes the Israeli approach of desalinization at scale, many people who can actually think for themselves question the impact of that approach, and recognize what an utter environmental disaster it could produce. This is not an area of judgment where one gets to call #oops as an excuse, regardless of which side the local mafia may be taking.
Industry Insights
“Software eats the world” is a catchphrase used by A16z. While I slightly agree with the title from this Datanami article, How Machine Learning Is Eating the Software World, its conclusions are pretty much the opposite of what we’ve observed with Apache Spark use cases in the field. Don't get me wrong – Reynold is a good friend, and IMO one of the most talented people working in distributed systems today. However, I have a hunch that the reporter munged the line.
Two key reasons why organizations adopt cloud-based notebooks are (1) to reduce their need for DevOps people to run clusters; and (2) to reduce the need for programmers to assist business people with queries for insights Big Data. Done and done. In other words, domain experts trump all in Data Science applications, while application developers (in relatively large supply, but relatively expensive) and expert systems engineers (in relatively short supply, extremely expensive) both become less of an existential bottleneck for new ventures. I’ll let you do the math on that one.
Some of the themes that I’ve been researching and illustrating over recent years include: Functional Programming for Big Data, Approximation Algorithms, Tensor Factorization, etc. Recognize that each of these point toward less emphasis on developers leveraging APIs, and meanwhile more emphasis on domain experts leveraging simple-to-use frameworks. That’s the bottom line of Apache Spark. Meanwhile, I have no doubt that A16z will continue to rake in loads of money – some of their partners are well-connected billionaires – just perhaps not as a consequence of their thesis. That ship is already sailing. Off, perhaps, toward the oh-not-so Great Pacific Garbage Patch.
Diversity
Speaking of VCs in SV… TechCrunch analysis recently found that female founders nearly doubled in 5 years. Par example, check out the recent Women in Data: Their Work and Achievements.
Meanwhile, I thoroughly enjoyed this gem by Karin Rubin: How women are conquering SP500… My feelings about the overall ethics of algorithmic trading are arguably mixed. However, if it’s going to happen, why not guide it based on diversity, since that demonstrates a #winning strategy?
Fun Stuff, friends in the news…
Check out Lumo Interactive Projector by Meghan Athavale and crew. It’s an interactive floor projector, transforming a floor into games that kiddos can design themselves.
Also, this bit Our Coming Robot Overlords about David, Amanda, and Zeno Hanson – friends back in Texas.
And, what William Barker called one of his most honest interviews, ever.
Upcoming Events
- 2015–06–04: Graph Analytics in Spark @BloomReach, Mountain View
- 2015–06–07: Apache Spark: Intro Workshop for Developers with Andy Petrella, @Scala Days, Amsterdam
- 2015–06–08: Data Science at scale and graph analysis with Dean Wampler, Amsterdam Spark Meetup
- 2015–06–09: GraphX: Graph analytics for insights about developer communities, @Scala Days, Amsterdam
- 2015–06–15: Spark Summit 2015, SF
- 2015–06–15: Apache Spark Community Event, @GalvanizeU SF
- 2015–07–23: Microservices, containers, and machine learning, @OSCON, PDX
- 2015–08–03: ICT Labs Summer School, @SICS Stockholm
- 2015–08–14: Scala By The Bay, SF
- 2015–09–30: Strata + Hadoop World NY, NYC
Will just leave you with…
This article. Wonderful, on so many levels.
