Newsletter Updates for August 2013

Lots of talks, lots conferences, lots of writing. Here are the latest updates about scheduled events, along with pointers to some of the best content that I've been studying lately.

YARN has a Docker-sized hole. Brace yourselves, in this post some friends at [ list the Hadoop vendors, other than MapR ] may get upset. Good friends all, I wish them the best. However, some things must be said. More about that in a moment. (And if I don't surface before Nowruz, please notify my family.)

It's been a busy month: Portland, Austin, Chicago. While we often go out for beers following a talk, Aaron Betik @Nike had the brilliant insight to hold our PDX meetup inside a brewery. Three birds, one stone. Many thanks to Aaron, Nike, Thomas Lockney @JanRain, Todd Johnson @Urban Airship for putting together that event. And to everyone at O'Reilly Media for such a great OSCON!

Girish Kalathagiri and I wrote a paper about ensemble modeling for the first-ever PMML Workshop at KDD 2013 in Chicago. Fun to be on the ground floor, and I look forward to this expanding into a full conference. Many thanks to Bob Grossman, Walt Wells, et al., for their efforts arranging the workshop.

Mike Zeller from Zementis explained about a big spike in industry usage over the past year. PMML has become a pervasive standard for describing analytics workflows. I was particularly impressed with solutions from Augustus for Python, and with the recent updates to Knime – both presented at our workshop. In the broader scope of Enterprise data workflows, I've been learning much about the Python stack from Continuum Analytics – Anaconda, Wakari, etc. – plus excellent work on IPython Notebook, Pandas, Scikit-Learn, and related projects. Augustus (from Open Data Group) fits well within that context to provide PMML for model creation and model scoring. Especially given Continuum's support for compiling and optimizing apps and attention to low-latency use cases. Similarly, Knime has a brilliant commercial integration in the context of ParAccel by Actian. I'm particularly impressed with Actian's use of Knime and Eclipse to provide a UI for Enterprise data workflows: great UX plus operationalizing. I got to spend some time recently with the teams at Continuum and Actian. Highly recommended.

So, the takeaway is, if you're working with an analytics stack that does not incorporate PMML support as a core feature, run. It's time to switch.

–––

In Austin, I got to meet with directors of the new Business Analytics program at the UT/Austin McCombs School of Business. Brilliant, a mere glance at their syllabus and schedule proved this program to promote the kind of aggressively fast-paced intellect which I crave. Which the industry craves for data scientists, too – in quantity. I'm looking forward to visiting their program again soon.

In Chicago, I got to meet with several of the current fellows in the Data Science for Social Good. Most substantive discussions I've had since Los Angeles about Open Data and its applications in practice. There's a new conference emerging in the midst of those discussions. Looking forward to final projects from this summer's DSSG fellowship!

(cluster computing, à la friends @Hazelcast)

Meanwhile, we have "Intro to Data Science" workshops and talks coming up in Denver, SF, NYC. We'll book-end each trip with office hours and on-site demos for Mesos.

Speaking of the workshops, a few great new titles related to that material, which I recommend:

Analyzing the Analyzers, by Harlan Harris, Sean Murphy, Marck Vaisman; O'Reilly (2013): including an excellent analysis of the skills, experience, and viewpoints of data practitioners in industry.

Mondrian in Action, by William D. Back, Nicholas Goodman, Julian Hyde; Manning (2013): beyond the Pentaho analytics, this team has some of the most comprehensive insights available about Enterprise SQL usage.

Storm Real-Time Processing Cookbook, by Quinton Anderson; Packt (2013): including clear, concise sample apps that integrate the kinds of frameworks we use in production (Storm, Kafka, Cascalog, etc.) and not mere code snippets shown in isolation.

–––

One of the main themes in my workshops and lectures is that Apache Hadoop is almost never used in isolation. However, that perspective has been taken to task by the architects of Hadoop:

"Hadoop is the kernel of a distributed operating system, and all the other components around the kernel are now arriving on the stage."

– Doug Cutting @Cloudera (2011-12-19).

"DistributedShell is the new WordCount."

– Arun Murthy @Hortonworks (2013-08-12).

Check out Arun's slide deck there. Seriously, Hadoop as a service endpoint? Wow, that's like Enterprise Java Beans redux. A kinder, gentler version of EJB. Or something. The notion of having to write 40 lines of Java to execute a Bash command line – now that's impressive! Just wow. I'd pay good money to be in the room when Cloudera and Hortonworks sales teams attempt to pitch this flavor of JVM nonsense to IT execs at Schwab. Their reality bubble leaves me wondering: have Doug or Arun even looked at a modern kernel, i.e., even bothered to notice Linux kernel commits, since their Java coding days back in 2006?

Because I'm quite certain that the people making Docker have. I'm quite certain that the people making OpenVZ have, too. Moreover, my friends who work in Ops for their profession are generally well aware of Docker and OpenVZ. However, YARN? Not so much…

Let's consider a historical trajectory:

• COBOL, circa 1959… DoD accountants trying to tell Ops how to do their job
• EJB, circa 1999… IBM/Sun/Oracle attorneys trying to tell Ops how to do their job
• OpenStack, circa 2010… NASA app developers trying to tell Ops how to do their job

Notice a trend? And now, for the latest contender:

• YARN, circa 2013… Yahoo! data engineers trying to tell Ops how to do their job

In my experiences with real companies – companies with substantial amounts of revenue, that is, not the Silicon Valley definitions – anyone outside of Ops trying to tell then how to do their job better have a CxO in their title. Preferably with a vowel in-between. In other words, keep the dilettantes the #&!% out of the Ops pit, to keep your employer from going out of business.

Meanwhile, what's the commercial reality of Big Data today? Well let's compare and contrast a couple big players in the space:

Actian is profitable, with a base of 10K customers, and north of $150MM annualized revenue. BTW, I'm a big fan of Amazon AWS Redshift, how about you? Runs great in production at scale. Love the economics of that.

Hortonworks recently took a B round, with $70MM in funding to-date. The company currently sells training and support; not clear how long it will take to become profitable. Oh, and their CTO recently left the building.

Hmmm… May need to check with some of my b-school friends about how to compare those fundamentals. Meanwhile, here are a few notes from Google about their experiences and benchmarks working with Linux kernels and very large-scale distributed computing.

–––

My first full-time job as a engineer was at a start-up in Sunnyvale in the summers of 1983-84. Our team ported Unix to a 32-bit minicomputer, and I wrote the sort package. If you've shopped at Ace Hardware or Pep Boys, your transactions probably went through our code. I'm grateful for that experience and learned a bit about operating systems by helping implement part of a popular embedded commercial distro.

At the time, many companies were totally absorbed in COBOL – still a good living back then. Those shops seemed oblivious to the changes underway: Unix minicomputers on the high end, workstations in the mid-range, PCs on the low end (e.g., Apple II running VisiCalc). These would soon wipe out COBOL programming like a forest fire raging through a stand of dry pines.

About that time I attended a seminar by one of my CS profs and his colleagues, describing a start-up based on work at Stanford University Network, aka SUN, which had commercialized a new network-enabled minicomputer. They considered that work transformational. Read: disruptive. They were right. That same year, Steve Jobs gave a standing-room-only lecture on campus about what he'd learned from Xerox PARC, aka the Lisa just prior to the Macintosh. He considered that work transformational. He was right. Meanwhile, a guy named Larry Ellison was busy hiring the bulk of our CS grads for his company, Oracle. You know the rest of the story. We could see big changes ahead circa 1983-84.

Over time, I've noticed how disruptive changes in computer technology tend to happen faster at the hardware and OS level, while the popular programming languages struggle to keep pace. Legacy frameworks encounter even more difficulty. I recall lectures at Stanford CS teaching pretty much the opposite: that it's simpler to evolve new technology at the language and application layer. While that notion may hold power in academia, its narrative conceit is that computer programming languages become tied to culture in industry, and culture has inertia.

That leaves us where we are today with Java vis-a-vis Big Data. For better or for worse, YHOO circa 2006 made a big bet on Java as the principal language for Big Data frameworks. YHOO circa 2006 didn't last, but the frameworks that emerged from it persist. These days the Big Data vendors are thoroughly occupied selling Apache Hadoop to the Global 1000 as a glorious path into a shiny, data-imbued future. In other words, recreating the Global 1000 in the image of YHOO circa 2006. Some excellent work came out of Yahoo! from the mid-/late- 2000s, and I greatly admire my friends who were there and made that happen. Even so, Hadoop is based on work at GOOG circa 2002 – now a few generations behind. Living (literally) with GOOG in my backyard, my neighbors who work there smirk whenever the word "Hadoop" gets mentioned.

When I see talented people who have their heads stuck inside IntelliJ, who cannot think outside of a Java API, it seems sad. It reminds me of those poor souls circa 1983 pouring over COBOL punch cards and teletype output. Sure, there's excellent software written in Java – java.util.concurrent comes to mind immediately.

I'm obviously quite a fan of JVM-based functional programming languages such as Clojure and Scala. However, when the "thought leaders" go around talking about Hadoop as an operating system, re-defining HA/low-latency service endpoints to be based on Hadoop and Java – it's COBOL all over again. Hold on tight.

Hadoop is not an operating system. 

Linux is an operating system. Unix is an operating system which had sophisticated features even sooner – arguably, as of the Linux 3.x kernels the playing field has perhaps become more leveled? Windows is also an operating system, albeit geared toward different usage. When people try to pitch "Hadoop as an operating system", they are trying to sell you snake oil.

The only thing that nonsense will buy the IT industry is even more of a guaranteed revenue stream for people building zero-day exploits in Beijing. Let's just suppose, hypothetically, that you run part of IT at Morningstar or Schwab. Imagine somebody trying to pitch you on JVM for low-latency services and cluster management. Are you going to bet your EVP's bonus on snake oil? Didn't think so.

The lesson is that enormous changes are afoot in terms of multi-core processors, large memory spaces, etc. These changes have huge impact on algorithm work for handling data at scale – without going bankrupt. Hadoop emerged in a day when spinny disks were king, multi-core was rare, and large memory spaces were expensive: that world is gone. Meanwhile, the modern kernels have kept pace with those industry changes. 

So, what is my point? Trying to resolve OS issues in the application layer is almost always a recipe for disaster. Caveat emptor.

I give Hadoop three years before it gets displaced. The lesson of Spark, in my analysis, is that rewriting Hadoop to be 100x better isn't hugely difficult, given the available building blocks for data center computing, based on the modern kernel. Meanwhile, the prognosis for Hadoop? Three years. On the outside.

Many thanks,

Paco

Newsletter Updates for July 2013

Lots of talks, lots conferences, lots of writing. Here are my latest updates about scheduled events, along with pointers to some of the best content that I've been studying lately.

Got to speak at Hadoop Summit last month, about the Pattern project. Great audience, lots of discussion about deploying predictive models at scale on Apache Hadoop clusters. BTW, one of the most compelling talks at Hadoop Summit this year was by Kevin Coogan, founder of AmalgaMood in DC. Kevin discussed their technology that leverages social signals and Open Data in predictive analytics when financial markets are not responding to what might otherwise be called analysts' consensus. Chaos, in other words. The Q&A for that talk in particular was enlightening and compelling: what a great way to scurry VCs up to the audience microphone.

We had several other excellent events recently: Seattle, Santa Clara, Los Angeles -- many thanks to hosts Surf Incubator, White Pages, and Factual. Plus, we had some private brown bags at LinkedIn, MapR, and other firms.  My takeaway: great to meet many amazing people, a wealth of talent, and overall so much dedication to learning about this field. To that point, there's lots of opportunity in Data Science roles and along with that a big need for people who are adept at working across disciplines. People need enough programming background to leverage distributed systems, which enable the compelling use cases. People also need enough quantitative background to leverage the math required for high ROI apps at scale. I find that many people attending the workshops have expertise in one field, and want to augment with the other field -- which is ideal for learning from each other. Upcoming events: Portland, Austin, Chicago. Nike is helping to sponsor the Portland meet up, at Widmers -- so we won't have far to go for beers afterwards. And during. See the calendar at http://liber118.com/pxn/ for more details. BTW, if you want a 20% discount for OSCON, please use OS13FOS for a discount code. If you have a city or venue to suggest for upcoming workshops and talks, please let me know @pacoid

Toward a general thesis for Cluster Computing...

In other news, the Mesos open source project has graduated into a top-level Apache project. I recently took a position as Chief Scientist at a new company related to that project, called Mesosphere, in San Francisco.

There's a general thesis emerging, namely that we run large-scale apps based on cluster computing. Because the data has become too big to fit on one computer anymore. Or, for that matter, the apps have become too complex to be handled by one computer, one person, one model. We require multi-disciplinary teams, leveraging cluster computing. Three areas of technology innovations get applied: Big Data, Data Science, and Cloud Computing. At a high level, applications generally leverage an abstraction layer, such as Cascading. At a low level, the more advanced organizations are leveraging cluster schedulers such as Mesos -- and, arguably, YARN coming along too. For an excellent overview, see the Wired article by Cade Metz, Return of the Borg: How Twitter Rebuilt Google's Secret Weapon.

I foresee a general trend of smarter clusters, leading into higher ROI on Big Data apps. On the one hand, multi-tenancy in clusters helps balance the utilization curves and cut costs. On the other hand, reducing the "wire tax" of moving data products from batch clusters to web app clusters will help enable new areas of algorithm development, by reducing critical latencies. Companies such as Twitter and Airbnb have both built their tech stacks using these components, Cascading and Mesos. I spot a trend.

My intent is to show sample apps that leverage both layers. Also, one advantage of Mesos is that it manages resources for many different kinds of frameworks and apps: Apache Hadoop, Spark, MPI, Memcached, Nginx, Redis, Ruby on Rails, Python, etc. That's perfect for Big Data use cases that blend multiple frameworks. Stay tuned.

Drilling down into the math...

My lectures tend to emphasize a division between the rigor and formalisms of statistical theory versus the relatively ad-hoc praxis of what we categorize as machine learning. At top schools, grad students in one of those fields receive high salaries and VC funding straight out of school, while grad students in the other... not so much. That's a shame, because mission-critical apps at scale rely on both disciplines. Machine learning allows you to make billion-dollar mistakes, while statistics help you avoid billion-dollar mistakes. Take a look at any good search engine team and you'll see how both disciplines become necessary in practice. Together.

The Abacus Bracelet

Another point is that machine learning approaches are a subcategory within optimization theory. As I'm researching industry use cases for Data Science, Big Data, Cloud Computing, etc., it becomes clear that more emphasis on optimization is crucial for long-term industry evolution. Hanging around seminars at Stanford's Systems Optimization Lab, the math innovations presented and their applications have huge implications for industry. As a case in point, John Deere probably won't be building a Facebook competitor any time soon; however, they must tackle hard problems at enormous scale in optimization. Mathematicians are responding to that demand. Given that 40% of the world's population works directly in agriculture, plus the urgency of global climate change, etc., I tend to find Deere's domain more compelling than yet-another social network, ad network, social game, etc.

Enough soap box. Instead I'd like to recommend an excellent resource in this area, notably Rob Zinkov's blog series Convex Optimized at http://zinkov.com/

My current homework, thanks to Rob, focuses on Alternating Direction Method of Multipliers (ADMM). This builds on the previous theme of exploiting sparsity and matrix factorization atop Hadoop. It also addresses the need for more emphasis on general approaches in optimization theory, and less on the nuances of machine learning algorithms. Much study will be required before my sample apps begin to emerge. Meanwhile here's to broadcast and gather as a more interesting pair of verbs than map and reduce. 

Some well-known Big Data vendors are struggling to market Apache Hadoop as an OS. It's not really. Hadoop may be a distributed file system plus some distributed computing, but calling it an operating system would be a great way to fail a CS midterm. More to the point, MapReduce as an abstraction is multiple layers removed from the needs of actual workloads, plus it's already 11+ years old. A better focus for these vendors might be to engage Professor Boyd, et al., to build distributed computing frameworks for commodity hardware based on ADMM principles, which support a wide range of commercial ML problems directly.

Doubtful that will happen. For example, that might require (gasp) supporting MPI features! Some engineer would need to prioritize studying math (gasp) in lieu of lobbying for 150 new commits on an Apache project! People could eventually recognize commercially interesting problems in Enterprise IT which are (gasp) not readily expressed as SQL! No, that won't happen. The Global 1000 is far too busy tooling up on Hadoop. Instead, I'd bet money on Anaconda, Spark, Titan, GraphLab, etc., leap-frogging the Hadoop-centric segment of the industry -- once the world beyond Silicon Valley wakes up to the realities of math emerging circa 2010. YMMV.

BTW, I must apologize, but it's become impossible to keep pace with email. While traveling, I find that Twitter works best for what must get said quickly. Semipublicly. I'll check Twitter often -- but email infrequently.

Many thanks,

Paco

Newsletter Updates for June 2013

Lots of talks, lots conferences, lots of writing. Here are my latest updates about scheduled events, along with pointers to some of the best content that I've been studying lately.

First, the Austin trip was awesome. Many thanks to all who attended the events -- and especially to those who participated in the workshop for excellent feedback. We've learned much from your suggestions, to use in the upcoming workshops.

Hadoop Summit is next week. I'm looking forward to catching up with friends who will be in town or (for those already in town) stepping out of the cubicles :) Come check out our talk about the Pattern project for PMML in Cascading. Other events are listed below for meetups and workshops in Seattle, Santa Clara, Los Angeles, followed by OSCON in PDX. Please tell your friends in those cities. I also have an "official" home page now with a newsletter sign-up, event calendar, links, etc., at: http://liber118.com/pxn/

An update on the O'Reilly book about Cascading: it will now be in print  July 22 -- just in time for OSCON. Speaking of which, we should really plan for a meetup, drinkup, birds-of-a-feather, or something in PDX.

In other news, Cascalog also has a new home page at http://cascalog.org/  Paul, Soren, Sam, Bruce, et al., nice work! This is a much needed resource for the developer community.

Now for a few "extras" in the newsletter... I got to attend a Stanford talk earlier this month by Tim Davis, from University of Florida. For the second time in one week, I heard the phrase: "There's no such thing as RAM anymore." That will become a theme for the architecture of algorithms on distributed systems. Following up on the sparse matrix techniques mentioned in my previous newsletter, this lecture was about Sparse Cholesky update/downdate, LU factorization, QR factorization, software architectures based on GPUs for HPC parallel processing, etc. In other words, even if his name doesn't quite ring a bell, you've probably used his software daily: core libraries for linear algebra optimization in R, Matlab, Mathematica, plus some of the core algorithms for Google Street View, 3D Earth, as well as many of the Verilog vendors. Professor Davis curates an extensive collection of sparse matrices, which I highly recommend. Not only are these visualization beautiful, but the examples represent important edge cases for sparse matrix factorization, used to evaluate new work on algorithms. Also note the museum exhibit coming up in October.

Speaking of sparse matrix factorization and "No such thing as RAM," I thoroughly enjoyed at day at National Instruments in Austin, giving a talk about Big Data trends, and also learning about machine learning techniques at microsecond speeds on FPGAs. There is a form of convergence afoot in the industry, between the technology pyrotechnics of NI and other firms working on sensor arrays for the "Internet of Things", and what we've been doing with large-scale cluster computing. Check out what NI has to say about the practice of Big Analog Data™ -- I have yet to see people not drop their jaws reading those stats.  Oddly enough, some of the most important techniques for machine learning algorithms at microsecond speeds have familiar cousins at petabyte scale, so I have a hunch there are many opportunities ahead based on this area of convergence.

Speaking of "Internet of Things" and real-world data, one of the most astounding projects that I've encountered in a long while is Protei. This is a truly innovative data platform: a multi-hull drone sail boat, built much like an eel, which changes how we clean up marine oil spills, recover "islands" of plastic waste, collect vital data from the radioactive waters off the coast from Fukushima, etc.  For another amazing innovation, Paragon Science received well-deserved press in the article, "Doctors and Social Oncology: The MDs most mentioned by their peers (breast cancer edition)".  I got to speak with Dr. Steve Kramer in Austin, and I'm quite impressed by capabilities of this technology for complex graph analysis and visualization.

In other news, I was grateful to attend the recent collaboration among Facebook, Twitter, and LinkedIn for the #Analytics@WebScale conference. We saw Facebook's first public announcement of Presto, a new approach to handling ad-hoc queries at very large scale which is now displacing Hive. Congrads to Martin, David, and teammates on the Presto project -- looking forward to this work becoming released as open source later this year! And in a new twist on the term "cloud computing", Facebook data is getting so large, so complex that they've even begun to experience weather conditions within their data centers :)

Finally, I learned a lot form the GOTO Chicago conference last month. Nathan Marz gave a talk on "Runaway complexity in Big Data systems... and a plan to stop it", with a video released.  Dean Wampler (from the new firm Concurrent Thought) and Amanda Laucher discuss functional programming an another video from the conference. Good stuff!

If you have a city or venue to suggest for upcoming workshops and talks, please let me know @pacoid

Thanks!

Paco