Ceteri

big data meets big quandary

2011-02-06T03:32:00.007-06:00

This past week was the initial Strata conference here in Silicon Valley. While I did get invites to preliminary planning, I did not get to attend the conf itself, due to focus on other important responsibilities last week. On one hand, I missed the opportunity to pay $1095 to hear Greenplum sales reps babble, or witness O'Reilly seek to define a field which many of us have worked hard to define

sample src + data for getting started on hadoop

2010-08-05T10:09:00.003-05:00

Monday, 19 July 2010... I had a wonderful opportunity to present at the Silicon Valley Cloud Computing meetup, on the topic "Getting Started on Hadoop".The talk showed examples of Hadoop Streaming, based on Python scripts  running on the AWS Elastic MapReduce service.We started with a brief history of MapReduce, including the concepts leading up to the framework as well as open source projects

tech start-up self-eval

2010-07-15T14:42:00.020-05:00

There have been some excellent discussions on Quora recently about how to structure start-ups, how to create the right kind of workplace cultures, best practices, anti-patterns, etc. All the better since Quora draws such an experienced, articulate audience, and makes highly effective use of their [topics, questions, follows, answers, comments, votes] format in a social network context. About that

you're still using cassandra?

2010-07-04T14:10:00.028-05:00

I have a hard time understanding why so many projects seem to be adopting use of Cassandra for key-value stores. It fits some, but probably not many. I'll share a hunch about why that may be the case... Email search may have been an original use case at Facebook, strictly speaking, but arguably (and I'm happy to crawl out on a limb here) Cassandra seems to fall into a more general category of

10 years ago

2010-01-01T01:30:00.003-06:00

Ten years ago today...I'd recently cashed out on a tech start-up in Vancouver and remodeled my purple house in Austin. It featured a gorgeous center-island kitchen, which I'd designed using 3D modeling software. All primed and ready for dinner parties. Those would come, but later.A friend was running the wine section of Central Market, so I leveraged his expert advice to invest in a

ایران سبز

2009-06-27T13:27:00.003-05:00

I love how there is a forest in Deylaman which resembles Muir Woods. A connection resonates there, between a poignant Green symbol of the Bay Area/California/Pacific Coast, where the United Nations charter was originally signed, and a vision of Green Iran.

counting bodies like sheep to the rhythm of the war drums

2009-06-19T23:15:00.034-05:00

Perspectives of a second generation Iranian —I am no expert. However, some things seem clear...As much as I'd wish there to be change, why must a tinderbox scenario in Iran prevail? It promises a bloodbath, a weakness for foreign opportunists to exploit, an enormous loss of the young and educated, another reason for neocons to beat their chests and fuel their dreams, counting bodies like sheep to

upcoming talks

2009-06-03T23:58:00.006-05:00

I will be speaking at three events over the next two weeks. These talks will focus on how ShareThis leverages AWS and MapReduce, based on Cascading and AsterData as vendors, plus several open source projects including Bixo and Katta.Tue 2009-06-09, 7:00pm-9:00pmScaleCamphttp://scalecamp.eventbrite.com/Santa Clara Marriott2700 Mission College BlvdSanta Clara, CA 95054Will discuss "How ShareThis

westward ho

2009-02-10T22:39:00.007-06:00

Some friends of mine live in the Westport area of Kansas City, across the street from a marker which notes the Oregon Trail, the Santa Fe Trail, and the California Trail. For many people in North America during the 19th century, their resolve to migrate west was tested there.View Larger MapSo it is for us, with two little girls ready to travel. Our oldest is eager to live in hotels again, because

election visualizations

2008-11-03T14:19:00.002-06:00

I use the following tools to watch the 2008 US presidential election take shape. These tools show great applications of geographic analytics, prediction markets, data visualization, etc.CNN Electoral Map CalculatorFiveThirtyEight Electoral ProjectionsRealClearPolitics Electoral CollegeCS Monitor Patchwork NationElection Night TimetableBetfair Next US President OddsMy one wish would be to see an

ceteris paribus and the basted egg heuristic

2008-10-15T07:48:00.003-05:00

Years ago, I had a job working as a short-order cook in a reasonably nice restaurant. One thing you learn quickly in that kind of work is what you can cook simply, reliably, while handling N other orders at the same time. Perhaps this falls into the general category of load-balancing or speculative execution, but some orders work better than others.An example of a simple order is a "basted egg".

cloud computing - methodology notes

2008-09-28T00:09:00.013-05:00

A couple companies ago, one of my mentors -- Jack Olson, author of Data Quality -- taught us team leaders to follow a formula for sizing software development groups. Of course this is simply a guidance, but it makes sense:9:3:1 for dev/test/docIn other words, a 3:1 ratio of developers to testers, and then a 9:1 ratio of developers to technical writers. Also figure in how a group that size (13)

hadoop in the cloud - patterns for automation

2008-08-26T18:29:00.004-05:00

A very good use for Hadoop is to manage large "batch" jobs, such as log file analysis. A very good use for elastic server resources -- such as Amazon EC2, GoGrid, AppNexus, Flexiscale, etc. -- is to handle large resource needs which are periodic but not necessarily 24/7 requirements, such as large batch jobs.Running a framework like Hadoop on elastic server resources seems natural. Fire up a few

shared memory: humdog

2008-08-13T21:33:00.006-05:00

News came today that a long-time friend had died over the weekend: Carmen Hermosillo, aka "Humdog". I am very sad to hear. She is dearly missed.Humdog generally kept Skype running on her computer all the time, which now shows "last seen on 2008-08-12 11:18 GMT-7". Someone must have turned off her computer just then.Thinking back, it must have been about 1991 when I first "met" Carmen online, on

a methodology for cloud computing architecture - part 5

2008-07-08T19:22:00.005-05:00

Diagram, redux.Errata: fixed issues regarding 3Tera and CohesiveFT, both of which deserved better positioning in the stack than my cloudy thoughts had heretofore imagined.Again, many thanks to vendors who are contacting me, and kudos in general to discussions on the "cloud-computing" group.One thing jumps out from this diagram now: items 6 and 12 are placeholders. What vendors or projects might

a methodology for cloud computing architecture - part 4

2008-07-07T22:25:00.006-05:00

[Ed: the diagram shown below has been updated; see previous diagram and the discussion update.]The diagram won't go away -This time around, I'd like to take a few notes from both the Globus VWS and the GoGrid / Flexiscale discussions on the "cloud-computing" email list...Suppose you just happened to have a bunch of hardware, and wanted to leverage that as a grid – in other words, as "disposable

a methodology for cloud computing architecture - part 3

2008-07-05T22:06:00.010-05:00

[Ed: the diagram shown below has been updated; see previous diagram and the discussion update.]One more iteration on that diagram.This time we'll represent some structured approaches running atop a Hadoop cluster. Parallel R being one that addresses our department's needs the most directly...FWIW, I'm more interested in what can be accomplished with batch jobs and non-relational database

a methodology for cloud computing architecture - part 2

2008-06-30T23:08:00.009-05:00

[Ed: the diagram shown below has been updated; see previous diagram and the discussion update.]Thanks to many people who gave their feedback – publicly and privately – about the diagram in my previous post. Apologies to RightScale and others for inverting their placement. I definitely want to recommend John Willis for an excellent overview of the vendors in this space.Will keep attempting to

a methodology for cloud computing architecture

2008-06-29T10:22:00.008-05:00

[Ed: the diagram shown below has been updated; see previous diagram and the discussion update.]I got asked to join to a system architecture review recently. The subject was a plan for data center redundancy. The organization had one data center, and included some relatively big iron:a large storage system appliance, with high capex (its name starts with an "N", let you guess)a large RDBMS with

hadoop, part 3: graph-based nlp

2008-03-12T21:17:00.000-05:00

I've been evaluating graph-based methods for language analytics. At my previous employer, we used statistical parsing and non-parametric methods as a basis for semantic analysis of texts. In other words, use a web crawler to grab some text content from a web page, run it through a natural language parser (after a wee bit o' clean-up), then use the resulting parse trees to extract meaning from the

first day at the new job

2008-02-22T22:31:00.001-06:00

As some of my friends have recently noticed (from profiles on LinkedIn, etc.) I've got a new employer.Of course, many folks have asked me, "How does one get hired by S.P.E.C.T.R.E., anyway?" Well, it's a long, involved process – lemme tell you. I actually got hired years ago, and have been moonlighting there on the sly. Hey, it happens. Silicon Valley was built out of that kind of "

hadoop, part 2: jyte cred graph

2008-02-20T13:35:00.000-06:00

Following from Part 1 in the series, this article takes a look at an initial application in Hadoop, using a simplified PageRank algorithm with graph data from jyte.com to illustrate a gentle introduction to distributed computing. The really cool part is that this runs quickly on my MacBook laptop, the same Java code can run and scale without modification on Amazon AWS, on an IBM mainframe, and (

hadoop, part 1: approach

2008-02-20T02:19:00.000-06:00

Recently I started putting together a collection of code examples based on the Hadoop open source project at Apache. These examples are available as open source on Google Code at:http://code.google.com/p/ceteri-mapred/This project is intended to explore applications of MapReduce programming techniques for handling large data sets within a distributed computing framework. It also presents more

which candidate "gets" open source?

2008-02-08T13:19:00.000-06:00

opensville

a sample chart

2008-02-07T17:52:00.000-06:00

Been wanting to try out the recently released Google Chart API. It looks great for implementing things on web pages like Tufte's sparklines.So here is a sample chart, with my respective ethnicity. Our family is a mixed bag, part which looks like a cross-section of early North America, and part looks like a cross-section of Central Asia, sampled right along the Silk Route. These things

predictions for 2008

2008-01-02T17:27:00.000-06:00

Happy New Years! Here are my predictions for 2008. We'll see how well these obtain by year-end.1. Spock.com becomes a tech-sector headline.Spock.com positions itself in the "ecology" of popular Web 2.0 site usage in-between Wikipedia and LinkedIn. This makes it an attraction for Google -- either to acquire or destroy, to preserve its position in contextual advertising -- which in turn pushes

jarad on vh1

2007-12-16T19:26:00.000-06:00

My nephew Jarad DeRochey makes a cameo in Gotti's Way, episode 4, beginning at about 01:41 -Hint: visualize my daughter Laurel (below) as a UFC fighter.

wash warm, tumble dry

2007-12-06T00:36:00.000-06:00

Some time ago, I became fascinated by the use of sentence fragments and fractional grammar in English. One observes this especially on product packaging in the US.For example, consider the statement:shake before openingWTF!?! Does that indicate that one must tremble prior to unfolding? Just what do all those fragments tell us?On a related note... My first laptop was a Mac PB180, which I bought

less war

2007-12-04T22:24:00.000-06:00

Stratfor has an excellent series of stories right about the recent NIE.Read it and weep, Neocons. There are only two political groups in the world now which promote the notion of Iran stepping across (arbitrary) lines in the sand regarding non-proliferation. Dick Cheney leads one of those groups. An interested reader can identify the other from here in about two clicks. May the ghosts of all

generations

2007-09-28T01:35:00.000-05:00

My mom, Cherie - circa 1942, in Rifle, CO. My daughter, Laurel - circa 2007, in Avila Beach, CA.Their visual resemblance at that age feels uncanny to me. Laurel has a wee bit o' the Persian side of our family asserting, more like my brother and sister do.Sixty five years. Thirteen hundred kilometers. Yours truly, gratefully in-between.Laurel's photo by Flo Leibert

the western lands

2007-09-10T03:01:00.000-05:00

Out on the Western Slope of Colorado, great mountains spill their bowels into a vast desert. Standing at one point along the Colorado River, one's view up the mountain is filled with snow-capped peaks, while downriver the view opens into a desert scene of mesa, scattered now with vineyards emerging out of near-Martian dust. Arriving out of France, out of Canada, out of Iowa, my people settled

schwa meets darpa

2007-02-22T17:01:00.000-06:00

Engadget ran a story today entitled "US Army to arm UAVs with xenon-based paralysis inducer". The article describes that technology as a "non-lethal crowd-control device".Wikipedia's entry for Schwa describes it as the "underground conceptual artwork" of Bill Barker. A true original among the popular alien image fetish / conspiracy theory "parody" themes of the 1990s. One of Bill's original

a headcase blog

2007-02-14T20:23:00.000-06:00

HeadCase has been moving steadily from stealth to not-so-much-stealth to "Hey, over here!" mode, right on schedule.I'd like to point out that we have a public blog now, featuring articles from my close friend and business partner, Mark Stephan Meadows, a.k.a. Pighed.For those who've visited the Purple House, and experienced an abruptly provocative canvas near an abnormally long table, that's who

spinning iran

2007-02-14T14:54:00.001-06:00

If you read just one article today, do just one thing to understand the world a little better, I recommend that it be to read Spinning Iran on Rafe Colburn's blog.Bush's rhetoric about "War with Iran" is more neocon (read: pro-Israeli lobby) nonsense. The Newsweek "photo gallery" is like a page out of 1984. US diplomats are negotiating with Iranian diplomats now, brokered via Russia and Qatar.

ogoglio open

2007-02-13T10:28:00.000-06:00

A quick note that Ogoglio, which is exploring the "new business district" of Metaverse, has uploaded an open source release.This is one to watch. In fact, watch the screencasts which Trevor is publishing about progress on Ogoglio on a fairly regular basis.A term has emerged in the Ogoglio notes: "distributed self". That seems quite succinct. In recent discusssions among HeadCase and friends,

don't mess with soccer moms

2007-02-02T00:35:00.000-06:00

So the lesson for "Big Box" stores and Corporate America is that if you pay off a corrupt city manager to bypass public reviews, and then threaten the elected members of the city council with litigation to keep them quiet, you can in fact proceed to build a 220,085 s.f. superstore operating 24/7 in the middle of an otherwise quiet, low-crime, residential area. In fact, you may get the public

attack at cia headquarters

2007-01-28T13:27:00.000-06:00

I can neither confirm nor deny that members of our party were in any way responsible for several unfortunate events which occurred last Friday evening at the Culinary Institute of America (California Branch) in St Helena. Because I plead the Fifth, my comments may not be construed to support any of the alleged acts, including but not limited to: (a) the disappearance of a wrought-iron rooster

01/11 - rip dear raw

2007-01-17T21:50:00.000-06:00

won't shed nary a tearknowing that you're laughingfrom the other side.http://robertantonwilson.blogspot.com/2007/01/raw-essence.htmlhttp://www2.blogger.com/profile/06504517528789477168http://subgenius.com//bigfist/pics13/RAW/index.htmlyour not-so-gentle prodding as "mark cruise" on our FringeWare email list, and those $23 worth of margaritas at the airport may be dearly remembered.

bay-nays foster

2007-01-11T16:15:00.000-06:00

My daughters call bananas "bay-nays". This recipe, with which I've been experimenting since 1996, is approximately as old as my first Java code. It's also the only dish that I've ever served to (otherwise seemingly civil) adults which caused them to fight and steal at a dinner party ;) Be sure to use dark chocolate and light, fresh flour ... and in the name of gopod, prepare enough to avoid the

wal-mart's peanut butter manifesto and "Slavery 2.0"

2007-01-04T08:35:00.000-06:00

Is the world's largest multinational corporation, Wal-Mart, on the decline? Or is it simply riding the political ebb and flow as it tracks a neoconservative agenda to undermine people of color and drive a wedge through the Democratic Party?Interesting news comes today that a judge in Chelan, WA has declared the building permit illegal for a new Wal-Mart "supercenter" being constructed there.The

a rhetorical answer

2006-12-29T15:15:00.000-06:00

Five things you didn't know about me:I'm Eurasian. My DNA analysis shows up as about 90% Caucasian and the rest is Far East Asian. Half of my family comes from Central Asia, where a guy named Timur-e-Lang had much to do with our blend. One thing about mixed race is that different physical features assert at different times as a child grows. When I was about 5-7 years old, more Asian features

bateriophage

2006-12-25T22:15:00.000-06:00

Couldn't resist making of note of "T4bateriophage" (image) by Wes Unruh from Key23, etc. He was the author of Metamagical Grafitti, which I enjoyed thoroughly. Thanks for the cites, Wu, from my Corporate Metabolism series. Excellent work on formulations applying sublation. I'm particularly fond of track 15, "Infected Nativity", on the related work DrOwning Street Memo by the Philip K Nixon

three laws of avatarics

2006-12-20T13:40:00.000-06:00

Susan Wu, an associate at CRV, wrote an excellent critique recently in the article Second Life: Incredible innovator, but probably not sustainable:Their high technical barriers to participation and the fact that SL is a closed standards system ultimately deters them from reaching mass market adoption.She followed a few days later with an example from the other end of the spectrum:APIs, like open

have a sale on diapers and raise the price of beer

2006-11-19T13:04:00.000-06:00

In the wake of successful Internet business models pushed by Google, Amazon, O'Reilly, - and really just anybody with a clue - to paraphrase Weird Al, "It's all about the dataset."One of my professors in grad school, Jeff Ullman, introduced us to a delightful form of torture known as Data Structures and Algorithms - as well as Database Theory, which previewed a little known, upcoming new standard

night at the movies

2006-11-08T10:57:00.000-06:00

Last night our family celebrated a sacred and rather nasty American tradition of consuming leftover Halloween candy while watching election returns. For a few brief hours, CNN eclipsed the role of HBO as the premium movie channel on cable television.We got to watch Arnold Schwarzenegger rake in his best-ever box office returns for a sequel, followed by a startlingly impressive trailer for the

sufi coffee shop

2006-11-07T14:13:00.000-06:00

On a recent trip to Palo Alto, I had missed nearly a full day and a half without an actual meal. There were too many hassles enroute and then lots of exciting work to discuss, once I got on the ground.By the time I had a chance to call home and then get a little food, not much around my hotel was open (or looked any good). I wandered, frustrated and hungry. Amazingly, here in the heart of

aws rocks the free world

2006-11-05T11:41:00.000-06:00

Amazon Web Services (AWS) is perhaps one of the most fundamental and far-reaching changes in the computer industry ... since perhaps when, recently, SecondLife hit the cover of BusinessWeek magazine. In fact, this news will be the cover story of BusinessWeek next week.I knew about a friend working on this project a couple of years ago, and helped a little (on the sly) with documenation. So when

in praise of seitan

2006-11-05T11:40:00.001-06:00

Recently my wife asked me how to cook eggplant. First, we had received some nice egg-shaped eggplant with our Greenling delivery last week. Secondly, there's a popular eggplant dish in Persian cuisine which I like to fix for the family, and Erin was perhaps hinting :)The Persian word for eggplant is "bademjan". I remember a similar word used to describe a popular eggplant salad in Russia -

hillary disgusts me

2006-10-30T16:48:00.000-06:00

I really cannot understand how anyone could support Hillary Clinton in politics and still have a clear conscience.This disgusts me.It would be difficult to identify a single entity which has been more damaging - politically, socially, economically, ethically, or morally - than Wal-Mart.That is not even touching on her close association with HMOs.I hope that Howie Hawkins wins the Senate race in

greenling

2006-10-30T15:57:00.000-06:00

I prefer my food slow, local, and organic.Eating at restaurants too often - in other words, more than a few times per week - just does not feel wholesome to me. Eating fast food often at all just feels wrong.For one thing, I cannot stand the size of portions which are typical at American restaurants. It's absurd. A friend of mine in Japan remarked that it's great when he visits because he can

buzz about headcase

2006-10-24T11:17:00.000-05:00

WSJ recently ran an article about HeadCase (subscription only). Nice to see the press.Customer-service systems are another possibility, with avatars that could answer questions online -- and know when to give way to a human. "What if the system was smart enough to sense you swearing at it and brought on a human immediately?" Ms. Cox says.Our investors are listed, both True Ventures and Burnt

congrads, anousheh ansari

2006-09-25T20:50:00.000-05:00

Congrads to Anousheh Ansari, first female private explorer in space, and first astronaut of Iranian descent (not to mention, a Trustee of Ansari X Prize, etc.)http://googleblog.blogspot.com/2006/09/our-call-from-space.html

hairstyle

2006-08-31T23:44:00.000-05:00

got a new hairstyle... rarely been thanked like that before by my hairstylist afterwards for encouraging gratuitous experimentation.it's my old hairstyle, circa 1994, just after testifying in the Motorola trial finished up... back when FringeWare was still a little corner of Europa Books in Austin.it's got this "fallen mohawk" thing going on, long on top, shaved up on back and sides. kinda like

iran, leading the blogosphere

2006-08-13T22:28:00.000-05:00

Oh, this is amazing...Mahmoud Ahmadinejad, president of Iran, has a blog.As my wife pointed out over my shoulder, now the real political leaders of the world not only have speech-writers, they must have blog-writers.My paternal family name notwithstanding, I'm no fan of the Imams. Even so, their chess game played on the entire world over the past few decades (during all of my adult life, at least

unstealthify

2006-08-10T00:46:00.000-05:00

HeadCase Humanufacturing, Inc., has aweb siteanemail addressand of course agooglism. tags: interactive narrative, secondlife

bedtime story

2006-07-01T01:20:00.000-05:00

When asked to make up a bedtime story, my 2.5 year old daughter composed and recited the following:One on a timeSomething, somethingAll by myself....then giggled profusely - as a full body endeavor - in the way that only a toddler can.I am amazed.

have software agents declined?

2006-06-27T23:23:00.000-05:00

Have efforts toward developing effective software agents declined?During the 1990s, it seemed as if "personal assistants", "intelligent agents", "bot", - whatever you want to call them - were poised to take over the Internet.For what it's worth, Wikipedia does a fine job of defining what an "agent" is - or rather, what is not an agent.Looking at a reference search site, such as Google Scholar,

first day at the new job

2006-06-16T01:59:00.000-05:00

It was my first full day at the new job. Still in stealth, we're making good progress. As the Technical Director, my day starts out with a quick peek through Gmail. What's come in from our Executive Producer in Australia before she called it a night? There's a blurb from Google Calendar, describing my agenda for Skype meetings throughout the day. Any news from LA while I was alseep? A few