Why Research and Product development should not be separated

Today, I am going to write about research vs product development in Software/internet media industries. And I am going to start from an examination of the current favorite company-Google.

When people write about google, they always write about how Sergey Brin and Larry Page started their company based on an idea, as if Page ranking in itself makes Google the best search engine on the planet. From another viewpoint, they talk about how Google redefined the advertising industry.

I am a Software engineer and the kind of person who wants to understand how Larry and Sergey made their jump from having a clever algorithm up their sleeves, to making it massively scalable and recruiting other great engineers. If you open the hood of Google, and take away all the applications such as GMail, GMaps etc, what you will see is a beautiful, fault-tolerant distrivuted computing architecture built by some great engineers. The group who built this is called the Systems Infrastrucure group inside Google. This is the group that built MapReduce, Google File System, Big Table, SawZall and a bunch of other stuff. This is the layer on which, a number of other google engineers write their applications.

You will see that a number of these papers are written by people from HP's Palo Alto Labs.
Sanjay Ghemawat, Jeffrey Dean, Alan Eustace. In fact, a number of top people in the early days of Google came from HP labs(which was Compaq labs before HP acquired Compaq). And which was DEC labs before Compaq acquired DEC.

Larry and Sergey may be smart but I always wonder how they could recruit so many parallel computing, distributed computing folks in their early days. Imagine you are the founder(s) of a small startup, with no revenue plan in 1999. How would you convince these researchers that they should work for you?

I can only speculate. But I think that the fact that DEC labs had no importance in the eyes of the HP management definitely helped. I can't help noticing that 1999 is also the year , when Carly Fiorina go the top job at HP. So many a top brains at HP labs must have preferred working at Google to a research lab of no importance to HP(Carly was a sales person afterall..).

I think this story has been repeated a number of times before. Successful companies establish research arms to solve their problem of being bogged down by the success of their existing products(The Innovators dilemma) . As time goes by, your product development organization becomes (a) dumber and dumber and bloated and misses the next big thing(b) you research arm becomes frustrated because of your inability to turn any breakthrough into successful products.

Then these researchers take their talent somewhere to solve real problems.