‘Patternization’, Machines and Experts

A scene from the movie ‘Back to the Future – II’
Marty McFly: [Reading the newspaper from 2015] “Within two hours of his arrest, Martin McFly Jr. was tried, convicted and sentenced to fifteen years in the state penitentiary.”? Within two hours?
Doc: The justice system works swiftly in the future now that they’ve abolished all lawyers.

Bill Gates made this prediction in 2014 about low-skilled jobs, “Software substitution, whether it’s for drivers or waiters or nurses … it’s progressing. … Technology over time will reduce demand for jobs, particularly at the lower end of skill set. … 20 years from now, labor demand for lots of skill sets will be substantially lower. I don’t think people have that in their mental model.”

Many people have predicted the eventual taking over of low-skilled jobs by machines in the not-so-distant future. But what about the experts? How safe are they?

The Case of Loan Officers

Loan officers have been an integral part of the lending process of banks. They earlier used their intuition to assess whether a borrower will repay the loan. Then they were guided by tools like the FICO score (credit score) and other metrics like debt-income ratio to make choices. Now they are in danger of being replaced. Completely. Peer-to-Peer marketplaces like Lending Club and Prosper ask applicants to fill information online and then check credit reports and assign a grade with a fixed interest rate. It crunches numbers to quickly make lending decisions. Because it cuts out expensive employees (yes, these loan officers), it can pass those savings to both lenders and borrowers. Apparently, many believe in this vision since LendingClub has/had investments by Google, BlackRock, T. Rowe Price Group etc.

Darin Inc. another online peer-to-peer lender has zero loan officers. A startup called Upstart, founded by ex-Googlers facilitates loans based on your education, area of study and job history and not just based on your FICO score and years of credit. The company positions it as an attractive alternative to millennials. Here again, the loan officer is conspicuous by his absence. Carl Benedict Frey and Michael Osborne predict in a research paper that loan officers have a 98% probability of being completely replaced in the next decade or two.

So, can machines replace other experts like it is replacing loan officers?

There are two main major factors impacting the jobs of experts:
1. Development of Patterns or what I call ‘Patternization’
2. Technology Improvements


In any field, when problems are loosely defined, it takes expertise to solve a problem. For example, some doctors treating cancers are very expensive because they have very high expertise. They formulate hypotheses and test them. A mediocre doctor will fail more often than a highly trained doctor. But over time, invariably, patterns emerge in solving these problems. When these patterns emerge, a mediocre doctor for example, is often as good as the best doctor in the world. Who would perform better in curing strep throat? The performance of a mediocre doctor would be almost the same as that of the best doctor in curing strep throats for which clear medications are established. Similar for any disease for which medicine exists that has been empirically proven to cure most of the time. When this happens, the best doctors move ‘upmarket’ focusing on complex diseases and the small diseases can be treated by nurses and other staff whose wages are lower. This in turn, causes the prices of the treatment to crash.

MinuteClinic has built a thriving business on the above logic. It established the first walk-in-clinic and treats common diseases which can be easily diagnosed. Patients do not wait in long queues. The business enjoys a customer satisfaction rating of about 95%.

The above is an example of ‘Patternization’. Whenever entrepreneurs have found patterns in solutions and have understood them, they can devise solutions that can be implemented at a far lesser cost than when done by experts. To reiterate, at this point, experts move upmarket and focus on more complex problems. For example, the first chip designers were pushed upmarket when software could emulate the patterns of chip designing. The first ‘experts’ who assembled PCs were no longer needed when patterns emerged on how to assemble them reliably every time and how the different parts fit with each other. Many accountants are not needed as before because software from Intuit takes care of it. A lot of companies that helped build websites no longer exist, as WordPress and companies like Wix and Weebly make it easier for people to build good enough websites by themselves.

When patternization occurs, experts move to places where their intuition and expertise is rewarded.

Abhijit Bannerjee and Esther Duflo recount an experiment in their remarkable book, ‘Poor Economics’. A NGO called Pratham ran a program called Balasakhi (children’s friend). The program took the twenty children (from very poor neighborhoods) in each classroom who needed the most help. They then worked with a young woman from the community who is the ‘balasakhi’. These ‘balasakhis’ with barely ten years of schooling have produced far greater results in learning outcomes compared to Government schools with highly paid teachers. Perhaps there is a pattern that produces results in teaching that does not need very expensive teachers to produce certain minimum outcomes.

Technology Improvements – When Technology Patternizes Jobs

Technology improvements often also work on the aforementioned point – develop patterns. But with the huge amount of data being generated, they can develop far more reliable patterns and can do so at a faster rate.

IBM’s Deep Blue could analyze 200 million positions per second and beat World Chess Champion Garry Kasparov. IBM computer Watson beat the top Jeopardy (a TV quiz show) champions Ken Jennings and Brad Rutter. Typical questions in the round were like: “And anytime you feel the pain, hey” this guy “refrain, don’t carry the world upon your shoulders.” Ken Jennings’ response after Watson’s win: “I, for one welcome our new computer overlords”.

Technology improvements do NOT immediately replace experts. They usually augment their outcomes. And they do so by developing better patterns.

Watson for example, has moved on from Jeopardy to working in Industries solving complex challenges along with human beings. It is being deployed in Healthcare, Retail, Finance and the Public Sector. Oncologists at the Memorial Sloan-Kettering Cancer Center are using Watson to provide chronic care and cancer treatment diagnostics. Watson accesses knowledge from 600,000 medical evidence reports, two million pages of text from 42 medical journals and 1.5 million patient records and clinical trials in the field of oncology. Watson then compares each patient’s individual symptoms, family history, genetic makeup, diet, vital signs, exercise routine etc. to diagnose and recommend a patient. In a nutshell, technology augments the work of experts.

Palantir builds software that do not replace experts but instead augments them. The company’s philosophy is that computers using artificial intelligence alone cannot defeat an intelligent adversary. They instead believe that a combination of human experts and computers make a much better team and build breakthrough software with that mental model. They call this ‘Augmented Intelligence’. This argument seems to have gained some support from results in chess. Garry Kasparov who was beaten by a machine introduced a new format called Advanced Chess or Centaur Chess, where teams consisting of both man and computer play against each other. Centaur Teams or ‘Man plus Machine’ teams are said to perform consistently better than just pure chess AI engines. Wired made a pertinent argument from this result: “If AI can help humans become better chess players, it stands to reason that it can help us become better pilots, better doctors, better judges, better teachers”.

Merck in association with Kaggle, organized the ‘Merck Molecular Activity Challenge’. A team from university of Toronto won the challenge. The team worked on data sets describing the chemical structure of thousands of different molecules. They have proved that a deep learning computer could develop its own rules to narrow down the thousands of unique molecules to those with the greatest potential to be effective. This could save thousands of hours of valuable ‘expert’ hours and ensure that their work yields faster results. Drug discoveries can occur faster than it happens now. Government subsidies in drug research can be more efficient and yield world changing outcomes.

Now let your imagination go wild to think about how the same ‘man + machine’ combinations as in Centaur Chess and Palantir can industries like Consulting, Finance, Law, Software Development, Marketing, General Management, Venture Capital etc. The results can be far more impressive and desirable.

Symantec Clearwell’s eDiscovery Platform analyzes and sorts more than 570,000 legal documents in just two days. It uses language analysis to identify general concepts in documents. It is clear that the software can be used for pre-trial research, save months of para-legal work (and replace hundreds of them?) and can augment the capabilities of a lawyer. A reminder that this ‘discovery’ from 570,000 documents can be done by just one lawyer.

In all the above examples, machines have developed better patterns and have produced far better outcomes in ‘collaboration’ with experts.

So, which is it? Are experts going to be replaced or are they simply going to become better and more valuable with the help of machines?

Patterns in Non-Routine Jobs

David Autor, in his paper, ‘The Polarization of Job Opportunities in the US Labor Market’ suggests that routine jobs, be it cognitive or manual are being automated. So experts, by definition since they are doing non-routine jobs, which often involve intuition are safe from being displaced?

In my opinion the ‘replace’ or ‘augment’ question about routine largely depends again on one important factor – have patterns already been identified on how the non-routine work is done. Once patterns emerge and are identified, machines will eventually replace experts in those jobs.

Loan officers are at such a high risk of being displaced simply because their jobs have largely become ‘patternized’. We saw in previous examples on how machines have made deep inroads into typically what you would consider non-routine jobs. Measurement Incorporated which won a Hewlett Prize in 2012 has developed technology which could grade essays (a non-routine job clearly) as good as a human grader.

As more and more non-routine parts of an expert’s job becomes ‘patternized’, the experts will be forced to move ‘upmarket’ and work on other parts of his job where patterns are yet to be discovered. But make no mistake. Machines are gunning for those other non-routine parts too faster than ever.

Machines will start ‘patternizing’ expert jobs faster than ever.

If you are a successful teacher, can the way you teach and motivate students be patternized? If you are a headhunter, can the way good candidates are found be patternized – for example by looking at the online crumbs left by the candidate? If you are a marketer, can the way you allocate resources across brands be patternized?

To summarize and make some inferences/predictions:
1. Experts are needed as long as their jobs are not ‘patternized’. Their work can be done at a lower cost if their jobs are ‘patternized’.
2. Machines are ‘patternizing’ parts of the jobs which experts used to do.
3. In the near future, machines will augment experts’ capabilities that will result in far better outcomes for consumers.
4. But all the while, machines will find patterns to perform a part of an expert’s job making it viable to let him ‘move up’ to focus on more intuitive jobs.
5. Fewer ‘Superstar’ experts will be needed in the future.
6. If you are an entrepreneur, there exists real opportunity in ‘patternization’.
7. If you are an employee, it makes sense to analyze if machines would soon find patterns to do your job good enough. Is there a pattern in playing chess? Is there a pattern in the way you do financial analysis? Is there a pattern in how you arrive at good consulting recommendations? Is there a pattern in how you make good venture capital investments?
8. If you are an employee, it is probably a good idea to join a startup that attacks these ‘experts’ and make them irrelevant.
9. I leave the implications about inequality as an exercise to the reader!


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s