Retailers and natural language processing, a match made in heaven?

Written language is a uniquely human ability. Imagining that one-day machines will be able to comprehend human language was a wild and crazy idea even only a decade ago.

However, progress has been made and now we can celebrate the remarkable accomplishments in this area, be it having the ability to translate calls instantly or winning in “Jeopardy!”. With all these developments in machines’ comprehension of language, one can't help but wonder, "Where will human skills fit in the future?"

Machines examine text at a scale and speed unattainable by human specialists. A machine can examine millions of product descriptions and discover all the potential subjects they could fit, but at this moment in time, there is still a need for someone to make the final decision whether or not these changes should go live.

When it comes down to the volumes of information that can be examined, machines have a definitive edge over people in speed. The question is, nonetheless, with such speed and scale, how do we make sure that the machines comprehend language with the high-quality precision. This really is where the power of human-machine cooperation really comes into play.

You'll find two schools of thought in the natural language processing universe: machine driven and human-driven.

The machine-driven strategy is described in great detail in the paper “The Unreasonable Effectiveness of Data,” released by Google researchers in 2009. What the paper describes are models that claimed that natural language processing can be done much more efficiently and cost effectively through statistical means rather than by methods that rely on human experts annotating text.

The “human-driven strategy” relies on rules created by expert linguists to annotate or structure language models. Traditionally handcrafting features often require expensive human labor and often rely on expert knowledge. However, this does not scale well past a certain volume of data, as there simply not enough trained and qualified people to do these tasks manually.

The competitive edges that the machine driven strategy provides are scalability, efficiency, and speed. Binding human judgment to the NLP algorithms is essential to enhancing the quality of machine work.

Machines are like infants learning the best way to speak. Sometimes they learn something new that may be wrong, it's our job to teach machines what is incorrect and what is correct, ignoring this important step can and almost certainly will lead to disastrous results.

At SentiSum we believe, the first step in unlocking useful insight from your unstructured data is to model it in a way that easily understood by your business team as well as the data team. We work in close partnership to maximise the value that our algorithms can generate for your business.