The science behind the Chatbot

A Chatbot is a computer program that behaves like a human, that is a classic definition of artificial intelligence. That classic definition is actually an oversimplification as there are a couple of AI technologies at the heart of the chatbot:

Natural Language Processing (NLP)

While natural language processing covers a plethora of topics, we can understand it best with the analogy of a person learning a new language. A person who has just been introduced to a language can understand the meaning of words they know when they read or hear them. Similarly, a Chatbot understands only the words that are fed into its lexicon. While feeding the entire English dictionary into a Chatbot program is very easy, telling it how different words can combine to make sentences is much trickier. Making a machine learn grammar is even tougher (not that it’s easy for humans to learn grammar!). A bot, unlike humans, can’t understand different contextual usages of a word if they are not programmed into it. Natural language processing is the name given to the ability of computer programs to make sense of common written and spoken language. The better a program understands sentences, the higher the quality of the replies it can provide to the user.

ne of the best examples of NLP today is its application in Google Search. Type something like, “How is the weather in Seattle” in Google and Google will provide the weather information in a nice card at the top of the page. Type, “what are the chances of rain in Seattle” and it will give you the same result. This is an excellent example of natural language processing in practice. The NLP programming of the Google bot is mature enough to understand that although the question and wording are slightly different, both cases require it to pull up the same set of information.

To achieve good NLP, chatbots are fed all thinkable variations of the frequently asked questions and answers manually. Algorithms are then written to help the bot fit in synonyms and scope is left for fuzzy matching as people generally ask the same questions in a number of ways.

Machine Learning (ML)

Machine learning is the ability of computers to learn by themselves without any hard coding. It’s worth mentioning that it is likely the vast majority of chat bots on the market today are not using machine learning. Most are using a retrieval based model where possible questions and answers are hard coded into a flowchart-like structure. Thus most of today’s chat bots are not really learning to answer questions themselves using machine learning but are still learning to understand questions better with the help of NLP. For chatbots that are using the machine learning like IBM’s Watson or Google Assistant, the difference is they have a lot more data to work with and train. Some chat bot today and those developed in the near future will use a process called “supervised learning” for machine learning.

To put it into simple terms, answers given by the bots for different questions are rated on accuracy, usually by a human operator and fed back to the bot. The bot thus keeps learning about the right and wrong answers for questions and continuously keeps improving its answers. It stores the correct answers in its memory and rejects the wrong ones.

Machine learning is perhaps the most important aspect of Chatbots, without machine learning the bot will remain as smart or as stupid as it was initially until its code is changed manually.

Sentiment Analysis

Although machine learning is essential for a chat bot to expand its response lexicon over time and NLP assists in understanding the questions and statements entered by the user, these two alone are not enough for a chat bot to have full contextual understanding in a conversation.

For anyone that has worked any length of time in customer service, the way you respond to customers depends vastly on their emotional state at the time. We live in an age where consumers can easily air their grievances with companies over public channels, so understanding when customers are overheating and offering that extra concession is essential to successfully managing customer relationships.

While NLP takes a sentence and breaks it down into codified elements to allow the chat bot to understand the content, sentiment analysis seeks to rate the context of a sentence or series of sentences on an emotional scale from perfectly positive to perfectly negative. When a user adds the sentence, everyone is going to hear about the kind of service your company has provided me, it is of no consequence in NLP. However, sentiment analysis will take the statement into context with the preceding conversation and decide whether to respond with thank you, or go into damage control mode.

Are you curious to hear more about Chatbots? We’d gladly help you discover what chatbots can do for business.