Artificial Intelligence and Natural Language Processing

In computer science, Artificial Intelligence or AI – sometimes referred to as machine intelligence – is “intelligence-like” features demonstrated by machines. “Real AI” – sometimes referred to as Strong AI or True AI or Artificial General Intelligence (AGI) or Super Intelligence – is still a hypothetical concept.

In everyday language, the term ”Artificial Intelligence” is often used to describe machines – or computers – that mimic cognitive functions, that we associate with the human mind, such as ”learning” and ”problem solving”.

Modern machine capabilities that are referred to as AI – include successfully understanding human language, advanced image recognition, competing at the highest level in strategic games, and operating self-driving vehicles.

Artificial Intelligence is a term in constant flux, and the concept of Artificial Intelligence is sometimes criticized for being inflated. In popular usage, the term is often used to represents advanced analytics that rely on Machine Learning.

Machine Learning (ML) uses mathematical algorithms that learn from experience and leverage pattern recognition from previous evidence, or from training data.

There are different types of Machine Learning.

  • Supervised Learning is typically task driven, uses labeled data, gives direct feedback, and predicts the next value.
  • Unsupervised Learning is typically data driven, identifies clusters, and finds hidden structure in data.
  • Reinforcement Learning learns from mistakes, through a decision process and a reward system, and learns series of best actions.
  • Deep Learning is part of a broader family of machine learning methods based on artificial neural networks with representation learning. Learning can be supervised, semi-supervised or unsupervised.

Data can be structured or unstructured.

  • Structured data is highly organized – typically in rows and columns – and formatted in a way so it’s easily labeled searchable in relational databases.
  • Unstructured data – such as language and images – has no pre-defined format or organization, making it much more difficult to collect, process, and analyze.

Unstructured language data is a challenge, since real world language usage – such as social media, communication, voice to text, personal notes, et cetera – is quite different from editorial and encyclopedic language.

There is renewed focus on the crucial difference between models that actually aim to mimic the brain, i.e. unsupervised learning vs. training, and on the efficiency of the unsupervised learning process/mechanism. 

Human language understanding – and other learning processes – is based on memory and association. Context gives meaning to a concept. The human brain does not require labeled – or annotated – training data to function.

It is starting to become known that, in order to solve many – if not most – real world natural language processing tasks, a machine must handle the extreme variability in natural language usage – by learning and understanding the meaning of single- and multi-term expressions, on-the-fly, from observing actual language usage. 

There are many new initiatives that approach the problems with unstructured language data by fundamental design. The language AI used to generate our data is unsupervised, with continuous learning, and without a retrain/re-deploy cycle.