Data mining and knowledge discovery How To Improve The Search Experience For Your Users?

At long last, organizations have a really powerful tool that boosts employees’ efficiency by speeding up their content search experience.

In the previous article in this series, Dawid discussed a great number of benefits of implementing a Knowledge Mining solution. This time, I would like to explain how Azure services make it work wonders.

Before we discuss the architecture specifics, we need to be clear on how the process works. Without further ado, let’s explain its three basic steps.

Knowledge Mining orchestration steps

In the previous article on AI-driven search, we have discovered that using Knowledge Mining you can extract information from structured and unstructured data stored in different sources. To do this, the solution uses a range of pre-trained and custom AI services.

In a nutshell, Knowledge Mining works by orchestrating the entire enrichment pipeline. There are three main phases to the process. We call them:

1. Ingestion
2. Enrichment
3. Exploration & Analysis

Here is a visualization of this structure:

Knowledge Mining orchestration steps (source: Microsoft)

What does each of this phases do? Let’s go over them one by one.

Ingestion

Ingestion is the first phase of Knowledge Mining. Here, both structured and unstructured data is processed.

What is the difference between these two types of data?

Structured data has a defined data model, and it typically resides in a relational database like Azure SQL. Unstructured data, on the other hand, does not have a predefined data model. It can come from sources such as NoSQL databases or file stores. File types include PDFs, images, Word documents, PowerPoint presentations, and more.

What about data ingestion? It is a process in which raw data, structured or unstructured, coming from various sources, is aggregated into a persistent, centralized data store.

At the end of the ingestion phase, documents are cracked. In other words, the solution extracts or creates text content from non-text sources. In this process, optical character recognition (OCR) is very useful. It proves especially helpful when we want to extract data from images or PDF files.

Enrichment

Enrichment is the second phase of Knowledge Mining, utilizing AI. In this process, the solution identifies patterns, obtains information, and gains understanding from texts coming from images, PDF files, and other unstructured data sources.

Knowledge Mining performs enrichment on individual documents as a sequence of calls to AI models. What’s interesting, you can use Azure cloud-based AI services, but you may also build your own custom models, which you will use in this phase.

Exploration & Analysis

In the final stage, Knowledge Mining exposes the newly enriched, structured documents. They are now ready for exploration and analysis.

During exploration, the solution reviews the added enrichment to learn more about the collected data. The results are then available via search indexes or end-user and line-of-business applications, such as customer relationship management (CRM) or enterprise resource planning (ERP) systems.

After exploration, it’s time for analysis. Typically, this process involves applying analytics tools, such as Power BI. They serve to explore and gain a deeper understanding of the gathered information.

The building blocks of an Azure-based Knowledge Mining solution

Let’s talk about the role the Azure cloud plays in building an intelligent search solution.

Microsoft Azure provides a number of useful services which make the solution work smoothly and effectively. Here are some examples.

Azure Cognitive Search

Azure Cognitive Search is a search-as-a-service solution. It gives developers the tools to provide a rich search experience.

Let me give you an example. Imagine a mobile app that you use to shop. You are looking for a specific product, so you immediately use the search box. There, you have a list of similar products. So, you decide to apply filtering by price. Still, the list of products is endless and your search experience leaves a lot to be desired.

You know very well that implementing your own search engine can be time-consuming. In such cases, Azure Cognitive Search speeds up the whole process and increases the quality of the search experience.

In general, there are two basic approaches that Azure Cognitive Search uses to ingest data and populate an index. We call them pull data and push data. Dawid has already touched upon them in the previous post. Let me just remind you briefly how they work.

Click to find out more

Pull Data

In this case, Azure Cognitive Search pulls data into the index from supported Azure data sources, such as:

• Blob Storage
• Cosmos DB
• Azure SQL Database 

Push Data

The second approach is quite different. The push model relies on custom applications to send documents directly into a search index. This is done programmatically. Applications can use either Azure Cognitive Search REST API or Azure Search SDK for .NET to send data into the index.