Web mining: What exactly is it?

One pillar of big data analytics is the availability of analyzable data. If this data does not come from internal company sources, it must be obtained from other sources. The Internet, or more specifically the World Wide Web, provides an abundance of freely accessible information.Web mining is needed to make this usable.

Definition of web mining

Web mining is the transfer of data mining techniques for the (partially) automatic extraction of information from the Internet, especially the World Wide Web. Web mining adopts procedures and methods from the fields of information retrieval, machine learning, statistics, pattern recognition and data mining.

In addition to copyright-protected sources, which may only be evaluated for research purposes, there are also a large number of open source sources or sources from the public sector that can be used for commercial analysis. However, much of this information was not created for machine processing and therefore cannot be used for analysis without prior transformation. Web mining, or more precisely web content mining, provides various methods for getting hold of such data and preparing it for further analysis.

Data protection is an important issue when storing personal and personally identifiable data. It must be ensured that no data is stored that can be linked to individuals. If such data is required for the specific use case, it must be ensured that it is anonymized before storage or further processing. By anonymizing the data at an early stage and not storing non-anonymized data, data protection and thus the protection of personal information is ensured. This will become even more important for all companies and authorities in light of the GDPR coming into force in May.

How is information available on the web?

Information on the World Wide Web can be available in many different ways. A single website contains many different types of information representation. The Empolis website, for example, contains textual information as well as images, graphics, videos and PDF documents.

Looking at the entire World Wide Web and the large social networks, with their content provided directly by end users, the list of possible data types becomes almost infinite. This poses major challenges for web mining, as in addition to supporting different communication protocols, individual information extraction methods must also be used for most data types.

The aim of all methods is to convert information that is primarily intended for processing and consumption by humans into a form that allows effective and fully automated analysis of the data obtained. This usually involves a conversion of the original data representation, as the requirements of an information-processing system are different from those of a view optimized for humans.

How does web mining work?

In the following, we look at the example of a single page of a website. The starting point is an HTML document that contains the technical and content description of the website. If you take a look at the structure of the HTML document, you quickly realize that a large part of the stored information is of a technical nature and has no relevance for web mining. These are primarily style sheets that control the visual appearance of the page and JavaScripts to optimize the user experience. They also contain a lot of control information that determines the structure of the page.

At first glance, such control information could be dispensed with and eliminated in the course of information extraction. However, the structure of the stored information can be decisive for its correct interpretation. The reader will assign a different significance to a sentence that appears in small print at the bottom of the page than to a sentence that prominently announces the beginning of a news article in bold letters. Accordingly, use case-specific sensitivity is required when reducing the information.

In addition to the information that can be derived directly from the content created for humans, HTML documents often contain other so-called meta-information such as title, summary, author and much more, which is explicitly intended for machine processing. What at first glance appears to be a treasure for every web miner quickly becomes a nightmare for the department responsible for developing the web mining tools. Not only are there various open standards for the annotation of meta information, but they are often implemented incorrectly.

Another challenge for web mining is the growing range of available frameworks for designing websites, each of which extends the documents with additional attributes to control the display according to a regularly undocumented scheme. This not only makes HTML structures more complex, but also makes them more difficult to interpret.

This also poses considerable challenges for current extraction methods. In addition, more and more websites do not include their content in the HTML, but load it dynamically via JavaScript at the time the website is displayed. In such cases, information can only be obtained through computationally intensive display in a browser or through extraction processes specially optimized for the website.

If this view is now extended to the entire website, links between the individual pages and beyond the website are added. Depending on the use case, these references can also have a relevant information content.

Web mining: the basis for big data analytics

Web mining is a powerful basis for data procurement in the context of big data analytics. Growing technical requirements, due to increasingly dynamic websites and web services, stand side by side with the necessary technical expertise for the targeted extraction and interpretation of the required information.

Web content mining is already an important technology in the context of big data and will maintain or even expand this position in the future.