Big data analytics in the financial industry

The world of finance is a grateful field of application for data engineers and their powerful algorithms: Number-driven, processes that have been IT-supported for decades and provide vast amounts of historical data and a defined set of KPIs against which the good from the bad, the profitable from the loss-making contracts can be distinguished - beautiful financial world.

The gaps in the data

Only a closer look at the databases of financial companies reveals this: There's hardly anything in there! Of course, every transaction is recorded with the time and place, sender and recipient and a reference to the underlying contract. But the context in which a transaction took place is rarely described in a structured way. What events could lead to a borrower defaulting? What political developments in Asia could jeopardize the supply chain of a policyholder in Europe? What are the costs of a 6-hour outage of an Irish data center?

In fact, the information needed to answer such questions is regularly available at banks and insurance companies. There is hardly a major credit risk that has not been thoroughly assessed in extensive expert reports, hardly an industrial risk that has not been inspected and described in detail by engineers before an insurance contract is concluded. However, all too often only the data from these descriptions flows into the collection of structured information that can be mapped to already known key performance indicators. The majority of the information in such appraisal reports, on the other hand, is hardly ever read, but merely archived properly.

And on closer inspection, even these expert reports contain only fractions of the information that could be used to assess credit, insurance or contract risks. Press releases, news in social media, local and economic news, announcements from supervisory authorities, statements from NGOs or even - often underestimated - the weather forecast provide much more information on financial risks than individual experts could take into account in their reports. And this is regularly provided free of charge and in real time. Why don't we use this information to better assess risks, recognize dangers earlier and react more quickly to damage?

Challenges and potential of AI in the financial world

The answer to this is very simple: because it makes work!

Everyone is talking about AI. And the discourse in politics and feature articles about the power of artificial intelligence almost gives the impression that all we have to do is list the desired data sources, press a button and wait for meaningful reports and - if necessary - a warning to be generated directly on the screen of the decision-maker responsible. Even better: the machine itself makes the right decision without human intervention.

That would be nice and, yes, that's where we want to be. But that's not where we are. No one today offers a standard software product that can distinguish a risk assessment from a loss report without further effort, extract a classified list of causes of loss from the full text of a loss report or determine the probability of occurrence of all the hazards mentioned in the risk assessment.

So is none of this possible? - Yes, of course it is possible. The data is there, the technologies are there and we also have the knowledge of how to apply the technologies to the data. What is still missing are all the knowledge models, training data and linguistic rules to recognize the right information, extract it reliably and make it available in the right analyses. And a dose of realism: a text does not turn into an error-free database if you simply send it through a classifier - it's not that simple. On the other hand, even information that is not explicitly named in texts can be recognized. The question of whether something can be realized is usually not one of information theory, but one of the relationship between costs and benefits.