Is ChatGPT the solution for everything? We would like to provide an overview to simplify the introduction to the topic and to support considerations and discussions.
History and relevant components of ChatGPT
The technological basis of ChatGPT is Large Language Models (LLM). These predict the probability of word sequences, sentences, etc.
LLMs can not only support basic activities in Natural Language Processing (NLP) (e.g. concept recognition, negation recognition, relation recognition), but also perform more complex activities such as:
LLM can thus have similarities with pre-processing pipelines, databases, recommender systems, virtual agents, etc. - with the impressive extra that they can be intuitively operated by speech.
LLMs are a further development ofartificial neural networks. Artificial neural networks were invented back in 1943 and are motivated by the structure of the human brain. An artificial neuron has any number of numerical inputs, calculates a mathematical function and delivers a numerical output. The mathematical function is described by weights (parameters), a weighted sum of the inputs and a linear activation function (Russel & Norvig, 2002). See also Figure 1.
Neurons can perform calculations between input and output, i.e. solve problems (e.g. logic A and B / A or B). The problem-solving ability of such neural networks is limited. For example, the problem "A and B" and the problem "A or B" can be solved, but not the problem "A exclusive-or B".
This is why deep neural networks are usually used; they consist of:
In the simplest case, the neurons in a lower layer are fed forward and fully connected to the neurons in the layer above (feedforward layer).
A training procedure assigns initial weights, compares the calculated output with the desired output on sample data and then adjusts weights depending on their influence (Rumelhart et al. 1986).
Traditional NLP applications used words in a very "one-dimensional" way. A word was either present in a sentence or text, "1", or not, "0".
With deep learning, words can be represented as numerical vectors (embeddings) by many dimensions that describe their semantics. In such spaces of vectors, words with similar meanings lie close together. In addition, relative properties are also obtained, e.g. if you subtract the vector for "woman" from the vector for "queen" and add the vector for "man", you obtain the vector for "king" (Mikolov et al. 2013). See also Figure 2.
Not only words, but also letters or parts of words can be represented as vectors (tokens). Special algorithms divide texts accordingly (tokenizer). Word sequences, sentences, documents and any text can also be represented as a set of vectors (encoder). The probability of subsequent texts can then be calculated (decoder). The context of a word (previous and subsequent words) can be taken into account (Attention). The architecture of the so-called transformer is particularly efficient due to its parallel computability (Vaswani et al., 2017).
Language models that can calculate the probability of texts can thus be trained directly on any collection of texts (non-supervised, because not specifically manually annotated). Training is carried out, for example, from right to left (Radford et al., 2018) or bidirectionally (Devlin et al., 2018). Input words are converted into word vectors and compared with other relevant words before/after the respective word in a deep neural network consisting in particular of so-called attention layers. A new representation is then obtained in each case, which can now be used further depending on the application, e.g. to translate a word into another language or to predict the next word in a generated text. See also Figure 3.
For example, deep learning has shown impressive results in text comprehension tasks, similar to those previously shown on images (Ruder, 2018).
These language models can then be used iteratively to create new texts by iteratively asking the language model for the most probable word (generative artificial intelligence).
In addition, these language models are an effective basis (pretraining) for transferring learned semantics to other areas (transfer learning) and improving specific skills through targeted training - and thus subsequent further refinement of the parameters (fine-tuning). This targeted learning usually takes place with manually annotated data (supervised). The more text data (= amount of data) and the more careful the training (= computing power), the more pronounced these properties become (scaling). For example, LLM can perform NLP activities with hardly any contextual information - just a few preceding words (prompts) (Brown et al., 2020).
LLMs such as GPT-2 / 3.5 / 4 (OpenAI), LLaMA (Meta), BERT (Google), Alpaca (Stanford) are more or less openly trained, exchanged and refined by various institutions and used as a basis for further refinements in various NLP applications (Foundation Language Models).
The open language model LLaMA was trained in its largest version with 65 billion parameters (weights) on 1.4 trillion word parts with 2048 A100 GPUs from NVIDIA, each with 80 GB of graphics memory, in 21 days. Meta estimates that the development and training cost a total of 2.638 MWh of energy.
Many LLMs, such as in ChatGPT, are also trained tounderstand instructions and conversations (instruction)with considerable manual effort.
This is done in two ways: Firstly, by manually creating pairs of input (request) and desired output (response) for supervised training. Secondly, through another language model that evaluates (rewards) responses to requests. This reward model again requires manually created pairs of input (request+response) and evaluation (or ranking in the case of several possible responses). The second training can then be carried out automatically by randomly selecting queries, evaluating them by the reward model and using them to further train the large language model (RLHF, Reinforcement Learning from Human Feedback).
Strengths and weaknesses of LLM
Only with time will you get a comprehensive picture of LLM's strengths and weaknesses.
Use cases with (and without) added value
Here too, it will only be possible to evaluate the benefits of large language models in specific use cases over time.
Some use cases with potential for LLM:
Basically any NLP application can benefit from LLM, for example chatbots, voicebots and virtual agents. Virtual agents often consist of separate components for natural language processing (NLP), dialog management, knowledge base and natural language generation (NLG). Large language models, especially when trained on specific tasks, interactions or conversations, promise simple architectures, all-encompassing text understanding and intuitive interactions with humans.
It remains to be seen how well this approach will prove itself in applications:
Can explicit information be extracted from text data (possibly even combined with images, audio, video) with higher quality, with a higher level of detail, with less effort, and used for automation?
Can hidden information, including knowledge, be recognized earlier and used with added value?
Can activities be carried out faster, more reliably, more comprehensibly, etc. in the interaction between man and machine? Perhaps even interdisciplinary, across workflows or processes, and with existing media breaks (from voice, to text, to video, to voice again).
There is enough under-considered text data, necessary knowledge and suboptimal workflows and processes in industry, public authorities, medicine, science and much more.
As an example, some questions from medical research and development projects at Empolis:
Medical information for examinations and treatments takes up a lot of time and resources, and often too many patients' questions remain unanswered.
In diagnostic disciplines such as radiology, diagnostic reports are either written or dictated in free text or questionnaires are filled out. Depending on user preference, more structured information or coherent texts are required to support the fast and error-free exchange of information between medical staff.
Many decisions on aftercare options such as rehab, home care, etc. are made too late, delaying discharges and the optimal further treatment of patients.
If these conditions are not met, the sustainable added value of LLM is likely to be low.
It needs to be evaluated in more detail how well general LLM can be used directly and where its limits lie. These limits can be shifted through refinements and targeted training. This is where the trade-off between cost and benefit is relevant.
Promising combinations of LLM with other technologies
Access (on-premise or cloud) and refinement of LLM is facilitated by various providers (e.g. Hugging Face, SpaCy), platforms and the open source commitment of many people (Patel et al., 2023).
When it comes to using LLM in your own applications, this can become complex. The offer is almost unmanageable. Depending on the provider, customer data (whether for training or usage) is in turn used to further develop the language models. Due to the effort involved in storing information, it must be carefully evaluated whether the effort is worthwhile. Quality optimization, tests, etc. are essential and must be carefully planned and implemented.
ChatGPT is another impressive example of the power of deep learning in natural language processing (NLP). Are ChatGPT and other LLMs the solution to everything? No, because the starting point for the application of deep learning has not changed: Large amounts of data, ideally elaborately curated by many people, and great computing power are required for impressive results.
LLM then enables far more intelligent systems. However, not in isolation, but in combination with other systems: from databases, knowledge bases and test procedures to knowledge graphs.
