2402 00854 SymbolicAI: A framework for logic-based approaches combining generative models and solvers

One promising approach towards this more general AI is in combining neural networks with symbolic AI. In our paper “Robust High-dimensional Memory-augmented Neural Networks” published in Nature Communications,1 we present a new idea linked to neuro-symbolic AI, based on vector-symbolic architectures. Each approach—symbolic, connectionist, and behavior-based—has advantages, but has been criticized by the other approaches. Symbolic AI has been criticized as disembodied, liable to the qualification problem, and poor in handling the perceptual problems where deep learning excels. In turn, connectionist AI has been criticized as poorly suited for deliberative step-by-step problem solving, incorporating knowledge, and handling planning. Finally, Nouvelle AI excels in reactive and real-world robotics domains but has been criticized for difficulties in incorporating learning and knowledge.

Similarly, Allen’s temporal interval algebra is a simplification of reasoning about time and Region Connection Calculus is a simplification of reasoning about spatial relationships. Cognitive architectures such as ACT-R may have additional capabilities, such as the ability to compile frequently used knowledge into higher-level chunks. A more flexible kind of problem-solving occurs when reasoning about what to do next occurs, rather than simply choosing one of the available actions. This kind of meta-level reasoning is used in Soar and in the BB1 blackboard architecture. Our chemist was Carl Djerassi, inventor of the chemical behind the birth control pill, and also one of the world’s most respected mass spectrometrists. We began to add to their knowledge, inventing knowledge of engineering as we went along.

IBM, MIT and Harvard release “Common Sense AI” dataset at ICML 2021

Enter Tim Rocktäschel, a Research Scientist at Facebook AI Research London and a Lecturer in the Department of Computer Science at University College London. Much of Tim’s work has been focused on ways to make RL agents learn with relatively little data, using strategies known as sample efficient learning, in the hopes of improving their ability to solve more general problems. These dynamic models finally enable to skip the preprocessing step of turning the relational representations, such as interpretations of a relational logic program, into the fixed-size vector (tensor) format. They do so by effectively reflecting the variations in the input data structures into variations in the structure of the neural model itself, constrained by some shared parameterization (symmetry) scheme reflecting the respective model prior. It wasn’t until the 1980’s, when the chain rule for differentiation of nested functions was introduced as the backpropagation method to calculate gradients in such neural networks which, in turn, could be trained by gradient descent methods. For that, however, researchers had to replace the originally used binary threshold units with differentiable activation functions, such as the sigmoids, which started digging a gap between the neural networks and their crisp logical interpretations.

Neuro-symbolic artificial intelligence can be defined as the subfield of artificial intelligence (AI) that combines neural and symbolic approaches. By neural we mean approaches based on artificial neural networks—sometimes called connectionist or subsymbolic approaches—and in particular this includes deep learning, which has provided very significant breakthrough results in the recent decade, and is fueling the current general interest in AI. By symbolic we mean approaches that rely on the explicit representation of knowledge using formal languages—including formal logic—and the manipulation of language items (‘symbols’) by algorithms to achieve a goal. Mostly, neuro-symbolic AI utilizes formal logic as studied in the knowledge representation and reasoning subfield of AI, but the lines blur, and tasks such as general term rewriting or planning, that may not be framed explicitly in formal logic, bear significant similarities and should reasonably be included. Neuro-symbolic AI has a long history; however, it remained a rather niche topic until recently, when landmark advances in machine learning—prompted by deep learning—caused a significant rise in interest and research activity in combining neural and symbolic methods. In this overview, we provide a rough guide to key research directions, and literature pointers for anybody interested in learning more about the field.

Modelling Symbolic Knowledge Using Neural Representations

In addition, several artificial intelligence companies, such as Teknowledge and Inference Corporation, were selling expert system shells, training, and consulting to corporations. However, the black-box nature of classic neural models, with most confirmations on their learning abilities being done empirically rather than analytically, renders some direct integration with the symbolic systems, possibly providing the missing capabilities, rather complicated. However, in the meantime, a new stream of neural architectures based on dynamic computational graphs became popular in modern deep learning to tackle structured data in the (non-propositional) form of various sequences, sets, and trees. Most recently, an extension to arbitrary (irregular) graphs then became extremely popular as Graph Neural Networks (GNNs).

• The idea was based on the, now commonly exemplified, fact that logical connectives of conjunction and disjunction can be easily encoded by binary threshold units with weights — i.e., the perceptron, an elegant learning algorithm for which was introduced shortly.
• However, there is a principled issue with such approaches based on fixed-size numeric vector (or tensor) representations in that these are inherently insufficient to capture the unbound structures of relational logic reasoning.
• Section 5 discusses the future research directions, after which Section 6 concludes this survey.
• Literature references within this text are limited to general overview articles, but a supplementary online document referenced at the end contains references to concrete examples from the recent literature.

Indeed, neuro-symbolic AI has seen a significant increase in activity and research output in recent years, together with an apparent shift in emphasis, as discussed in Ref. [2]. Below, we identify what we believe are the main general research directions the field is currently pursuing. It is of course impossible to give credit to all nuances or all important recent contributions in such a brief overview, but we believe that our literature pointers provide excellent starting points for a deeper engagement with neuro-symbolic AI topics. Knowledge-based systems have an explicit knowledge base, typically of rules, to enhance reusability across domains by separating procedural code and domain knowledge. A separate inference engine processes rules and adds, deletes, or modifies a knowledge store. For instance, one prominent idea was to encode the (possibly infinite) interpretation structures of a logic program by (vectors of) real numbers and represent the relational inference as a (black-box) mapping between these, based on the universal approximation theorem.

From a more practical perspective, a number of successful NSI works then utilized various forms of propositionalisation (and “tensorization”) to turn the relational problems into the convenient numeric representations to begin with [24]. However, there is a principled issue with such approaches based on fixed-size numeric vector (or tensor) representations in that these are inherently insufficient to capture the unbound structures of relational logic reasoning. Consequently, all these methods are merely approximations of the true underlying relational semantics. This is easy to think of as a boolean circuit (neural network) sitting on top of a propositional interpretation (feature vector). However, the relational program input interpretations can no longer be thought of as independent values over a fixed (finite) number of propositions, but an unbound set of related facts that are true in the given world (a “least Herbrand model”).

Program tracing, stepping, and breakpoints were also provided, along with the ability to change values or functions and continue from breakpoints or errors. It had the first self-hosting compiler, meaning that the compiler itself was originally written in LISP and then ran interpretively to compile the compiler code. Our researchers are working to usher in a new era of AI where machines can learn more like the way humans do, by connecting words with images and mastering abstract concepts. These old-school parallels between individual neurons and logical connectives might seem outlandish in the modern context of deep learning. However, given the aforementioned recent evolution of the neural/deep learning concept, the NSI field is now gaining more momentum than ever.

Title:Deep Symbolic Learning: Discovering Symbols and Rules from Perceptions

Consequently, also the structure of the logical inference on top of this representation can no longer be represented by a fixed boolean circuit. This idea has also been later extended by providing corresponding algorithms for symbolic knowledge extraction back from the learned network, completing what is known in the NSI community as the “neural-symbolic learning cycle”. And while the current success and adoption of deep learning largely overshadowed the preceding techniques, these still have some interesting capabilities to offer. In this article, we will look into some of the original symbolic AI principles and how they can be combined with deep learning to leverage the benefits of both of these, seemingly unrelated (or even contradictory), approaches to learning and AI. How to explain the input-output behavior, or even inner activation states, of deep learning networks is a highly important line of investigation, as the black-box character of existing systems hides system biases and generally fails to provide a rationale for decisions. Recently, awareness is growing that explanations should not only rely on raw system inputs but should reflect background knowledge.

For instance, when confronted with unseen situations during training, machines may struggle to make accurate decisions in medical diagnosis. Another crucial consideration is the compatibility of purely perception-based models with the principles of explainable AI (Ratti & Graves, 2022). Neural networks, being black-box systems, are symbolic learning unable to provide explicit calculation processes. In contrast, symbolic systems offer enhanced appeal in terms of reasoning and interpretability. For example, through deductive reasoning and automatic theorem proving, symbolic systems can generate additional information and elucidate the reasoning process employed by the model.

Therefore, an urgent need arises to provide a comprehensive survey that encompasses popular methods and specific techniques (e.g., model frameworks, execution processes) to expedite advancements in the neural-symbolic field. Distinguishing itself from the aforementioned surveys, this paper emphasizes classifications, techniques, and applications within the domain of neural-symbolic learning systems. Using symbolic knowledge bases and expressive metadata to improve deep learning systems.

Attempting these hard but well-understood problems using deep learning adds to the general understanding of the capabilities and limits of deep learning. It also provides deep learning modules that are potentially faster (after training) and more robust to data imperfections than their symbolic counterparts. The above paper introduces the current research status and research methods of neural-symbolic learning systems in detail. The deep learning hope—seemingly grounded not so much in science, but in a sort of historical grudge—is that intelligent behavior will emerge purely from the confluence of massive data and deep learning.

Background knowledge can also be used to improve out-of-sample generalizability, or to ensure safety guarantees in neural control systems. Other work utilizes structured background knowledge for improving coherence and consistency in neural sequence models. Symbolic reasoning and deep learning are two fundamentally different approaches to building AI systems, with complementary strengths and weaknesses. Despite their clear differences, however, the line between these two approaches is increasingly blurry.

The true resurgence of neural networks then started by their rapid empirical success in increasing accuracy on speech recognition tasks in 2010 [2], launching what is now mostly recognized as the modern deep learning era. Shortly afterward, neural networks started to demonstrate the same success in computer vision, too. We’ve relied on the brain’s high-dimensional circuits and the unique mathematical properties of high-dimensional spaces.

In the next article, we will then explore how the sought-after relational NSI can actually be implemented with such a dynamic neural modeling approach. Particularly, we will show how to make neural networks learn directly with relational logic representations (beyond graphs and GNNs), ultimately benefiting both the symbolic and deep learning approaches to ML and AI. This section introduces the methods used in neural-symbolic learning systems in three main categories. We aim to distill the representative ideas that provide evidence for the integration between neural networks and symbolic systems, identify the similarities and differences between different methods, and offer guidelines for researchers. The main characteristics of these representative methods are summarized in Table 3. To date, neural networks have demonstrated remarkable accomplishments in perception-related tasks, such as image recognition (Rissati, Molina, & Anjos, 2020).