TECH

AI models can already write code and solve complex tasks. The next challenge is proving they are correct
AI models can already write code, generate text, and solve complex tasks—sometimes with impressive speed. However, one problem continues to challenge experts: certain answers may seem convincing but are actually incorrect.
As this technology begins to enter critical processes and take on tasks autonomously, new concerns arise; in certain contexts, it is not enough for an answer to *seem* right—it must be demonstrated to be correct.
For Amazon Web Services (AWS), the solution to this problem may lie in neuro-symbolic AI, an approach that combines the learning capabilities of generative models with formal mathematical verification methods. "The idea is to use the best of both worlds," explains Leonardo de Moura, a Senior Principal Applied Scientist at AWS.
On one hand, language models operate based on probabilities, capable of producing answers that—while appearing correct—do not necessarily come with a guarantee of accuracy.
On the other hand, automated reasoning follows a different logic. Instead of seeking a probable answer, it allows for the verification of properties defined mathematically. However, this approach only works when it is possible to precisely define what "correct" means.
For instance, in a data compression program, one can mathematically establish a clear property: if a file is compressed and then decompressed, the result must be exactly the original file. In other words, it is possible to specify what the program should do and verify whether that property is met, the manager explains.
However, not all problems work this way. "Imagine, for example, that I want to prove a mathematical model identifies a cat. How do we mathematically define what a cat is?" he asks. For concepts that are vaguer and harder to formalize, formal verification techniques cannot offer the same guarantees. That is why automated reasoning finds particularly relevant applications in areas such as the verification of programs, hardware, or mathematical models of specific processes, notes Leonardo de Moura.
“Automated reasoning has many applications, but it doesn't completely solve the problem,” the executive points out. The answer involves combining the capabilities of generative models with logic-based reasoning systems...“Language models, despite sometimes making mistakes, do incredible things. So, what can we do? The model generates a result, and we use the logic component to verify whether that result is correct or not”... Leonardo de Moura, Senior Principal Applied Scientist at AWS.
How does it work in practice? Returning to the compression program example: instead of asking an AI system simply to write it, one can mathematically specify what needs to be guaranteed.
In other words, the system has the “freedom” to create the program, but generating code that merely *appears* to work isn't enough; it must also construct a proof that the program satisfies the defined property.
Instead of asking for just one thing—like “write the program”—we are asking: “write the program *and* prove that it is correct.”
Combining the ability to generate solutions with formal verification increases confidence in the results, but can the neuro-symbolic AI approach eliminate hallucinations?
According to Leonardo de Moura, the main limitation remains the ability to mathematically define what “correct” means. In the case of the compression program, the property is clear and can be formalized. In other cases, such as identifying a cat, the task is much more difficult.
There are also limits inherent to logic itself. “There are problems that are undecidable,” the expert explains, adding that in such situations, it is impossible to guarantee the existence of a proof for every true statement. However, he acknowledges that these are “extreme cases.”
The next stage of AI...Although the combination of generative AI and automated reasoning is already being used by AWS in areas such as hardware verification, cryptography, and agent control, Leonardo de Moura believes the most significant impact is yet to come.
He argues that, in the coming years, this approach could transform software development, envisioning an AI that is not only capable of writing programs but can also demonstrate how they meet the defined security and functional properties.
He explains that programmers today are sometimes "afraid to implement an optimization because they fear introducing a bug into the code." The alternative would be to manually verify that the changes preserve all properties—a process that can be very time-consuming.
"It is a very time-consuming task. But AI is great at performing tasks that are mechanical and repetitive. It has infinite patience"... "We will have software that is more secure and developed faster, because we are verifying that the code possesses the desired properties. Users will have greater confidence in the software they are using. (...) And I don't think this will happen ten years from now; it will happen much sooner," asserts Leonardo de Moura.
by mundophone





