AI Atlas:
Using Comgra to Visualize AI
Rudina Seseri
It is no secret that AI has become increasingly complex in recent years. Even beyond the myriad individual techniques and breakthroughs published each day, enterprise AI systems are often created using ensembles of many different components – from attention mechanisms to convolutional and recurrent neural network layers – based on the desired end business use case. These complex constructions make it challenging to understand how such systems function internally, creating a significant hurdle in AI development known as the “black box” problem. Put simply, it is extremely difficult to properly explain the decision-making of large neural networks and thus it is costly and time-consuming to troubleshoot errors and improve faulty or biased models.
In response, researchers worldwide have dedicated enormous resources towards techniques that improve AI explainability – I even wrote about one such technique, MILAN, in one of my earliest AI Atlases. Today, I return to the topic of interpretable AI with a look at Comgra, a tool recently published by researchers in Germany that adds a visual interface to AI development while unlocking new capabilities in model testing and deployment.
🗺️ What is Comgra?
Comgra, short for Computation Graph Analysis, is an open-source tool that works with the popular AI development framework PyTorch. Its purpose is to help users understand the inner workings of neural networks by extracting information about the model’s internal behaviors and presenting it in an easy-to-navigate graphical interface (shown below). This visualization enables users to track the various stages of model training, identify problem areas, and test hypotheses without the need to repeatedly re-run the model. In other words, Comgra greatly aids users in conceptualizing how an AI model processes information and where adjustments may be needed.
🤔 What is the significance of Comgra, and what are its limitations?
The key value of Comgra lies in its ability to demystify complex neural networks. By organizing large volumes of data into an accessible format, the tool enables users to explore a model from multiple angles, whether overall trends or specific data points. This flexibility makes it a powerful resource for anyone involved in improving AI models, particularly in terms of debugging and optimizing the underlying architecture. Furthermore, the visual component of Comgra introduces a novel approach to analyzing AI beyond sifting through thousands of lines of code, lowering the entry barrier for business and non-technical users to understand the products and services they support.
- Debugging: Comgra can quickly identify anomalies and irregularities in a model’s behavior, helping developers to troubleshoot and correct errors before they become pervasive.
- Optimization: Developers can leverage Comgra to design more efficient models and architectures by deriving insights on how various components of a neural network interact.
- Interpretability: Researchers interested in understanding why AI models make certain decisions can use Comgra to track the specific parameters and activations that influence the outcome. This has enormous value when identifying bias within an AI system or when auditing mistakes to prevent them from happening again.
However, while Comgra is a unique and extremely promising tool, it is not an all-in-one solution for AI development. The researchers behind Comgra have identified a need for improvement in areas including:
- Pre-defined metrics: Comgra requires users to decide in advance which data they want to track before the model begins training. If critical data points are overlooked, the model may need to be re-run, which can be time-consuming and resource-intensive.
- Real-time adaptability: Comgra cannot dynamically adjust its approach based on previous results, limiting the tool’s ability to adapt to unexpected issues or outliers that arise mid-training.
- Anomaly detection: Although Comgra aids in identifying irregularities, its anomaly detection capabilities are still under development and the tool does not provide automatic, intelligent analysis of potential problem areas.
🛠️ Use cases of Comgra
Comgra is best suited for scenarios where understanding internal AI model behavior is crucial, making it valuable for improving accuracy and efficiency in areas such as:
- Language models: By using Comgra, developers can diagnose issues with natural language AI systems and identify whether an error stems from faulty weighting in a specific neural network layer or traces back to an input, enabling targeted adjustments to improve overall model performance.
- Fraud detection: Comgra could track outlier results from AI deployments in industries such as financial services by isolating situations where a model’s internal calculus deviates from expected patterns. This would assist in pinpointing the root causes of false positives and improve accuracy in detecting fraud.
- Image recognition: Comgra could be leveraged to identify biases in AI processing. This would enable engineers to build fine-tuned improvements for underserved user segments, ultimately increasing the effectiveness of personalization.
Stay up-to-date on the latest AI news by subscribing to Rudina’s AI Atlas.