Built with Google AI: Achieve Observability and Responsible AI for ML Models with Google Cloud and Fiddler AI

Real-world impact

Top U.S. banks, fintech, e-commerce, hiring, and other Fortune 500 companies have used Fiddler to explain the predictions of their models, monitor deployed models, perform root cause analysis when issues are detected, and thereby manage the risk associated with their models.

Model risk management: The opaque nature of AI leads to risk exposure and costly inefficiencies. A top-five bank partners with Fiddler to implement explainable monitoring, mitigate risk, and solve operational inefficiencies. In particular, the bank needed to monitor ML models deployed to production across several lines of business due to regulatory auditing and compliance reasons. Furthermore, the bank had multiple redundant ML serving systems across lines of business, resulting in undue maintenance overhead and lack of centralized best practices and oversight mechanisms. By partnering with Fiddler, data scientists were able to upload trained models to Fiddler to allow model validators to review and test models for risk and bias. Once models are validated and then deployed in production, Fiddler presents insights on model performance and behavior over time. The oversight and consistency provided by this centralized platform helped eliminate both the risk of untracked production models and the cost and overhead required to build and maintain multiple serving systems across lines of business. Combined, the bank projects these savings to be several millions of dollars per year.

Fraud detection: Fraud detection is critical to maintain user trust and operational efficiency in e-commerce platforms. Traditional fraud detection systems rely on rule-based logic backed by operations personnel manually reviewing potential fraudulent cases, often resulting in many unmanageable rules and exceptions. Fiddler helped a large e-commerce company to transition from such a traditional system to an ML-based fraud detection system. The fraud detection team’s data scientists partnered with Fiddler to build an in-house buyer fraud classification model with improved accuracy and fewer false positives compared to the earlier rule-based system. Fiddler’s explanations and insights help the agents understand why certain transactions were classified as fraudulent. Using Fiddler, the team increased the speed of their fraudulent transaction review process considerably and enabled agents to provide better feedback to internal stakeholders and customers.

Model drift detection and resolution: Timely detection and resolution of drifts in the behavior of ML models is a crucial problem faced by any company that has deployed ML models in production. The lack of reliable tools to detect and resolve such issues can significantly impact business metrics such as revenue and profit as well as result in reputational risk and poor quality of service. Fiddler partnered with a company that offers a specialized large scale business analytics and data science platform for e-commerce companies. The company expanded the number of ML models in production without having sufficient tooling for monitoring the deployed models. As a result, timely insights into model performance were sometimes missed, causing a reactive rather than proactive allocation of resources to production issues. The Fiddler MPM platform helped this company to improve their model lifecycle management and data understanding capabilities by adopting automation and standardization tools. In particular, Fiddler helped the company obtain visibility into:

• model predictions, changes, and other events

• a record of how models have been performing over time; a single source of truth for model predictions

• a centralized tool to analyze and debug model performance issues

• and standardized model interpretations.

Fiddler also helped the team to decide when and how often to retrain their models, and more broadly, to refine their model maintenance and governance procedures.

Better together: Fiddler AI + Google Cloud

Fiddler AI running on Google Cloud enables Fortune 500 companies and other companies adopting AI/ML models to validate their models before deployment, detect and resolve issues encountered by deployed models, perform root cause analysis powered by explainable AI, and thereby ensure that models continue to retain performance, comply with regulations, and satisfy responsible AI principles.

LLMs and other generative AI-based applications have recently opened up exciting new possibilities and are set to turbocharge enterprise use cases across industries. However, there is an acute need for appropriate AI safety and monitoring mechanisms for Generative AI models and applications. Fiddler can seamlessly extend visibility into these advanced AI deployments on Google Cloud with robustness auditing, embedding, input, and output monitoring and other responsible AI functionality in one single pane of glass.

Learn more about Google Cloud’s open and innovative generative AI partner ecosystem and about Fiddler on Google Cloud.

The post Built with Google AI: Achieve Observability and Responsible AI for ML Models with Google Cloud and Fiddler AI appeared first on AIPressRoom.