Delivery Hero takes on voucher fraud with Google Cloud AI

The team chose Vertex AI for its ML model development environment for its scalability and tight integration with BigQuery — the primary data warehouse at Delivery Hero — and other Google Cloud services. The models are trained in Vertex AI Pipelines which stores their metadata in Vertex AI Model Registry. Once trained and analyzed, the models are built into a FastAPI docker image by Cloud Build.

CI/CD for ML

In order to enable fast iterations of the model development, all workflows were deeply integrated with GitHub Actions CI/CD, thus allowing users to train models and build images while following software engineering and MLOps best practices:

• Version Control (VC) – all changes tracked for both model and data sides (data is passed to Vertex Pipelines as a snapshot of a BigQuery table stored on GCS as a set of parquet files, with parameter `DATASET_VERSION` as a suffix).

• Continuous Integration (CI) – reliable trunk-based development in a GitHub repository submitting workflows to shared GCP Vertex AI environments without a need to trigger the pipelines from the local machine. As a result, the users (data scientists) can run experiments right within the Pull Requests (PR) with explicit experiment tracking.

• Continuous Deployment (CD) – enabling users to release new models to production confidently and independently without much assistance from the engineering side (“Friday night release”).

• Continuous Testing (CT) – visibility of model quality metrics and artifacts lineage integrated with CI/CD for better collaboration between data scientists , ML engineers , stakeholders, and decisionmakers.

One of the reasons for implementing CT for the models was the need to enable the development and maintenance of the multiple models trained while sharing the same code base but being trained on different subsets of data. A typical example within Delivery Hero is to maintain tens of models, one per country, and deploy them in a regional cluster (EMEA, APAC, etc.). Thus, the decision on each model has to be taken individually, however, the development, evaluation and sometimes deployment iterations are shared across the models.

Implementation details

For the Incentive Fraud use-case, it was possible to implement such ML Operations workflows using the GCP Vertex AI Model Registry at the backend, with the use of an internal Python package ml-utils developed by the team. This package provides a single CLI or Python API interface to link together the entities of Kubeflow pipelines (used internally by GCP Vertex AI Pipelines): Pipelines (or Pipeline Runs), Experiments (grouping of the pipelines), and Models and Datasets (output artifacts of the pipelines). Internally, ml-utils loads the large json definitions of the Kubeflow Pipeline Runs, finds the required artifacts and downloads them in a pre-defined format. More importantly, it provides an abstraction layer to the Models enforcing naming conventions, and provides functionality to query the Vertex ML Metadata in order to search for the models by wildcards.

The picture below illustrates the use of the described CI/CD workflow based on Vertex AI with the use of the custom tool ml-utils:

The post Delivery Hero takes on voucher fraud with Google Cloud AI appeared first on AIPressRoom.