QA / QC, Moderation, Review

1. Checklist / Questions to ask your team

   • Have you listed all assumptions for models, data and software both for training and deployed app and gave a test scenarios to valid

   • Have you calculated a ML test score that uses: which includes but not limited to, latency on calling ML API endpoints, Memory/CPU usage when performing prediction, Disk utilization (if applicable), basic date science statistical measures(median & mean prediction values , min/mx prediction values and standard deviation over a given time frame

   • Do you have tools in place for continually test and validate algorithms, so that they do not discriminate against users based on race, gender, nationality, age, religious beliefs, etc.

   • Do you test test for poisoned data, you might need to syn size a poisoned data set

   • Do you do ethical reviews throughout the MLOps life cycle

   • Do you have measurement ethical principles such as environmental, security, transparency and explainability, privacy protection, and diversity and inclusiveness ie. record in AI BOM

   • List out the different testing approaches in building, deploy/release and after deployment phases:

     • Building : Unit tests, acceptance tests, benchmarking, UI tests, soak tests, smoke tests, etc

     • Deploy/Release: Config tests, Load, Monitoring (Data Science) tests, Feature Flagging test

     • After Deployment: Logs , Monitoring,(Input changes, Predictions and System) ) Tracing, A/B Tests, Auditing, Security tests
       

   2. External Resources - Tools to use

   • NIST Face Recognition Vendor Test (FRVT)

   • LFAI: Adversarial Robustness Toolkit

   • Prometheus & Grafana

   • model monitoring, eg. Seldon, Data Robot, MLFlow, superwise.ai and hydrosphere.io and KF Serving

   3. Case Studies

   4. Further Readings

   • ML Test Score

   • Google Best practices

   • Holistic Adversarial Robustness of Deep Learning Models