System Subtainability

  1. Checklist / Questions to ask your team

    • Di you have a method to detect and alert any model drift ie. keep track of performance degradation and accuracy of model

    • Are you monitoring your input for skews?

    • Have you tested arbitration when system failures or introduces bias

    • Have you considered using A|B test models to determine if a challenger model is ready to replace a deployed model?

    • Do you capture input data (features) and output predictions while in production

    • Have you consider long-term impact on human and environment?

    • Do you have a method to detect, report and asses AI harm incidents

    2. External Resources - Tools to use

    • Feature monitoring libraries: Great Expectations and Deequ

    • System Logs

    • Test Score Card to include model accuracy, results/predictions features and raw inputs

    3. Templates (Samples)

    4. Case Studies

    5 Further Readings