Internal Audit

1. Checklist / Questions to ask your Development team

• Have you established a baseline for audit (EBA (ICO 2020)

• Have you completed an audit of models and data prior to deployment and established acceptable thresholds

• Who will be performing the audit? Do they have the skills to be successful?

• Is the process of generating the analysis well documented and reproducible if we discover issues in the future?

• Do you have a set of unit tests where each unit test verifies whether a predefined specification is satisfied (ie. accuracy over 95%)

• If you are using facial recognition data, log the angle relative to the camera to be captured in sBOM

• If possible, create unit tests using at least 3 different model algorithms

• Can you explain in understandable terms a decision that the model made in cases where a justification is needed? and are you saving these details in a log file?

• Have you communicated the shortcomings, limitations, and biases of the model/data to relevant stakeholders in ways that can be generally understood?

• Document and make available an API that allows third parties to query the algorithmic system and assess its response.= if applicable

• Make sure that if data is needed to properly audit your algorithm, such as in the case of a machine-learning algorithm, that sample (e.g., training) data is made available and documented in the AI BOM

• External Resources - Tools to use

  • AuditAI

  • Aequitas: Bias and Fairness Audit Toolkit

  • Templates(Sample)

    • AI Audit Score card

    • AI Bias Score card

• Further Readings

  • Ethics‑based auditing of automated decision‑making systems:
    intervention points and policy implications

  • Algorithmic impact assessment cast study healthcare

  • Auditing Blackbox models

  • Auditing AI Models for Verified Deployment under Semantic Specifications

  • Defining an End to End Framework for Auditing