Skip to content

ML Testing for Everyone. Find issues before they become problems.

License

Notifications You must be signed in to change notification settings

ETIQ-AI/ml-testing

Repository files navigation

Etiq AI logo

Etiq AI Demo Repository

Etiq is an ML testing platform for data scientists and ML engineers. Supports both pre-production and in-production, with Etiq Test, Monitor, and Explain. This repo constains demo notebooks and curated testing metrics.
Explore the documentation

Raise an issue · Slack · Blog · Contact

Etiq AI

In order to start working with the Etiq AI platform you will need to sign-up to our dashboard: https://dashboard.etiq.ai/ and then install the python package.

pip install etiq

This repository provides a number of notebooks and examples that show how easy it is to use and work with the Etiq.

Docs

For a quickstart and install just go to our docs: https://docs.etiq.ai/quickstart

Getting started

To set-up your testing suite as you build your model, you can just use etiq directly from your jupyter notebook or any other python based IDE. To start just import etiq once you've installed it, log to the dashboard and create a project. You can run the library without connecting to the dashboard, but then your results will not be stored. Only the results of the tests gets stored, the details of the models or datasets underneath are not stored.

import etiq

Load the config you'll use for your tests from whereever you store it. A config is in a simple json format & it's where you can set your tests details - metrics you want to use, what kind of scans you want to run, what thresholds you want to put for passing a test:

etiq.load_config("./config_demo.json")

A config is in a simple json format & it's where you can set your tests details - metrics you want to use, what kind of scans you want to run, what thresholds you want to put for passing a test:

    {"dataset": {
            "label": "income",
            "bias_params": {
                "protected": "gender",
                "privileged": 1,
                "unprivileged": 0,
                "positive_outcome_label": 1,
                "negative_outcome_label": 0
            },
            "train_valid_test_splits": [0.0, 1.0, 0.0],
            "remove_protected_from_features": true

        },
        "scan_accuracy_metrics": {
            "thresholds": {
                "accuracy": [0.8, 1.0],
                "true_pos_rate": [0.6, 1.0],
                "true_neg_rate":  [0.6, 1.0]
            }
        },
        "scan_bias_metrics": {
            "thresholds": {
                "equal_opportunity": [0.0, 0.2],
                "demographic_parity": [0.0, 0.2]
            }
        },
        "scan_leakage": {
            "leakage_threshold": 0.85
           }
    }

Next, log your dataset, model, shanpshot - as applicable

from etiq import Model

dataset = etiq.BiasDatasetBuilder.dataset(my_test_dataset)
bias_params = etiq.BiasDatasetBuilder.bias_params()

model = Model(model_architecture=my_trained_model, model_fitted=my_model_fit)

snapshot = project.snapshots.create(name="My New Snapshot",
                                    dataset=dataset,
                                    model=model,
                                    bias_params=bias_params)

Run the scans you want to run:

snapshot.scan_accuracy_metrics()

snapshot.scan_bias_metrics()

You can retrieve the results either via the interface or via the dashboard.

The view below shows the list of snapshots and aggregated view of the scans and model health:

dashboard_screenshot

Each snapshot links to a detailed view where all the scans associated with that snapshot are displayed:

snapshot_screenshot

For testing in production, we will release an Etiq + Airflow demo example shortly. But in the meantime, just reach out: [email protected]

We also recommend integrating tests with your Github commits. For help just email us [email protected]

Example Notebooks & Configs

We have pre-set example notebooks and config files.

Legacy library

As the change from v1.2.4 to v1.3 we have Legacy notebooks in case you are still on 1.2.4 and you can find the dashboard for this version at https://legacy-dashboard.etiq.ai

Bugs and feature requests

If you discover a bug or have a great idea that you would like to see in the Etiq AI tool then please share it either by:

Documentation

Etiq AI provide a full set of documentation covering all aspects of the Etiq AI Library here.

The Good Data Science Club

Are you interested in Ethics, Fairness & Bias in Artificial intelligence and Machine Learning?

If so we are part of The Good Data Science Club community. We'd love to discuss all things AI/ML here and you can click to join here.

Copyright and license

All Code and documentation copyright 2019–2022 Etiq AI

Releases

No releases published

Packages

No packages published

Languages