Opponent exploitation in imperfect game using Neural Networks

This is the dissertation project of my MSc in Artificial Intelligence at Heriot-Watt University

The subject of my dissertation is "opponent exploitation in imperfect games using Neural Networks", in other words, how can you exploit your opponent in games such as poker where you are not aware of all the information using a neural network.

The dissertation is available here.

Setup

The recommended way to setup the python environment is with conda.

Environment

conda env create --name opponent-exploitation --file=environments.yml
conda activate opponent-exploitation

Install the CFR C-binding

Makes sure to have c++ python headers in your c++ compiler

cd ./c-src/
python setup.py install

If you have any errors, it is likely because you do not have correctly installed the python c++ headers or that you are not targeting std:c++17

Run

To run the program use this command (makes sure to have the conda environement setup and activated)

> python -m src --help
usage: opponent-modelling-nn [-h] --env {kuhn,texas} {train,report} ...

positional arguments:
  {train,report}

optional arguments:
  -h, --help          show this help message and exit
  --env {kuhn,texas}  the poker environment to use (default: None)

You have two available environment, the kuhn poker varient or the limit texas hold'em.

Training

> python -m src --env kuhn train --help
usage: opponent-modelling-nn train [-h] [--report] [-id SAVE_ID] [-m MODEL_SIZE] [--nb-hidden-layers-om NB_HIDDEN_LAYERS_OM] [--layers-activation-om {relu,linear} [{relu,linear} ...]]
                                   [--layers-size-om LAYERS_SIZE_OM [LAYERS_SIZE_OM ...]] [--nb-hidden-layers-dm NB_HIDDEN_LAYERS_DM]
                                   [--layers-activation-dm {relu,linear} [{relu,linear} ...]] [--layers-size-dm LAYERS_SIZE_DM [LAYERS_SIZE_DM ...]] [-nb TRAINING_NB]
                                   [-new NEW_INDIVIDUAL_PER_ITERATION] [-i ITERATION_NB] [-r MUTATION_RATE] [--nb-games NB_GAMES] [--suboptimal-agents-nb SUBOPTIMAL_AGENTS_NB]

optional arguments:
  -h, --help            show this help message and exit
  --report              run report directly after (default: False)
  -id SAVE_ID, --save-id SAVE_ID
                        the save id (default: None)

Neural Network:
  -m MODEL_SIZE, --model-size MODEL_SIZE
                        size of the opponent model (default: 5)
  --nb-hidden-layers-om NB_HIDDEN_LAYERS_OM
                        opponent modelling number of hidden layers (default: 1)
  --layers-activation-om {relu,linear} [{relu,linear} ...]
                        opponent modelling activation function of the hidden layer(s) (default: ['linear'])
  --layers-size-om LAYERS_SIZE_OM [LAYERS_SIZE_OM ...]
                        opponent modelling size of the hidden layer(s) (default: [10])
  --nb-hidden-layers-dm NB_HIDDEN_LAYERS_DM
                        descision making number of hidden layers (default: 1)
  --layers-activation-dm {relu,linear} [{relu,linear} ...]
                        descision making activation function of the hidden layer(s) (default: ['linear'])
  --layers-size-dm LAYERS_SIZE_DM [LAYERS_SIZE_DM ...]
                        descision making size of the hidden layer(s) (default: [10])

Evolutionary Algorithm:
  -nb TRAINING_NB, --training-nb TRAINING_NB
                        number of training individual (default: 20)
  -new NEW_INDIVIDUAL_PER_ITERATION, --new_individual-per-iteration NEW_INDIVIDUAL_PER_ITERATION
                        new individual per iteration (default: 15)
  -i ITERATION_NB, --iteration-nb ITERATION_NB
                        number of iterations (default: 100)
  -r MUTATION_RATE, --mutation-rate MUTATION_RATE
                        mutation rate (default: 0.1)
  --nb-games NB_GAMES   number of games between agents each iteration (default: 25)
  --suboptimal-agents-nb SUBOPTIMAL_AGENTS_NB
                        number of suboptimal agents (default: 3)

Report

> python -m src --env kuhn report --help
usage: opponent-modelling-nn report [-h] [-nn] [-b] [-r] [-all] id

positional arguments:
  id                    the save id

optional arguments:
  -h, --help            show this help message and exit
  -nn, --nn-summary
  -b, --best-response
  -r, --reward-over-time
  -all, --report-all    run all the reporting (default: False)

Report for limit texas hold'em has never been tested, feel free to raise an issue if you face any error.