Dataset.Anonymisation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ae552ad1",
   "metadata": {},
   "source": [
    "# Anonymisation of 'customer_information.csv' dataset for use by Imperial researchers and government (and calculation of K-anonymity)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0447d175",
   "metadata": {},
   "source": [
    "Anonymisation is the practice of removing identifying information from data in order to protect individuals' privacy. By anonymising data, we can ensure that sensitive information is kept secure. In this project, we aim to create an anonymised dataset by removing personally identifiable information from the original dataset whilst attempting to retain its utility and insights by using a combination of techniques such as pseudonymous identification and data perturbation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14bda890",
   "metadata": {},
   "source": [
    "## Helper functions\n",
    "The following helper functions are needed:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8fe288a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper functions\n",
    "\n",
    "# The following variable countries were hard-coded to fix unmatched territory errors\n",
    "northern_countries = [\"Svalbard & Jan Mayen Islands\"]\n",
    "southern_countries = [\"Micronesia\"]\n",
    "\n",
    "# Parse country into shortform \n",
    "def parse_country(country_name):\n",
    "    return coco.convert(country_name, to='name_short', include_obsolete=True)\n",
    "\n",
    "# Convert country of birth into Hemisphere (Northern or Southern) based on latitude coordinates\n",
    "def country_to_hemisphere(country_name):\n",
    "    try:  \n",
    "        if country_name in southern_countries: \n",
    "            return \"Southern Hemisphere\" \n",
    "        elif country_name in northern_countries:\n",
    "            return \"Northern Hemisphere\"\n",
    "        else:\n",
    "            return (\"Southern\" if Nominatim(user_agent=\"CDM\").geocode(parse_country(country_name)).latitude < 0 else \"Northern\") + \" Hemisphere\"\n",
    "    except Exception as e:\n",
    "        print(e)\n",
    "        return \"Error\"\n",
    "    \n",
    "# SHA hash function using a key and salt\n",
    "def hash(to_hash, key):\n",
    "    salt = os.urandom(16)\n",
    "    h = hashlib.sha256()\n",
    "    h.update(key)\n",
    "    h.update(salt)\n",
    "    h.update(to_hash.encode())\n",
    "    return to_hash, h.hexdigest(), salt.hex()\n",
    "\n",
    "# To encrypt and save as encrypted file; specify file to encrypt, encrypted file destination, and destination key location\n",
    "def encrypt(to_encrypt, file_destination, key_location):\n",
    "    key = Fernet.generate_key() # AES in CBC mode with a 128-bit key for encryption\n",
    "    fernet = Fernet(key)\n",
    "    \n",
    "    with open(key_location, 'wb') as f:\n",
    "        f.write(key)\n",
    "    \n",
    "    with open(to_encrypt, 'rb') as f:\n",
    "        plaintext = f.read()   \n",
    "    \n",
    "    encrypted = fernet.encrypt(plaintext)\n",
    "    with open(file_destination, 'wb') as e:\n",
    "        e.write(encrypted)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e3071a7",
   "metadata": {},
   "source": [
    "## Loading required data and creating the anonymised dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb7f54d0",
   "metadata": {},
   "source": [
    "The original dataset is loaded, an empty output Dataframe is initialised, and a postcode reference table is loaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b5970dab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Postcode</th>\n",
       "      <th>Region</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AB</td>\n",
       "      <td>Other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AL</td>\n",
       "      <td>England</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>B</td>\n",
       "      <td>England</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BA</td>\n",
       "      <td>England</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BB</td>\n",
       "      <td>England</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Postcode   Region\n",
       "0       AB    Other\n",
       "1       AL  England\n",
       "2        B  England\n",
       "3       BA  England\n",
       "4       BB  England"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in data to be anonymised\n",
    "original_data = pd.read_csv(\"Data/customer_information.csv\")\n",
    "\n",
    "# Reading in postcode_region.csv to map given postcode to countries in the UK - 'England' and 'Other'(includes Wales, Scotland, Northern Ireland)\n",
    "postcode_dictionary = pd.read_csv('Data/postcode_region.csv')\n",
    "\n",
    "# Create anon_data variable as initial data with unneeded direct identifiers dropped\n",
    "anon_data = pd.DataFrame()\n",
    "\n",
    "postcode_dictionary.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c06bf0b6",
   "metadata": {},
   "source": [
    "## Adding variables to the anonymised dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81ea4296",
   "metadata": {},
   "source": [
    "Assigning gender and case-control status as given"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "06a31d8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>CC.Status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Gender  CC.Status\n",
       "0      F          0\n",
       "1      M          0\n",
       "2      F          0\n",
       "3      F          0\n",
       "4      F          0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Assign gender\n",
    "anon_data['Gender'] = original_data['gender']\n",
    "\n",
    "# Assign case-control status\n",
    "anon_data['CC.Status'] = original_data['cc_status']\n",
    "\n",
    "anon_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2e22409",
   "metadata": {},
   "source": [
    "# Anonymisation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52a30590",
   "metadata": {},
   "source": [
    "## Pseudonymisation with hashed Sample ID"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612eaa75",
   "metadata": {},
   "source": [
    "Next, a unique Sample ID is created from the National Insurance Number to link the anonymised data with the reference data containing sensitive information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8ae51be7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>CC.Status</th>\n",
       "      <th>Sample.ID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>717341b0d3455426af5043290ec12b0681175d0b2da21a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>dd3daf05200664054f9162a61dd76d64802bff56437e20...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Gender  CC.Status                                          Sample.ID\n",
       "0      F          0  717341b0d3455426af5043290ec12b0681175d0b2da21a...\n",
       "1      M          0  ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...\n",
       "2      F          0  dd3daf05200664054f9162a61dd76d64802bff56437e20...\n",
       "3      F          0  a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...\n",
       "4      F          0  74e8b816673182819be3ba63b609ff22fb9b8f8a28426f..."
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Clean NIN formatting and assign Sample ID as a hashed form of the NIN\n",
    "key = os.urandom(16)\n",
    "original_data[\"national_insurance_number\"], anon_data['Sample.ID'], salts = zip(*original_data[\"national_insurance_number\"].apply(\n",
    "    lambda x: hash(re.sub(r'(.{2})(?!$)','\\\\1 ', x.replace(' ', '') ), key)))\n",
    "\n",
    "anon_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "da04a73e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Hashed.NIN</th>\n",
       "      <th>Salt</th>\n",
       "      <th>Key</th>\n",
       "      <th>NIN</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>717341b0d3455426af5043290ec12b0681175d0b2da21a...</td>\n",
       "      <td>ae064c90392c2d5c545da74d82c2ae52</td>\n",
       "      <td>ae823853bfce7c66904c0de363de9b2f</td>\n",
       "      <td>ZZ 19 48 92 T</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...</td>\n",
       "      <td>af9e92525c444b408b30a920c3e3fd1a</td>\n",
       "      <td>ae823853bfce7c66904c0de363de9b2f</td>\n",
       "      <td>ZZ 75 35 13 T</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>dd3daf05200664054f9162a61dd76d64802bff56437e20...</td>\n",
       "      <td>b735f1a9a61d9716d1dbed1db632d9c4</td>\n",
       "      <td>ae823853bfce7c66904c0de363de9b2f</td>\n",
       "      <td>ZZ 94 71 96 T</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...</td>\n",
       "      <td>dcb27a563ea605225fc2bccdb4e7c6ea</td>\n",
       "      <td>ae823853bfce7c66904c0de363de9b2f</td>\n",
       "      <td>ZZ 39 69 47 T</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...</td>\n",
       "      <td>8907a93f3a4de5bdf6dc3fd1d6de8337</td>\n",
       "      <td>ae823853bfce7c66904c0de363de9b2f</td>\n",
       "      <td>ZZ 30 98 91 T</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          Hashed.NIN  \\\n",
       "0  717341b0d3455426af5043290ec12b0681175d0b2da21a...   \n",
       "1  ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...   \n",
       "2  dd3daf05200664054f9162a61dd76d64802bff56437e20...   \n",
       "3  a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...   \n",
       "4  74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...   \n",
       "\n",
       "                               Salt                               Key  \\\n",
       "0  ae064c90392c2d5c545da74d82c2ae52  ae823853bfce7c66904c0de363de9b2f   \n",
       "1  af9e92525c444b408b30a920c3e3fd1a  ae823853bfce7c66904c0de363de9b2f   \n",
       "2  b735f1a9a61d9716d1dbed1db632d9c4  ae823853bfce7c66904c0de363de9b2f   \n",
       "3  dcb27a563ea605225fc2bccdb4e7c6ea  ae823853bfce7c66904c0de363de9b2f   \n",
       "4  8907a93f3a4de5bdf6dc3fd1d6de8337  ae823853bfce7c66904c0de363de9b2f   \n",
       "\n",
       "             NIN  \n",
       "0  ZZ 19 48 92 T  \n",
       "1  ZZ 75 35 13 T  \n",
       "2  ZZ 94 71 96 T  \n",
       "3  ZZ 39 69 47 T  \n",
       "4  ZZ 30 98 91 T  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a reference table between NIN and respective hashed NIN\n",
    "reference_table = pd.DataFrame()\n",
    "reference_table['Hashed.NIN'] = anon_data['Sample.ID']\n",
    "reference_table['Salt'] = salts\n",
    "reference_table['Key'] = key.hex()\n",
    "reference_table['NIN'] = original_data['national_insurance_number']\n",
    "\n",
    "reference_table.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be409014",
   "metadata": {},
   "source": [
    "## Banding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0841c30a",
   "metadata": {},
   "source": [
    "### Date of birth and education level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "32d0a350",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>CC.Status</th>\n",
       "      <th>Sample.ID</th>\n",
       "      <th>Birthyear</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>717341b0d3455426af5043290ec12b0681175d0b2da21a...</td>\n",
       "      <td>[1975, 1995)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>dd3daf05200664054f9162a61dd76d64802bff56437e20...</td>\n",
       "      <td>[1975, 1995)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...</td>\n",
       "      <td>[1955, 1975)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Gender  CC.Status                                          Sample.ID  \\\n",
       "0      F          0  717341b0d3455426af5043290ec12b0681175d0b2da21a...   \n",
       "1      M          0  ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...   \n",
       "2      F          0  dd3daf05200664054f9162a61dd76d64802bff56437e20...   \n",
       "3      F          0  a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...   \n",
       "4      F          0  74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...   \n",
       "\n",
       "      Birthyear  \n",
       "0  [1975, 1995)  \n",
       "1  [1995, 2015)  \n",
       "2  [1975, 1995)  \n",
       "3  [1995, 2015)  \n",
       "4  [1955, 1975)  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Birth years are extracted\n",
    "birthyears = pd.DatetimeIndex(original_data['birthdate']).year\n",
    "\n",
    "# Banding the birth years into 20-year intervals\n",
    "anon_data['Birthyear'] = pd.cut(birthyears, np.arange(birthyears.min(), birthyears.max()+20, 20), right=False)\n",
    "\n",
    "anon_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92cc6575",
   "metadata": {},
   "source": [
    "### Full postcode to countries within the UK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e93c7596",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>CC.Status</th>\n",
       "      <th>Sample.ID</th>\n",
       "      <th>Birthyear</th>\n",
       "      <th>Postcode</th>\n",
       "      <th>UK.Country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>717341b0d3455426af5043290ec12b0681175d0b2da21a...</td>\n",
       "      <td>[1975, 1995)</td>\n",
       "      <td>LS</td>\n",
       "      <td>England</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "      <td>M</td>\n",
       "      <td>England</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>dd3daf05200664054f9162a61dd76d64802bff56437e20...</td>\n",
       "      <td>[1975, 1995)</td>\n",
       "      <td>SO</td>\n",
       "      <td>England</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "      <td>B</td>\n",
       "      <td>England</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...</td>\n",
       "      <td>[1955, 1975)</td>\n",
       "      <td>TQ</td>\n",
       "      <td>England</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Gender  CC.Status                                          Sample.ID  \\\n",
       "0      F          0  717341b0d3455426af5043290ec12b0681175d0b2da21a...   \n",
       "1      M          0  ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...   \n",
       "2      F          0  dd3daf05200664054f9162a61dd76d64802bff56437e20...   \n",
       "3      F          0  a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...   \n",
       "4      F          0  74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...   \n",
       "\n",
       "      Birthyear Postcode UK.Country  \n",
       "0  [1975, 1995)       LS    England  \n",
       "1  [1995, 2015)        M    England  \n",
       "2  [1975, 1995)       SO    England  \n",
       "3  [1995, 2015)        B    England  \n",
       "4  [1955, 1975)       TQ    England  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Assign UK country derived from postcode by comparing to the postcode_dictionary reference table\n",
    "anon_data['Postcode'] = original_data['postcode'].apply(lambda x: re.search('[a-zA-Z]*', x).group(0))\n",
    "anon_data = pd.merge(anon_data, postcode_dictionary, on='Postcode', how='left')\n",
    "anon_data = anon_data.rename(columns={'Region': 'UK.Country'})\n",
    "\n",
    "anon_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9daf80f7",
   "metadata": {},
   "source": [
    "### Education level and country of birth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "82f9f648",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>CC.Status</th>\n",
       "      <th>Sample.ID</th>\n",
       "      <th>Birthyear</th>\n",
       "      <th>Postcode</th>\n",
       "      <th>UK.Country</th>\n",
       "      <th>Education.Level</th>\n",
       "      <th>Location.of.Birth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>717341b0d3455426af5043290ec12b0681175d0b2da21a...</td>\n",
       "      <td>[1975, 1995)</td>\n",
       "      <td>LS</td>\n",
       "      <td>England</td>\n",
       "      <td>Higher</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "      <td>M</td>\n",
       "      <td>England</td>\n",
       "      <td>BasicOther</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>dd3daf05200664054f9162a61dd76d64802bff56437e20...</td>\n",
       "      <td>[1975, 1995)</td>\n",
       "      <td>SO</td>\n",
       "      <td>England</td>\n",
       "      <td>Higher</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "      <td>B</td>\n",
       "      <td>England</td>\n",
       "      <td>BasicOther</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...</td>\n",
       "      <td>[1955, 1975)</td>\n",
       "      <td>TQ</td>\n",
       "      <td>England</td>\n",
       "      <td>BasicOther</td>\n",
       "      <td>Southern Hemisphere</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Gender  CC.Status                                          Sample.ID  \\\n",
       "0      F          0  717341b0d3455426af5043290ec12b0681175d0b2da21a...   \n",
       "1      M          0  ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...   \n",
       "2      F          0  dd3daf05200664054f9162a61dd76d64802bff56437e20...   \n",
       "3      F          0  a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...   \n",
       "4      F          0  74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...   \n",
       "\n",
       "      Birthyear Postcode UK.Country Education.Level    Location.of.Birth  \n",
       "0  [1975, 1995)       LS    England          Higher  Northern Hemisphere  \n",
       "1  [1995, 2015)        M    England      BasicOther  Northern Hemisphere  \n",
       "2  [1975, 1995)       SO    England          Higher  Northern Hemisphere  \n",
       "3  [1995, 2015)        B    England      BasicOther  Northern Hemisphere  \n",
       "4  [1955, 1975)       TQ    England      BasicOther  Southern Hemisphere  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Assign education level as banded education level\n",
    "anon_data['Education.Level'] = original_data['education_level'].map(lambda x: \"Higher\" if x in [\"bachelor\", \"masters\", \"phD\"] else \"BasicOther\")\n",
    "\n",
    "# Assign hemisphere of birth depending on country of birth\n",
    "anon_data['Location.of.Birth'] = original_data['country_of_birth'].apply(lambda x: country_to_hemisphere(x))\n",
    "\n",
    "anon_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f95bf1a9",
   "metadata": {},
   "source": [
    "## Data perturbation (adding Gaussian noise)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "144a2b16",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>CC.Status</th>\n",
       "      <th>Sample.ID</th>\n",
       "      <th>Birthyear</th>\n",
       "      <th>Postcode</th>\n",
       "      <th>UK.Country</th>\n",
       "      <th>Education.Level</th>\n",
       "      <th>Location.of.Birth</th>\n",
       "      <th>Weight</th>\n",
       "      <th>Height</th>\n",
       "      <th>Countries.Visited</th>\n",
       "      <th>Avg.Alcohol</th>\n",
       "      <th>Avg.Cigarettes</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>717341b0d3455426af5043290ec12b0681175d0b2da21a...</td>\n",
       "      <td>[1975, 1995)</td>\n",
       "      <td>LS</td>\n",
       "      <td>England</td>\n",
       "      <td>Higher</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "      <td>73.5</td>\n",
       "      <td>1.58</td>\n",
       "      <td>39.0</td>\n",
       "      <td>4.3</td>\n",
       "      <td>208.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M</td>\n",
       "      <td>0</td>\n",
       "      <td>ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "      <td>M</td>\n",
       "      <td>England</td>\n",
       "      <td>BasicOther</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "      <td>66.9</td>\n",
       "      <td>1.52</td>\n",
       "      <td>43.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>dd3daf05200664054f9162a61dd76d64802bff56437e20...</td>\n",
       "      <td>[1975, 1995)</td>\n",
       "      <td>SO</td>\n",
       "      <td>England</td>\n",
       "      <td>Higher</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "      <td>92.0</td>\n",
       "      <td>1.84</td>\n",
       "      <td>7.0</td>\n",
       "      <td>6.3</td>\n",
       "      <td>61.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "      <td>B</td>\n",
       "      <td>England</td>\n",
       "      <td>BasicOther</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "      <td>58.9</td>\n",
       "      <td>1.84</td>\n",
       "      <td>33.0</td>\n",
       "      <td>4.4</td>\n",
       "      <td>261.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>F</td>\n",
       "      <td>0</td>\n",
       "      <td>74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...</td>\n",
       "      <td>[1955, 1975)</td>\n",
       "      <td>TQ</td>\n",
       "      <td>England</td>\n",
       "      <td>BasicOther</td>\n",
       "      <td>Southern Hemisphere</td>\n",
       "      <td>101.9</td>\n",
       "      <td>1.74</td>\n",
       "      <td>29.0</td>\n",
       "      <td>4.9</td>\n",
       "      <td>348.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Gender  CC.Status                                          Sample.ID  \\\n",
       "0      F          0  717341b0d3455426af5043290ec12b0681175d0b2da21a...   \n",
       "1      M          0  ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...   \n",
       "2      F          0  dd3daf05200664054f9162a61dd76d64802bff56437e20...   \n",
       "3      F          0  a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...   \n",
       "4      F          0  74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...   \n",
       "\n",
       "      Birthyear Postcode UK.Country Education.Level    Location.of.Birth  \\\n",
       "0  [1975, 1995)       LS    England          Higher  Northern Hemisphere   \n",
       "1  [1995, 2015)        M    England      BasicOther  Northern Hemisphere   \n",
       "2  [1975, 1995)       SO    England          Higher  Northern Hemisphere   \n",
       "3  [1995, 2015)        B    England      BasicOther  Northern Hemisphere   \n",
       "4  [1955, 1975)       TQ    England      BasicOther  Southern Hemisphere   \n",
       "\n",
       "   Weight  Height  Countries.Visited  Avg.Alcohol  Avg.Cigarettes  \n",
       "0    73.5    1.58               39.0          4.3           208.0  \n",
       "1    66.9    1.52               43.0          1.5            50.0  \n",
       "2    92.0    1.84                7.0          6.3            61.0  \n",
       "3    58.9    1.84               33.0          4.4           261.0  \n",
       "4   101.9    1.74               29.0          4.9           348.0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add gaussian noise to weight, height, countries visited, average number of drinks in alcohol units per week and average cigrettes smoked per week.\n",
    "weight_noise = np.random.normal(0,1,1000)*5\n",
    "anon_data['Weight'] = round(original_data['weight']+weight_noise, 1)\n",
    "\n",
    "height_noise = np.random.normal(0,1,1000)/5\n",
    "anon_data['Height'] = round(original_data['height']+height_noise, 2)\n",
    "\n",
    "countries_noise = np.random.normal(0,1,1000)*5\n",
    "anon_data['Countries.Visited'] = round(original_data['n_countries_visited']+countries_noise)\n",
    "\n",
    "alcohol_noise = np.random.normal(0,1,1000)\n",
    "anon_data['Avg.Alcohol'] = round(original_data['avg_n_drinks_per_week']+alcohol_noise, 1)\n",
    "\n",
    "smoking_noise = np.random.normal(0,1,1000)*20\n",
    "anon_data['Avg.Cigarettes'] = round(original_data['avg_n_cigret_per_week']+smoking_noise)\n",
    "\n",
    "anon_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "452032a6",
   "metadata": {},
   "source": [
    "# Calculating K-anonymity using quasi-identifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7ff6ebf",
   "metadata": {},
   "source": [
    "The following code groups the quasi-identifiers specified, calculates the k-value, and returns a count of the \"unique\" rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "f3a90bed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The anonymised dataset is 2-anonymous; there are 0 unique quasi-identifier permutations.\n",
      "\n",
      "Least-frequent quasi-identifier permutations, in ascending order:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Birthyear</th>\n",
       "      <th>Location.of.Birth</th>\n",
       "      <th>UK.Country</th>\n",
       "      <th>Education.Level</th>\n",
       "      <th>Count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>47</td>\n",
       "      <td>M</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "      <td>Southern Hemisphere</td>\n",
       "      <td>Other</td>\n",
       "      <td>Higher</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>31</td>\n",
       "      <td>M</td>\n",
       "      <td>[1955, 1975)</td>\n",
       "      <td>Southern Hemisphere</td>\n",
       "      <td>Other</td>\n",
       "      <td>Higher</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>46</td>\n",
       "      <td>M</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "      <td>Southern Hemisphere</td>\n",
       "      <td>Other</td>\n",
       "      <td>BasicOther</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>19</td>\n",
       "      <td>F</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "      <td>Other</td>\n",
       "      <td>Higher</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>39</td>\n",
       "      <td>M</td>\n",
       "      <td>[1975, 1995)</td>\n",
       "      <td>Southern Hemisphere</td>\n",
       "      <td>Other</td>\n",
       "      <td>Higher</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index Gender     Birthyear    Location.of.Birth UK.Country Education.Level  \\\n",
       "0     47      M  [1995, 2015)  Southern Hemisphere      Other          Higher   \n",
       "1     31      M  [1955, 1975)  Southern Hemisphere      Other          Higher   \n",
       "2     46      M  [1995, 2015)  Southern Hemisphere      Other      BasicOther   \n",
       "3     19      F  [1995, 2015)  Northern Hemisphere      Other          Higher   \n",
       "4     39      M  [1975, 1995)  Southern Hemisphere      Other          Higher   \n",
       "\n",
       "   Count  \n",
       "0      2  \n",
       "1      2  \n",
       "2      2  \n",
       "3      2  \n",
       "4      3  "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking counts of quasi-identifiers permutations\n",
    "df_count = anon_data.groupby(['Gender', 'Birthyear', 'Location.of.Birth', \n",
    "                            'UK.Country', 'Education.Level']).size().reset_index(name = 'Count') \n",
    "counts_exceeding_0 = df_count.sort_values(\"Count\")[df_count.sort_values(\"Count\")['Count'] > 0].reset_index()\n",
    "\n",
    "# Calculate the k-value and the number of unique quasi-identifier permutations\n",
    "print(\"The anonymised dataset is \" + str(counts_exceeding_0['Count'][0]) + \"-anonymous; there are \" +\n",
    "      str(len(df_count[df_count['Count']==1])) + \" unique quasi-identifier permutations.\\n\")\n",
    "\n",
    "# Printing the final grouped output \n",
    "print(\"Least-frequent quasi-identifier permutations, in ascending order:\")\n",
    "counts_exceeding_0.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bf9b0cd",
   "metadata": {},
   "source": [
    "# Viewing and saving the anonymised dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "3ec90c01",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sample.ID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Birthyear</th>\n",
       "      <th>Location.of.Birth</th>\n",
       "      <th>UK.Country</th>\n",
       "      <th>Weight</th>\n",
       "      <th>Height</th>\n",
       "      <th>Education.Level</th>\n",
       "      <th>Avg.Alcohol</th>\n",
       "      <th>Avg.Cigarettes</th>\n",
       "      <th>Countries.Visited</th>\n",
       "      <th>CC.Status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>717341b0d3455426af5043290ec12b0681175d0b2da21a...</td>\n",
       "      <td>F</td>\n",
       "      <td>[1975, 1995)</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "      <td>England</td>\n",
       "      <td>73.5</td>\n",
       "      <td>1.58</td>\n",
       "      <td>Higher</td>\n",
       "      <td>4.3</td>\n",
       "      <td>208.0</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...</td>\n",
       "      <td>M</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "      <td>England</td>\n",
       "      <td>66.9</td>\n",
       "      <td>1.52</td>\n",
       "      <td>BasicOther</td>\n",
       "      <td>1.5</td>\n",
       "      <td>50.0</td>\n",
       "      <td>43.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>dd3daf05200664054f9162a61dd76d64802bff56437e20...</td>\n",
       "      <td>F</td>\n",
       "      <td>[1975, 1995)</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "      <td>England</td>\n",
       "      <td>92.0</td>\n",
       "      <td>1.84</td>\n",
       "      <td>Higher</td>\n",
       "      <td>6.3</td>\n",
       "      <td>61.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...</td>\n",
       "      <td>F</td>\n",
       "      <td>[1995, 2015)</td>\n",
       "      <td>Northern Hemisphere</td>\n",
       "      <td>England</td>\n",
       "      <td>58.9</td>\n",
       "      <td>1.84</td>\n",
       "      <td>BasicOther</td>\n",
       "      <td>4.4</td>\n",
       "      <td>261.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...</td>\n",
       "      <td>F</td>\n",
       "      <td>[1955, 1975)</td>\n",
       "      <td>Southern Hemisphere</td>\n",
       "      <td>England</td>\n",
       "      <td>101.9</td>\n",
       "      <td>1.74</td>\n",
       "      <td>BasicOther</td>\n",
       "      <td>4.9</td>\n",
       "      <td>348.0</td>\n",
       "      <td>29.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           Sample.ID Gender     Birthyear  \\\n",
       "0  717341b0d3455426af5043290ec12b0681175d0b2da21a...      F  [1975, 1995)   \n",
       "1  ff878150b526edf6a15e3bcc58e6a30bd2852f5dbef10c...      M  [1995, 2015)   \n",
       "2  dd3daf05200664054f9162a61dd76d64802bff56437e20...      F  [1975, 1995)   \n",
       "3  a17d9326ef0ada45cfbc51fdf59c70129813f7c4bf2421...      F  [1995, 2015)   \n",
       "4  74e8b816673182819be3ba63b609ff22fb9b8f8a28426f...      F  [1955, 1975)   \n",
       "\n",
       "     Location.of.Birth UK.Country  Weight  Height Education.Level  \\\n",
       "0  Northern Hemisphere    England    73.5    1.58          Higher   \n",
       "1  Northern Hemisphere    England    66.9    1.52      BasicOther   \n",
       "2  Northern Hemisphere    England    92.0    1.84          Higher   \n",
       "3  Northern Hemisphere    England    58.9    1.84      BasicOther   \n",
       "4  Southern Hemisphere    England   101.9    1.74      BasicOther   \n",
       "\n",
       "   Avg.Alcohol  Avg.Cigarettes  Countries.Visited  CC.Status  \n",
       "0          4.3           208.0               39.0          0  \n",
       "1          1.5            50.0               43.0          0  \n",
       "2          6.3            61.0                7.0          0  \n",
       "3          4.4           261.0               33.0          0  \n",
       "4          4.9           348.0               29.0          0  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Re-order columns\n",
    "anon_data = anon_data[['Sample.ID', 'Gender', 'Birthyear', 'Location.of.Birth', 'UK.Country', 'Weight', \n",
    "                        'Height', 'Education.Level', 'Avg.Alcohol', 'Avg.Cigarettes', 'Countries.Visited', 'CC.Status']]\n",
    "\n",
    "# View the anonymised dataset\n",
    "anon_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e8a4c92",
   "metadata": {},
   "source": [
    "## Creating CSV files for the anonymised data and the reference table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "5f1fd897",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output the files into .csv format\n",
    "output_name = \"anon_dataset\"\n",
    "anon_data.to_csv(output_name + \".csv\", sep=\",\", index=None)\n",
    "\n",
    "reference_table.to_csv(\"reference_table.csv\", sep=\",\", index=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b6bbb9d",
   "metadata": {},
   "source": [
    "## Encrypting the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "e2aa7e74",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encrypt csv and delete original file\n",
    "encrypt(output_name + \".csv\", output_name + \"_encrypted.csv\", \"key.key\")\n",
    "os.remove(output_name + \".csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "b37c51be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip freeze > requirements.txt"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.6 ('hda')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "caa27b422802b4fe8c2ce0d4ce59ab3cbffba11e0630735fb9c7a91a54a91b7c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}