Skip to content
LumenaryAcademy
Frontier 85 min

Responsible data science

Fairness you can measure and the trade-offs you cannot escape, privacy beyond removing names, consent and minimisation, documentation, and why so much applied work fails to reproduce.

Code tested · 6AI-draftedAwaiting review by a data scientist

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

Accurate for whom?

A lender's new credit model is more accurate than the old scorecard, cheaper to run and approves loans in seconds. Then someone splits the results by group and finds that, among applicants who went on to repay, one group was approved 76% of the time and the other 61%. Is the model unfair? Should it be changed, and to satisfy which definition of fair? Can the lender even check without collecting more sensitive data?

None of these questions is answered by accuracy, and all of them are part of the job. The record of applied data science contains enough harm done by careful, well-meaning people to make the point. A widely used US health-care algorithm predicted which patients needed extra care using their past costs as the target; because less had been spent on Black patients with the same needs, it under-referred them, and changing the target fixed most of the gap (Obermeyer and colleagues, 2019). Data sets released "anonymously" have repeatedly been re-identified. And a large share of published results built on machine learning fail when someone else tries to reproduce them.

This lesson treats responsibility as part of the craft, with code: measuring fairness and facing its trade-offs, protecting privacy properly, collecting only what you need, documenting data and models so others can judge them, and avoiding the errors behind the reproducibility crisis. The methods underneath are in Lumenary Statistics; here we put them to work.

DefinitionsH

Fairness, measured

Take a classifier that approves (Y^=1\hat Y = 1) or declines (Y^=0\hat Y = 0) applicants, an outcome YY (1 if the person repays, 0 if not), and a protected attribute AA such as sex, ethnicity or religion. The common group fairness criteria compare simple rates across the values of AA.

Definition— group fairness criteria

  • Demographic parity: equal approval rates, P(Y^=1∣A=a)P(\hat Y = 1 \mid A = a) the same for every group.
  • Equal opportunity: equal true positive rates, TPR=P(Y^=1∣Y=1,A=a)\text{TPR} = P(\hat Y = 1 \mid Y = 1, A = a): people who would repay are approved equally often in every group.
  • Equalised odds: equal true positive rates and equal false positive rates, FPR=P(Y^=1∣Y=0,A=a)\text{FPR} = P(\hat Y = 1 \mid Y = 0, A = a).
  • Predictive parity: equal positive predictive value, PPV=P(Y=1∣Y^=1,A=a)\text{PPV} = P(Y = 1 \mid \hat Y = 1, A = a): an approval means the same chance of repayment in every group. (Its stronger cousin, calibration within groups, asks the same of every score level.)

Each criterion protects something different. Demographic parity looks only at outcomes and ignores YY, which suits situations where the recorded outcome is itself suspect. Equal opportunity protects creditworthy people from being turned away more often because of their group. Predictive parity protects the meaning of the decision, which is what a lender (or a judge reading a risk score) relies on. In practice, the gaps are reported as differences or ratios; a gap of zero is rarely achievable, so the question becomes which gaps matter and how large is acceptable.

ComputationC

A lending model, audited

The data below are synthetic: 20,000 applicants in two groups, A and B, defined by a protected characteristic. Each applicant has an unseen capacity to repay; the lender sees only a documented income score and the length of their credit history, both noisy reflections of that capacity. In this synthetic world group B's capacity is lower on average (for reasons outside the data, such as less stable incomes), so its repayment rate is lower. The model is a logistic regression that is never shown the group. It approves applicants whose predicted probability of repaying is at least 0.75.

python
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

# Synthetic loan applicants from two groups, A and B, defined by a protected characteristic.
rng = np.random.default_rng(11)
n = 20_000
group = np.where(rng.random(n) < 0.35, "B", "A")
capacity = rng.normal(np.where(group == "A", 0.4, -0.2), 1.0)       # ability to repay (unseen)
income = capacity + rng.normal(0, 0.8, n)                           # documented income score
history = np.clip(rng.normal(24 + 8 * capacity, 10), 0, None)       # months of credit history
repaid = rng.random(n) < 1 / (1 + np.exp(-(1.3 + 1.4 * capacity)))
df = pd.DataFrame({"group": group, "income": income, "history": history, "repaid": repaid})
train, test = df.iloc[: n // 2], df.iloc[n // 2 :].copy()
model = LogisticRegression().fit(train[["income", "history"]], train["repaid"])  # group not used
test["score"] = model.predict_proba(test[["income", "history"]])[:, 1]

def rates(d, approve):
    """Approval rate, TPR, FPR and PPV of an approval rule, where 'positive' means repaid."""
    y = d["repaid"].to_numpy()
    return pd.Series({"repay rate p": y.mean(), "approved": approve.mean(), "TPR": approve[y].mean(),
                      "FPR": approve[~y].mean(), "PPV": y[approve].mean()})

table = pd.DataFrame({g: rates(d, d["score"].to_numpy() >= 0.75) for g, d in test.groupby("group")})
print("approve if the model's P(repay) >= 0.75; test applicants:", test["group"].value_counts().to_dict())
print(table.round(3).to_string())
print(f"\ndemographic parity difference (approval rate, A - B): {table.loc['approved', 'A'] - table.loc['approved', 'B']:.3f}")
print(f"equalised odds differences: TPR {table.loc['TPR', 'A'] - table.loc['TPR', 'B']:.3f}, "
      f"FPR {table.loc['FPR', 'A'] - table.loc['FPR', 'B']:.3f}")

# The model never saw the group, but its inputs carry it: how well do they predict the group?
auc = roc_auc_score(test["group"] == "B", -model.decision_function(test[["income", "history"]]))
print(f"AUC for telling group B from A using the model's score alone: {auc:.2f}")

# Chouldechova's identity, checked on each group: FPR = p/(1-p) * (1-PPV)/PPV * TPR
for g in "AB":
    p, ppv, tpr = table.loc["repay rate p", g], table.loc["PPV", g], table.loc["TPR", g]
    print(f"group {g}: FPR from the identity {p / (1 - p) * (1 - ppv) / ppv * tpr:.3f}, observed {table.loc['FPR', g]:.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.
group Agroup B00.510.6850.4950.7590.6120.3980.2380.8820.850approval rateTPRFPRPPVsingle threshold, P(repay) at least 0.75
The single-threshold lending model on synthetic test data, by group (numbers from the block above). Predictive value is close across groups; approval rates and both error rates are not.

Three things stand out. First, leaving the protected attribute out did not make the model blind to it: the score alone separates the groups with an AUC of 0.64, because income and credit history differ between groups. Removing a column removes the label, not the information, which is why "we don't use group" is not an audit. Second, the model approves 68.5% of group A and 49.5% of group B, a demographic parity gap of 19 points, and its true positive rates differ by 15 points: a group B applicant who would repay is approved 61% of the time, against 76% for group A. Third, its predictive value is close (88% against 85%): an approval means roughly the same thing in both groups. The last lines confirm, on these data, an identity worth proving.

DefinitionsH

Why you cannot have everything

Theorem— Chouldechova's identity

For any classifier and any group with prevalence p=P(Y=1)p = P(Y = 1) strictly between 0 and 1 in which the classifier has at least one true positive (so TPR>0\text{TPR} > 0 and PPV>0\text{PPV} > 0),

FPR=p1−p⋅1−PPVPPV⋅TPR.\text{FPR} = \frac{p}{1 - p} \cdot \frac{1 - \text{PPV}}{\text{PPV}} \cdot \text{TPR}.

Consequently, if two groups have different prevalences, no classifier can have equal PPV, equal TPR and equal FPR in both, except one that makes no false positives at all (PPV=1\text{PPV} = 1 in both).

Proof

Write the group's counts as true positives TPTP, false positives FPFP, false negatives FNFN and true negatives TNTN, with NN people in all. Then TP+FN=pNTP + FN = pN and FP+TN=(1−p)NFP + TN = (1-p)N. By definition TPR=TP/(pN)\text{TPR} = TP / (pN), so TP=TPR⋅pNTP = \text{TPR} \cdot pN. From PPV=TP/(TP+FP)\text{PPV} = TP/(TP + FP) we get FP=TP⋅(1−PPV)/PPVFP = TP \cdot (1 - \text{PPV}) / \text{PPV}. Dividing by (1−p)N(1-p)N gives FPR=FP/((1−p)N)=TPR⋅p1−p⋅1−PPVPPV\text{FPR} = FP/((1-p)N) = \text{TPR} \cdot \frac{p}{1-p} \cdot \frac{1 - \text{PPV}}{\text{PPV}}. If PPV and TPR are equal across groups, FPR is proportional to the odds p/(1−p)p/(1-p), which differ when prevalences differ, so FPR must differ unless the common factor (1−PPV)/PPV(1 - \text{PPV})/\text{PPV} is zero, that is, unless PPV=1\text{PPV} = 1 and the classifier makes no false approvals at all.

∎

Kleinberg, Mullainathan and Raghavan proved a closely related result for calibrated risk scores. These theorems were prompted by a real argument: a 2016 investigation found that a recidivism risk score used in US courts had higher false positive rates for Black defendants, and the score's makers replied that it had equal predictive value across groups. Both observations were correct, and the identity shows they had to be, given different recorded reoffending rates.

So choosing a fairness criterion means choosing which gaps to accept. The block below compares three policies on the same model: one threshold for everyone; separate thresholds that equalise approval rates (demographic parity), keeping the total approval rate unchanged; and separate thresholds that equalise true positive rates (equal opportunity).

python
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression

# The same synthetic applicants and model as in the previous block.
rng = np.random.default_rng(11)
n = 20_000
group = np.where(rng.random(n) < 0.35, "B", "A")
capacity = rng.normal(np.where(group == "A", 0.4, -0.2), 1.0)
income = capacity + rng.normal(0, 0.8, n)
history = np.clip(rng.normal(24 + 8 * capacity, 10), 0, None)
repaid = rng.random(n) < 1 / (1 + np.exp(-(1.3 + 1.4 * capacity)))
df = pd.DataFrame({"group": group, "income": income, "history": history, "repaid": repaid})
train, test = df.iloc[: n // 2], df.iloc[n // 2 :].copy()
model = LogisticRegression().fit(train[["income", "history"]], train["repaid"])
test["score"] = model.predict_proba(test[["income", "history"]])[:, 1]

def evaluate(d, thresholds):
    """Per-group threshold, approval rate, TPR, FPR and PPV; and the overall counts of
    approved borrowers who default and of declined applicants who would have repaid."""
    cols = {}
    for g, part in d.groupby("group"):
        y, ok = part["repaid"].to_numpy(), part["score"].to_numpy() >= thresholds[g]
        cols[g] = [thresholds[g], ok.mean(), ok[y].mean(), ok[~y].mean(), y[ok].mean()]
    ok = d["score"].to_numpy() >= d["group"].map(thresholds).to_numpy()
    y = d["repaid"].to_numpy()
    return cols, int((ok & ~y).sum()), int((~ok & y).sum())

a, b = test[test["group"] == "A"], test[test["group"] == "B"]
share = (test["score"] >= 0.75).mean()                       # keep the total approval rate
tpr = (test.loc[test["repaid"], "score"] >= 0.75).mean()     # keep the overall TPR
policies = {
    "single": {"A": 0.75, "B": 0.75},
    "parity": {"A": a["score"].quantile(1 - share), "B": b["score"].quantile(1 - share)},
    "opportunity": {"A": a.loc[a["repaid"], "score"].quantile(1 - tpr),
                    "B": b.loc[b["repaid"], "score"].quantile(1 - tpr)},
}
table, counts = {}, {}
for name, th in policies.items():
    cols, bad, missed = evaluate(test, th)
    for g in "AB":
        table[(name, g)] = cols[g]
    counts[name] = (bad, missed)
print(pd.DataFrame(table, index=["threshold", "approved", "TPR", "FPR", "PPV"]).round(3).to_string())
for name, (bad, missed) in counts.items():
    print(f"{name:12} approved borrowers who default: {bad}; declined applicants who would repay: {missed}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Each policy fixes one gap and opens another. Demographic parity equalises approval rates at 61.8%, but now group B's creditworthy applicants are approved more often than group A's (73% against 69%), and B's predictive value falls to 81.5%. Equal opportunity equalises TPR at 71% and, in these data, nearly equalises FPR too, so it comes close to equalised odds; the price, exactly as the theorem says, is a wider PPV gap (88.9% against 82.4%). The cost to the lender is modest in this example: about 40 more defaults among 10,000 applicants under either group-aware policy.

What should the lender do? The numbers cannot say. Some honest observations help.

  • Group-specific thresholds use the protected attribute directly, which the law forbids in many places and for many decisions (treating applicants differently because of their group), even when the aim is to reduce a disparity. Check with a lawyer before building one.
  • Better data often beats a better trade-off. If group B's incomes are less documented, recording informal income may raise both accuracy and fairness at once. Look for the cause of a gap before choosing which gap to accept.
  • The decision can change, not just the threshold. A smaller first loan, a guarantor or a payment holiday may serve applicants near the threshold better than a yes or a no.
  • Monitor after deployment, by group, because the data and populations change (data science in production).

Open-source toolkits such as Fairlearn and AIF360 compute these metrics and some mitigations; they are helpers for the arithmetic, not substitutes for the judgement.

ComputationC

Privacy: why removing names is not enough

Removing names and ID numbers does not make data anonymous. Latanya Sweeney estimated from 1990 US census data that 87% of the population was likely unique on just postcode, sex and date of birth (Sweeney, 2000); a later re-analysis with 2000 census data put it nearer 63% (Golle, 2006), still a majority. Anyone who knows those three facts about a person, and has a second data set that lists them with a name, can link the two. Richer data are even more identifying: a few dated film ratings were enough to pick out individuals in a large "anonymised" ratings data set (Narayanan and Shmatikov, 2008).

Definition— k-anonymity

The quasi-identifiers of a table are the columns that, combined, could identify someone through other data (age, sex, area, dates). A table is k-anonymous if every combination of quasi-identifier values that occurs is shared by at least kk rows, so each person is hidden in a crowd of at least kk.

The block measures kk for a synthetic table a lender might be asked to share with researchers, then generalises it, then shows the weakness of kk-anonymity, and finally shows two other tools: a keyed hash for pseudonyms, and a differentially private count.

python
import hashlib
import hmac
import numpy as np
import pandas as pd

# A synthetic table a lender might be asked to share with researchers.
rng = np.random.default_rng(5)
n = 2000
districts = ["North", "South", "East", "West", "Central", "Hillside", "Lakeside", "Riverside"]
people = pd.DataFrame({
    "national_id": [f"{x:08d}" for x in rng.integers(10_000_000, 40_000_000, n)],
    "age": rng.integers(18, 70, n),
    "sex": rng.choice(["F", "M"], n),
    "district": rng.choice(districts, n),
    "loan_kes": rng.choice([5_000, 10_000, 20_000, 50_000], n),
})
# After a flood in Riverside, almost every borrower there fell behind.
people["defaulted"] = rng.random(n) < np.where(people["district"] == "Riverside", 0.97, 0.15)

def k_anonymity(d, quasi):
    """k = the size of the smallest group of rows sharing the same quasi-identifier values."""
    sizes = d.groupby(quasi, observed=True).size()
    return int(sizes.min()), int((sizes == 1).sum())

quasi = ["age", "sex", "district"]
print("exact age, sex, district:   k = %d, people unique in the table: %d" % k_anonymity(people, quasi))

# Minimise and generalise: drop what the research does not need, coarsen what it does.
shared = people.drop(columns=["national_id"]).copy()
shared["age"] = pd.cut(shared["age"], [17, 29, 44, 70], labels=["18-29", "30-44", "45-69"])
print("age band, sex, district:    k = %d, people unique in the table: %d" % k_anonymity(shared, quasi))

# k-anonymity does not stop an attribute being learnt when a whole group shares it.
groups = shared.groupby(quasi, observed=True)["defaulted"].agg(["size", "mean"]).rename(columns={"mean": "default rate"})
print(groups[groups["default rate"] == 1].reset_index().to_string(index=False))

# Pseudonymise the key with a keyed hash (the key would live in a secrets store, not in code).
key = b"demo-key-for-this-lesson-only"
pseudo = [hmac.new(key, i.encode(), hashlib.sha256).hexdigest()[:12] for i in people["national_id"][:3]]
print("pseudonyms:", pseudo)

# A differentially private count: add Laplace noise with scale 1/epsilon.
true_count = int(people.query("district == 'North' and defaulted").shape[0])
for eps in [1.0, 0.1]:
    noisy = true_count + rng.laplace(0, 1 / eps, 5)
    print(f"defaults in North: true {true_count}; five releases at epsilon {eps}:", np.round(noisy).astype(int))
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

With exact ages, 166 of the 2,000 borrowers are the only person of their age and sex in their district: k=1k = 1. Coarsening age to three bands raises kk to 21. But kk-anonymity protects identity, not attributes: all 34 men aged 18 to 29 in Riverside defaulted, so anyone who knows a young man from Riverside is in the table learns that he defaulted without identifying his row. Refinements such as l-diversity require variety in sensitive values within each group, but every such rule can be defeated by an attacker with the right background knowledge, and generalising enough to stop them all usually destroys the data's usefulness.

Differential privacy takes a different approach: instead of transforming the data, it adds carefully calibrated random noise to every released statistic, so that the output barely depends on whether any one person's record is included. Its guarantee holds whatever the attacker knows, and the parameter ε\varepsilon sets the trade-off between privacy and accuracy. The theory and its use in official statistics, including the US 2020 census, are taught in privacy and official statistics; open-source libraries such as OpenDP implement it.

ExplanationH

Datasheets and model cards

A model or data set that others will use needs a description of what it is, how it was made, and where it should not be used. Two short formats have become standard. A datasheet for a data set (Gebru and colleagues) answers questions about motivation, composition, collection, preprocessing, intended uses, distribution and maintenance: who is in it, who is missing, how consent was obtained. A model card (Mitchell and colleagues) records a model's intended use, its training and evaluation data, its performance overall and by group, its ethical considerations and its known limitations. A minimal card for the lending model, with this lesson's numbers:

yaml
model: loan-approval-logistic v1.0 (synthetic lesson example)
intended_use: rank applications for a first microloan; a person reviews every decline
not_for: setting interest rates; any use outside the lending population it was trained on
inputs: documented income score, months of credit history (protected attribute not used)
training_data: 10,000 synthetic applicants; test data: 10,000 more from the same population
decision_rule: approve if predicted P(repay) >= 0.75
performance_by_group:   # test data
  A: {approval_rate: 0.685, TPR: 0.759, FPR: 0.398, PPV: 0.882}
  B: {approval_rate: 0.495, TPR: 0.612, FPR: 0.238, PPV: 0.850}
known_limitations:
  - score predicts group membership (AUC 0.64): inputs act as proxies
  - real outcomes are observed only for approved applicants (selective labels)
  - equal TPR, FPR and PPV cannot all hold while repayment rates differ
review: monthly, by group; next full re-audit after 6 months or any change of inputs

The value is less in the document than in the questions it forces. Filling in "who is missing from the data?" or "what should this not be used for?" is often where a problem is first noticed.

ComputationC

The reproducibility crisis in applied work

Kapoor and Narayanan surveyed reviews of machine-learning-based research and found data leakage (information from the test data reaching the model during training) in at least 294 papers across 17 fields, from medicine to political science; in one field they re-examined, civil-war prediction, complex models lost their claimed advantage over decades-old logistic regression once the leakage was fixed (Kapoor and Narayanan, 2023). The causes were mostly mundane: preprocessing on the whole data set before splitting, duplicates across training and test sets, features that would not be available at prediction time, and test sets drawn from a different population than the one the model would serve.

The block shows one classic case, which has misled research in genomics and elsewhere. With 120 cases and 5,000 measurements of pure noise, choosing the "best" 20 features using all the data before cross-validating makes a model look accurate; doing the selection inside each training fold gives the honest answer.

python
import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline

# 120 "patients", 5,000 measured "biomarkers", and an outcome that is pure coin-flipping.
rng = np.random.default_rng(3)
X = rng.normal(size=(120, 5000))
y = rng.integers(0, 2, 120)

# Leaky: choose the 20 features most associated with y using ALL the data, then cross-validate.
chosen = SelectKBest(f_classif, k=20).fit(X, y).get_support()
leaky = cross_val_score(LogisticRegression(max_iter=1000), X[:, chosen], y, cv=5).mean()

# Honest: the selection is part of the model, so it is redone inside every training fold.
pipe = make_pipeline(SelectKBest(f_classif, k=20), LogisticRegression(max_iter=1000))
honest = cross_val_score(pipe, X, y, cv=5).mean()
print(f"accuracy with selection before splitting: {leaky:.2f}")
print(f"accuracy with selection inside each fold: {honest:.2f}   (chance is 0.50)")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The leaky procedure reports 88% accuracy on coin flips. The honest one reports 57%, which with 120 cases (a standard error of about 0.05) is consistent with chance. Nothing in the leaky code looks wrong line by line; the error is in the order of operations. The cure is structural: put every step that learns from data (scaling, imputation, feature selection) inside a pipeline fitted on training folds only, and keep a final test set untouched until the end.

Leakage is one source of irreproducibility; analytic flexibility is the other. Many reasonable choices, tried until one "works", produce results that do not replicate (see multiple testing and reproducibility, and the forking-paths simulation in data science with AI assistants and agents). The REFORMS checklist, a consensus of researchers across fields, turns these lessons into 32 questions to answer before publishing a machine-learning-based result. Pre-registration of the analysis, sharing code and data where possible, and an independent replication before acting on a surprising finding are the habits that make results trustworthy.

Optional Christian reflection · Serving people, not metrics

A reflection from the Christian tradition that inspires Lumenary Academy. It is not part of the mathematics and you are welcome to skip it.

The parable of the shepherd who leaves ninety-nine sheep to look for the one that is lost (Luke 15:3–7) is not about statistics, but it is a striking picture of attention to the one case an average hides. A model's accuracy is an average over people, and each row of the lending table stands for someone hoping for a loan. Jesus' saying that "the sabbath was made for man, and not man for the sabbath" (Mark 2:27) can be read the same way: measures exist to serve people, not people to serve measures. Whatever one believes, the discipline it suggests is practical. Before optimising a number, ask whom it serves, and whom it might leave behind.

Common misconceptionsH

Common misconceptions

Misconception

"If the model doesn't use the protected attribute, it can't be unfair." The lesson's model never saw the group, yet its score separated the groups (AUC 0.64) and its error rates differed by about 15 points. Other inputs carry the information. Leaving the attribute out also makes the disparity harder to measure.

Misconception

"There is one correct definition of fairness, and we just need to find it." When base rates differ, equal error rates and equal predictive values are mathematically incompatible. Which to prioritise depends on the decision, the harms, the law and the people affected; the data scientist's job is to make the trade-off visible, not to settle it alone.

Misconception

"Hashed or pseudonymised data are anonymous." A keyed hash hides the ID from those without the key, but the remaining columns can still single people out, and the law treats such data as personal. An unkeyed hash of an ID number can simply be reversed by hashing every possible ID.

Misconception

"k-anonymous data cannot reveal anything about an individual." It hides which row is whose, not what a group has in common: the young men of Riverside were hidden in a crowd of 34 who had all defaulted.

Exercises

Exercises

1. Responsible data science

Warm-upHuman

An audit finds that among applicants who went on to repay, 76% of group A were approved but only 61% of group B. Which fairness criterion does this violate?

Choose one

2. Responsible data science

CoreHuman

A lender approved 260 of 400 applicants from group A and 120 of 250 applicants from group B. What is the demographic parity difference (approval rate of A minus approval rate of B), in percentage points?

3. Responsible data science

CoreHuman

In one group, 75% of applicants would repay (p=0.75p = 0.75). A model approves 60% of those who would repay (TPR = 0.6), and 90% of the people it approves do repay (PPV = 0.9). Using Chouldechova's identity, what percentage of the people who would not repay does it approve (the FPR)? Give a whole number.

4. Responsible data science

CoreComputational

Write group_rates(df, threshold). df has columns group, score and repaid (True if the person repaid). Approving everyone with score >= threshold, return a DataFrame indexed by group (sorted) with columns approved, TPR, FPR and PPV, in that order, where 'positive' means repaid.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

5. Responsible data science

StretchComputational

Write privacy_report(df, quasi, sensitive) returning a pair of integers (k, l): k is the size of the smallest group of rows that share the same values of the quasi-identifier columns quasi (so the table is k-anonymous), and l is the smallest number of distinct values of the column sensitive within any such group (so it is l-diverse). A group with l = 1 reveals its sensitive value to anyone who knows a person is in it.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

6. Responsible data science

CoreHuman

A team replaces every national ID number with an HMAC pseudonym (the key is kept by a different team) and keeps age, sex, district, loan amount and default status. Under the GDPR and similar laws, what is the resulting table?

Choose one

7. Responsible data science

CoreHuman

A team predicts hospital readmission from patient records. Which of these procedures leak information from the test data into training? Select all that apply.

Choose all that apply

Further reading and resources