Dataset espionage

Feature Scatter

Synthetic two-dimensional classification dataset with a linear boundary and no label noise.

Provenance

uv run ./random_classification_data_generator.py --number-of-data-points 200 --feature1-name "SecretHandshakeQuality" --feature1-mean 70 --feature1-stddev 10 --feature2-name "AccentThickness" --feature2-mean 30 --feature2-stddev 8 --target-column-name "AgentStatus" --primary-key-name "AgentID" --table-name "espionage_agents"  --splits-table-name "espionage_splits" --random-seed 42  --class-deciding-expression "SecretHandshakeQuality - AccentThickness > 35" --false-class-name "Loyal"  --true-class-name "DoubleAgent" --noise 0.0 --holdout 0.2 --validation 0.5 --output-db-file $(TEMPLATES_DIR)/$(ESPIONAGE_DATASET).sqlite --config-file  configs/$(ESPIONAGE_DATASET).config.json --use-uuid

Best Validation Round Rank

Test scores by release date

Regression slope: 0.0006, intercept: -12.7634, p-value: 0.066381

Debug

merge info: 33 rows
filter ollama: 31 rows
drop NA release/accuracy: 31 rows
compute KT and sort
added baselines
create figure
scatter models: [(datetime.date(2024, 1, 25), -0.09772102067959801), (datetime.date(2024, 7, 18), -0.1555197331793902), (datetime.date(2024, 10, 22), -0.09772102067959801), (datetime.date(2024, 10, 22), -0.07147167409197236), (datetime.date(2024, 11, 20), -0.04671885117881216), (datetime.date(2024, 11, 20), -0.09772102067959801), (datetime.date(2024, 12, 5), -0.04671885117881216), (datetime.date(2024, 12, 5), -0.023301077408850783), (datetime.date(2025, 1, 30), -0.09772102067959801), (datetime.date(2025, 1, 30), -0.25983377939935925), (datetime.date(2025, 2, 5), -0.3974001770526203), (datetime.date(2025, 2, 5), -0.09772102067959801), (datetime.date(2025, 2, 19), -0.04671885117881216), (datetime.date(2025, 2, 19), -0.09772102067959801), (datetime.date(2025, 3, 25), -0.023301077408850783), (datetime.date(2025, 4, 14), -0.04671885117881216), (datetime.date(2025, 4, 14), -0.04671885117881216), (datetime.date(2025, 4, 16), -0.09772102067959801), (datetime.date(2025, 4, 16), -0.1256595574906267), (datetime.date(2025, 5, 14), -0.09772102067959801), (datetime.date(2025, 5, 14), -0.09772102067959801), (datetime.date(2025, 5, 14), -0.023301077408850783), (datetime.date(2025, 5, 14), -0.0010816804642877792), (datetime.date(2025, 5, 14), -0.04671885117881216), (datetime.date(2025, 5, 14), -0.0010816804642877792), (datetime.date(2025, 5, 14), -0.023301077408850783), (datetime.date(2025, 5, 14), -0.023301077408850783), (datetime.date(2025, 5, 14), -0.023301077408850783), (datetime.date(2025, 5, 14), -0.023301077408850783), (datetime.date(2025, 5, 14), -0.07147167409197236), (datetime.date(2025, 5, 14), -0.0010816804642877792)]
xlim_min 2024-01-01
scatter ensembles: 5
draw ensemble trend line
xlim_max 2025-05-14
axhline bayesian rule list at -0.081
annotate bayesian rule list at edge 20222.0
mark baseline crossing
savefig website/dataset/espionage/release_scores_espionage.png

Model Scores

Train vs Validation

Regression slope: 1.0346, intercept: 0.0022, p-value: 1.0514e-11

Validation vs Train

Regression slope: 0.7749, intercept: 0.1669, p-value: 1.0514e-11

Ensemble Max Scores

Baseline

Baseline Models

Logistic Regression

take the accentthickness and multiply by 1.580, take the secrethandshakequality and multiply by -1.676, add those up and if the number is greater than 61.565 then predict DoubleAgent, otherwise predict Loyal.

Decision Tree

Dummy

Always predict 0

RuleFit

Bayesian Rule List

CORELS

1. {'col': 'X9', 'index_col': 9, 'cutoff': 0.5, 'val': 0.265625, 'flip': False, 'val_right': 0.75, 'num_pts': 160, 'num_pts_right': 32}
2. {'col': 'X0', 'index_col': 0, 'cutoff': 0.5, 'val': 0.15841584158415842, 'flip': False, 'val_right': 0.6666666666666666, 'num_pts': 128, 'num_pts_right': 27}
3. {'col': 'X8', 'index_col': 8, 'cutoff': 0.5, 'val': 0.013888888888888888, 'flip': False, 'val_right': 0.5172413793103449, 'num_pts': 101, 'num_pts_right': 29}
4. {'col': 'X1', 'index_col': 1, 'cutoff': 0.5, 'val': 0.0, 'flip': False, 'val_right': 0.07142857142857142, 'num_pts': 72, 'num_pts_right': 14}
5. {'val': 0.0, 'num_pts': 58}

EBM