Dataset potions

Feature Scatter

Synthetic two-dimensional classification dataset with a linear boundary and 20% label noise.

Provenance

uv run ./random_classification_data_generator.py --number-of-data-points 200 --feature1-name "FizzIntensity"  --feature1-mean 40  --feature1-stddev 12  --feature2-name "ColourShift"  --feature2-mean 15  --feature2-stddev 5  --target-column-name "PotionEffectiveness"  --primary-key-name "PotionBatchID"  --table-name "magic_potions"  --splits-table-name "potion_splits"  --random-seed 42   --class-deciding-expression "3 * FizzIntensity + ColourShift > 140"  --false-class-name "Ineffective"  --true-class-name "Effective"  --noise 0.20  --holdout 0.2  --validation 0.5  --output-db-file $(TEMPLATES_DIR)/$(POTIONS_DATASET).sqlite  --config-file configs/$(POTIONS_DATASET).config.json --use-uuid

Best Validation Round Rank

Test scores by release date

Regression slope: 0.0002, intercept: -3.1906, p-value: 0

Debug

merge info: 35 rows
filter ollama: 33 rows
drop NA release/accuracy: 33 rows
compute KT and sort
added baselines
create figure
scatter models: [(datetime.date(2024, 1, 25), -0.1256595574906267), (datetime.date(2024, 7, 18), -0.18758554574618913), (datetime.date(2024, 10, 22), -0.1555197331793902), (datetime.date(2024, 10, 22), -0.2222090105096017), (datetime.date(2024, 11, 20), -0.2222090105096017), (datetime.date(2024, 11, 20), -0.2222090105096017), (datetime.date(2024, 12, 5), -0.25983377939935925), (datetime.date(2024, 12, 5), -0.2222090105096017), (datetime.date(2025, 1, 30), -0.18758554574618913), (datetime.date(2025, 1, 30), -0.2222090105096017), (datetime.date(2025, 2, 5), -0.2222090105096017), (datetime.date(2025, 2, 5), -0.18758554574618913), (datetime.date(2025, 2, 19), -0.1256595574906267), (datetime.date(2025, 2, 19), -0.18758554574618913), (datetime.date(2025, 2, 27), -0.2222090105096017), (datetime.date(2025, 2, 27), -0.34654747821528553), (datetime.date(2025, 3, 25), -0.1256595574906267), (datetime.date(2025, 4, 14), -0.1256595574906267), (datetime.date(2025, 4, 14), -0.34654747821528553), (datetime.date(2025, 4, 16), -0.1256595574906267), (datetime.date(2025, 4, 16), -0.09772102067959801), (datetime.date(2025, 5, 14), -0.25983377939935925), (datetime.date(2025, 5, 14), -0.3010299956639812), (datetime.date(2025, 5, 14), -0.1555197331793902), (datetime.date(2025, 5, 14), -0.18758554574618913), (datetime.date(2025, 5, 14), -0.34654747821528553), (datetime.date(2025, 5, 14), -0.52144068276802), (datetime.date(2025, 5, 14), -0.1555197331793902), (datetime.date(2025, 5, 14), -0.2222090105096017), (datetime.date(2025, 5, 14), -0.34654747821528553), (datetime.date(2025, 5, 14), -0.1555197331793902), (datetime.date(2025, 5, 14), -0.1256595574906267), (datetime.date(2025, 5, 14), -0.2222090105096017)]
xlim_min 2024-01-01
scatter ensembles: 2
draw ensemble trend line
xlim_max 2025-05-14
axhline decision trees at -0.166
annotate decision trees at edge 20222.0
mark baseline crossing
savefig website/dataset/potions/release_scores_potions.png

Model Scores

Train vs Validation

Regression slope: 1.3376, intercept: -0.1567, p-value: 0.00042013

Validation vs Train

Regression slope: 0.2503, intercept: 0.5030, p-value: 0.00042013

Ensemble Max Scores

Baseline

Baseline Models

Logistic Regression

take the colourshift and multiply by -0.008, take the fizzintensity and multiply by -0.107, add those up and if the number is greater than 4.645 then predict Effective, otherwise predict Ineffective.

Decision Tree

Dummy

Always predict 1

RuleFit

Bayesian Rule List

CORELS

1. {'col': 'X4', 'index_col': 4, 'cutoff': 0.5, 'val': 0.21875, 'flip': True, 'val_right': 0.65625, 'num_pts': 160, 'num_pts_right': 128}
2. {'col': 'X8', 'index_col': 8, 'cutoff': 0.5, 'val': 0.125, 'flip': False, 'val_right': 0.5, 'num_pts': 32, 'num_pts_right': 8}
3. {'col': 'X9', 'index_col': 9, 'cutoff': 0.5, 'val': 0.0, 'flip': True, 'val_right': 0.15789473684210525, 'num_pts': 24, 'num_pts_right': 19}
4. {'val': 0.0, 'num_pts': 5}

EBM