This article explores how machine learning models can predict the severity of cyber security incidents. As cyber insurance becomes increasingly important, companies need better ways to assess and price cyber risks accurately.
Project Overview
This research aims to develop machine learning models that can accurately predict the severity of cyber security incidents. Our goal is to create tools that help cyber insurance companies better assess risks, set appropriate premiums, and manage claims more effectively. By predicting how many people will be affected by a data breach, insurers can make more informed decisions about policy pricing and risk management.
To achieve this, we analyze real-world data from the Information Commissioner's Office (ICO), a UK-based independent authority dedicated to upholding information rights. The ICO's database contains detailed records of actual data security incidents and their consequences, providing valuable insights into how cyber breaches unfold in practice. Using this comprehensive dataset, we develop and test predictive models that can generalize to new, unseen incidents, addressing a critical need for robust risk assessment tools in the cyber insurance industry.
Main Challenges
Predicting cyber incident severity presents several significant technical challenges that traditional machine learning approaches struggle to address effectively. The most critical obstacles include handling the ordered nature of severity levels (ordinality), dealing with highly imbalanced data where severe incidents are rare, and creating meaningful features from complex incident data. These challenges require specialized approaches and careful consideration to build models that can accurately predict real-world cyber risks.
Raw Data Analysis
The dependent variable "No. Data Subjects Affected" (highlighted in the table below) serves as a proxy measure for incident severity, representing the ordinal scale of individuals impacted by each breach. While direct severity metrics are unavailable in public datasets, the number of affected data subjects provides a meaningful approximation of incident magnitude, as larger breaches typically correlate with greater financial losses, regulatory penalties, and reputational damage.
| Column | Data Type | Unique Values | Top 5 Values |
|---|---|---|---|
| BI Reference | Object | 14,835 | N/A |
| Quarter | Categorical | 4 | Q3 (n=14,777), Q2 (n=14,082), Q1 (n=13,875), Q4 (n=12,760) |
| Data Subject Type | Categorical | 9 | Employees (n=17,856), Customers/prospects (n=15,605), Users (n=5,721), Unknown (n=3,987), Children (n=3,916) |
| Data Type | Categorical | 16 | Basic personal identifiers (n=19,199), Economic/financial data (n=7,627), Identification data (n=6,122), Official documents (n=4,830), Health data (n=4,448) |
| Decision Taken | Categorical | 4 | No Further Action (n=21,793), Investigation Pursued (n=18,411), Informal Action Taken (n=14,629), Not Yet Assigned (n=661) |
| Incident Type | Categorical | 7 | Phishing (n=19,471), Ransomware (n=18,828), Other cyber incident (n=7,364), Unauthorised access (n=6,007), Malware (n=2,403) |
| Sector | Categorical | 23 | Education and childcare (n=10,424), Retail and manufacture (n=9,652), Finance insurance and credit (n=6,622), General business (n=4,200), Legal (n=4,022) |
| Time Taken to Report | Categorical | 4 | 24 hours to 72 hours (n=25,123), 72 hours to 1 week (n=12,224), More than 1 week (n=9,132), Less than 24 hours (n=9,015) |
| No. Data Subjects Affected (Target Variable) | Categorical | 7 | Unknown (n=18,282), 100 to 1k (n=13,018), 1k to 10k (n=9,554), 10 to 99 (n=6,332), 1 to 9 (n=3,910) |
Data Processing Pipeline
Source: The dataset is sourced from the ICO's data security incident trends (ICO dataset).
The data processing pipeline transforms raw ICO incident reports into model-ready datasets through four key stages. Initial data cleaning addresses duplicate entries, missing values, and inconsistent reporting formats across the 55,494 incident records. Feature engineering creates predictive variables such as temporal features (time since reporting), data type severity scores, and data subject breadth metrics that capture incident complexity. Feature preprocessing involves encoding categorical variables, scaling numerical features, and ensuring data consistency for machine learning algorithms. Finally, class imbalance handling applies techniques like SMOTE oversampling and weighted loss functions to address the scarcity of high-severity incidents, ensuring models can effectively learn from rare but critical cyber events.
Duplicate Management Strategy
The ICO dataset contains inherent complexities due to repeated reporting and incident updates that create duplicate entries, which can significantly skew analysis and compromise model training accuracy if left unaddressed. To manage this challenge, we implement a comprehensive two-level deduplication strategy that systematically groups entries by BI Reference, Year, and Quarter while comparing critical fields including Data Subject Type, Incident Type, and Decision Taken. When these key fields differ for the same incident, the system creates separate incident records to preserve data integrity. The ID assignment logic ensures traceability by maintaining the original BI Reference for first occurrences while assigning incremental suffixes (such as ABC123A1, ABC123A2) to subsequent variations. This approach handles two primary split scenarios: time period splits where identical BI References appear across different quarters, and field value splits where core incident characteristics vary within the same reporting period. This deduplication framework ensures that each unique cyber incident is properly represented while maintaining the granular detail necessary for accurate severity prediction modeling.
Feature Engineering Details
Feature engineering involves creating new features from existing data to improve model performance. For this study, we developed several key features to capture important aspects of cyber incidents.
Temporal Features: We engineered Years Since Start (years elapsed since 2019) to normalize temporal data for scale-sensitive models, and Quarter Encoding using one-hot encoding to preserve the cyclical nature of quarters rather than simple numerical encoding.
Data Type Score: This severity assessment assigns scores (1-10) based on potential impact if compromised. The scoring system reflects both PII sensitivity and desirability to attackers: Economic and financial data (10), Identification data (9), Genetic or biometric data (9), Health data (8), Criminal convictions or offences (8), Official documents (7), Sex life data (6), Location data (5), Data revealing racial or ethnic origin (4), Religious or philosophical beliefs (3), Political opinions (2), Trade union membership (2), and Basic personal identifiers (1). For each incident, we sum scores of unique affected data types and standardize the result. For example, an incident affecting "Economic and financial data" (10) and "Health data" (8) receives a standardized score based on the sum of 18.
Data Subject Type Count: This feature counts distinct types of data subjects affected (employees, customers, patients), as incidents affecting multiple subject types indicate broader impact and potentially greater severity.
Categorical Variable Encoding: We applied dummy variable encoding (one-hot) to Sector, Incident Type, and Decision Taken, using a drop-first strategy to prevent multicollinearity while removing zero-variance columns that provide no model information.
Ordinal Variable Encoding: We preserved natural ordering in "No. Data Subjects Affected" and "Time Taken to Report" using ordered encoding, replacing categories with numerical values that respect severity magnitude relationships.
Missing Value Imputation: We used Multivariate Imputation by Chained Equations (MICE) for the target variable, which iteratively imputes missing values using predictive models (linear regression for continuous, logistic for binary, proportional odds for ordinal data) while accounting for inter-variable relationships and uncertainty. Distribution validation ensured imputation quality and data integrity.
Selected Models
This study employs four specialized machine learning models designed to handle ordinal classification tasks for predicting cyber security incident severity. Each model addresses the unique challenges of ordered categorical data while offering distinct advantages for cyber insurance applications.
| Model | Description | Pros | Cons |
|---|---|---|---|
| Ordered Logistic Regression | Immediate-Threshold (IT) ordinal logistic model that treats ordinal classification as a series of binary classification problems between adjacent severity levels. | Handles ordinal data directly; provides interpretable coefficients; computationally efficient; extends naturally from binary logistic regression. | Relies on proportional odds assumption; assumes equal intervals between categories; affected by class imbalance; may underperform on high-severity incidents. |
| Ordinal Random Forest | Combines random forests with cumulative probability estimation to respect ordinal structure while maintaining non-parametric flexibility for complex relationships. | Handles ordinal data without strong assumptions; non-parametric; captures complex relationships; robust to outliers; provides probability estimates. | Less interpretable than regression models; computationally intensive; still affected by class imbalance; requires careful hyperparameter tuning. |
| CatBoost | Uses ordered boosting with symmetric decision trees and native categorical feature handling, employing ordered target encoding to prevent data leakage. | Handles categorical features effectively; robust to overfitting; high predictive accuracy; efficient implementation; automatic feature preprocessing. | Less interpretable than linear models; doesn't directly optimize ordinal-specific loss function; computationally intensive for large datasets. |
| Neural Network with CORN | CORN (Conditional Ordinal Regression for Neural Networks) reformulates ordinal regression as binary classification tasks for adjacent classes, modeling conditional probabilities P(Y>m|Y≥m,x) to ensure rank consistency without requiring monotonic thresholds. This approach guarantees proper probability distributions while enabling flexible neural network architectures. | Specifically designed for ordinal data; learns complex non-linear relationships; flexible architecture; handles high-dimensional data; ensures rank consistency by design. | Computationally expensive; less interpretable; dependent on architecture and hyperparameters; requires careful regularization; sensitive to class imbalance. |
The CORN methodology represents a significant advancement in ordinal neural networks by addressing rank consistency issues that plague traditional approaches. Unlike standard ordinal regression that requires monotonic threshold parameters, CORN models conditional probabilities recursively, ensuring that predicted probabilities naturally sum to 1.0 and maintain proper ordering without post-processing constraints. This makes it particularly suitable for our use case as maintaining the natural ordering of severity levels is crucial for accurate risk assessment.
Hyperparameter Optimization and Model Evaluation Summary
A robust hyperparameter tuning strategy was implemented for each ordinal model, leveraging custom tuner classes and 5-fold cross-validation to optimize performance and generalization. Both the hyperparameter optimization process and the final model evaluation utilized the Weighted MAE loss function as the primary optimization criterion. The following results present the optimal model performance on the held-out test dataset, representing the models' ability to predict cyber incident severity on previously unseen data.
The confusion matrices display prediction accuracy across six ordinal severity classes: Class 1 (1-9 data subjects affected), Class 2 (10-99 subjects), Class 3 (100-1k subjects), Class 4 (1k-10k subjects), Class 5 (10k-100k subjects), and Class 6 (100k+ subjects). Each matrix shows true classes (rows) versus predicted classes (columns), where diagonal elements represent correct predictions and off-diagonal elements indicate misclassifications. For ordinal data, predictions close to the diagonal (adjacent class errors) are more acceptable than distant misclassifications that severely violate the natural ordering of incident severity.
Model evaluation was grounded in metrics reflecting class imbalance and ordinality: Mean Absolute Error (MAE), accuracy, class-frequency-weighted F1, Weighted MAE (emphasizing rare classes), and Closeness Evaluation Measure (CEM). The Weighted MAE metric was specifically chosen as it penalizes misclassifications of rare, high-severity incidents more heavily than common, low-severity cases, making it particularly suitable for cyber insurance applications where accurately predicting catastrophic events is paramount.
While the Ordinal Neural Network delivered the strongest general results, CatBoost Ordinal demonstrated superior balance in capturing minority, high-severity classes, reliably identifying extreme outcomes (very large breaches). This makes CatBoost uniquely valuable for cyber insurance actuarial and underwriting applications, where properly weighting catastrophic loss scenarios is paramount.
Optimal hyperparameter configurations discovered through Weighted MAE optimization for each ordinal model, displaying the best-performing parameter settings identified via 5-fold cross-validation.
Class Error Distribution Analysis
The error distribution plots provide crucial insights into ordinal prediction quality by showing not just accuracy, but how "close" incorrect predictions are to the true severity classes. These histograms aggregate prediction errors from all 5-fold cross-validation iterations, combining results from multiple independently trained models to ensure robust evaluation. Each subplot displays the percentage distribution of absolute errors (|predicted_class - true_class|) for one model, where Error = 0 represents perfect predictions, Error = 1 indicates acceptable adjacent-class mistakes (e.g., predicting "10-99 affected" when true is "100-1k affected"), and Error ≥ 2 reveals problematic distant misclassifications that violate ordinal relationships.
Each histogram includes statistical summaries showing Mean Absolute Error, Median Absolute Error, Perfect Predictions percentage, and Within ±1 Class accuracy. Models with tighter error distributions concentrated at low error values demonstrate superior ordinal awareness and are particularly valuable for cyber insurance applications where maintaining severity order is crucial for accurate risk assessment.
Cross-validation aggregated error distributions showing absolute prediction errors for each model. Higher concentrations at Error = 0-1 indicate better ordinal prediction quality and respect for severity class ordering.
Comprehensive performance metrics (Accuracy, Weighted F1, Weighted MAE, and CEM) aggregated across all cross-validation folds, enabling robust model comparison for cyber incident severity prediction.
Most Relevant Features for Severity Prediction
Understanding which features most significantly influence cybersecurity breach severity prediction is crucial for both model interpretability and practical application. Analysis consistently highlights several key predictive features:
- Data Type Score is the most important feature, reflecting the severity and sensitivity of compromised data types and their potential impact on affected individuals.
- Years Since Start captures temporal changes, suggesting incident severity has evolved due to changing threat landscapes and reporting practices.
- Data Subject Type Count measures breach breadth across different stakeholder groups, with wider impact linking to higher severity.
- Time Taken to Report and Incident Types (especially Phishing and Ransomware) indicate that detection delays and specific attack vectors increase severity likelihood.
- Sector and regulatory features show less predictive power, suggesting data characteristics outweigh organizational context.
Two caveats are worth noting in interpreting the strong influence of data type. First, there may be a selection effect in reporting: incidents involving highly sensitive categories (e.g., financial or health data) are more clearly identifiable as reportable breaches, potentially inflating their representation relative to less sensitive incidents. Second, our subjective classification scheme for assigning severity scores to data types may, by coincidence, align closely with attacker incentives. If cyber criminals disproportionately target high-value data, then both our scoring and the reporting patterns will reinforce each other, amplifying the apparent predictive power of these variables. These dynamics suggest that the observed importance of data type could partly reflect underlying reporting and targeting biases, rather than purely intrinsic relationships.
While predicting the number of data subjects affected provides a useful proxy for incident severity, care must be taken not to assume a simplistic relationship of Loss = (# of subjects) × (average $ loss per subject). Both the number of affected individuals and the per-subject cost are strongly conditioned by the type of data compromised. Modeling these factors independently risks obscuring their interaction and underestimating extreme loss scenarios. Future work on loss quantification should jointly model subject counts and subject types to capture these dependencies and better reflect the true distribution of cyber losses.
Conclusion
This research demonstrates that ordinal classification approaches can effectively capture the natural ordering of cyber incident severity, with key insights emerging for risk assessment practitioners. The prominence of data type sensitivity and stakeholder breadth as primary predictors suggests that incidents involving sensitive information (financial, health, biometric data) and affecting multiple stakeholder groups consistently correlate with higher severity outcomes.
The findings highlight that what data is compromised and who is affected matters more than organizational context or sector-specific factors for predicting incident magnitude. This insight can inform risk prioritization decisions, whether for insurance underwriting, regulatory compliance planning, or cybersecurity investment allocation. However, practitioners should be aware of potential reporting biases toward high-sensitivity data types and the importance of modeling data characteristics jointly rather than independently when estimating losses.
As cyber threats continue to evolve, these findings provide a quantitative foundation for evidence-based risk assessment, emphasizing the critical importance of data classification and stakeholder impact analysis in predicting the true scope of cybersecurity incidents.
References
Academic Publications:
Rennie, J. D., & Srebro, N. (2005). Loss functions for preference levels: Regression with discrete ordered labels. Proceedings of the IJCAI multidisciplinary workshop on advances in preference handling. Retrieved from https://people.csail.mit.edu/jrennie/papers/ijcai05-preference.pdf
Shi, X., Cao, W., & Raschka, S. (2019). Deep neural networks for rank-consistent ordinal regression based on conditional probabilities. arXiv preprint arXiv:1907.02436. Retrieved from https://arxiv.org/pdf/1907.02436
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: Synthetic minority oversampling technique. Journal of Artificial Intelligence Research, 16, 321-357.
Cruz-Ramírez, M., Hervás-Martínez, C., Sánchez-Monedero, J., & Gutiérrez, P. A. (2014). Metrics to guide a multi-objective evolutionary algorithm for ordinal classification. Neurocomputing, 135, 21-31. https://doi.org/10.1016/j.neucom.2013.05.058
Kato, T., Kashima, H., Sugiyama, M., & Asai, K. (2008). Multi-task learning via conic programming. arXiv preprint arXiv:1012.2609. Retrieved from https://arxiv.org/pdf/1012.2609
Costa, J., & Silva, C. (2020). Ordinal classification with distance regularization for robust brain age prediction. Pattern Recognition Letters, 140, 180-186. https://doi.org/10.1016/j.patrec.2020.10.005
Software Documentation and Libraries:
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., ... & Duchesnay, E. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825-2830. OneHotEncoder documentation. Retrieved from https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A. V., & Gulin, A. (2018). CatBoost: unbiased boosting with categorical features. Advances in Neural Information Processing Systems, 31. CatBoost Classifier documentation. Retrieved from https://catboost.ai/docs/en/concepts/python-reference_catboostclassifier
Lemaître, G., Nogueira, F., & Aridas, C. K. (2017). Imbalanced-learn: A python toolbox to tackle the curse of imbalanced datasets in machine learning. Journal of Machine Learning Research, 18(1), 559-563. SMOTE implementation. Retrieved from https://imbalanced-learn.org/dev/references/generated/imblearn.over_sampling.SMOTE.html
Fabian, P. (2016). MORD: Ordinal regression in Python. MORD LogisticIT documentation. Retrieved from https://pythonhosted.org/mord/reference.html#mord.LogisticIT
Research Software and Repositories:
Raschka Research Group. (2021). CORAL PyTorch: Ordinal regression with deep neural networks. GitHub repository. Retrieved from https://github.com/Raschka-research-group/coral-pytorch
Raschka Research Group. (2021). CORAL PyTorch tutorials: Ordinal classification examples. Retrieved from https://raschka-research-group.github.io/coral-pytorch/tutorials/pytorch_lightning/ordinal-coral_cement/
Ck37. (2020). CORAL Ordinal: Conditional ordinal regression for neural networks. GitHub repository. Retrieved from https://github.com/ck37/coral-ordinal
EvALLTEAM. (2021). CEM-Ord: Closeness Evaluation Measure for ordinal classification. GitHub repository. Retrieved from https://github.com/EvALLTEAM/CEM-Ord
Hovinh. (2021). Closeness Evaluation Measure: Implementation and examples. GitHub repository. Retrieved from https://github.com/hovinh/closeness-evaluation-measure/tree/main
Okasag. (2019). OrderedForest: Ordered random forest implementation. GitHub repository. Retrieved from https://github.com/okasag/OrderedForest
ORF Lab. (2021). ORF-py: Ordered Random Forest in Python. Retrieved from https://orf-lab.github.io/orf-py/#
Institutional Publications:
INESC TEC. (2020). Advanced methods in ordinal classification for cybersecurity applications. INESC TEC Digital Repository. Retrieved from https://repositorio.inesctec.pt/server/api/core/bitstreams/01f5e712-d91a-4b81-ad84-f8486614c1da/content
Association for Computational Linguistics. (2020). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. ACL Anthology. Retrieved from https://aclanthology.org/2020.acl-main.363.pdf
