literature review on machine learning classification

• Research article
• Open access
• Published: 15 February 2021

Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making

• Alan Brnabic 1 &
• Lisa M. Hess   ORCID: orcid.org/0000-0003-3631-3941 2  

BMC Medical Informatics and Decision Making volume  21 , Article number:  54 ( 2021 ) Cite this article

28k Accesses

48 Citations

3 Altmetric

Metrics details

Machine learning is a broad term encompassing a number of methods that allow the investigator to learn from the data. These methods may permit large real-world databases to be more rapidly translated to applications to inform patient-provider decision making.

This systematic literature review was conducted to identify published observational research of employed machine learning to inform decision making at the patient-provider level. The search strategy was implemented and studies meeting eligibility criteria were evaluated by two independent reviewers. Relevant data related to study design, statistical methods and strengths and limitations were identified; study quality was assessed using a modified version of the Luo checklist.

A total of 34 publications from January 2014 to September 2020 were identified and evaluated for this review. There were diverse methods, statistical packages and approaches used across identified studies. The most common methods included decision tree and random forest approaches. Most studies applied internal validation but only two conducted external validation. Most studies utilized one algorithm, and only eight studies applied multiple machine learning algorithms to the data. Seven items on the Luo checklist failed to be met by more than 50% of published studies.

Conclusions

A wide variety of approaches, algorithms, statistical software, and validation strategies were employed in the application of machine learning methods to inform patient-provider decision making. There is a need to ensure that multiple machine learning approaches are used, the model selection strategy is clearly defined, and both internal and external validation are necessary to be sure that decisions for patient care are being made with the highest quality evidence. Future work should routinely employ ensemble methods incorporating multiple machine learning algorithms.

Peer Review reports

Traditional methods of analyzing large real-world databases (big data) and other observational studies are focused on the outcomes that can inform at the population-based level. The findings from real-world studies are relevant to populations as a whole, but the ability to predict or provide meaningful evidence at the patient level is much less well established due to the complexity with which clinical decision making is made and the variety of factors taken into account by the health care provider [ 1 , 2 ]. Using traditional methods that produce population estimates and measures of variability, it is very challenging to accurately predict how any one patient will perform, even when applying findings from subgroup analyses. The care of patients is nuanced, and multiple non-linear, interconnected factors must be taken into account in decision making. When data are available that are only relevant at the population level, health care decision making is less informed as to the optimal course of care for a given patient.

Clinical prediction models are an approach to utilizing patient-level evidence to help inform healthcare decision makers about patient care. These models are also known as prediction rules or prognostic models and have been used for decades by health care professionals [ 3 ]. Traditionally, these models combine patient demographic, clinical and treatment characteristics in the form of a statistical or mathematical model, usually regression, classification or neural networks, but deal with a limited number of predictor variables (usually below 25). The Framingham Heart Study is a classic example of the use of longitudinal data to build a traditional decision-making model. Multiple risk calculators and estimators have been built to predict a patient’s risk of a variety of cardiovascular outcomes, such as atrial fibrillation and coronary heart disease [ 4 , 5 , 6 ]. In general, these studies use multivariable regression evaluating risk factors identified in the literature. Based on these findings, a scoring system is derived for each factor to predict the likelihood of an adverse outcome based on a patient’s score across all risk factors evaluated.

With the advent of more complex data collection and readily available data sets for patients in routine clinical care, both sample sizes and potential predictor variables (such as genomic data) can exceed the tens of thousands, thus establishing the need for alternative approaches to rapidly process a large amount of information. Artificial intelligence (AI), particularly machine learning methods (a subset of AI), are increasingly being utilized in clinical research for prediction models, pattern recognition and deep-learning techniques used to combine complex information for example genomic and clinical data [ 7 , 8 , 9 ]. In the health care sciences, these methods are applied to replace a human expert to perform tasks that would otherwise take considerable time and expertise, and likely result in potential error. The underlying concept is that a machine will learn by trial and error from the data itself, to make predictions without having a pre-defined set of rules for decision making. Simply, machine learning can simply be better understood as “learning from data.” [ 8 ].

There are two types of learning from the data, unsupervised and supervised. Unsupervised learning is a type of machine learning algorithm used to draw inferences from datasets consisting of input data without labelled responses. The most common unsupervised learning method is cluster analysis, which is used for exploratory data analysis to find hidden patterns or grouping in data. Supervised learning involves making a prediction based on a set of pre-specified input and output variables. There are a number of statistical tools used for supervised learning. Some examples include traditional statistical prediction methods like regression models (e.g. regression splines, projection pursuit regression, penalized regression) that involve fitting a model to data, evaluating the fit and estimating parameters that are later used in a predictive equation. Other tools include tree-based methods (e.g. classification and regression trees [CART] and random forests), which successively partition a data set based on the relationships between predictor variables and a target (outcome) variable. Other examples include neural networks, discriminant functions and linear classifiers, support vector classifiers and machines. Often, predictive tools are built using various forms of model aggregation (or ensemble learning) that may combine models based on resampled or re-weighted data sets. These different types of models can be fitted to the same data using model averaging.

Classical statistical regression methods used for prediction modeling are well understood in the statistical sciences and the scientific community that employs them. These methods tend to be transparent and are usually hypothesis driven but can overlook complex associations with limited flexibility when a high number of variables are investigated. In addition, when using classic regression modeling, choosing the ‘right’ model is not straightforward. Non-traditional machine learning algorithms, and machine learning approaches, may overcome some of these limitations of classical regression models in this new era of big data, but are not a complete solution as they must be considered in the context of the limitations of data used in the analysis [ 2 ].

While machine learning methods can be used for both population-based models as well as for informed patient-provider decision making, it is important to note that the data, model, and outputs used to inform the care of an individual patient must meet the highest standards of research quality, as the choice made will likely have an impact on both the long- and short-term patient outcomes. While a range of uncertainty can be expected for population-based estimates, the risk of error for patient level models must be minimized to ensure quality patient care. The risks and concerns of utilizing machine learning for individual patient decision making have been raised by ethicists [ 10 ]. The risks are not limited to the lack of transparency, limited data regarding the confidence of the findings, and the risk of reducing patient autonomy in choice by relying on data that may foster a more paternalistic model of healthcare. These are all important and valid concerns, and therefore the role of machine learning for patient care must meet the highest standards to ensure that shared, not simply informed, evidence-based decision making be supported by these methods.

A systematic literature review was published in 2018 that evaluated the statistical methods that have been used to enable large, real-world databases to be used at the patient-provider level [ 11 ]. Briefly, this study identified a total of 115 articles that evaluated the use of logistic regression (n = 52, 45.2%), Cox regression (n = 24, 20.9%), and linear regression (n = 17, 14.8%). However, an interesting observation noted several studies utilizing novel statistical approaches such as machine learning, recursive partitioning, and development of mathematical algorithms to predict patient outcomes. More recently, publications are emerging describing the use of Individualized Treatment Recommendation algorithms and Outcome Weighted Learning for personalized medicine using large observational databases [ 12 , 13 ]. Therefore, this systematic literature review was designed to further pursue this observation to more comprehensively evaluate the use of machine learning methods to support patient-provider decision making, and to critically evaluate the strengths and weaknesses of these methods. For the purposes of this work, data supporting patient-provider decision making was defined as that which provided information specifically on a treatment or intervention choice; while both population-based and risk estimator data are certainly valuable for patient care and decision making, this study was designed to evaluate data that would specifically inform a choice for the patient with the provider. The overarching goal is to provide evidence of how large datasets can be used to inform decisions at the patient level using machine learning-based methods, and to evaluate the quality of such work to support informed decision making.

This study originated from a systematic literature review that was conducted in MEDLINE and PsychInfo; a refreshed search was conducted in September 2020 to obtain newer publications (Table 1 ). Eligible studies were those that analyzed prospective or retrospective observational data, reported quantitative results, and described statistical methods specifically applicable to patient-level decision making. Specifically, patient-level decision making referred to studies that provided data for or against a particular intervention at the patient level, so that the data could be used to inform decision making at the patient-provider level. Studies did not meet this criterion if only a population-based estimates, mortality risk predictors, or satisfaction with care were evaluated. Additionally, studies designed to improve diagnostic tools and those evaluating health care system quality indicators did not meet the patient-provider decision-making criterion. Eligible statistical methods for this study were limited to machine learning-based approaches. Eligibility was assessed by two reviewers and any discrepancies were discussed; a third reviewer was available to serve as a tie breaker in case of different opinions. The final set of eligible publications were then abstracted into a Microsoft Excel document. Study quality was evaluated using a modified Luo scale, which was developed specifically as a tool to standardize high-quality publication of machine learning models [ 14 ]. A modified version of this tool was utilized for this study; specifically, the optional item were removed, and three terms were clarified: item 6 (define the prediction problem) was redefined as “define the model,” item 7 (prepare data for model building) was renamed “model building and validation,” and item 8 (build the predictive model) was renamed “model selection” to more succinctly state what was being evaluated under each criterion. Data were abstracted and both extracted data and the Luo checklist items were reviewed and verified by a second reviewer to ensure data comprehensiveness and quality. In all cases of differences in eligibility assessment or data entry, the reviewers met and ensured agreement with the final set of data to be included in the database for data synthesis, with a third reviewer utilized as a tie breaker in case of discrepancies. Data were summarized descriptively and qualitatively, based on the following categories: publication and study characteristics; patient characteristics; statistical methodologies used, including statistical software packages; strengths and weaknesses; and interpretation of findings.

The search strategy was run on September 1, 2020 and identified a total of 34 publications that utilized machine learning methods for individual patient-level decision making (Fig.  1 ). The most common reason for study exclusion, as expected, was due to the study not meeting the patient-level decision making criterion. A summary of the characteristics of eligible studies and the patient data are included in Table 2 . Most of the real-world data sources included retrospective databases or designs (n = 27, 79.4%), primarily utilizing electronic health records. Six analyses utilized prospective cohort studies and one utilized data from a cross sectional study.

Prisma diagram of screening and study identification

General approaches to machine learning

The types of classification or prediction machine learning algorithms are reported in Table 2 . These included decision tree/random forest analyses (19 studies) [ 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 ] and neural networks (19 studies) [ 24 , 25 , 26 , 27 , 28 , 29 , 30 , 32 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 ]. Other approaches included latent growth mixture modeling [ 45 ], support vector machine classifiers [ 46 ], LASSO regression [ 47 ], boosting methods [ 23 ], and a novel Bayesian approach [ 26 , 40 , 48 ]. Within the analytical approaches to support machine learning, a variety of methods were used to evaluate model fit, such as Akaike Information Criterion, Bayesian Information Criterion, and the Lo-Mendel-Rubin likelihood ratio test [ 22 , 45 , 47 ], and while most studies included the area under the curve (AUC) of receiver-operator characteristic (ROC) curves (Table 3 ), analyses also included sensitivity/specificity [ 16 , 19 , 24 , 30 , 41 , 42 , 43 ], positive predictive value [ 21 , 26 , 32 , 38 , 40 , 41 , 42 , 43 ], and a variety of less common approaches such as the geometric mean [ 16 ], use of the Matthews correlation coefficient (ranges from -1.0, completely erroneous information, to + 1.0, perfect prediction) [ 46 ], defining true/false negatives/positives by means of a confusion matrix [ 17 ], calculating the root mean square error of the predicted versus original outcome profiles [ 37 ], or identifying the model with the best average performance training and performance cross validation [ 36 ].

Statistical software packages

The statistical programs used to perform machine learning varied widely across these studies, no consistencies were observed (Table 2 ). As noted above, one study using decision tree analysis used Quinlan’s C5.0 decision tree algorithm [ 15 ] while a second used an earlier version of this program (C4.5) [ 20 ]. Other decision tree analyses utilized various versions of R [ 18 , 19 , 22 , 24 , 27 , 47 ], International Business Machines (IBM) Statistical Package for the Social Sciences (SPSS) [ 16 , 17 , 33 , 47 ], the Azure Machine Learning Platform [ 30 ], or programmed the model using Python [ 23 , 25 , 46 ]. Artificial neural network analyses used Neural Designer [ 34 ] or Statistica V10 [ 35 ]. Six studies did not report the software used for analysis [ 21 , 31 , 32 , 37 , 41 , 42 ].

Families of machine learning algorithms

Also as summarized in Table 2 , more than one third of all publications (n = 13, 38.2%) applied only one family of machine learning algorithm to model development [ 16 , 17 , 18 , 19 , 20 , 34 , 37 , 41 , 42 , 43 , 46 , 48 ]; and only four studies utilized five or more methods [ 23 , 25 , 28 , 45 ]. One applied an ensemble of six different algorithms and the software was set to run 200 iterations [ 23 ], and another ran seven algorithms [ 45 ].

Internal and external validation

Evaluation of study publication quality identified the most common gap in publications as the lack of external validation, which was conducted by only two studies [ 15 , 20 ]. Seven studies predefined the success criteria for model performance [ 20 , 21 , 23 , 35 , 36 , 46 , 47 ], and five studies discussed the generalizability of the model [ 20 , 23 , 34 , 45 , 48 ]. Six studies [ 17 , 18 , 21 , 22 , 35 , 36 ] discussed the balance between model accuracy and model simplicity or interpretability, which was also a criterion of quality publication in the Luo scale [ 14 ]. The items on the checklist that were least frequently met are presented in Fig.  2 . The complete quality assessment evaluation for each item in the checklist is included in Additional file 1 : Table S1.

Least frequently met study quality items, modified Luo Scale [ 14 ]

There were a variety of approaches taken to validate the models developed (Table 3 ). Internal validation with splitting into a testing and validation dataset was performed in all studies. The cohort splitting approach was conducted in multiple ways, using a 2:1 split [ 26 ], 60/40 split [ 21 , 36 ], a 70/30 split [ 16 , 17 , 22 , 30 , 33 , 35 ], 75/25 split [ 27 , 40 ], 80/20 split [ 46 ], 90/10 split [ 25 , 29 ], splitting the data based on site of care [ 48 ], a 2/1/1 split for training, testing and validation [ 38 ], and splitting 60/20/20, where the third group was selected for model selection purposes prior to validation [ 34 ]. Nine studies did not specifically mention the form of splitting approach used [ 15 , 18 , 19 , 20 , 24 , 29 , 39 , 45 , 47 ], but most of those noted the use of k fold cross validation. One training set corresponded to 90% of the sample [ 23 ], whereas a second study was less clear, as input data were at the observation level with multiple observations per patient, and 3 of the 15 patients were included in the training set [ 37 ]. The remaining studies did not specifically state splitting the data into testing and validation samples, but most specified they performed five-fold cross validation (including one that generally mentioned cohort splitting) [ 18 , 45 ] or ten-fold cross validation strategies [ 15 , 19 , 20 , 28 ].

External validation was conducted by only two studies (5.9%). Hische and colleagues conducted a decision tree analysis, which was designed to identify patients with impaired fasting glucose [ 20 ]. Their model was developed in a cohort study of patients from the Berlin Potsdam Cohort Study (n = 1527) and was found to have a positive predictive value of 56.2% and a negative predictive value of 89.1%. The model was then tested on an independent from the Dresden Cohort (n = 1998) with a family history of type II diabetes. In external validation, positive predictive value was 43.9% and negative predictive value was 90.4% [ 20 ]. Toussi and colleagues conducted both internal and external validation in their decision tree analysis to evaluate individual physician prescribing behaviors using a database of 463 patient electronic medical records [ 15 ]. For the internal validation step, the cross-validation option was used from Quinlan’s C5.0 decision tree learning algorithm as their study sample was too small to split into a testing and validation sample, and external validation was conducted by comparing outcomes to published treatment guidelines. Unfortunately, they found little concordance between physician behavior and guidelines potentially due to the timing of the data not matching the time period in which guidelines were implemented, emphasizing the need for a contemporaneous external control [ 15 ].

Handling of missing values

Missing values were addressed in most studies (n = 21, 61.8%) in this review, but there were thirteen remaining studies that did not mention if there were missing data or how they were handled (Table 3 ). For those that reported methods related to missing data, there were a wide variety of approaches used in real-world datasets. The full information maximum likelihood method was used for estimating model parameters in the presence of missing data for the development of the model by Hertroijs and colleagues, but patients with missing covariate values at baseline were excluded from the validation of the model [ 45 ]. Missing covariate values were included in models as a discrete category [ 48 ]. Four studies removed patients from the model with missing data [ 46 ], resulting in the loss of 16%-41% of samples in three studies [ 17 , 36 , 47 ]. Missing data from primary outcome variables were reported among with 59% (men) and 70% (women) within a study of diabetes [ 16 ]. In this study, single imputation was used; for continuous variables CART (IBM SPSS modeler V14.2.03) and for categorical variables the authors used the weighted K-Nearest Neighbor approach using RapidMiner (V.5) [ 16 ]. Other studies reported exclusion but not specifically the impact on sample size [ 29 , 31 , 38 , 44 ]. Imputation was conducted in a variety of ways for studies with missing data [ 22 , 25 , 28 , 33 ]. Single imputation was used in the study by Bannister and colleagues, but followed by multiple imputation in the final model to evaluate differences in model parameters [ 22 ]. One study imputed with a standard last-imputation-forward approach [ 26 ]. Spline techniques were used to impute missing data in the training set of one study [ 37 ]. Missingness was largely retained as an informative variable, and only variables missing for 85% or more of participants were excluded by Alaa et al. [ 23 ] while Hearn et al. used a combination of imputation and exclusion strategies [ 40 ]. Lastly, missing or incomplete data were imputed using a model-based approach by Toussi et al. [ 15 ] and using an optimal-impute algorithm by Bertsimas et al. [ 21 ].

Strengths and weaknesses noted by authors

Publications summarized the strengths and weaknesses of the machine learning methods employed. Low complexity and simplicity of machine-based learning models were noted as strengths of this approach [ 15 , 20 ]. Machine learning approaches were both powerful and efficient methods to apply to large datasets [ 19 ]. It was noted that parameters in this study that were significant at the patient level were included, even if at the broader population-based level using traditional regression analysis model development they would have not been significant and therefore would have been otherwise excluded using traditional approaches [ 34 ]. One publication noted the value of machine learning being highly dependent on the model selection strategy and parameter optimization, and that machine learning in and of itself will not provide better estimates unless these steps are conducted properly [ 23 ].

Even when properly planned, machine learning approaches are not without issues that deserve attention in future studies that employ these techniques. Within the eligible publications, weaknesses included overfitting the model with the inclusion of too much detail [ 15 ]. Additional limitations are based on the data sources used for machine learning, such as the lack of availability of all desired variables and missing data that can affect the development and performance of these models [ 16 , 34 , 36 , 48 ]. The lack of all relevant variables was noted as a particular concern for retrospective database studies, where the investigator is limited to what has been recorded [ 26 , 28 , 29 , 38 , 40 ]. Importantly and as observed in the studies included in this review, the lack of external validation was stated as a limitation of studies included in this review [ 28 , 30 , 38 , 42 ].

Limitations can also be on the part of the research team, as the need for both clinical and statistical expertise in the development and execution of studies using machine learning-based methodology, and users are warned against applying these methods blindly [ 22 ]. The importance of the role of clinical and statistical experts in the research team was noted in one study and highlighted as a strength of their work [ 21 ].

This study systematically reviewed and summarized the methods and approaches used for machine learning as applied to observational datasets that can inform patient-provider decision making. Machine learning methods have been applied much more broadly across observational studies than in the context of individual decision making, so the summary of this work does not necessarily apply to all machine learning-based studies. The focus of this work is on an area that remains largely unexplored, which is how to use large datasets in a manner that can inform and improve patient care in a way that supports shared decision making with reliable evidence that is applicable to the individual patient. Multiple publications cite the limitations of using population-based estimates for individual decisions [ 49 , 50 , 51 ]. Specifically, a summary statistic at the population level does not apply to each person in that cohort. Population estimates represent a point on a potentially wide distribution, and any one patient could fall anywhere within that distribution and be far from the point estimate value. On the other extreme, case reports or case series provide very specific individual-level data, but are not generalizable to other patients [ 52 ]. This review and summary provides guidance and suggestions of best practices to improve and hopefully increase the use of these methods to provide data and models to inform patient-provider decision making.

It was common for single modeling strategies to be employed within the identified publications. It has long been known that single algorithms to estimation can produce a fair amount of uncertainty and variability [ 53 ]. To overcome this limitation, there is a need for multiple algorithms and multiple iterations of the models to be performed. This, combined with more powerful analytics in recent years, provides a new standard for machine learning algorithm choice and development. While in some cases, a single model may fit the data well and provide an accurate answer, the certainty of the model can be supported through novel approaches, such as model averaging [ 54 ]. Few studies in this review combined multiple families of modeling strategies along with multiple iterations of the models. This should become a best practice in the future and is recommended as an additional criterion to assess study quality among machine learning-based modeling [ 54 ].

External validation is critical to ensure model accuracy, but was rarely conducted in the publications included in this review. The reasons for this could be many, such as lack of appropriate datasets or due to the lack of awareness of the importance of external validation [ 55 ]. As model development using machine learning increases, there is a need for external validation prior to application of models in any patient-provider setting. The generalizability of models is largely unknown without these data. Publications that did not conduct external validation also did not note the need for this to be completed, as generalizability was discussed in only five studies, one of which had also conducted the external validation. Of the remaining four studies, the role of generalizability was noted in terms of the need for future external validation in only one study [ 48 ]. Other reviews that were more broadly conducted to evaluate machine learning methods similarly found a low rate of external validation (6.6% versus 5.9% in this study) [ 56 ]. It was shown that there was lower prediction accuracy by external validation than simply by cross validation alone. The current review, with a focus on machine learning to support decision making at a practical level, suggests external validation is an important gap that should be filled prior to using these models for patient-provider decision making.

Luo and others suggest that k -fold validation may be used with proper stratification of the response variable as part of the model selection strategy [ 14 , 55 ]. The studies identified in this review generally conducted 5- or tenfold validation. There is no formal rule for the selection for the value of k , which is typically based on the size of the dataset; as k increases, bias will be reduced, but in turn variance will increase. While the tradeoff has to be accounted for, k  = 5–10 has been found to be reasonable for most study purposes [ 57 ].

The evidence from identified publications suggests that the ethical concerns of lack of transparency and failure to report confidence in the findings are largely warranted. These limitations can be addressed through the use of multiple modeling approaches (to clarify the ‘black box’ nature of these approaches) and by including both external and high k-fold validation (to demonstrate the confidence in findings). To ensure these methods are used in a manner that improves patient care, the expectations of population-based risk prediction models of the past are no longer sufficient. It is essential that the right data, the right set of models, and appropriate validation are employed to ensure that the resulting data meet standards for high quality patient care.

This study did not evaluate the quality of the underlying real-world data used to develop, test or validate the algorithms. While not directly part of the evaluation in this review, researchers should be aware that all limitations of real-world data sources apply regardless of the methodology employed. However, when observational datasets are used for machine learning-based research, the investigator should be aware of the extent to which the methods they are using depend on the data structure and availability, and should evaluate a proposed data source to ensure it is appropriate for the machine learning project [ 45 ]. Importantly, databases should be evaluated to fully understand the variables included, as well as those variables that may have prognostic or predictive value, but may not be included in the dataset. The lack of important variables remains a concern with the use of retrospective databases for machine learning. The concerns with confounding (particularly unmeasured confounding), bias (including immortal time bias), and patient selection criteria to be in the database must also be evaluated [ 58 , 59 ]. These are factors that should be considered prior to implementing these methods, and not always at the forefront of consideration when applying machine learning approaches. The Luo checklist is a valuable tool to ensure that any machine-learning study meets high research standards for patient care, and importantly includes the evaluation of missing or potentially incorrect data (i.e. outliers) and generalizability [ 14 ]. This should be supplemented by a thorough evaluation of the potential data to inform the modeling work prior to its implementation, and ensuring that multiple modeling methods are applied.

This review found a wide variety of approaches, methods, statistical software and validation strategies that were employed in the application of machine learning methods to inform patient-provider decision making. Based on these findings, there is a need to ensure that multiple modeling approaches are employed in the development of machine learning-based models for patient care, which requires the highest research standards to reliably support shared evidence-based decision making. Models should be evaluated with clear criteria for model selection, and both internal and external validation are needed prior to applying these models to inform patient care. Few studies have yet to reach that bar of evidence to inform patient-provider decision making.

Availability of data and materials

All data generated or analyzed during this study are included in this published article and its supplementary information files.

Abbreviations

Artificial intelligence

Area under the curve

Classification and regression trees

Logistic least absolute shrinkage and selector operator

Steyerberg EW, Claggett B. Towards personalized therapy for multiple sclerosis: limitations of observational data. Brain. 2018;141(5):e38-e.

Fröhlich H, Balling R, Beerenwinkel N, Kohlbacher O, Kumar S, Lengauer T, et al. From hype to reality: data science enabling personalized medicine. BMC Med. 2018;16(1):150.

Article   PubMed   PubMed Central   Google Scholar  

Steyerberg EW. Clinical prediction models. Berlin: Springer; 2019.

Book   Google Scholar  

Schnabel RB, Sullivan LM, Levy D, Pencina MJ, Massaro JM, D’Agostino RB Sr, et al. Development of a risk score for atrial fibrillation (Framingham Heart Study): a community-based cohort study. Lancet. 2009;373(9665):739–45.

D’Agostino RB, Wolf PA, Belanger AJ, Kannel WB. Stroke risk profile: adjustment for antihypertensive medication. Framingham Study Stroke. 1994;25(1):40–3.

Article   CAS   PubMed   Google Scholar  

Framingham Heart Study: Risk Functions 2020. https://www.framinghamheartstudy.org/ .

Gawehn E, Hiss JA, Schneider G. Deep learning in drug discovery. Mol Inf. 2016;35:3–14.

Article   CAS   Google Scholar  

Vamathevan J, Clark D, Czodrowski P, Dunham I, Ferran E, Lee G, et al. Applications of machine learning in drug discovery and development. Nat Rev Drug Discov. 2019;18(6):463–77.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Marcus G. Deep learning: A critical appraisal. arXiv preprint arXiv:180100631. 2018.

Grote T, Berens P. On the ethics of algorithmic decision-making in healthcare. J Med Ethics. 2020;46(3):205–11.

Article   PubMed   Google Scholar  

Brnabic A, Hess L, Carter GC, Robinson R, Araujo A, Swindle R. Methods used for the applicability of real-world data sources to individual patient decision making. Value Health. 2018;21:S102.

Article   Google Scholar  

Fu H, Zhou J, Faries DE. Estimating optimal treatment regimes via subgroup identification in randomized control trials and observational studies. Stat Med. 2016;35(19):3285–302.

Liang M, Ye T, Fu H. Estimating individualized optimal combination therapies through outcome weighted deep learning algorithms. Stat Med. 2018;37(27):3869–86.

Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res. 2016;18(12):e323.

Toussi M, Lamy J-B, Le Toumelin P, Venot A. Using data mining techniques to explore physicians’ therapeutic decisions when clinical guidelines do not provide recommendations: methods and example for type 2 diabetes. BMC Med Inform Decis Mak. 2009;9(1):28.

Ramezankhani A, Hadavandi E, Pournik O, Shahrabi J, Azizi F, Hadaegh F. Decision tree-based modelling for identification of potential interactions between type 2 diabetes risk factors: a decade follow-up in a Middle East prospective cohort study. BMJ Open. 2016;6(12):e013336.

Pei D, Zhang C, Quan Y, Guo Q. Identification of potential type II diabetes in a Chinese population with a sensitive decision tree approach. J Diabetes Res. 2019;2019:4248218.

Neefjes EC, van der Vorst MJ, Verdegaal BA, Beekman AT, Berkhof J, Verheul HM. Identification of patients with cancer with a high risk to develop delirium. Cancer Med. 2017;6(8):1861–70.

Mubeen AM, Asaei A, Bachman AH, Sidtis JJ, Ardekani BA, Initiative AsDN. A six-month longitudinal evaluation significantly improves accuracy of predicting incipient Alzheimer’s disease in mild cognitive impairment. J Neuroradiol. 2017;44(6):381–7.

Hische M, Luis-Dominguez O, Pfeiffer AF, Schwarz PE, Selbig J, Spranger J. Decision trees as a simple-to-use and reliable tool to identify individuals with impaired glucose metabolism or type 2 diabetes mellitus. Eur J Endocrinol. 2010;163(4):565.

Bertsimas D, Dunn J, Pawlowski C, Silberholz J, Weinstein A, Zhuo YD, et al. Applied informatics decision support tool for mortality predictions in patients with cancer. JCO Clin Cancer Inform. 2018;2:1–11.

Bannister CA, Halcox JP, Currie CJ, Preece A, Spasic I. A genetic programming approach to development of clinical prediction models: a case study in symptomatic cardiovascular disease. PLoS ONE. 2018;13(9):e0202685.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Alaa AM, Bolton T, Di Angelantonio E, Rudd JHF, van der Schaar M. Cardiovascular disease risk prediction using automated machine learning: a prospective study of 423,604 UK Biobank participants. PLoS ONE. 2019;14(5):e0213653.

Baxter SL, Marks C, Kuo TT, Ohno-Machado L, Weinreb RN. Machine learning-based predictive modeling of surgical intervention in glaucoma using systemic data from electronic health records. Am J Ophthalmol. 2019;208:30–40.

Dong Y, Xu L, Fan Y, Xiang P, Gao X, Chen Y, et al. A novel surgical predictive model for Chinese Crohn’s disease patients. Medicine (Baltimore). 2019;98(46):e17510.

Hill NR, Ayoubkhani D, McEwan P, Sugrue DM, Farooqui U, Lister S, et al. Predicting atrial fibrillation in primary care using machine learning. PLoS ONE. 2019;14(11):e0224582.

Kang AR, Lee J, Jung W, Lee M, Park SY, Woo J, et al. Development of a prediction model for hypotension after induction of anesthesia using machine learning. PLoS ONE. 2020;15(4):e0231172.

Karhade AV, Ogink PT, Thio Q, Cha TD, Gormley WB, Hershman SH, et al. Development of machine learning algorithms for prediction of prolonged opioid prescription after surgery for lumbar disc herniation. Spine J. 2019;19(11):1764–71.

Kebede M, Zegeye DT, Zeleke BM. Predicting CD4 count changes among patients on antiretroviral treatment: Application of data mining techniques. Comput Methods Programs Biomed. 2017;152:149–57.

Kim I, Choi HJ, Ryu JM, Lee SK, Yu JH, Kim SW, et al. A predictive model for high/low risk group according to oncotype DX recurrence score using machine learning. Eur J Surg Oncol. 2019;45(2):134–40.

Kwon JM, Jeon KH, Kim HM, Kim MJ, Lim S, Kim KH, et al. Deep-learning-based out-of-hospital cardiac arrest prognostic system to predict clinical outcomes. Resuscitation. 2019;139:84–91.

Kwon JM, Lee Y, Lee Y, Lee S, Park J. An algorithm based on deep learning for predicting in-hospital cardiac arrest. J Am Heart Assoc. 2018;7(13):26.

Scheer JK, Smith JS, Schwab F, Lafage V, Shaffrey CI, Bess S, et al. Development of a preoperative predictive model for major complications following adult spinal deformity surgery. J Neurosurg Spine. 2017;26(6):736–43.

Lopez-de-Andres A, Hernandez-Barrera V, Lopez R, Martin-Junco P, Jimenez-Trujillo I, Alvaro-Meca A, et al. Predictors of in-hospital mortality following major lower extremity amputations in type 2 diabetic patients using artificial neural networks. BMC Med Res Methodol. 2016;16(1):160.

Rau H-H, Hsu C-Y, Lin Y-A, Atique S, Fuad A, Wei L-M, et al. Development of a web-based liver cancer prediction model for type II diabetes patients by using an artificial neural network. Comput Methods Programs Biomed. 2016;125:58–65.

Ng T, Chew L, Yap CW. A clinical decision support tool to predict survival in cancer patients beyond 120 days after palliative chemotherapy. J Palliat Med. 2012;15(8):863–9.

Pérez-Gandía C, Facchinetti A, Sparacino G, Cobelli C, Gómez E, Rigla M, et al. Artificial neural network algorithm for online glucose prediction from continuous glucose monitoring. Diabetes Technol Therapeut. 2010;12(1):81–8.

Azimi P, Mohammadi HR, Benzel EC, Shahzadi S, Azhari S. Use of artificial neural networks to decision making in patients with lumbar spinal canal stenosis. J Neurosurg Sci. 2017;61(6):603–11.

Bowman A, Rudolfer S, Weller P, Bland JDP. A prognostic model for the patient-reported outcome of surgical treatment of carpal tunnel syndrome. Muscle Nerve. 2018;58(6):784–9.

Hearn J, Ross HJ, Mueller B, Fan CP, Crowdy E, Duhamel J, et al. Neural networks for prognostication of patients with heart failure. Circ. 2018;11(8):e005193.

Google Scholar  

Isma’eel HA, Cremer PC, Khalaf S, Almedawar MM, Elhajj IH, Sakr GE, et al. Artificial neural network modeling enhances risk stratification and can reduce downstream testing for patients with suspected acute coronary syndromes, negative cardiac biomarkers, and normal ECGs. Int J Cardiovasc Imaging. 2016;32(4):687–96.

Isma’eel HA, Sakr GE, Serhan M, Lamaa N, Hakim A, Cremer PC, et al. Artificial neural network-based model enhances risk stratification and reduces non-invasive cardiac stress imaging compared to Diamond-Forrester and Morise risk assessment models: a prospective study. J Nucl Cardiol. 2018;25(5):1601–9.

Jovanovic P, Salkic NN, Zerem E. Artificial neural network predicts the need for therapeutic ERCP in patients with suspected choledocholithiasis. Gastrointest Endosc. 2014;80(2):260–8.

Zhou HF, Huang M, Ji JS, Zhu HD, Lu J, Guo JH, et al. Risk prediction for early biliary infection after percutaneous transhepatic biliary stent placement in malignant biliary obstruction. J Vasc Interv Radiol. 2019;30(8):1233-41.e1.

Hertroijs DF, Elissen AM, Brouwers MC, Schaper NC, Köhler S, Popa MC, et al. A risk score including body mass index, glycated haemoglobin and triglycerides predicts future glycaemic control in people with type 2 diabetes. Diabetes Obes Metab. 2018;20(3):681–8.

Oviedo S, Contreras I, Quiros C, Gimenez M, Conget I, Vehi J. Risk-based postprandial hypoglycemia forecasting using supervised learning. Int J Med Inf. 2019;126:1–8.

Khanji C, Lalonde L, Bareil C, Lussier MT, Perreault S, Schnitzer ME. Lasso regression for the prediction of intermediate outcomes related to cardiovascular disease prevention using the TRANSIT quality indicators. Med Care. 2019;57(1):63–72.

Anderson JP, Parikh JR, Shenfeld DK, Ivanov V, Marks C, Church BW, et al. Reverse engineering and evaluation of prediction models for progression to type 2 diabetes: an application of machine learning using electronic health records. J Diabetes Sci Technol. 2016;10(1):6–18.

Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011;13(2):217–24.

Lu CY. Observational studies: a review of study designs, challenges and strategies to reduce confounding. Int J Clin Pract. 2009;63(5):691–7.

Morgenstern H. Ecologic studies in epidemiology: concepts, principles, and methods. Annu Rev Public Health. 1995;16(1):61–81.

Vandenbroucke JP. In defense of case reports and case series. Ann Intern Med. 2001;134(4):330–4.

Buckland ST, Burnham KP, Augustin NH. Model selection: an integral part of inference. Biometrics. 1997;53:603–18.

Zagar A, Kadziola Z, Lipkovich I, Madigan D, Faries D. Evaluating bias control strategies in observational studies using frequentist model averaging 2020 (submitted).

Kang J, Schwartz R, Flickinger J, Beriwal S. Machine learning approaches for predicting radiation therapy outcomes: a clinician’s perspective. Int J Radiat Oncol Biol Phys. 2015;93(5):1127–35.

Scott IM, Lin W, Liakata M, Wood J, Vermeer CP, Allaway D, et al. Merits of random forests emerge in evaluation of chemometric classifiers by external validation. Anal Chim Acta. 2013;801:22–33.

Kuhn M, Johnson K. Applied predictive modeling. Berlin: Springer; 2013.

Hess L, Winfree K, Muehlenbein C, Zhu Y, Oton A, Princic N. Debunking Myths While Understanding Limitations. Am J Public Health. 2020;110(5):E2-E.

Thesmar D, Sraer D, Pinheiro L, Dadson N, Veliche R, Greenberg P. Combining the power of artificial intelligence with the richness of healthcare claims data: Opportunities and challenges. PharmacoEconomics. 2019;37(6):745–52.

Download references

Acknowledgements

Not applicable.

No funding was received for the conduct of this study.

Author information

Authors and affiliations.

Eli Lilly and Company, Sydney, NSW, Australia

Alan Brnabic

Eli Lilly and Company, Indianapolis, IN, USA

Lisa M. Hess

You can also search for this author in PubMed   Google Scholar

Contributions

AB and LMH contributed to the design, implementation, analysis and interpretation of the data included in this study. AB and LMH wrote, revised and finalized the manuscript for submission. AB and LMH have both read and approved the final manuscript.

Corresponding author

Correspondence to Lisa M. Hess .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

Authors are employees of Eli Lilly and Company and receive salary support in that role.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Table S1. Study quality of eligible publications, modified Luo scale [14].

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Brnabic, A., Hess, L.M. Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making. BMC Med Inform Decis Mak 21 , 54 (2021). https://doi.org/10.1186/s12911-021-01403-2

Download citation

Received : 07 July 2020

Accepted : 20 January 2021

Published : 15 February 2021

DOI : https://doi.org/10.1186/s12911-021-01403-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Machine learning
• Decision making
• Decision tree
• Random forest
• Automated neural network

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

• General enquiries: [email protected]

A Systematic Literature Review on Machine Learning and Deep Learning Methods for Semantic Segmentation

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Proceedings

Information

American Journal of Data Mining and Knowledge Discovery

• Processing Charges

Machine Learning for Text Classification on Twitter: A Literature Review

Muneer Alsurori

Information Technology Science, Ibb University, Ibb, Yemen

Rahuf Alwan

Wafa Algumaei

Somia Alturki

Entsar Alkahtany

Add to Mendeley

This literature review examines the application of machine learning (ML) techniques for text classification on Twitter. With the immense volume of data generated on social media platforms like Twitter, there is a need for automated methods to extract valuable information. ML, known for its ability to learn patterns and relationships in large datasets, has gained significant attention in this context. The purpose of this review is to explore the background and aim of ML for text classification on Twitter, the methods employed, the results obtained, and the conclusions drawn. The review begins by discussing the background and aim, emphasizing the vast amount of data available on Twitter and the need for automated techniques to extract useful information from this data. It highlights the significance of ML in addressing this challenge, particularly in tasks such as sentiment analysis, topic modeling, and spam detection, which play a crucial role in social media analysis. Next, the review provides an overview of the methods used in various studies on text classification using Twitter data. It explores the latest approaches and techniques employed in ML, including feature extraction methods like bag-of-words, n-grams, and word embeddings. It also discusses the preprocessing steps involved in preparing Twitter data for classification tasks. subsequently, the review presents the results obtained from different studies in the field. It discusses the performance metrics used to evaluate the effectiveness of ML models, highlighting measures such as accuracy, precision, recall, and F1-score. The review also discusses variations in performance across different classification tasks, providing insights into the strengths and limitations of the approaches used.

Machine Learning, Text Classification, Twitter Data, NLP

Alsurori, M., Enan, A., Alwan, R., Algumaei, W., Alturki, S., et al. (2023). Machine Learning for Text Classification on Twitter: A Literature Review. American Journal of Data Mining and Knowledge Discovery , 8 (1), 11-17. https://doi.org/10.11648/j.ajdmkd.20230801.12

Alsurori, M.; Enan, A.; Alwan, R.; Algumaei, W.; Alturki, S., et al. Machine Learning for Text Classification on Twitter: A Literature Review. Am. J. Data Min. Knowl. Discov. 2023 , 8 (1), 11-17. doi: 10.11648/j.ajdmkd.20230801.12

Alsurori M, Enan A, Alwan R, Algumaei W, Alturki S, et al. Machine Learning for Text Classification on Twitter: A Literature Review. Am J Data Min Knowl Discov . 2023;8(1):11-17. doi: 10.11648/j.ajdmkd.20230801.12

Cite This Article

• Author Information

Verification Code/Mostafa Zaman Chowdhury

The verification code is required.

Verification code is not valid.

Science Publishing Group (SciencePG) is an Open Access publisher, with more than 300 online, peer-reviewed journals covering a wide range of academic disciplines.

Learn More About SciencePG

• Special Issues
• AcademicEvents
• ScholarProfiles
• For Authors
• For Reviewers
• For Editors
• For Conference Organizers
• For Librarians
• Article Processing Charges
• Special Issues Guidelines
• Editorial Process
• Peer Review at SciencePG
• Open Access
• Ethical Guidelines

Important Link

• Manuscript Submission
• Propose a Special Issue
• Join the Editorial Board
• Become a Reviewer

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 27 March 2024

Research on predicting the driving forces of digital transformation in Chinese media companies based on machine learning

• Zhan Wang 1 ,
• Xu Zhao 3 ,
• Yuxuan Wang 3 &
• Zihan Xiao 3  

Scientific Reports volume  14 , Article number:  7286 ( 2024 ) Cite this article

25 Accesses

Metrics details

• Information technology
• Socioeconomic scenarios
• Sustainability

Chinese media companies are facing opportunities and challenges brought about by digital transformation. Media economics takes the evaluation of the business results of media companies as the main research topic. However, overcoming the internal differences in the industry and comprehensively predicting the digital transformation of Chinese media companies from multiple dimensions has become an important issue to be understood. Based on the “TOE-I” theoretical framework, this study innovatively uses machine learning methods to predict the digital transformation of Chinese media companies and to analyze specific modes of the main driving factors affecting the digital transformation, using data from China’s A-share-listed media companies from 2010 to 2020. The study found that environmental drivers can most effectively and accurately predict the digital transformation of Chinese media companies. Therefore, under sustained and stable economic and financial policies, guiding inter-industry competition and providing balanced digital infrastructure conditions are keys to bridging internal barriers in the media industry and promoting digital transformation. In the process of transformation from traditional content to digital production, media companies should focus on policy changes, economic benefits, the decision-making role of core managers, and the training and preservation of digital technology talent.

Similar content being viewed by others

Global prediction of extreme floods in ungauged watersheds

Grey Nearing, Deborah Cohen, … Yossi Matias

The role of artificial intelligence in achieving the Sustainable Development Goals

Ricardo Vinuesa, Hossein Azizpour, … Francesco Fuso Nerini

The impact of artificial intelligence on employment: the role of virtual agglomeration

Yang Shen & Xiuwu Zhang

Introduction

In the digital economy era, media has become ubiquitous in our lives. With the help of digital technologies such as mobile Internet, big data, cloud computing, the Internet of Things (IoT) and artificial intelligence, companies involved in the media industry are facing new development opportunities. The 14th Five-Year Plan (2021–2025) in the National Economic and Social Development and Vision 2035 of the People’s Republic of China lists digital transformation as a national development strategy. This plan clearly outlines the implementation of the digital cultural industry, in accelerating the development of new cultural enterprises, cultural forms, and cultural consumption patterns, and strengthening digital creativity, network audio-visuals, digital publishing, digital entertainment, and the online broadcast industry. COVID-19 has had a serious negative impact on China’s economic and social development. However, driven by digital technology, the traditional media industry pattern has undergone fundamental changes, and the digital transformation and development of media has become an inevitable trend 1 . In 2022, China’s media industry output fell 2.11% at a value of 290,825 billion yuan, presenting an overall decline but a local rise. Internet advertising, Internet marketing services, mobile data and Internet business, online games and other traditional high-output areas show different degrees of negative growth. The total revenue of radio and television advertising, book sales, and newspaper industry was less than that of the network audio-visual field 2 .

Although media companies are part of the media industry, their driving forces and the pressures of digital transformation differ due to their different businesses. China’s media industry has certain ideological attributes, especially political attachments represented in publishing, radio, and television mainstream media 3 , 4 . Based on guidance promoting the amalgamation of traditional and emerging media in 2014, the Chinese government provides huge external policy support for media companies whose main businesses are in publishing, radio, and television. Advertising and film industries have also been affected by the digital transformation from business processes to organizations and business models. Hence, it is necessary to understand how to overcome the internal differences in the industry and make a comprehensive evaluation and prediction of the digital transformation of Chinese media companies.

Existing research has largely focused on the correlation between single-dimension characteristics and media digital transformation, only making predictions within a sample. There is little comprehensive consideration of the driving forces in the digital transformation of media companies 5 , 6 , 7 . This study is based on the theory of TOE (Technology-Organization-Environment), taking the media industry category as an important dimension in predicting the digital transformation of media companies, and building an out-of-sample prediction model for digital transformation (i.e. a “TOE-I” model) of media companies from the dimensions of technology, organization, environment, and media industry classifications. We aim to analyze the differences in the prediction ability of digital transformation behavior of different types of elements in varying dimensions and to identify the main factors and patterns of influence that drive media companies to participate in digital transformation. Our highlight can be divided into two aspects: (1) the advanced empirical research methods based on machine learning, and (2) the innovative theoretical framework which is applicable to the prediction of Chinese media companies’ digital transformation.

This research innovatively adopted the ensemble learning method in machine learning to explore the main cause behind the digital transformation of Chinese media companies. The advantages of this method are as follows: Firstly, the existing literature regarding media companies’ digital motivation has tended to use multiple linear regression, which cannot accurately reflect the complex relationship between variables due to the nonlinear characteristics of normal relationships. In contrast, ensemble learning can effectively handle the nonlinear relationship and possible interactions between variables; thus the resulting model can better reflect the information. Furthermore, the ensemble learning method does not need to preset the functional form of variables in the model but fits the relationship between variables as much as possible according to the training set data. This is more suitable for predictive analysis than traditional linear research methods 8 . Secondly, ensemble learning can make use of relative importance and partial dependence plots and other means to analyze different variables on the prediction ability of digital behavior, and can describe the specific variables affecting the data transformation of media companies to facilitate comparative analysis between variables. Thirdly, although machine learning methods are applied more in the field of natural sciences than in social science 9 , the advanced learning power and self-correction ability of machine learning are suitable for quantitative analysis of causal relationships between economic variables. They can also produce more accurate estimates of control variables, such as fixed effects outside the sample. Therefore, compared to traditional multiple linear regression, machine learning has the advantages of flexibility, accuracy, and foresight.

The media has undergone significant changes in a short period, due to technological changes. From publishing, broadcasting, and television, to the digital platform-based Internet and marketing, there is a significant gap between the business of companies in the media industry and the production of content and products. As mentioned earlier, the media industry represented by radio, television, and newspapers has strong political attributes that differ from the advertising, marketing, and gaming industries that actively integrate into international capital operations and look at the global market. Therefore, following existing literature practices 10 , 11 , 12 , this article innovatively divides the driving forces of the digital transformation of Chinese media companies into the following four dimensions of driving forces (adding “I” to “TOE”): (1) Technical, (2) Organizational, (3) Environmental, and (4) Industrial. This article explores the factors that drive or hinder the digital transformation of media companies from these perspectives. Based on the research results, corresponding policy implications are proposed.

This study is based on the “TOE-I” framework. Part 1 is the introduction of our research and in Part 2 we conduct the literature review mainly focused on the application of machine learning in media economics and the basic “TOE” framework. Part 3 is the methodology including the research design, data sources and variable definitions and empirical results and analysis. Part 4 is the conclusion of our main findings and we provide suggestions for policy makers and companies.

Literature review

The influencing factors of enterprise digital transformation.

Current relevant research into enterprises’ digital transformation largely focuses on two perspectives: the driving and hindering factors of digital transformation, and the impact of digital transformation on all aspects of the enterprise. Regarding the drivers of digital transformation, Verhoef et al. 13 focus on the response of enterprises to the changes in digital technology, increased digital competition, and the consequential digitalized customer behavior. While emerging digital technologies reduce labor costs, competition among companies intensifies and consumer preferences change accordingly. Yan et al. 14 found that mixed ownership reform is the catalyst for digital transformation and is also the key driving force for the sustainable development of China’s state-owned enterprises. In addition, digitalization has become an important factor affecting the decision-making processes of entrepreneurs, and digital strategy is an important part of the corporate strategy of enterprises 15 . Regarding the factors hindering the digital transformation of enterprises, Roman and Rusu 16 found that a lack of technology and capital negatively influences the digital transformation of enterprises, and that digital infrastructure has become an external factor affecting the digital transformation of enterprises. Looking at the impact of digital transformation, there is a focus on the relationship between digital transformation and the performance of enterprises 17 . Driven by the pursuit of profits, digital transformation has promoted the financialization of enterprises, especially among companies with poor internal and external governance 18 . The digital transformation of enterprises significantly promotes mergers and acquisitions by reducing internal organizational costs and is more significant among private enterprises 19 .

There are also many studies on digital transformation in different industries and different types of enterprises. For instance, Lange et al. 20 collected data from semi-structured interviews and concluded capital to be a driver for digital start-ups in massive and rapid business scaling (MRBS). Roman and Rusu 16 established an econometric-based model and highlighted the relationship between the performance of SMEs and digital transformation indicators. Ardolino et al. 21 focused on how digital capabilities (IoT, cloud computing, and predictive analytics) support the service transformation of industrial enterprises.

Application of machine learning in the field of media economics

Media economics is a discipline at the intersection of economics, management, and communication. It has shifted from the traditional media industry represented by the printing, television, and film industries to the new media research period with the Internet, digital platforms, and mobile communication media as the main focus 22 . The study of media economics under the corporate paradigm, and the evaluation of the operational results of media organizations has always been an issue. To evaluate business performance, Huang 23 put forward an evaluation system to measure the financialization level of a media company, including the index of the ownership structure, shareholder value, financial asset holding ratio, and financial investment rate. Sheng et al. 24 established an evaluation system on the performance of media organizations’ mergers and acquisitions to evaluate the value-creation ability after mergers and acquisitions. Xie and Li 5 looked at the evaluation of the competitiveness of listed media companies during the big data era.

At the core of artificial intelligence technology, machine learning technology has been widely used in the field of journalism and communication; for example, in the mode reformation of content production 25 , 26 , the prediction and discovery of social media trends, and the emotional analysis of users 27 , 28 , 29 . These methods are occasionally used in predicting and analyzing the operation and development of media companies. Pan and Wang 6 used machine learning methods to conduct text analysis of the annual report information of media companies to identify the relationship between digital transformation and the value of cultural enterprises. Shi and Wang 30 focused on the advertising industry, combining artificial neural network (ANN) algorithms to achieve intelligent evaluation and predictive analysis of advertising publishing and click results, and to optimize the resource utilization efficiency of the advertising industry. Sun et al. 31 used text mining and natural language processing (NLP) technology to conduct an emotional analysis on negative reports on the operation and financial status of media companies and established a warning mechanism for adverse impact on financial status.

In summary, this study has found that in the field of media economics, research is focused on macro perspective industry characteristics such as the operation and management of integrated media and the development and operation models of new media formats. From the perspective of micro market entities, however, there is still a lack of theoretical and empirical research on the transmission path to the industry change brought by digital transformation within media companies. Analysis and research on the driving factors of the implementation of digital transformation strategies in media companies through the predictive ability of machine learning is rare. This research aims to find indicators to measure the degree of digital transformation in the media industry and applies the machine learning method to identify the key elements driving the digital transformation of the media industry in China.

The “TOE-I” prediction model

The theoretical framework of TOE (Technology, Organization, Environment) was first proposed by Tornatzky and Fleischer to comprehensively study and analyze the influencing factors that may cause interference when enterprises adopt innovative technologies 32 . At the technology level, the TOE framework considers the influence of the internal technical level and technical support-related factors within an organization; that is, whether the enterprise can apply existing technologies, which is the basis for enterprises to adopt innovative technology 33 , 34 . At the organizational level, to achieve the future application of innovative technologies within the organization, the focus is on the composition of specialties and responsibilities of organizational personnel at different levels 35 , 36 . Environmental factors represent the macro external characteristics of the specific environment where the organization operates, such as government policies 37 , competitive pressure, and the business environment 38 . The TOE framework systematically considers technology and organizational factors both inside and outside organizations so it has strong systematization and operability.

However, industrial segmentation in different industries faces various digital transformation challenges in the digital era. Some scholars have proposed that to determine the specific factors in these three backgrounds and establish the potential relationships between these factors, the TOE framework serves as a basic framework to integrate other relative elements 33 . Influenced by technological changes, media has changed dramatically in a short period of time. From publishing to radio and television then to the Internet and marketing on a digital platform, there is a big gap between the business and the produced content of the media industry. Therefore, based on the characteristics of Chinese media companies, this study takes “I”-Industry as an important dimension to predict their digital transformation and integrates it into the TOE framework.

Based on existing literature practices 10 , 11 , 12 , this study divides the driving force of the digital transformation of Chinese media companies into the following four dimensions: (1) Technical driving force. Digital technology is the basis of the digitalization of the media industry and should be applied in all fields, particularly the production and operation process of the media industry 10 , 12 . The technical upgrading of enterprises is directly reflected in the investment in technical research and development and in the scale of technical personnel 39 , 40 . (2) Organizational driving force. The heterogeneity of corporate internal governance subjects, such as the characteristics of senior executives, enterprise organization, and governance structure will lead to different behavior in digital transformation among media companies 41 . Based on previous research 6 , 7 , the characteristics of the organizational driving force include the size, knowledge level, and social resources of the senior management team, as well as the revenue ability, debt repayment ability and continuous growth ability. (3) Environmental driving force. In the tide of media globalization, Chinese media companies represented by advertising and games expand their overseas markets, while media companies such as radio, television, and publishing (combining social and corporate attributes) are influenced by policy. Therefore, this study includes the opening rate of monetary policy, financial support, and competition pressure in the industry, as well as the level of protection of intellectual property by local government as environmental driving factors. (4) Industrial driving force. Publishing, radio and television, advertising and film, games and digital media face different industry bases and characteristics in the digital transformation. Mainstream media, represented by publishing and radio and television, actively develop new forms and content based on new media platforms. They also undertake political tasks in guiding public opinion and “narrating Chinese stories well” 3 , 4 . Big data and intelligent algorithms have continued to erode the boundaries of the traditional advertising industry, causing collective concerns in the advertising industry 42 . Within the industrial driving force dimension, China’s listed media companies are subdivided into six industries of games, advertising marketing, film and television cinema, digital media, publishing, and television broadcasting in predicting the driving force of the industry.

This article therefore aims to use the TOE model and takes “Industry” as one of the influencing factors based on the particularity of the Chinese media industry to explain why Chinese media companies conduct digital transformation, thereby filling the theoretical gap in the interaction mechanism between companies and industry characteristics in the Chinese media industry. To obtain a more accurate model, the study chose machine learning methods. Based on the practices of other research, we innovatively used ensemble learning models other than text analysis, such as Random Forest Regression (RFR) and Gradient Boosting Regression (GBR) models to expand the application of machine learning methods in the media field.

Research design

Research methods.

This study uses an integrated machine learning method to construct and integrate multiple base learners to achieve more accurate prediction effects than using a single one. According to the degree of independence among the base learners, the method of Nie et al. 43 and Parzinger et al. 44 selected the Gradient Boosting Regression (GBR) and Random Forest Regression (RFR) in serial and parallelization methods, then compared them with multiple linear regression and LASSO in a linear research method. The integrated machine learning method effectively illustrates the non-linear relationships and interactions between the variables in the linear relationship, so that it performs well in out-of-sample prediction tasks 8 . Therefore, this study predicts that integrated machine learning methods will outperform linear research methods in predicting the degree of digital transformation of media companies.

Model design

The model performance is examined from two perspectives: the model interpretation ability and the prediction error. In terms of model interpretation ability, Chen et al. 45 and Ghazwani and Begum 46 illustrate that ensemble learning can adjust itself based on the deviation between the model fitting value and the observation value in the previous calculation and can self-check the accuracy of the model. Therefore, we believe that the difference between the estimated values of the model and the observations can be used as a standard to evaluate the interpretation ability of the prediction model. The following two indicators are used: (1) Intra-sample goodness of fit ( \({\text{R}}_{\text{Is}}^{2}\) ) to evaluate the fitting effect of each research method on the training set sample. With the higher within-sample goodness of fit, the model is also more interpretable to the training set samples. (2) Out-of-sample goodness of fit ( \({\text{R}}_{\text{oos}}^{2}\) ) to measure the universality of the model. In addition, this study measures the generalization ability of the model from the perspective of variance, and chooses the explanatory variance to measure the dispersion of the actual value \(\text{(}{\text{EVS}}_{\text{oos}}\) ).

In terms of model prediction error, we followed the practice of Chen et al. 47 in selecting the out-of-sample mean variance ( \({\text{MSE}}_{\text{oos}}\) ) to investigate the deviation degree between the predicted and actual value. The out-of-sample mean square error is positively correlated with the accuracy of the model prediction. To avoid the large deviation value in the test set, which leads to estimated mean square error inconsistency, the average absolute error ( \({\text{MAE}}_{\text{oos}}\) ) and absolute median \(({\text{MedAE}}_{\text{oos}})\) differences were used to evaluate the accuracy of the model prediction. The specific methods of each evaluation index are shown in Table 1 .

In interpreting the model results, the integrated machine learning method includes multiple learners so it cannot be directly explained as much as a single learner 48 . To solve this problem, we used a relative importance and partial dependency graph to interpret the practical significance in the ensemble machine learning model. Relative importance refers to the degree of importance of one variable relative to the others in the process of model fitting. According to the method of Supsermpol et al. 49 , the relative importance of the variable can be assessed by measuring the decrease of the variable after its introduction. If the relative importance of a variable is high, it has a stronger influence in predicting the digital transformation of media companies. The partial dependency graph illustrates measurement of the influence of the changing degree of a certain variable on the digital transformation of a media company, assuming that other features are unchanged. Moreover, it is displayed in the form of images, which have more visual features. The single variable is more accurate in predicting the degree of digital transformation of media companies 50 .

Data sources and variable definitions

Data source.

This study selected media companies listed on A-shares in 2010–2020 as the initial sample, with company data from Wind and CSMAR databases. To exclude the interference of any special observation samples to the prediction results, the data were processed as follows: (1) exclusion of enterprises with ST, PT, and other abnormal listing status to avoid interference with the overall prediction effect due to abnormal operation of the enterprise itself; (2) elimination of samples with missing data; and (3) continuous variables in the data were treated by 1% and 99% quantile to avoid extreme outlier interference. A final set of 395 observations were obtained. The classification of the media industry adopts the 2021 SHENYIN&WANGUO classification method in the CSMAR database.

Variable definition

The digital transformation index (Digitaltransindex) in the CSMAR database was selected as the response variable. According to the CSMAR variable, the response variable using the annual report of enterprise digital transformation-related word frequency statistics can effectively reflect the enterprise digital transformation and transformation degree. It is divided into five parts: artificial intelligence (AI), blockchain (BD), cloud computing (CC), big data (BD) and the application of digital technology level (ADT). Table 2 shows the detailed calculations.

Based on the existing research of digital transformation drivers, this study selected the driving force characteristics of the model from the following four dimensions, as shown in Fig.  1 :

Four dimensions in TOE-I model.

This study draws on Yang and Xu 39 and Li 9 to select the intensity of R&D expenses and the proportion of technical personnel as the measurement indicators of innovation ability and absorption ability, as shown in Table 3 .

In terms of organizational dimensions, there are two main factors that affect the strategic decisions in a company’s digital transformation. The first is the leadership style of the company’s CEO and management. The attitude of the executive team towards risk, as well as the decision-making style and decision-making power of the management are closely related to the implementation level of the digital transformation strategy. The second factor lies in the internal operations and cash flow of the company. The implementation of digital transformation strategy requires a large amount of capital participation so the use and fundraising of internal and external funds of enterprises should be listed as influencing factors. Referring to Bernile et al. 51 , Schoar and Zuo 52 , and Bandiera et al. 53 , Manager Number, Education Level, Social Network, ROA, Growth, TobinQ, Lev, Top Ten Holders’ Rate, Duality, and IndDirector Ratio were selected as these variables 51 , 52 , 53 . Detailed calculations are shown in Table 4 .

Also, with reference to Xu et al. 54 , Sun and Zheng 55 , Wu and Ma 56 , this study took Financial Support, Infrastructure Score, Monetary Policy, IP Protection, and HhiD as variables to measure the environmental characteristics of media companies. The above indicators reflect the overall business environment of the media industry and the support from governments in different regions for innovative development in the media industry, as shown in Table 5 .

The fourth dimension is the industry classification. According to the revised version of the SHENYIN&WANGUO classification 2021, the media industry is subdivided into six categories: games, advertising and marketing, film and television cinema, digital media, publishing, and TV broadcasting, with a total of 141 listed companies, as shown in Table 6 .

Similarly, this study draws on Li et al. 57 , 58 , Zhao et al. 59 , Hanelt et al. 60 in taking Past Revenue, Cash Flow Ratio, Firm Age, Firm Size, and SOE as benchmark variable groups, as shown in Table 7 .

Empirical results and analysis

Descriptive statistics.

As shown in Table 8 , the mean value of the digital transformation index of media companies is 493.9037975, and the standard deviation is 297.7080399, indicating that the degree of digital transformation differs significantly among industries, and the characteristics of other variables have no outliers.

The prediction effect of the model constructed based on the machine learning method on the digital transformation index of media companies

Table 9 shows the prediction results of the models constructed by different integrated machine learning methods on the degree of digital transformation of media companies. The results in column (1) show that the within-sample goodness of fit of the multiple linear regression and LASSO model is lower than that of the GBR and RFR, indicating that the within-sample fitting effect of the integrated learning method is superior. In addition, the results of columns (2) and (3) in Table 9 show that the out-of-sample goodness of fit and interpretable variance of GBR have the highest values, 0.59123809 and 0.55601084, respectively, followed by RFR. Four indicators of both methods are higher than 0.5, indicating that machine learning methods can better predict the degree of digital transformation of media companies. It is clear that in column (4) the out-of-sample mean square error of the GBR and the RFR is smaller than the multiple linear regression and LASSO. Finally, columns (5) and (6) indicate that the GBR and RFR have low mean absolute errors, 0.57720771 and 0.58604578, respectively. This indicates that the model improvement effect is not obvious after excluding the deviation value.

In conclusion, the GBR and RFR in the ensemble machine learning method fit better to the data, thus constructing a more accurate prediction research model. This study further discusses the driving forces of the digital transformation of media companies and the key factors.

Differences in the driving force dimensions of media companies' digital transformation prediction ability

To explore the different driving force dimensions of media company digital transformation prediction ability, this study first constructed the listed years (Firm Age) and company size (Size) using control characteristics such as benchmark model calculation and comparison to add different driving force combinations of prediction performance. As the research conclusions obtained based on different evaluation indicators are largely the same, this study analyzes the out-of-sample goodness of fit and the research results are as shown in Table 10 .

Firstly, we considered the difference in the ability of single-dimensional drivers to predict the digital transformation of media companies. In comparison with other driving forces, the addition of environmental driving force features to the benchmark model achieves the best prediction effect. Taking RFR as an example, after adding technology, organization, environment, and industry drivers to the benchmark model, the predicted value increased by 92.86%, 100.53%, 145.17%, and 128.04%, respectively. Secondly, we considered differences in the ability of different combinations of drivers to predict the digital transformation of media companies. A combination including environmental driving force dimensions has the best performance: when the two types of driving force characteristics are combined, the benchmark model adds the environmental driving force and the industry driving force can obtain a higher model interpretation ability. When the technology driving force, environmental driving force and industry driving force are added to the benchmark model, the highest model interpretation ability is achieved. The results show that the environment driving force is more accurate in predicting the digital transformation degree, indicating that stable monetary policy, comprehensive infrastructure construction, government financial support for digital transformation, and good industry concentration are key elements in driving the digital transformation of Chinese media companies.

Differential analysis of the prediction ability of digital transformation by key factors under different driving forces

Based on the GBR and RFR, the relative importance of variables in the machine learning model is clear. Figures  2 and 3 report the relative importance ranking of the variables. Table 11 shows the variables ranked in the top 15 of the GBR and RFR methods, indicating that these characteristics are the key elements affecting the digital transformation of Chinese media companies.

Relative importance ranking based on GBR.

Relative importance ranking based on RFR.

Prediction model of digital transformation of media companies by important driving factors

Following the prediction method of GBR and RFR (The order in Figures 4 – 7 is as follows), among the many factors that affect the digital transformation of media companies, this study found that monetary policy, industry competition pressure, the proportion of technical size, return on assets of enterprises, age of the listed companies and industry classification have the best effect on predicting the digital transformation of media companies.

Monetary policy

Figure  4 is a partial dependency graph of monetary policy. The agent variable of monetary policy is the growth rate of M2. As shown in the figure, when the growth rate of M2 is less than 12.5%, there is no obvious impact on the degree of digital transformation of enterprises. However, when the growth rate is higher than 12.5%, the degree of digital transformation in enterprises shows a downward trend. Therefore, this study holds that the impact of monetary policy on the digital transformation of enterprises is not monotonous, and that enterprise managers should pay attention to the external environment at all times and adjust the process of the digital transformation of media companies timely.

Partial dependence on monetary policy.

Industry competition pressure

Figure  5 shows the HhiD of the industry as a tool to measure the level of competition among companies. Industry competition reflects the intensity in the competition for limited resources among companies. When the index is less than 0.05%, its impact on the digital transformation of media companies is significant and monotonous. When the index is 0.05% − 0.3%, the degree of digital transformation of media companies slowly decreases under its action, and when it reaches 0.3%, the impact effect is small. This shows that when the HhiD is high, media companies have low entry barriers and stable profit flow, thus enterprises have little or no demand to achieve differentiation in homogeneous competition.

Partial dependence chart of industry competitiveness.

The proportion of technical size

Figure  6 shows a partial dependence diagram of the proportion of technical size. This study selected the proportion of technical size as the agent variable. When the proportion of technical size is less than 10%, its influence on the digital transformation index of media companies shows a rapid increasing trend. When the proportion approaches 20%, the impact on the digital transformation index is highest. Above this, the impact effect is significantly reduced. Therefore, media companies should determine the proportion of technical size according to the requirements of digital transformation.

Partial dependence of technical size.

Return on assets of enterprises

Figure  7 shows a partial dependency graph of the return on assets of an enterprise. As shown, when the ROA of a media company is negative, this indicates problems in the operation of the enterprise. Managers will put more time and energy into business activities, rather than focusing on digital transformation, so the impact is small. When the return on equity of media companies is positive, this shows a temporary trend of rapid increase, followed by stability, indicating that its impact on the intensity of digital transformation is fluctuating.

Partial dependency of the ROA.

Age of the listed companies

Figure  8 shows the age of the listed company, used to describe the companies’ characteristics in the benchmark variables. The results indicated that when the establishment period of a media company is less than 15 years, the impact on the digital transformation of companies is not significant. When the establishment period of a media company reaches 15 years, the effect on the digital transformation is minimal, and then shows a significant increasing trend until the establishment period reaches 25 years. Media company management experience affects the strength of the digital transformation. When the company age is small, and lacks operating experience, the digital transformation is relatively low. In contrast, established companies with greater ability and resources can effectively support the digital transformation reform process, making full use of information advantage and achieving scale effects.

Partial dependence on the age of listed companies.

Industry classification

This research innovatively integrates the dimension of the industry driving force using the theoretical framework of TOE and forms the “TOE-I” model to predict the intensity of digital transformation of Chinese media companies. Firstly, through the two integrated machine learning methods of GBR and RFR, the model is shown to have a significant prediction effect on the advertising industry and the film and television industry, indicating that “TOE-I” can be better applied to the digital transformation prediction of the advertising industry and the film and television industry. Secondly, the prediction ability of radio and television, digital media, and the publishing industry is weak. As traditional mainstream media, radio, television and publishing media extend China’s mainstream media to the Internet field, the stronger the political attribute, the stronger the uncertainty of digital transformation, and the more difficult it is to accurately predict using the “TOE-I” model. Thirdly, game companies have usually been established more recently and are based on digital technology, so taking the digital transformation index as the measurement standard, the effect of digital transformation prediction is not significant enough. See Table 12 for details.

Robustness test

First, changing the training set division method. This study used the 8:2 proportion random classification to determine the training set and the test set, which weakens the randomness to some extent. Therefore, the K-fold method was used for further random division in the robustness test. In machine learning, K-fold cross-validation is a common method of model evaluation. It can help us accurately evaluate the performance of machine learning models and provides more reliable results particularly if the data is limited. The validation steps of the K-fold method were as follows:

The original dataset was split randomly into K subsets of similar size, taking K values of 10.

One subset was selected as the validation set and the remaining K-1 subset as the training set.

The model was trained using the training set and evaluated on the validation set.

Steps 2 and 3 were repeated until each subset is used as a validation set.

The results of K times of evaluation were integrated to obtain the final model evaluation index.

Based on this, K-fold cross validation can be repeated through the process of more stable evaluation results to reduce the contingency caused by different data divisions. For small data sets, K-fold cross validation can better evaluate the performance of the model, reducing the data caused by overfitting or underfitting problems.

As shown in Table 13 , after replacing the training and test sets using the K-fold test, the correlation findings are compared to Table 9 with no change.

Second, changing the measure of the intensity of digital transformation. Drawing on Xiao 61 , this study used different entries of enterprise digital transformation strength, eliminated the term “digital technology application” at the application level, and only retained the terms “artificial intelligence”, “blockchain”, “cloud computing”, and “big data” at the basic digital technology level. Add 1 to the total occurrence frequency and take the natural logarithm as the replacement variable for robustness testing. Using new response variables and refitting the model, the results were consistent with the main test, and the specific tests are shown in Table 14 .

Previous research has focused on the correlation between a single factor of a single dimension feature and the digital transformation of media industry, only conducting predictions within the sample, and lacking a comprehensive consideration of the driving forces in the digital transformation of media companies 5 , 6 , 7 . In this study, the driving forces of the digital transformation of Chinese media companies are divided into four dimensions: technology, organization, environmental, and industry drivers (i.e., the “TOE-I” model). The purpose of classifying the driving forces of digital transformation is to explore the differences in the prediction ability of media companies concerning digital transformation. This research analyzed the key driving factors of the digital transformation of media companies and the specific mode influenced by the above factors of the digital transformation of the Chinese media industry.

This study innovatively used ensemble learning methods, taking relative importance indicators and partial dependency graphs to help realize the research purpose. By comparing the fitting effect of the combination of different dimensions of driving forces in the benchmark model, we found that the environmental driving force can predict the digital transformation behavior of the Chinese media industry effectively and accurately, showing environmental drivers to be the dominant factor in influencing Chinese media companies’ strategies for digital transformation. Compared to linear methods such as multiple linear regression, the ensemble learning method achieved better performance in both model interpretation ability and minimization of prediction error, with the RFR method having the best predictive performance. The driving factors of (1) monetary policy, competition pressure in the industry, and the infrastructure index in the environmental driving force, (2) the equity concentration, enterprise value, executive team knowledge level and social networks in the organizational driving force, and (3) advertising, film and television industries in the industry driving forces all have significant predictive effects on the digital transformation of media industry in China.

Based on the above conclusions, the following policy implications are suggested:

Policy makers are supposed to provide stable monetary policies. Media companies like game companies and Internet advertising and marketing companies that think globally should be provided with stable economic and financial policies to facilitate their digital transformation. As shown in Fig.  4 , the intensity of digital transformation is highest when the M2 growth rate is 12.5%. Therefore, government managers should provide a stable monetary policy to promote the digital transformation of media companies. Moreover, as demonstrated by the prediction ability of the digital transformation, the dimension of the external environmental driving force has the greatest impact on the digital transformation of media companies. Therefore, in addition to providing a stable monetary policy, competition between industries should be guided to provide the matching infrastructure conditions for digital transformation.

Managers should maintain stable profit sources to promote digital transformation in media companies; for example for companies in radio, television, and newspapers with strong political attributes and little focus on income output and income. This study indicates that ensuring a positive cash flow of enterprises is an important driving factor for the digital transformation. Attention should also be paid to the decision-making role of core managers in the process of digital transformation. This research adopted two machine learning prediction methods—GBR and RFR that showed enterprise core managers to be an important influence in the prediction factors of China media companies (refer to ranking 2 of GBR and ranking 3 of RFR in Table 11 ). Thus, to ensure digital transformation, media companies should focus on the decision-making role of core managers. Furthermore, media companies should pay attention to the cultivation and preservation of digital technical talent. In the gradual conversion from traditional content and news production to digital, platform-based production, the cultivation and preservation of technical talent are crucial. Figure  6 illustrates that when the proportion of technical personnel in the company is 20%, digital transformation is the highest. Therefore, media companies should recruit digital technical personnel to maintain this level.

Within the media industry, it is necessary to seize the opportunity of technological change and pay close attention to policy changes. As shown in this study, the establishment of enterprises, the change of media, and the internal gap in the media industry all cause differences in the digital transformation of the media industry.

There is a gap in the application of empirical research and machine learning methods in existing media economics research, so the application of machine learning in the fields of media economics, journalism, and communication should be continuously promoted.

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Cui, B. & Yu, H. China media industry development report 2023. Media. 15 , 15–20 (2023).

Google Scholar  

Cui, B., Zhao, M. & Ding, M. Media Blue Book: China Media Industry Development Report (The Social Sciences Academic Press, 2023).

Yuan, Y. P. et al. Government digital transformation: Understanding the role of government social media. Govern. Inf. Q. 40 , 101775. https://doi.org/10.1016/j.giq.2022.101775 (2023).

Article   Google Scholar  

Waisbord, S. Continuities and breaks in digital journalism and media systems. Dig. Journal. https://doi.org/10.1080/21670811.2023.2254826 (2023).

Xie, Y. & Li, B. Evaluation of competitiveness portfolio of listed media groups in the big data era. Journal. Commun. Rev. 75 , 14086 (2022).

Pan, A. & Wang, X. How does digital transformation promote the high-quality development of cultural enterprises. J. Shenzhen Univ. 40 , 44–54 (2023).

Xing, M. & Chen, D. How does digital transformation leverage the performance of listed cultural companies? Based on “cost reduction and efficiency increasing” perspective. Cult. Ind. Res. 01 , 136–153 (2023).

Kleinberg, J., Ludwig, J., Mullainathan, S. & Obermeyer, Z. Prediction policy problems. Am. Econ. Rev. 105 , 491–495 (2015).

Article   PubMed   PubMed Central   Google Scholar  

Li, Y., Qian, Y., Li, Q. & Li, L. Evaluation of smart city construction and optimization of city brand model under neural networks. Comput. Sci. Inf. Syst. 20 , 573–593 (2023).

Zhang, Z. The theoretical framework, practical logic and realization path of cultural industry digitization. Soc. Sci. Front. 09 , 74–83 (2022).

Han, X., Guo, W. & Shi, L. Dimension framework and practice strategy of media innovation degree: From the perspectives of content production, market and technological innovation. Contemp. Commun. 06 , 49–54 (2022).

Zhang, Z. Research on business model innovation mechanism of cultural enterprises based on grounded theory. Theory J. 04 , 109–116 (2022).

ADS   Google Scholar  

Verhoef, P. C. et al. Digital transformation: A multidisciplinary reflection and research agenda. J. Bus. Res. 122 , 889–901 (2021).

Yan, L., Ling, X., Wang, Z. & Xu, Y. Can mixed-ownership reform boost the digital transformation of state-owned enterprises?. Econ. Anal. Policy. 79 , 647–663 (2023).

Kraus, S. et al. Digital transformation: An overview of the current state of the art of research. SAGE Open. https://doi.org/10.1177/21582440211047576 (2021).

Roman, A. & Rusu, V. D. Digital technologies and the performance of small and medium enterprises. Stud. Bus. Econ. 17 , 190–203 (2022).

Zeng, H., Ran, H., Zhou, Q., Jin, Y. & Cheng, X. The financial effect of firm digitalization: Evidence from China. Technol. Forecast. Soc. Chang. https://doi.org/10.1016/j.techfore.2022.121951 (2022).

Wu, K. & Lu, Y. Corporate digital transformation and financialization: Evidence from Chinese listed firms. Financ. Res. Lett. 57 , 104229. https://doi.org/10.1016/j.frl.2023.104229 (2023).

Tu, W. & He, J. Can digital transformation facilitate firms’ M & A: Empirical discovery based on machine learning. Emerg. Mark. Financ. Trade. 59 , 113–128 (2023).

Lange, F., Tomini, N., Brinkmann, F., Kanbach, D. K. & Kraus, S. Demystifying massive and rapid business scaling: An explorative study on driving factors in digital start-ups. Technol. Forecast. Soc. Chang. 196 , 122841. https://doi.org/10.1016/j.techfore.2023.122841 (2023).

Ardolino, M. et al. The role of digital technologies for the service transformation of industrial companies. Int. J. Prod. Res. 56 , 2116–2132 (2018).

Yu, G. & Liu, Y. Understanding generative Al: An examination of a milestone in the history of the internet’s development. Media Observ. 09 , 36–44 (2023).

Huang, H. Research on the effect mechanism of Chinese internet platform company’s financialization and capital expansion. Media Econ. Manag. Res. 01 , 71–101 (2022).

Sheng, H., Zhang, J. & Zhang, K. The characteristics and performance of M & A of press and publication media enterprises: Configuration analysis of fs OCA. Journal. Res. 04 , 104–118 (2022).

Guzman, A. L. & Lewis, S. C. Artificial intelligence and communication: A human–machine communication research agenda. New Media Soc. 22 (1), 70–86 (2020).

Opdahl, A. L. et al. Trustworthy journalism through AI. Data Knowl. Eng. 146 , 102182. https://doi.org/10.1016/j.datak.2023.102182 (2023).

Goldani, M. H., Safabakhsh, R. & Momtazi, S. Convolutional neural network with margin loss for fake news detection. Inf. Process. Manag. 58 , 102418. https://doi.org/10.1016/j.ipm.2020.102418 (2021).

Duan, H. K., Vasarhelyi, M. A., Codesso, M. & Alzamil, Z. Enhancing the government accounting information systems using social media information: An application of text mining and machine learning. Int. J. Account. Inf. Syst. 48 , 100600. https://doi.org/10.1016/j.accinf.2022.100600 (2022).

Wang, J. & Liu, Y. L. Deep learning-based social media mining for user experience analysis: A case study of smart home products. Technol. Soc. 73 , 102220. https://doi.org/10.1016/j.techsoc.2023.102220 (2023).

Shi, B. & Wang, H. An AI-enabled approach for improving advertising identification and promotion in social networks. Technol. Forecast. Soc. Chang. 188 , 122269. https://doi.org/10.1016/j.techfore.2022.122269 (2022).

Sun, J., Li, N. & Zhao, M. Analysis and machine learning negative media coverage and corporate financial distress prediction: Based on text. Collect. Essays Financ. Econ. 9 , 80–90 (2023).

Tornatzky, L. G. & Fleischer, M. The processes of technological innovation. Lexington, MA: Lexington Books (1990).

Awa, H. O., Ojiabo, O. U. & Orokor, L. E. Integrated technology organization-environment (T-O-E) taxonomies for technology adoption. J. Enterpr. Inf. Manag. 30 , 893–921 (2017).

Muchenje, G. & Seppanen, M. Unpacking task-technology fit to explore the business value of big data analytics. Int. J. Inf. Manag. 69 , 102619. https://doi.org/10.1016/j.ijinfomgt.2022.102619 (2023).

Delke, V., Schiele, H., Buchholz, W. & Kelly, S. Implementing Industry 40 technologies: Future roles in purchasing and supply management. Technol. Forecast. Soc. Change. 196 , 122847. https://doi.org/10.1016/j.techfore.2023.122847 (2023).

Hadwer, A. A., Tavana, M., Gillis, D. & Rezania, D. A systematic review of organizational factors impacting cloud-based technology adoption using technology-organization-environment framework. Internet Things. 15 , 100407. https://doi.org/10.1016/j.iot.2021.100407 (2021).

Shi, X., Zhang, Y., Wu, Y. & Wu, H. Political turnover and firm innovation in China: The moderating role of innovation and entrepreneurship environment. J. Asian Econ. 88 , 101651. https://doi.org/10.1016/j.asieco.2023.10165 (2023).

Luo, Y., Cui, H., Zhong, H. & Wei, C. Business environment and enterprise digital transformation. Financ. Res. Lett. 57 , 104250. https://doi.org/10.1016/j.frl.2023.104250 (2023).

Yang, J. & Xu, N. Research on the influence of dynamic capability and top managerial social capital on the digital transformation of enterprises. J. Technol. Econ. 42 , 97–109 (2023).

Zhang, M., Wang, D., Zeng, N. & Fu, S. How to become a sturdy grass withstanding strong wind? A study on the antecedent configurations of entrepreneurial ecosystem resilience based on WSR. Methodol. Manag. Rev. 35 , 89–102 (2023).

Chen, P. & Hao, Y. Digital transformation and corporate environmental performance: The moderating role of board characteristics. Corpor. Soc. Respons. Environ. Manag. 29 , 1757–1767 (2022).

Yu, G. & Fang, K. Will algorithmic content delivery result in information cocoons? A positive analysis based on media diversity and trust in information source. Shandong Soc. Sci. 11 , 170–174 (2020).

Nie, P. et al. Prediction of home energy consumption based on gradient boosting regression tree. Energy Rep. 7 , 1246–1255 (2021).

Parzinger, M. et al. Comparison of different training data sets from simulation and experimental measurement with artificial users for occupancy detection: Using machine learning methods Random Forest and LASSO. Build. Environ. 223 , 109313. https://doi.org/10.1016/j.buildenv.2022.109313 (2022).

Chen, C. H., Tanaka, K., Kotera, M. & Funatsu, K. Comparison and improvement of the predictability and interpretability with ensemble learning models in QSPR applications. J. Cheminform. https://doi.org/10.1186/s13321-020-0417-9 (2020).

Ghazwani, M. & Begum, M. Y. Computational intelligence modeling of hyoscine drug solubility and solvent density in supercritical processing: Gradient boosting, extra trees, and random forest models. Sci. Rep. 13 , 10046. https://doi.org/10.1038/s41598-023-37232-8 (2023).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Chen, X., Cho, Y. H., Dou, Y. W. & Lev, B. Predicting future earnings changes using machine learning and detailed financial data. J. Account. Res. 60 , 467–515 (2022).

Aceña, V., de Diego, I. M., Fernández, R. & Moguerza, J. M. Minimally overfitted learners: A general framework for ensemble learning. Knowl. Based Syst. 254 , 109669. https://doi.org/10.1016/knosys.2022.109669 (2022).

Supsermpol, P., Huynh, V. M., Thajchayapong, S. & Chiadamron, N. Predicting financial performance for listed companies in Thailand during the transition period: A class-based approach using logistic regression and random forest algorithm. J. Open Innov. 9 , 100130. https://doi.org/10.1016/j.joitmc.2023.100130 (2023).

Friedman, J. H. Greedy function approximation: A gradient boosting machine. Ann. Statist. 29 , 1189–1232 (2001).

Article   MathSciNet   Google Scholar  

Bernile, G., Bhagwat, V. & Rau, P. R. What doesn’t kill you will only make you more risk-loving: Early-life disasters and CEO behavior. J. Financ. 72 , 167–206 (2017).

Schoar, A. & Zuo, L. Shaped by booms and busts: How the economy impacts CEO careers and management styles. Rev. Financ. Stud. 30 , 1425–1456 (2017).

Bandiera, O. et al. CEO behavior and firm performance. J. Polit. Econ. 128 , 1325–1369 (2020).

Xu, N., Yuan, Q., Jiang, X. & Kam, K. C. Founder’s political connections, second generation involvement, and family firm performance: Evidence from China. J. Corpor. Financ. 33 , 243–259 (2015).

Sun, Z. & Zheng, Y. City government micro-blogging development and driving factors in china based on the combination of big data and small data analysis for 228 cities (2011–2017). J. Public Manag. 18 , 77–89 (2021).

Wu, Q. & Ma, X. Analysis on the efficiency of coordinated poverty reduction of government finance and finance supporting agriculture under the background of rural revitalization: Based on our country’s inter-provincial panel data. Modern. Manag. 41 , 15–19 (2021).

Li, S., Li, X., Wang, S. & Tong, Y. Family firm succession and digital transformation: Promotion or inhibition?. J. Manag. World 39 , 171–191 (2023).

Li, H., Long, H. & Wu, F. Heterogeneous institutional investors and enterprise digital transformation. Financ. Forum. 26 , 37–46 (2021).

Zhao, X., Chen, Q. & Zhang, H. Firm investment and financial autonomy: A transaction cost economics and firm lifecycle approach. Manag. Decis. Econ. 44 , 2540–2567 (2023).

Hanelt, A., Bohnsack, R., Marz, D. & Antunes, C. A systematic review of the literature on digital transformation: Insights and implications for strategy and organizational change. J. Manag. Stud. 58 , 1159–1197 (2021).

Xiao, T., Wu, Y. & Qi, W. Does digital transformation help high-quality development of enterprises? Evidences from corporate innovation. Bus. Manag. J. 44 , 41–62 (2022).

Download references

Acknowledgements

The authors would like to thank all of the people who participated in the studies.

Funding was provided by Liaoning Provincial Office of Philosophy and Social Science (Grant No. L20CXW008) and Office of the Foundation of Liaoning Province Education Administration (Grant No.  JG22DB240).

Author information

Authors and affiliations.

College of Humanities and Communication, Dongbei University of Finance and Economics, Dalian, China

School of Information Science and Technology, Dalian University of Science and Technology, Dalian, China

Surrey International Institute, Dongbei University of Finance and Economics, Dalian, 116025, Liaoning, China

Xu Zhao, Yuxuan Wang & Zihan Xiao

You can also search for this author in PubMed   Google Scholar

Contributions

Zhan Wang, Yao Li, Xu Zhao, Yuxuan Wang and Zihan Xiao: contributed to conceptualization, methodology, analysis and writing; Yao Li, Xu Zhao: contributed to validation and resources; Zhan Wang, Xu Zhao and Yao Li: contributed to experiment design, and data collection; Zhan Wang, Xu Zhao, Yuxuan Wang and Zihan Xiao contributed to investigation, supervision, and review editing. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Xu Zhao .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Wang, Z., Li, Y., Zhao, X. et al. Research on predicting the driving forces of digital transformation in Chinese media companies based on machine learning. Sci Rep 14 , 7286 (2024). https://doi.org/10.1038/s41598-024-57873-7

Download citation

Received : 28 September 2023

Accepted : 22 March 2024

Published : 27 March 2024

DOI : https://doi.org/10.1038/s41598-024-57873-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Digital transformation
• Machine learning
• Chinese media companies
• Media economics

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Dela J Public Health
• v.9(4); 2023 Nov
• PMC10759980

Machine Learning Methods for Systematic Reviews:

Stephanie roth.

1 Medical Librarian, Lewis B. Flinn Medical Library, ChristianaCare

Alex Wermer-Colan

2 Academic Director, Loretta C. Duckworth Scholars Studio, Temple University Libraries

Associated Data

Search strategies and the deduplicated citations are deposited in TUScholarShare available at: https://scholarshare.temple.edu/handle/20.500.12613/4637

Appendices and supplementary materials such as the data extraction template, search strategies, and characteristics of included studies can be found at: https://osf.io/x84t5/

At the forefront of machine learning research since its inception has been natural language processing, also known as text mining, referring to a wide range of statistical processes for analyzing textual data and retrieving information. In medical fields, text mining has made valuable contributions in unexpected ways, not least by synthesizing data from disparate biomedical studies. This rapid scoping review examines how machine learning methods for text mining can be implemented at the intersection of these disparate fields to improve the workflow and process of conducting systematic reviews in medical research and related academic disciplines.

The primary research question that this investigation asked, “what impact does the use of machine learning have on the methods used by systematic review teams to carry out the systematic review process, such as the precision of search strategies, unbiased article selection or data abstraction and/or analysis for systematic reviews and other comprehensive review types of similar methodology?” A literature search was conducted by a medical librarian utilizing multiple databases, a grey literature search and handsearching of the literature. The search was completed on December 4, 2020. Handsearching was done on an ongoing basis with an end date of April 14, 2023.

The search yielded 23,190 studies after duplicates were removed. As a result, 117 studies (1.70%) met eligibility criteria for inclusion in this rapid scoping review.

Conclusions

There are several techniques and/or types of machine learning methods in development or that have already been fully developed to assist with the systematic review stages. Combined with human intelligence, these machine learning methods and tools provide promise for making the systematic review process more efficient, saving valuable time for systematic review authors, and increasing the speed in which evidence can be created and placed in the hands of decision makers and the public.

Machine learning refers to a wide range of computational methods involving the optimization of statistical and analytical processes towards enhanced pattern recognition and classification of common features across diverse datasets. At the forefront of machine learning has been experimentation and research involving data mining, especially text mining. Machine learning methods have shown promising applications in the field of information retrieval for identifying keywords, topics, and stylistic patterns across a body of texts.

In the last decade, sophisticated methods of machine learning have become increasingly possible at scale thanks to innovations in graphical processing units and related computing hardware.

The significance of these methodological evolutions for the field of information science and several other fields remains under-examined today. While systematic reviews are a growing practice in library and information science, evidence on the usefulness of machine learning methods for the improvement of automated searches and filtration of resources and other review methods is lagging recent advancements in the field. Furthermore, assessment of tools and their applications in real-world scenarios remains minimal, with library practitioners needing more guidance on what resources can enable more efficient searching and reviews.

This rapid scoping review, then, seeks to address a potential blind spot in the current review literature for the wide variety of machine learning approaches and methods that have been applied to the systematic review process. To this end, this paper sets out to examine the impact of machine learning and related language processing algorithms, with a focus on what impact these techniques can have on improving the efficiency of human workflows.

Completing a systematic review is no small endeavor, typically taking months of planning and over a year to complete. This was felt firsthand by the review team in undertaking this rapid scoping review. While the timeframe does not appear to be rapid, it was necessary for the review to become rapid to avoid spanning several more years due to the broad nature of the research question. To this end, this review investigates the effectiveness and impact of machine learning for each main stage of the systematic review process. The intended audience is librarians/information professionals who often are the ones guiding researchers through these stages; and to those who develop or are interested in developing future tools/software for systematic reviews.

This rapid scoping review is categorized by the general stages of the systematic review:

Protocol/Planning Stage

This stage involves investigating the feasibility of the review, gathering a team, and formulating/preregistering a protocol outlining the detailed methods the review will follow, including the predetermined inclusion/exclusion criteria for study selection. Working with a librarian is important at the onset of the review and during the protocol development stage.

Search Stage

This stage of the review involves working with a co-authored librarian or information specialist who will create, test, and provide the team with a comprehensive search strategy and translate this search strategy across several databases and grey literature sources. Additional methods may be employed such as citation chaining. Review team members with the most subject matter expertise will hand search the literature.

Screening Stage

Title/abstract screening.

This stage of the review involves screening the title/abstracts found in the search results for relevancy. This process is conducted by two independent and blinded reviewers. Studies are filtered by Yes/No, no reasons are recorded for exclusion.

Full Text Screening

This stage of the review involves screening only the full texts of the included studies (Yes responses) from the title/abstract phase. Full texts of these studies are gathered, read, and checked against the predefined inclusion/exclusion criteria. This process is also conducted by two independent and blinded reviewers. Studies are filtered by Yes/No, reasons for exclusion are provided at this phase.

Data Extraction Stage

This stage of the review involves extracting the data from the included studies from the results of the full text review stage, characteristics of studies are recorded and other important data that will inform the review and statistical analysis if a meta-analysis is required.

Appraisal/Synthesis and Analysis Stage

Critical appraisal.

This stage of the review involves analyzing the included studies for risk of bias and/or assessing the quality of the study designs and methods.

Synthesis/Writing

This stage of the review involves synthesizing the evidence from the included studies. Conclusions are drawn from the evidence and gaps are explored for further research. The written portion of the review can be completed without including a meta-analysis or other statistical analysis.

Meta-analysis/Analysis

This stage of the review is only for those that require a meta-analysis or statistical analysis. Not every review requires one. It is recommended to work with a biostatistician at the onset of the review to determine if one is necessary.

Since a rapid scoping review, PRISMA-ScR was utilized for the reporting of this review. A protocol was registered in the Open Science Framework as a preregistration ( https://osf.io/j8ydg/ ).

Selection Criteria

The following criteria had to be met for a study to meet inclusion criteria for this rapid scoping review: research methods studies, a focus on machine learning, text analysis and/or automation, a focus on all or one or more stage(s) of the systematic review process, and the use of a machine learning application to assist with any or all stages of the review process. This review takes into consideration the overall landscape of machine learning and text analysis in systematic reviews, especially in terms of emerging trends and methods, while also being attentive to the barriers to facilitation and widespread adoption when user-friendly tools are not readily available.

Systematic reviews including all evidence syntheses were excluded since this review focuses on papers about the methods used in systematic reviews. Studies about updating a systematic review were excluded since this review to examines machine learning in the context of doing a new systematic review. Editorials, book chapters and similar works were also excluded.

Search Methods

To identify studies to include or consider for this rapid scoping review a medical librarian (SR), developed detailed search strategies for each database. The search was developed for PubMed (NLM) and was translated to Embase (Elsevier), Scopus (Elsevier), LISTA (EbscoHost) and the Social Science Premium Collection (ProQuest).

An attempt to locate grey literature was carried out using a Google search and scanning the SuRe Info web resource ( https://sites.google.com/york.ac.uk/sureinfo/home ).

A handsearch was conducted by scanning reference lists and the following journals or conference proceedings of BMC Systematic Reviews, Journal of Clinical Epidemiology, International Conference on Evaluation and Assessment in Software Engineering, Journal of Biomedical Informatics, JAMIA, AMIA, Research Synthesis Methods, BMC Bioinformatics, Expert Systems with Applications, ESMAR Conf and MLA vConference .

The search includes a date restriction of 2003 to present. The date restriction is justified due to the slow growth of machine learning overall and the use of machine learning in systematic reviews is a present-day advancement. Although this is not an exact cut-off date, the authors did not see a reason to search further back in the literature, as there were no potential harms or risks to persons due to the nature of this rapid scoping reviews focus on research methods.

The full systematic review search was completed on December 4, 2020, and formal handsearching was done on an ongoing basis with an end date of April 14, 2023.

While this review began before PRISMA-S for searching was published, the search methods are reported with the exclusion of a formal peer review of the search strategies.

Details of the search are provided in the Supplementary Materials and are available in the Temple University institutional repository, TUScholarShare.

Original Search Results by Database:

• PubMed (NLM) (10,695 Results)
• Embase (Elsevier) (981 Results)
• Scopus (Elsevier) (5,831 Results)
• LISTA (EbscoHost) (2,921 Results)
• Social Science Premium Collection (ProQuest) (2,589 Results)

The search resulted in 23,190 studies (including 147 from grey literature sources and 38 from hand searching). The 4,440 duplicate studies were found and omitted using Endnote X.7 for the deduplication of records and 18,750 references were eligible to screen.

Study Selection

Titles and abstracts were screened independently and blinded by two reviewers (SR, AWC) to identify studies that potentially met inclusion criteria. Endnote X.7 was used to manage references and remove duplicates before importing into Abstrackr to ensure blinded screening among reviewers. Abstrackr was selected as a screening tool because it uses machine learning to rank studies by relevance. The two blinded reviewers only screened the first 6,995 titles and abstracts. This threshold was decided upon since there were no longer relevant articles. Any disagreements between reviewers of those that were screened, were resolved by a third reviewer (JP) who served as a tiebreaker. The full text of 283 potentially eligible studies were then reviewed for eligibility by two independent reviewers (SR, AWC). There were no disagreements identified between reviewers. DeepL was used to translate one non-English language study. It was determined that 166 studies were excluded for the following reasons: wrong study type, wrong outcome, duplicate study, and the lack of availability of a full text. It was determined that 117 studies (1.70%) met eligibility criteria and were included in the final analysis. Details of the study selection process are detailed in the PRISMA Flow Diagram ( Figure 1 ).

PRISMA Flow Diagram

Data Extraction

Data was extracted from each of the included studies using a custom data extraction form. Two review authors extracted data using Excel spreadsheets, there were no discrepancies between reviewers. Characteristics of included studies are available in the online supplementary materials

This rapid scoping review categorizes the results based on the stage of the systematic review and in order of the review process. A category was also created for multiple review stages; this category includes papers that included multiple tools/software and/or methods for more than one stage of the review or a single tool/software and/or method that could be used for multiple stages of the review process. Results by review stage:

No studies were identified for this stage of the review.

Utilization of machine learning tools for systematic review searching can have its own set of time savings advantages when employed correctly with the involvement of a librarian or information expert. Several studies evaluated new or existing machine learning tools to support the search stage of the systematic review (studies cited in the online supplementary materials). These tools include RCT tagger, Litsearchr, WordNet, Robot Search, Zettair, SLR.qub tool (Systematic Literature Review – query builder), TerMine, Leximancer and Paperfetcher. A tool included in this category is the automated deduplication tool, Deduklick.

Several studies addressed user-friendliness, an important aspect for adoption of machine learning for systematic reviews. While Grames reported on the user-friendliness of the R package, Litsearchr, another caveat to implementation identified was the need for basic experience in using the R coding language. One barrier to the adoption of machine learning in systematic reviews is not having basic knowledge of how-to code. Most researchers conducting systematic reviews will not have this specialized skill set. While not expected, having a basic to advanced understanding in how to code can offer several benefits. One benefit of utilizing code is that there is no cost, as once free systematic review tools move to become proprietary (e.g., Covidence). The availability and cost of tools included in this rapid scoping review are reported in the online supplementary materials.

Screening titles/abstracts is one of the most laborious and time-consuming stages of the systematic review process and it is here when systematic review teams are truly tested to see if they have both the capacity and the time to complete a full systematic review. By design, machine learning is a suitable method for ameliorating this problem. While there is plenty of literature in this area, many are computer science-based studies describing complex algorithms or methods that will likely not result in the development of a public-facing tool in the short term. Hamel et al., provides some guidance and a set of recommendations for implementing machine learning in the title/abstract screening stage of the systematic review. 1

Less than 20 studies in this category referenced an existing tool available for public use. Several examples of publicly available tools include: DistillerSR, 2 Rayyan, Colandr, Abstrackr, EPPI-Reviewer, ASReview, RobotAnalyst, SWIFT Review, MetaMap, RapidMiner, and SyRF. In some cases, a tool in development was mentioned that either could not be found or no longer exists and/or never made it to production for public use such as StArt, 3 , 4 Revis 5 and TWISTER. 6 It is not certain if they were later merged with another tool, if they are still in the development stage, or if they never were intended for public use.

A main barrier to adoption of machine learning in systematic reviews is user-friendliness. Only 11 studies 7 – 17 addressed user-friendliness when evaluating their tool or process.

A few projects in existence to assist with the screening stage of the review are not yet suitable for the public or for those without advanced coding skills. 18 – 20

The use of machine learning has shown to reduce the time and human effort devoted to the screening stage of a systematic review; 7 , 21 – 40 however, the availability of their data was absent, making it hard to replicate. Extensive research is being done in this area, but data and/or computer scientists need to work together and across disciplines to learn from prior work. Data transparency allows others to build upon their work. However, there seems to be more proprietary tools in this category, making it difficult.

Machine learning can have time saving advantages for the data extraction stage of the systematic review that involves manually reading and extracting relevant texts from the included and chart the data. Duc An Bui, et al 41 utilized a PDF text classification tool to see how it helped with the data extraction stage of the systematic review, focusing on PDFBox ( https:/PDFbox/pdfbox.apache.org/ ) in combination with an annotation tool called GATE. Two studies 42 , 43 referenced the tool, EXACT ( https://bio-nlp.org/EXACT/ ), a tool designed to assist in the data extraction of clinical trials from a trials registry, clinicaltrials.gov. Both studies report that the data extracted was accurate and had a significant reduction on workload.

Torres & Cruzes introduced a tool called, Textum which used machine learning to assist researchers in analyzing specific parts of a paper. It estimated to have an overall 80% reduction in the time spent analyzing the texts of a traditional review. 44 However, today, this tool could not be located online, and it is not clear if it ever existed for public use. Other studies noted machine learning methods might help with the data extraction stage of the systematic review, but they have not yet resulted in a tool for the public. 45 , 46 However, only one has shared data from their study, which is a barrier to further developing machine learning for this stage of the review. 45 One limitation to adoption of these tools is that only one study in this category mentioned being user-friendly. The AFLEX-tag tool was reported to have a user-centered design. 47 Unfortunately, this tool does not seem to be publicly available.

The appraisal, synthesis, and analysis stages of the systematic review are areas where machine learning can aid, but likely not be a replacement for, human input. Several tools in this category were based on coding algorithms, 48 with the exception of RobotReviewer ( https://www.robotreviewer.net/ ) , a tool publicly available that could assist with reducing the time spent on the risk of bias assessment stage, but not serve as a complete replacement for manual risk of bias assessment. 48 – 50

A couple of R packages were also designed to help with the final analysis stage of the systematic review, such as Robumeta 51 and PublicationBias 52 which can assist with the sensitivity analysis for publication bias in systematic reviews.

Lingo3G is a machine learning tool that could support scoping reviews utilizing clustering to generate themes and/or a set of codes across several studies more rapidly than manual methods for synthesizing and coding the literature. 53 Marshall et al. (2015) 48 described a novel method for using support vector machines (SVM) to help automate the risk of bias assessment of clinical trials in systematic reviews. The author’s goal was to pair it with another tool in the pipeline to semi-automate the screening of abstracts.

Millard et al. reported a tool that was found to reduce the amount of time required by human reviewers for the risk of bias assessment. 54 On average it was found that more than 33% of research articles could be labeled with higher certainty than that of a human reviewer. While the use of machine learning to assist with the risk of bias assessment was like the method introduced by Marshall et al., 48 , 49 one difference according to Millard et al. 54 is that their team tested its method using full text articles rather than only titles and abstracts.

Multiple Review Stages

In a case study by Clark et al., multiple tools were used to complete a full small-scale systematic review in just two weeks (2weekSR) for multiple stages of the review. 55 One of the tools they used for the 2weekSR, was the RobotReviewer ( https://www.robotreviewer.net/ ), a semi-automation tool that uses machine learning to help with the risk of bias assessment of randomized controlled trials. A suite of automation tools used for the completion of a 2weekSR included the Systematic Review Accelerator, 55 , 56 which is designed to speed up each stage of the systematic review process. Recently the 2weekSR was tested on large scale systematic reviews, 57 completed within a few weeks. While review team members for the 2weekSR had protected research time to work on these projects, the findings are still demonstrative of the time reduction benefit of using machine learning.

Similarly, Haddaway et al. evaluates the use of partial automation using computational methods to assist in the facilitation of conducting a mapping review. 58 Since mapping reviews (and similarly scoping reviews) often assess a greater volume of literature than a traditional systematic review, partial automation can offer several workload advantages for the review team for various stages of the review. Lagopoulos & Tsoumakas, explored similar advantages with the hybrid machine learning tool, Elastic ( https://www.elastic.co/what-is/elasticsearch-machine-learning).Elastic had assisted with the preparation, retrieval, and appraisal stages of the systematic review. Utilizing this technology was called a technology assisted review (TAR). This hybrid approach is one that doesn’t involve creating a Boolean query by information experts. It relies instead upon initial machine learning retrieval methods, inter-review ranking and intra-review ranking.

When Altena, A. J. et al. 59 examined multiple machine learning tools for multiple systematic review stages, one noteworthy tool that stood out was Swift-Review ( https://www.sciome.com/swift-review/ ). Swift-Review uses statistical text mining and machine learning methods to help with search refinement to mine for relevant terms and with literature prioritization to help rank order documents for manual screening. Interestingly, Altena reports a low uptake in utilizing tools like Swift-Review in systematic reviews despite the advantages they may provide. Barriers to adoptability were usability, licensing, the steep learning curve, lack of support, mismatch to the workflow and the lack of time needed to assess or evaluate a new tool.

It is important for researchers to evaluate how much time should be devoted to assessing, evaluating, and implementing new tools. Further complicating this effort is the fact that free tools often are not sustainable with little support for users, especially when they are created by an individual or as a side hobby. This can often make the effort for review teams to implement machine learning tools more time intensive which is a barrier to the time reduction they normally offer. Out of the studies in this category, only two studies by Fabbri et al. 3 , 4 address the user-friendliness of the machine learning tool for systematic reviews, StArt ( http://lapes.dc.ufscar.br/tools/start_tool ). It would be helpful if future studies adopt this approach.

Miranda et al., explores the development of a new tool with screenshots of how it is applied to assist with several stages of the systematic review (i.e., the search, article selection, and data extraction). However, this tool doesn’t appear to be available to the general public. 60

This rapid scoping review examines the use of machine learning and related language processing algorithms and its impact on improving the efficiency of human workflows that researchers have been developing to reduce the amount of time necessary to complete each stage of the systematic review when compared to manual methods. Completing a systematic review that is done adequately is a time-consuming task and takes a minimum of a year or more depending on several factors.

A few limitations of this review include not exploring machine learning research outside of systematic reviews, which may have led to omitting papers still potentially relevant to the systematic review process. While not a formal exclusion of the paper, the search did not explore all computer science and computational method sources, databases and/or journals.

This rapid scoping review provides important insights into the state of machine learning developments to reduce the time and human labor spent on conducting a new systematic review from start to finish. Included in this review were the barriers to adoption of machine learning for researchers and the lack or reproducibility of machine learning data for those developing new software or tools.

Testing and providing more robust insights into the implementation of these machine learning methods and/or tools, such as demonstrating and rating the user-friendliness for the general user, are especially important, but are not widely demonstrated in the current literature. The results of this review may help computer scientists and/or programmers to eliminate research waste by identifying what methods or tools are already being developed across several disciplines. Perhaps new collaborations will result in building new tools that can universally address the current inefficiencies of conducting a manual systematic review. Librarians and information specialists will be able to find new ways to partner with researchers to supplement laborious tasks with machine learning.

While machine learning can assist with the systematic review process at various stages, it is still an emerging field for wide-scale application and must rely upon human input to be successfully implemented. Those who are developing tools or machine learning algorithms should work to make sure their research is clear and transparent to allow others to build upon their work. Future machine learning tools will need to be built with the end-user in mind for ease of use and widespread adoption. The barriers to adoption of machine learning should be considered and addressed during development.

Based on the current research, the use of machine learning to make the systematic review process more efficient remains favorable. Nevertheless, the researcher will need to be able to evaluate this new technology and select those that are best suited for their systematic review team’s needs. Combined with human intelligence, machine learning looks promising for making the systematic review process more efficient, saving time for the review team, and increasing the speed in which evidence is created. However, more experimental research studies with reproducible and open datasets are needed in the future to prove its effectiveness.

While this is an evolving area, currently machine learning is not a replacement for human effort. Those with the most systematic review methodological expertise, librarians, information specialists, statisticians, are all still essential in the overall design and implementation of the systematic review. Machine learning has the potential now to accelerate the rate of review completion for researchers and/or librarians and other information experts who invest time in learning and adopting these new tools.

Acknowledgements

Many thanks to Ania Korsunska for her assistance in screening titles/abstracts and to Jenny Pierce, MS for her assistance in serving as a tiebreaker.

Data Availability Statement

Machine Learning Techniques in Supply Chain Management: An Exploratory Literature Review

• Conference paper
• First Online: 31 March 2024
• Cite this conference paper

• Samia Haman 24 ,
• Aniss Moumen 24 ,
• Kaoutar Jenoui 25 ,
• Brahim Elbhiri 25 &
• Younes El Bouzekri El Idrissi 24  

Part of the book series: Advances in Science, Technology & Innovation ((ASTI))

Included in the following conference series:

• International Conference on Advanced Technologies for Humanity

Supply chain processes demand the use of a technology that can handle their growing complexity. Therefore, this study aims to investigate how machine learning techniques contribute to the management of the supply chain, by analyzing the current literature. Using an exploratory literature review methodology, this study analyzes the publications available in scientific databases: Scopus, Web of Science, and Science Direct. We have found 94 references that we extracted and analyzed using three tools: Zotero, NVIVO, and SPSS. We present, in this paper, an analysis of the meta-data, then we discuss machine learning techniques used in the different departments of the company, finally we present the perspectives of our study.

• Supply chain management
• Artificial Intelligence
• Machine learning
• Exploratory literature review

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Durable hardcover edition
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Albadrani, A., Zohdy, M. A., & Olawoyin, R. (2020). An approach to optimize future inbound logistics processes using machine learning algorithms. In 2020 IEEE International Conference on Electro Information Technology (EIT) (pp. 402–406). IEEE.

Google Scholar  

Albadrani, A., Alghayadh, F., Zohdy, M. A., Aloufi, E., & Olawoyin, R. (2021). Performance and predicting of inbound logistics processes using machine learning. In 2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC) (pp. 0790–0795). IEEE.

Bakar, N. A., et al. (2016). Abridgment of traditional procurement and e-procurement: definitions, tools and benefits. Journal of Emerging Economies and Islamic Research, 4 (1), 74–91.

Dash, R., McMurtrey, M., Rebman, C., & Kar, U. K. (2019). Application of artificial intelligence in automation of supply chain management. Journal of Strategic Innovation and Sustainability, 14 (3), 43–53.

Dondo, R., Méndez, C. A., & Cerdá, J. (2009). Managing distribution in supply chain networks. Industrial & Engineering Chemistry Research, 48 (22), 9961–9978.

Article   CAS   Google Scholar  

Dossou, P. E. (2019). Development of a new framework for implementing industry 4.0 in companies. Procedia Manufacturing, 38 , 573–580.

Article   Google Scholar  

Duvenage, M. M. (2009). Design of a warehouse SCOR model to align supply chain activities .

Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4 , 237–285.

Ma, Q., Li, H., & Thorstenson, A. (2021). A big data-driven root cause analysis system: Application of Machine Learning in quality problem solving. Computers & Industrial Engineering, 160 , 107580.

Makkar, S., Devi, G., Solanki, V. K. (2019). Applications of machine learning techniques in supply chain optimization. In International Conference on Intelligent Computing and Communication Technologies (pp. 861–869). Springer, Singapore.

Mcmahon, M., Mumper, D., Ihaza, M., & Farrar, D. (2019). How smart is your manufacturing? build smarter with AI. In 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC) (Vol. 2, pp. 55–60). IEEE.

Moumen, A., Bouchama, E. H., & El Bouzekri El idirissi, Y (2020). Data mining techniques for employability: Systematic literature review. 2020 IEEE 2nd International Conference on Electronics, Control, Optimization and Computer Science (ICECOCS) , 2020.

Mohamed-Iliasse, M., Loubna, B., & Abdelaziz, B. (2020). Is machine learning revolutionizing supply Chain? 2020 5th International Conference on Logistics Operations Management (GOL) , 2020.

Reiner, G., & Hofmann, P. (2006). Efficiency analysis of supply chain processes. International Journal of Production Research, 44 (23), 5065–5087.

Rifqi, H., Zamma, A., Souda, S. B., & Hansali, M. (2021). Positive effect of industry 4.0 on quality and operations management. International Journal of Online & Biomedical Engineering, 17 (9).

Sankhye, S., & Hu, G. (2020). Machine learning methods for quality prediction in production. Logistics, 4 (4), 35.

Shukla, R. K., Garg, D., & Agarwal, A. (2011). Understanding of supply chain: A literature review. International Journal of Engineering Science and Technology, 3 (3), 2059–2072.

Yadav, G., Kumar, A., Luthra, S., Garza-Reyes, J. A., Kumar, V., & Batista, L. (2020). A framework to achieve sustainability in manufacturing organisations of developing economies using industry 4.0 technologies’ enablers. Computers in Industry, 122 , 103280.

Download references

Author information

Authors and affiliations.

Laboratory of Engineering Sciences, National School of Applied Sciences, Ibn Tofail University, Kenitra, Morocco

Samia Haman, Aniss Moumen & Younes El Bouzekri El Idrissi

Smartilab, EMSI, Rabat, Morocco

Kaoutar Jenoui & Brahim Elbhiri

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Samia Haman .

Editor information

Editors and affiliations.

Ecole Marocaine Des Science De L'ingénieur (EMSI), Rabat, Morocco

Brahim El Bhiri

Institut National de Statistique et d'Economie Appliquée (INSEA), Rabat-Institut, Rabat, Morocco

Rajaa Saidi

Mohammed Essaaidi

University of North Dakota (UND), Grand Forks, ND, USA

Naima Kaabouch

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Cite this paper.

Haman, S., Moumen, A., Jenoui, K., Elbhiri, B., El Bouzekri El Idrissi, Y. (2024). Machine Learning Techniques in Supply Chain Management: An Exploratory Literature Review. In: El Bhiri, B., Saidi, R., Essaaidi, M., Kaabouch, N. (eds) Smart Mobility and Industrial Technologies. ICATH 2022. Advances in Science, Technology & Innovation. Springer, Cham. https://doi.org/10.1007/978-3-031-46849-0_17

Download citation

DOI : https://doi.org/10.1007/978-3-031-46849-0_17

Published : 31 March 2024

Publisher Name : Springer, Cham

Print ISBN : 978-3-031-46848-3

Online ISBN : 978-3-031-46849-0

eBook Packages : Earth and Environmental Science Earth and Environmental Science (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research