Note: First published in The Intellectual Property Strategist and Law.com.

This article is Part Three of a Three-Part Article Series

Artificial intelligence is changing industry and society, and metrics at the US Patent and Trademark Office (USPTO) reflect its impact. In a recent publication, the USPTO indicated that from 2002 to 2018 the share of all patent applications relating to artificial intelligence grew from 9% to approximately 16%. See Inventing AI, Tracing the diffusion of artificial intelligence with U.S. patents, Office of the Chief Economist, IP Data Highlights (October 2020). For the foreseeable future, patent applications involving artificial intelligence technologies, including machine learning, will increase with the continued proliferation of such technologies. However, subject matter eligibility can be a significant challenge in securing patents on artificial intelligence and machine learning.

This three-part article series explores USPTO handling of Alice issues involving artificial intelligence and machine learning through a sampling of recent Patent Trial and Appeal Board (PTAB) decisions. See Alice Corp. v. CLS Bank Intl, 134 S. Ct. 2347 (2014). Some decisions dutifully applied USPTO guidelines on subject matter eligibility, including Example 39 thereof, to resolve appeal issues brought to the PTAB. In one case, the PTAB sua sponte offered eligibility guidance even with no Alice appeal issue before it. These decisions inform strategies to optimize patent drafting and prosecution for artificial intelligence and machine learning related inventions.

Part One can be viewed here.

Part Two can be viewed here.

Part Three

Machine Learning Is Little More Than Just Another, Known, Data Processing Technique

The PTAB can provide subject matter eligibility guidance on artificial intelligence related inventions even when not asked. Ex parte Kneuper, Appeal 2020-005835 (PTAB April 28, 2021) is a reminder to patent applicants about inherent unpredictability and risk in PTAB appeal, especially in relation to Alice. In Kneuper, the sole issue on appeal before the PTAB was whether the claims were properly rejected during examination under section 103. The independent claim at issue recited:

1. An aircraft flight planning apparatus comprising:a database including

a plurality of forecasting models configured to

generate predictions of a predetermined characteristic onwhich at least a portion of an aircraft flight plan is based,where the predetermined characteristic includes at least aportion of a weather forecast, and

at least one data matrix of test predictions for the

predetermined characteristic generated by each of theplurality of forecasting models, each of the at least onedata matrix of test prediction includes a plurality of testprediction data points; and

an aircraft flight planning controller coupled to thedatabase, the aircraft flight planning controller being configured

to receive analysis forecast data having at least one

analysis data point,

select a forecasting model, from the plurality of

forecasting models, based on a comparison between theat least one analysis data point and the plurality of testprediction data points of a respective forecasting model,and

provide a prediction of the predetermined

characteristic generated with the forecasting model,selected from the plurality of forecasting models, thatcorresponds to a test prediction data point that isrepresentative of the at least one analysis data point.

Id. at 2. Claim 4 in Kneuper depended from claim 1, and added the following limitation: wherein each of the plurality of forecasting models are machine learning models. Thus, claim 4 specifically covered machine learning models that generate predictions of a predetermined characteristic, including a portion of a weather forecast, on which at least a portion of an aircraft flight plan is based.

Prior to discussing the prior art issue on appeal, the PTAB warned:

Before delving into the merits of the art rejection, we would be remissif we failed to mention that Appellants claims appear to recite little morethan using computer software for data collection, analysis, and display.Such is generally considered an abstract idea in the form of a mental processunder our Guidelines for analysis under 35 U.S.C. 101 . . .

Id. at 3. The first three paragraphs of the decision reflect the PTABs uninvited, albeit active, skepticism regarding eligibility, a non-issue up to that point. Of note, that skepticism was not supported by any discussion of, for example, an abstract idea, specific limitations, additional limitations, prong one, prong two, an inventive concept, or Example 39. Without regard to the analytical framework that typically supports an Alice decision, or an opportunity for the patent applicant to make its case, the PTAB likely sealed the fate of the claims at issue with this directive to the examiner: In the event that Appellant continues prosecution after resolution of this appeal, the Examiner may want to evaluate the eligibility of this application under Section 101. This admonishment as to eligibility was signaled by the PTABs later observation in relation to section 103 that [a]t the end of the day, machine learning is little more than just another, known, data processing technique. The PTAB acknowledged but dismissed the fact that the specification in Kneuper referenced decision trees, random forest algorithms, polynomial fit, and k-nearest neighbors as suitable machine learning models.

Kneuper is not surprising. Experienced practitioners know that the PTAB is not shy raising issues without invitation. While there should be no doubt that such risk also applies to artificial intelligence and machine learning related inventions, the added unpredictability of Alice issues in particular exacerbates risk. In this regard, patent applicants should remember that claim limitations involving artificial intelligence and machine learning may be deemed so deficient in terms of eligibility as to warrant preemptive PTAB refusals.

Key Takeaways

Patent strategy on artificial intelligence and machine learning inventions should account for recent PTAB decisions. The decisions explored in this three-part article series show that claims reciting predictive capabilities of machine learning models, even when relatively detailed, may not satisfy USPTO guidelines on subject matter eligibility. Drafters accordingly should prepare patent applications to support claims that recite detail about implementation and training of the models. In addition, discussion in the specification about technological difficulties overcome by machine learning claim limitations may strengthen eligibility positions.

Other considerations addressed by the PTAB decisions are also relevant to patent strategy. As the decisions reflect variation regarding PTAB focus on the first prong versus the second prong of Step 2A, patent applicants should seize opportunities to present arguments under both. When machine learning claim limitations regarding implementation are detailed, the first prong and Example 39 more easily support eligibility. Such detailed claim limitations likewise may bolster arguments establishing a technical improvement under the second prong, especially when complemented with strong distinctions over prior art. Further, before appealing even non-Alice issues, patent applicants should be prepared for the PTAB proactively questioning the eligibility of claims relating to artificial intelligence and machine learning.

Read more from the original source:
Artificial Intelligence And Subject Matter Eligibility In U.S. Patent Office Appeals - Part Three Of Three - Lexology

Close-up of blue logo on sign with facade of headquarters buildings in background near the ... [+] headquarters of Apple Computers in the Silicon Valley, Cupertino, California, August 26, 2018. (Photo by Smith Collection/Gado/Getty Images)

Apple stock (NASDAQ: AAPL) has gained almost 4% over the last week at near all-time highs of about $154 per share, driven by anticipation surrounding the launch of the companys next generation of iPhones, which are likely due around the third week of September, and possibility of updates to other products including MacBooks, Apple Watches, and AirPods in the coming months. Bloomberg previously reported that Apples suppliers are prepping to build as many as 90 million new iPhones this year, a 20% bump over its initial production run for the iPhone 12, indicating that Apple anticipates robust demand for the device, despite it likely being an incremental upgrade over the iPhone 12 which saw a design overhaul.

So will Apple stock continue to trend higher over the coming weeks and months, or is a correction looking more likely? According to the Trefis Machine Learning Engine, which identifies trends in a companys historical stock price data, returns for Apple stock average 2.4% in the next month (21 trading days) period after experiencing a 3.8% rally over the last five trading days.

But how would these numbers change if you are interested in holding Apple stock for a shorter or a longer time period? You can test the answer and many other combinations on the Trefis Machine Learning to test AAPL Stock Chances Of Rise After A Fall And Vice-Versa. You can test the chance of recovery over different time intervals of a quarter, month, or even just one day!

MACHINE LEARNING ENGINE try it yourself:

IF AAPL stock moved by -5% over 5 trading days, THEN over the next 21 trading days, AAPL stock moves an average of 1.7%, with a 54% probability of a positive return over this period.

Also, given a -5% movement for the stock over 5 trading days, it has historically witnessed an excess return of 2.9% compared to the S&P500 over the next 21 trading days, with a 64.4% percent probability of a positive excess return.

Some Fun Scenarios, FAQs & Making Sense of AAPL Stock Movements:

Question 1: Is the average return for Apple stock higher after a drop?

Answer:

Consider two situations,

Case 1: Apple stock drops by -5% or more in a week

Case 2: Apple stock rises by 5% or more in a week

Is the average return for Apple stock higher over the subsequent month after Case 1 or Case 2?

AAPL stock fares better after Case 2, with an average return of 1.7% over the next month (21 trading days) under Case 1 (where the stock has just suffered a 5% loss over the previous week), versus, an average return of 2.2% for Case 2.

In comparison, the S&P 500 has an average return of 3.1% over the next 21 trading days under Case 1, and an average return of just 0.5% for Case 2 as detailed in our dashboard that details the average return for the S&P 500 after a fall or rise.

Try the Trefis machine learning engine above to see for yourself how Apple stock is likely to behave after any specific gain or loss over a period.

Question 2: Does patience pay?

Answer:

If you buy and hold Apple stock, the expectation is over time the near-term fluctuations will cancel out, and the long-term positive trend will favor you - at least if the company is otherwise strong.

Overall, according to data and Trefis machine learning engines calculations, patience absolutely pays for most stocks!

For AAPL stock, the returns over the next N days after a -5% change over the last five trading days is detailed in the table below, along with the returns for the S&P500:

Average Return

Question 3: What about the average return after a rise if you wait for a while?

Answer:

The average return after a rise is understandably lower than after a fall as detailed in the previous question. Interestingly, though, if a stock has gained over the last few days, you would do better to avoid short-term bets for most stocks.

AAPLs returns over the next N days after a 5% change over the last five trading days is detailed in the table below, along with the returns for the S&P500:

Average Return

Its pretty powerful to test the trend for yourself for Apple stock by changing the inputs in the charts above.

What if youre looking for a more balanced portfolio instead? Heres a high-quality portfolio thats beaten the market since 2016

See allTrefis Featured AnalysesandDownloadTrefis Datahere

Original post:
Apple Stock Hits All Time Highs Of $154. Will It Rally Further? - Forbes

With digital disruption affecting every industry, including healthcare, the capacity to collect, exchange, and deliver data has become critical.

FREMONT, CA: Through algorithmic procedures, machine learning applications can increase the accuracy of treatment protocols and health outcomes. For instance, deep learning, a subset of advanced machine learning that simulates how the human brain works, is rapidly employed in radiology and medical imaging. Deep learning applications can detect, recognize, and evaluate malignant tumors from images using neural networks that learn from data without supervision.

Increased processing speeds and cloud infrastructures enable machine learning programs to discover anomalies in images that are not visible to the human eye, assisting in diagnosing and treating disease.

Machine learning developments in healthcare will continue to alter the business. The machine learning applications now in effect are a diagnostic tool for diabetic retinopathy and predictive analytics for predicting breast cancer recurrence using medical information and photos.

Three areas in which machine learning in health informatics impacts healthcare are discussed in the following sections.

Recordkeeping: In health informatics, machine learning can help streamline recordkeeping, particularly electronic health records (EHRs). Using AI to optimize EHR management can improve patient care, cost savings in healthcare and administration, and operational efficiencies.

Natural language processing is one example. It enables clinicians to take and record clinical notes without relying on human methods.

Additionally, machine learning algorithms can simplify physician usage of EHR management systems by offering clinical decision assistance, automating image analysis, and integrating telehealth technology.

The integrity of Data: Gaps in healthcare data can result in erroneous predictions from machine learning algorithms, which can severely impact clinical decision-making.

Since healthcare data was initially meant for EHRs, it must be prepared before machine learning algorithms can utilize it efficiently.

Professionals in health informatics are accountable for data integrity. Health informatics experts collect, analyze, classify, and cleanse data.

Analytical Prediction: Combining machine learning, health informatics, and predictive analytics enhances healthcare processes, the transformation of clinical decision support tools, and patient outcomes. The promise of machine learning in transforming healthcare is in its ability to harness health informatics to forecast health outcomes via predictive analytics, resulting in more accurate diagnosis and treatment and improved clinician insights for tailored and cohort therapies.

Additionally, machine learning may bring value to predictive analytics by translating data for decision-makers, allowing them to identify process gaps and optimize overall healthcare business operations.

See Also:Top Healthcare Communication Solution Companies

Link:
The Role of Machine Learning in Health Informatics - Healthcare Tech Outlook

A critical step after implementing a machine learning algorithm is to find out how effective our model is based on metrics and datasets. Different performance metrics available are used to evaluate the Machine Learning Algorithms. As an example, to distinguish between different objects, we can use classification performance metrics such as Log-Loss, Average Accuracy, AUC, etc. If the machine learning model is trying to predict, then an RMSE or root mean squared error can be used to calculate the efficiency of the model.

In this article, we will be discussing the performance metrics used in classification and also explore the significant use of two, in particular, the AUC and ROC. Below is the outline of important points that we will be discussing in the article.

The metrics that one chooses to evaluate a machine learning model play an important role. The choice of metric influences how the performance of machine learning algorithms can be measured and compared. But, Metrics possess a slight difference from loss functions. Loss functions are meant to show the measure of model performance. Theyre used to train a machine learning model, maybe using a kind of optimization like Gradient Descent, and are usually differentiable in the models parameters. Metrics on the other hand are used to monitor and evaluate the performance of a model during training and testing, not needing to be differentiable. The importance of various characteristics in the result will also be influenced completely by the metric.

One of the basic classification metrics is the Confusion Matrix. It is a tabular visualization of the truth labels versus the models predictions. Each row of the confusion matrix represents instances in a predicted class and each column represents instances in an actual class. Confusion Matrix is not entirely a performance metric but provides a basis on which other metrics can evaluate the results. There are 4 classes of a Confusion Matrix. The True Positive signifies how many positive class samples the created model has predicted correctly. True Negative signifies how many negative class samples the created model predicted correctly. False Positive signifies how many negative class samples the created model predicted incorrectly and vice versa goes for False Negative.

Precision-recall and F1 scores are the metrics for which the values are obtained from a confusion matrix as they are based on true and false classifications. The recall is also termed as the true positive rate or sensitivity, and precision is termed as the positive predictive value in classification.

Accuracy in terms of Performance Metrics is the measure of correct prediction of the classifier compared to its overall data points. It is the ratio of the units of correct predictions and the total number of predictions made by the classifiers. These additional performance evaluations help out to derive more meaning from your model.

AUC ROC is used to visualize the performance of a classification model based on its rate or correct and incorrect classifications. Further in this article, we will discuss in detail the AUC-ROC.

ROC curve, also known as Receiver Operating Characteristics Curve, is a metric used to measure the performance of a classifier model. The ROC curve depicts the rate of true positives with respect to the rate of false positives, therefore highlighting the sensitivity of the classifier model. The ROC is also known as a relative operating characteristic curve, as it is a comparison of two operating characteristics, the True Positive Rate and the False Positive Rate, as the criterion changes. An ideal classifier will have a ROC where the graph would hit a true positive rate of 100% with zero false positives. We generally measure how many correct positive classifications are being gained with an increment in the rate of false positives.

ROC curve can be used to select a threshold for a classifier, which maximizes the true positives and in turn minimizes the false positives. ROC Curves help determine the exact trade-off between the true positive rate and false-positive rate for a model using different measures of probability thresholds. ROC curves are more appropriate to be used when the observations present are balanced between each class. This method was first used in signal detection but is now also being used in many other areas such as medicine, radiology, natural hazards other than machine learning. A discrete classifier returns only the predicted class and gives a single point on the ROC space. But for probabilistic classifiers, which give a probability or score that reflects the degree to which an instance belongs to one class rather than another, we can create a curve by changing the threshold for the score.

Area Under Curve or AUC is one of the most widely used metrics for model evaluation. It is generally used for binary classification problems. AUC measures the entire two-dimensional area present underneath the entire ROC curve. AUC of a classifier is equal to the probability that the classifier will rank a randomly chosen positive example higher than that of a randomly chosen negative example. The Area Under the Curve provides the ability for a classifier to distinguish between classes and is used as a summary of the ROC curve. The higher the AUC, it is assumed that the better the performance of the model at distinguishing between the positive and negative classes.

The area under the curve is one of the good ways to estimate the accuracy of the model. An excellent model poses an AUC near to the 1 which tells that it has a good measure of separability. A poor model will have an AUC near 0 which describes that it has the worst measure of separability. In fact, it means it is reciprocating the result and predicting 0s as 1s and 1s as 0s. When an AUC is 0.5, it means the model has no class separation capacity present whatsoever.

AUC-ROC is the valued metric used for evaluating the performance in classification models. The AUC-ROC metric clearly helps determine and tell us about the capability of a model in distinguishing the classes. The judging criteria being Higher the AUC, better the model. AUC-ROC curves are frequently used to depict in a graphical way the connection and trade-off between sensitivity and specificity for every possible cut-off for a test being performed or a combination of tests being performed. The area under the ROC curve gives an idea about the benefit of using the test for the underlying question. AUC ROC curves are also a performance measurement for the classification problems at various threshold settings.

The AUC-ROC curve of a test can also be used as a criterion to measure the tests discriminative ability, telling us how good the test is in a given clinical situation. The closer an AUC-ROC curve is to the upper left corner, the more efficient the test being performed will be. To combine the False Positive Rate and the True Positive Rate into a single metric, we can first compute the two former metrics with many different thresholds for the logistic regression, then plot them on a single graph. The resulting curve metric we consider is the area under this curve, which we call AUC-ROC.

Image Source

AUC-ROC can be easily performed in Python using Numpy. The metric can be implemented on different Machine Learning Models to explore the potential difference between the scores. Here I have inculcated the same on two models, namely logistic Regression and Gaussian Naive Bias.

Just as discussed above, you can apply a similar formula using Python,

Output :

A Deterministic AUC-ROC plot can also be created to gain a deeper understanding. Here, plotting for Logistic Regression ;

For Gaussian Naive Bayes,

The results may vary given the stochastic nature of the algorithms the evaluation procedure used or differences in numerical precision.

The AUC-ROC is an essential technique to determine and evaluate the performance of a created classification model. Performing this test only increases the value and correctness of a model and in turn, helps improve its accuracy. Using this method helps us summarize the actual trade-off between the true positive rate and the predictive value for a predictive model using different probability thresholds which is an important aspect of classification problems.

In this article, we understood what a Performance Metric actually is and explored a classification metric, known as the AUC-ROC curve. We determined why it should be used and how it can be performed using python through a simple example. I would like to encourage the reader to explore the topic further as it is an important aspect while creating a classification model.

Happy Learning!

Understanding ROC and AUC

An introduction to ROC analysis

More here:
Understanding the AUC-ROC Curve in Machine Learning Classification - - Analytics India Magazine

Background

Labor induction (IOL) is the artificial stimulation of uterine contractions during pregnancy prior to the onset of labor in order to promote a vaginal birth.1 Recent advances in obstetric and fetal monitoring techniques have resulted in the majority of induced pregnancies having favorable outcomes; however, adverse health outcomes resulting in low Apgar scores in neonates continue to exist.2 The Apgar score tool, developed by Virginia Apgar, is a test administered to newborns shortly after birth. This examination analyzes the heart rate, muscle tone, and other vital indicators of a baby to determine if extra medical care or emergency care is required.3 The test is usually administered twice: once at 1 minute after birth and again at 5 minutes.4 Apgar scores obtained 5 minutes after birth have become widely used in the prediction of neonatal outcomes such as asphyxia, hypoxic-ischemic encephalopathy, and cerebral palsy.5 Additionally, recent research has established that Apgar values <7 five minutes after birth are related with impaired cognitive function, neurologic disability, and even subtle cognitive impairment as determined by scholastic achievement at the age of 16.6 Perinatal morbidity and death can be decreased by identifying and managing high-risk newborns effectively.7 Accurate detection of low Apgar scores at 5 minutes following labor induction is hence one among the ways to ensure optimal health and survival of the newborn.8 Several studies based on statistical learning have shown relationship and the interplay of maternal and neonatal variables for low Apgar scores.9,10 However, no studies have been conducted to date that focus exclusively on modeling neonatal Apgar scores following IOL intervention. As machine learning is applied to increasingly sensitive tasks and on increasingly noisy data, it is critical that these algorithms are validated against neonatal healthcare data.11 In addition, myriad studies have reported the potential of ensemble learning algorithms in predictive tasks.12,13 In the current study, we assessed the performance metrics of the three powerful ensemble learning algorithms. Due to skewed or imbalanced distribution of the outcome of interest, we further assessed whether the synthetic minority oversampling technique (SMOTE), Borderline-SMOTE and random undersampling (RUS) techniques would impact the learning process of the models.

We analyzed data from the Kilimanjaro Christian Medical Centre (KCMC) birth registry for women who gave birth to singleton infants between 2000 and 2015. This facility serves a population of around 11 million people from the region and neighboring areas. The register collects data on the mothers health prior to and during pregnancy, as well as complications and the infants status. All induced women who delivered singleton infants vaginally during the study period and had complete birth records were eligible for this study. Women with multiple gestations, stillbirths were excluded. These exclusions were necessary to offset the effect of possible overestimation of the prevalence of low Apgar scores (Figure 1). More information about the KCMC birth registry database can be found elsewhere.14 The final sample comprised 7716 induced deliveries.

Figure 1 Schematic diagram for sample size estimation.

Abbreviations: CS, cesarean section; IOL, induction of labor.

The response variable was Apgar score at 5 minutes (coded 0 for normal, and 1 for low) which was computed using five criteria. The first criterion included the strength and regularity of newborns heart rate where babies with 100 beats per minute or more scored 2 points while those with less than 100 scored 1 point and those with 0 heart rate scored 0 points. The second criterion assessed lung maturity or breathing effort, awarding 2 points to newborns with regular breathing, 1 point to those with irregular breathing with 30 breaths per minute, and 0 points to those with no breath at all. Muscle tone and mobility make the third component, for which active neonates received 2 points, moderately active ones received 1 point, and those who limped received no point. The fourth factor is skin color and oxygenation, where infants with pink color receiving 2 points, those with bluish extremities receiving 1 point, and those with completely bluish color receiving 0 points. The final component assesses reflex responses to irritating stimuli, with crying receiving 2 points, whimpering receiving 1 point, and silence receiving 0 points. The investigator then added the scores for each finding and defined a number less than seven (<7) as low and >7 as normal Apgar score. The current study examined the predictors of low Apgar scores previously reported in literature such as parity, maternal age, gestational age, number of prenatal visits, induction method used, body mass index (BMI). The gestational age at birth was calculated using the last menstrual period date and expressed in whole weeks, with deliveries of less than 37 weeks classified as preterm, those between 37 and 41 weeks as term, and those of 41 weeks or more as postterm. Additional behavioral and neonatal risk factors included child sex, smoking and alcohol consumption during pregnancy, as well as the history of using any form of family planning method were also examined. These factors were categorized as yes or no, with yes indicating the occurrence of these outcomes. The categories of the covariates for some factors variables were selected following a preliminary examination of the data.

Boosting algorithms have received significant attention in recent years in data science and machine learning. Boosting algorithms combine several weak models to produce a strong or more accurate model.15,16 Boosting techniques such as AdaBoost, Gradient boosting, and extreme gradient boosting (XGBoost) are all examples of ensemble learning algorithms that are often employed, particularly in data science contests.17 AdaBoost is designed to boost the performance of weak learners. The algorithm constructs an ensemble of weak learners iteratively by modifying the weights of misclassified data in each iteration. It gives equal weight to each training set sample when training the initial weak learner.18 Weights are revised for each succeeding weak learner in such a way that samples misclassified by the current weak learner receive a larger weight. Additionally, the family of boosting algorithms are said to be advantageous for resolving class imbalance problems since they provide a greater weight to the minority class with each iteration, as data from this class is frequently misclassified in other ML algorithms.19 Gradient boosting (GB) constructs an additive model incrementally and it enables optimization of arbitrary differentiable loss functions. It makes use of the gradient descent algorithm to reduce the number of errors in sequential models.20 In contrast to conventional gradient boosting, XGBoost employs its own way of tree construction, with the similarity score and gain determining the optimal node splits. So, it is a decision-tree-based ensemble method that utilizes a gradient boosting framework.21 Figure 2 shows the basic mechanism of boosting-based algorithm in modelling process.

Figure 2 Basic mechanism for boosting-based algorithms.

Our dataset was imbalanced in terms of class frequency, as the positive class (low Apgar score newborns) had only 733 individuals (9.5%). If one of the target classes contains a small number of occurrences in comparison to the other classes, the dataset is said to be imbalanced.22,23 Numerous ways to deal with unbalanced datasets have been presented recently.2426 This paper presents two approaches for balancing the dataset including synthetic minority oversampling technique (SMOTE) and random undersampling (RUS) technique. In contrast to traditional boosting, which assigns equal weight to all misclassified cases, resampling methods (SMOTE or RUS) and boosting algorithms (AdaBoost, Gradient boosting, XGBoost) applied to several highly and somewhat imbalanced datasets have been shown to improve prediction on the minority class.27,28 Re-sampling is a preprocessing approach that balances the distribution of an unbalanced dataset before it is sent to any classifiers.29 Resampling methods are designed to change the composition of a training dataset for an imbalanced classification task. SMOTE begins by randomly selecting an instance of a minority class and determining its k nearest minority class neighbors. The synthetic instance is then formed by selecting one of the k closest neighbors at random in the feature space to form a line segment.30 Borderline-SMOTE begins by classifying observations belonging to the minority class. It considers any minority observation to be noise if all of its neighbors are members of the majority class and the minority observation is discarded while constructing synthetic data. Additionally, it resamples entirely from a few places designated as border points with both majority and minority class. Additionally, it resamples entirely from a few places designated as border points with both majority and minority class instances. Undersampling (RUS) approaches eliminate samples from the training dataset that belong to the majority class in order to more evenly distribute the classes. The strategy reduces the dataset by removing examples from the majority class with the goal of balancing the number of examples in each class.31 Figure 3 indicates the basic mechanism for both RUS and SMOTE techniques.

Figure 3 Mechanisms of resampling techniques used: (A) RUS random undersampling (B) SMOTE synthetic minority oversampling techniques.

Descriptive statistics were obtained using STATA version 14. Data preprocessing and the main analyses were performed using Python programming (version 3.8.0). The predictive models for low Apgar scores were generated with test and training sets using Python scikit-learn (version 0.24.0) packages for machine learning. The parameters to assess the predictive performance of the selected ensemble machine learning algorithms have been evaluated in equations (1) through (8). The dataset was firstly converted to comma-separated values (CSV) file and imported to Python tool. We used open-source libraries in Python including Scikit-learn, Numpy and Pandas. The python codes used to generate the results along with the outputs are attached herein (Supplementary File 1).

(1)

(2)

(3)

(4)

(5)

(6)

(7)

(8)

where TP, FP, TN, FN, FPrate, PPV and NPV representtrue positive, false positive, true negative, false negative, false-positive rate, positive predictive value and negative predictive value respectively.

The sociodemographic and obstetric characteristics of the participants are summarized in Table 1. A total of 7716 Singleton births were analyzed. Of these, 55% of the deliveries were from nulliparous women while majority (88%) of study participants were aged <35 years and about 80% of the total deliveries were at term. The proportion of neonates with low Apgar scores (<7) was found to be 9.5%.

Table 1 Demographic Information of the Study Participant (N=7716)

Prior to the use of resampling techniques, all models performed nearly identically. Of all the resampling techniques considered in the current study, borderline-SMOTE was shown to significantly improve the performance of all the models in terms of all the metrics under observation (Table 2). RUS and SMOTE exhibited little or no improvement on baseline performance in all instances of their respective ensemble models. Performance in terms of AUC metrics for AdaBoost, GB, and XGBoost has been shown in Figure 4.

Table 2 Predictive Performance for of Low Apgar Score Following Labor Induction Using Ensemble Learning

Figure 4 Receiver operating characteristic (ROC) curve diagrams for boosting-based ensemble classifiers comparing the performance by resampling methods.

In this paper, we trained and evaluated the performance of three ensemble-based ML algorithms on a rare event (9.5% for <7 Apgar score versus 90.5% for >7 Apgar score). We then demonstrated how the resampling techniques can affect the learning process of the selected models on the imbalanced data. Kubat et al proposed a heuristic under-sampling method for balancing the data set by removing noise and redundant instances of the majority class.32 Chawla et al oversampled the minority class using the SMOTE (Synthetic Minority Oversampling Technique) technique, which generated new synthetic examples along the line between the minority examples and their chosen nearest neighbors.33 In the current study, both sampling techniques (SMOTE and RUS) were seen to slightly improve the sensitivity of the minority class, with the largest improvement seen from using borderline-SMOTE technique. Improvement of sensitivity means the ratio of correct positive predictions, that is, neonates with <7 Apgar score, to the total positive examples is relatively high. In other words, with the improvement shown by XGBoost following the Borderline-SMOTE resampling techniques, the model was able to correctly identify 93% (an improvement from 20% baseline performance) of the neonates with a low Apgar score, while missing 7% only. On the other hand, all the models performed well (Specificity = 99%) in correctly identifying neonates with normal (>7) Apgar score without the application of resampling methods. This could be because the number of neonates with a normal Apgar score was significantly greater than those with a low Apgar score in this database (n=6983 vs n=733), making the negative class more likely to be predicted. Notable is the Positive Predicted Value (PPV) obtained with XGBoost using the Borderline-SMOTE resampling method, which indicates that 94% of neonates predicted to have a low Apgar score actually had one. Numerous studies have demonstrated the critical importance of maximizing models sensitivity as well as PPV particularly when dealing with class imbalanced datasets.34 Precision and sensitivity make it possible and desirable to evaluate a classifiers performance on the minority class, resulting in another metric called the F-score.35 The F-score is high when both sensitivity and precision are high.36 Again, the best F-score was obtained in all models when borderline-SMOTE was used. However, the best F-score was reached by borderline-SMOTE applied specifically on XGBoost classifier. In terms of AUROC, borderline-SMOTE demonstrated a considerable improvement in the ensemble learners learning process. Neither SMOTE nor RUS techniques could improve the learning process in this occasion. Numerous studies have identified reasons for ineffectiveness in these resampling techniques, the most frequently cited being class overlap in feature space, which makes it more difficult for the classifier to learn the decision boundary. Studies have established that, if there is an overlapping between the classes given the variables in the dataset, SMOTE would be generating synthetic points affecting the separability.37,38 In addition, studies have pointed out that Tomek links, which are pairs of opposing instances that are very close together prior to model building, could be generated as well as other points, therefore harming the classification.39,40

Researchers working on artificial intelligence particularly on computer-assisted decision-making in healthcare as well as developers who are interested in developing predictive models for decision support system for neonatal healthcare can obtain clues on the efficiency of the ensemble learners particularly when the data is imbalanced and the respective resampling techniques that are likely to improve such prediction and hence make an informed decision. In totality, based on historical registry data, these model predictions enable healthcare informaticians to make highly accurate guesses about the likely outcomes of the intervention.

As we examined data from a single tertiary institution, our findings may have good internal validity but limited generalizability or external validity. It is possible that the study will show different results for datasets collected from other tertiary hospitals in north Tanzania; thus, caution should be exercised when concluding the specific finding. Furthermore, because we only looked at AUROC, F-scores, precision, NPV, PPV, sensitivity and specificity as performance indicators for boosting-based algorithms, our findings may be rather limited. Future research may shed light on other performance metrics, particularly those for unbalanced data, such as informedness, markedness, and Matthews correlation coefficient (MCC). Additionally, the current study did not conduct variable selection or feature engineering, nor did it address confounding variables, which could have limited or reduced classifier performance by increasing the likelihood of model overfitting. It would have been interesting to investigate whether or not the impact of feature engineering and confounding effects would result in improved results for both the SMOTE and RUS methods.

We encourage further research into other strategies for improving the learning process in this neonatal outcome, such as the ADASYN (ADAptive SYNthetic) sampling approach and the use of other SMOTE variants such as Safe-Level-SMOTE, SVM-SMOTE and KMeans-SMOTE. The combination of hybrid methods, that is, executing SMOTE and RUS methods concurrently on these ensemble methods, is also worth trying.

Predicting neonatal low Apgar scores after labor induction using this database may be more effective and promising when borderline-SMOTE is executed along with the ensemble methods. Future research may focus on testing additional resampling techniques mentioned earlier, performing feature engineering or variable selection, and optimizing further the ensemble learning hyperparameters.

This study was approved by the Kilimanjaro Christian Medical University College (KCMU-College) research ethics committee (reference number 985). Because the interview was conducted shortly after the mother had given birth, consent was only obtained verbally before the interview and enrollment. Trained nurses provided the information to the participants about the birth registry project and the information that they would need from them. However, following the consent, the woman could still choose whether or not to respond to specific questions. The KCMC hospital provided administrative clearance to access the data, and the Kilimanjaro Christian Medical College Research Ethics and Review Committee (KCMU-CRERC) approved all consent procedures. The database used in the current study contained no personally identifiable information in order to protect the study participants confidentiality and privacy.

The Birth Registry, the Obstetrics & Gynecology Department, and Epidemiology & Applied Biostatistics Department of the Kilimanjaro Christian Medical University College provided invaluable assistance during this investigation. Thanks to the KCMC birth registry study participants and the Norwegian birth registry for supplying the limited dataset utilized in this investigation.

This work was supported by the Research on CDC-Hospital-Community Trinity Coordinated Prevention and Control System for Major Infectious Diseases, Zhengzhou University 2020 Key Project of Discipline Construction [XKZDQY202007], 2021 Postgraduate Education Reform and Quality Improvement Project of Henan Province [YJS2021KC07], and National Key R&D Program of China [2018YFC0114501].

The authors declare that they have no competing interest.

1. Rayburn WF, Zhang J. Rising rates of labor induction: present concerns and future strategies. Obstet Gynecol. 2002;100(1):164167.

2. Grobman WA, Gilbert S, Landon MB, et al. Outcomes of induction of labor after one prior cesarean. Obstet Gynecol. 2007;109(2):262269. doi:10.1097/01.AOG.0000254169.49346.e9

3. Casey BM, McIntire DD, Leveno KJ. The continuing value of the Apgar score for the assessment of newborn infants. New Eng J Med. 2001;344(7):467471. doi:10.1056/NEJM200102153440701

4. Finster M, Wood M, Raja SN. The Apgar score has survived the test of time. J Am Soc Anesthesiol. 2005;102(4):855857.

5. Leinonen E, Gissler M, Haataja L, et al. Low Apgar scores at both one and five minutes are associated with longterm neurological morbidity. Acta Paediatrica. 2018;107(6):942951. doi:10.1111/apa.14234

6. Ehrenstein V, Pedersen L, Grijota M, Nielsen GL, Rothman KJ, Srensen HT. Association of Apgar score at five minutes with long-term neurologic disability and cognitive function in a prevalence study of Danish conscripts. BMC Pregnancy Childbirth. 2009;9(1):17. doi:10.1186/1471-2393-9-14

7. Manning FA, Harman CR, Morrison I, Menticoglou SM, Lange IR, Johnson JM. Fetal assessment based on fetal biophysical profile scoring: IV. An analysis of perinatal morbidity and mortality. Am J Obstet Gynecol. 1990;162(3):703709. doi:10.1016/0002-9378(90)90990-O

8. Yeshaneh A, Kassa A, Kassa ZY, et al. The determinants of 5th minute low Apgar score among newborns who delivered at public hospitals in Hawassa City, South Ethiopia. BMC Pediatr. 2021;21:266. doi:10.1186/s12887-021-02745-6

9. Lai S, Flatley C, Kumar S. Perinatal risk factors for low and moderate five-minute Apgar scores at term. Eur J Obstet Gynecol Reprod Biol. 2017;210:251256. doi:10.1016/j.ejogrb.2017.01.008

10. Rogers JF, Graves WL. Risk factors associated with low Apgar scores in a lowincome population. Paediatr Perinat Epidemiol. 1993;7(2):205216. doi:10.1111/j.1365-3016.1993.tb00394.x

11. Ahmad MA, Eckert C, Teredesai A. Interpretable machine learning in healthcare. Proceedings of the 2018 ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, 15 August 2018. 559560.

12. Mung PS, Phyu S. Effective analytics on healthcare big data using ensemble learning. In: 2020 IEEE Conference on Computer Applications (ICCA); February 27, 2020; IEEE. 14.

13. Liu N, Li X, Qi E, Xu M, Li L, Gao B. A novel ensemble learning paradigm for medical diagnosis with imbalanced data. IEEE Access. 2020;8:171263171280. doi:10.1109/ACCESS.2020.3014362

14. Bergsj P, Mlay J, Lie RT, Lie-Nielsen E, Shao JF. A medical birth registry at Kilimanjaro Christian Medical Centre. East Afr J Public Health. 2007;4(1):14.

15. Robinson JW. Regression tree boosting to adjust health care cost predictions for diagnostic mix. Health Serv Res. 2008;43(2):755772. doi:10.1111/j.1475-6773.2007.00761.x

16. Park Y, Ho J. Tackling overfitting in boosting for noisy healthcare data. In: IEEE Transactions on Knowledge and Data Engineering; December 16, 2019.

17. Joshi MV, Kumar V, Agarwal RC. Evaluating boosting algorithms to classify rare classes: comparison and improvements. In Proceedings 2001 IEEE International Conference on Data Mining, 29 November 2001. IEEE; 257264.

18. Ying C, Qi-Guang M, Jia-Chen L, Lin G. Advance and prospects of AdaBoost algorithm. Acta Autom Sin. 2013;39(6):745758. doi:10.1016/S1874-1029(13)60052-X

19. Lee W, Jun CH, Lee JS. Instance categorization by support vector machines to adjust weights in AdaBoost for imbalanced data classification. Inf Sci (Ny). 2017;381:92103. doi:10.1016/j.ins.2016.11.014

20. Lusa L. Gradient boosting for high-dimensional prediction of rare events. Comput Stat Data Anal. 2017;113:1937. doi:10.1016/j.csda.2016.07.016

21. Wang H, Liu C, Deng L. Enhanced prediction of hot spots at protein-protein interfaces using extreme gradient boosting. Sci Rep. 2018;8(1):13.

22. Zhao Y, Wong ZS, Tsui KL. A framework of rebalancing imbalanced healthcare data for rare events classification: a case of look-alike sound-alike mix-up incident detection. J Healthc Eng. 2018;2018. doi:10.1155/2018/6275435

23. Li J, Liu LS, Fong S, et al. Adaptive swarm balancing algorithms for rare-event prediction in imbalanced healthcare data. PLoS One. 2017;12(7):e0180830. doi:10.1371/journal.pone.0180830

24. Zhu B, Baesens B, Vanden Broucke SK. An empirical comparison of techniques for the class imbalance problem in churn prediction. Inf Sci. 2017;408:8499. doi:10.1016/j.ins.2017.04.015

25. Gosain A, Sardana S. Handling class imbalance problem using oversampling techniques: a review. In: 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI); September 13, 2017; IEEE. 7985.

26. Amin A, Anwar S, Adnan A, et al. Comparing oversampling techniques to handle the class imbalance problem: a customer churn prediction case study. IEEE Access. 2016;26(4):79407957. doi:10.1109/ACCESS.2016.2619719

27. Elreedy D, Atiya AF. A comprehensive analysis of synthetic minority oversampling technique (SMOTE) for handling class imbalance. Inf Sci. 2019;1(505):3264. doi:10.1016/j.ins.2019.07.070

28. Prusa J, Khoshgoftaar TM, Dittman DJ, Napolitano A. Using random undersampling to alleviate class imbalance on tweet sentiment data. In: 2015 IEEE International Conference on Information Reuse and Integration; August 13, 2015; IEEE. 197202.

29. Chernick MR. Resampling methods. Wiley Interdiscip Rev Data Min Knowl Discov. 2012;2(3):255262.

30. Cheng K, Zhang C, Yu H, Yang X, Zou H, Gao S. Grouped SMOTE with noise filtering mechanism for classifying imbalanced data. IEEE Access. 2019;7:170668170681. doi:10.1109/ACCESS.2019.2955086

31. Triguero I, Galar M, Merino D, Maillo J, Bustince H, Herrera F. Evolutionary undersampling for extremely imbalanced big data classification under apache spark. In: 2016 IEEE Congress on Evolutionary Computation (CEC); July 24, 2016; IEEE. 640647.

32. Kubat M, Matwin S. Addressing the course of imbalanced training sets: one-sided selection. In: ICML. Vol. 97. Citeseer; 1997:179186.

33. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res. 2002;16:321357. doi:10.1613/jair.953

34. Sokolova M, Japkowicz N, Szpakowicz S. Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation. In: Australasian Joint Conference on Artificial Intelligence; December 4, 2006; Springer, Berlin, Heidelberg. 10151021.

35. Goutte C, Gaussier E. A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In: European Conference on Information Retrieval; March 21, 2005; Springer, Berlin, Heidelberg. 345359.

36. Guns R, Lioma C, Larsen B. The tipping point: f-score as a function of the number of retrieved items. Inf Process Manag. 2012;48(6):11711180. doi:10.1016/j.ipm.2012.02.009

37. Alahmari F. A comparison of resampling techniques for medical data using machine learning. J Inf Knowl Manag. 2020;19:113.

38. Vuttipittayamongkol P, Elyan E, Petrovski A. On the class overlap problem in imbalanced data classification, knowledge-based systems 212; 2021. Available from: http://www.sciencedirect.com/science/article/pii/S0950705120307607. Accessed August 31, 2021.

39. Zeng M, Zou B, Wei F, Liu X, Wang L. Effective prediction of three common diseases by combining SMOTE with Tomek links technique for imbalanced medical data. In 2016 IEEE International Conference of Online Analysis and Computing Science (ICOACS); May 28, 2016; IEEE. 225228.

40. Ning Q, Zhao X, Ma Z. A novel method for Identification of Glutarylation sites combining Borderline-SMOTE with Tomek links technique in imbalanced data. In: IEEE/ACM Transactions on Computational Biology and Bioinformatics; July 8, 2021.

Read more here:
Neonates with a low Apgar score after induction of labor | RMHP - Dove Medical Press