|
Forecasting with Computational Intelligence - An Evaluation of Support Vector Regression and Artificial Neural Networks for Time Series Prediction
Sven F. Crone, Stefan Lessmann and Swantje Pietsch
Abstract: Recently, novel algorithms of Support Vector Regression and Neural Networks have received increasing attention in time series prediction. While they offer attractive theoretical properties, they have demonstrated only mixed
results within real world application domains of particular
time series structures and patterns. Commonly, time series are
composed of a combination of regular patterns such as levels,
trends and seasonal variations. Thus, the capability of novel
methods to predict basic time series patterns is of particular
relevance in evaluating their initial contribution to forecasting.
This paper investigates the accuracy of competing forecasting
methods of NN and SVR through an exhaustive empirical
comparison of alternatively tuned candidate models on 36
artificial time series. Results obtained show that SVR and NN
provide comparative accuracy and robustly outperform
statistical methods on selected time series patterns.
The complete paper is availbale in the conference proceeding of the International Joint Conference on Neural Networks 2006.
(I am not responsible for the content of the linked pages)
Citation: Crone, S. F., Lessmann, S., Pietsch, S.: Forecasting with Computational Intelligence - An Evaluation of Support Vector Regression and Artificial Neural Networks for Time Series Prediction, IEEE World Congress on Computational Intelligence, Vancouver (Canada), 2006
Parameter Sensitivity of Support Vector Regression and Neural Networks for Forecasting
Sven F. Crone, Stefan Lessmann and Swantje Pietsch
Abstract: Support Vector Regression (SVR) and artificial
Neural Networks (NN) promise attractive features for time
series forecasting. Despite their attractive theoretical
properties, limited empirical studies using small or unbalanced
parameter setups yield inconsistent results regarding their
empirical accuracy. This paper investigates the accuracy of
different configurations of NN and SVR parameters, paying
particular attention to the common SVR kernels of polynomial,
radial basis functions, sigmoid and linear functions through an
exhaustive empirical comparison. We investigate the
forecasting performance of alternative parameter setups with
established benchmarks, evaluating all models on 36 artificial
time series with archetypical patterns of level, trend,
seasonality and trend-seasonality. As a result, we find that SVR
and NN outperform statistical methods on particular time
series patterns. Forecasting performance of SVR and NN is
impacted by choice of parameters, indicating NN and SVR with
the RBF kernels as robust choices on most time series
forecasting problems.
The complete paper is availbale in the conference post-proceeding of the DMIN 2006. (I am not responsible for the content of the linked pages)
Citation: Crone, S. F., Lessmann, S., Pietsch, S.: Parameter Sensitivity of Support Vector Regression and Neural Networks for Forecasting, The 2006 International Conference on Data Mining, Las Vegas (U.S.A.), 2006.
A Naïve Support Vector Regression Benchmark for the NN3 Forecasting Competition
Sven F. Crone and Swantje Pietsch
Abstract: Support Vector Regression is one of the promising
contenders in predicting the 111 time series of the NN3 Neural
Forecasting Competition. As they offer substantial degrees of
freedom in the modeling process, in selecting the kernel
function and its parameters, cost and epsilon parameters, issues
of model parameterization and model selection arise. In lack of
an established methodology or comprehensive empirical
evidence on their modeling, a number of heuristics and ad-hoc
rules have emerged, that result in selecting different models,
which show different performance. In order to determine a
lower bound for Support Vector Regression accuracy in the
NN3 competition, this paper seeks to compute benchmark
results using a naïve methodology with a fixed parameter gridsearch
and exponentially increasing step sizes for radial basis
function kernels, estimating 43,725 candidate models for each
of the 111 time series. The naïve approach attempts to mimic
many of the common mistakes in model building, providing
error as a lower bound to support vector regression accuracy.
The in-sample results parameters are evaluated to estimate the
impact of potential shortcomings in the grid search heuristic
and the interaction effects of the parameters.
The complete paper is availbale in the conference proceeding of the International Joint Conference on Neural Networks 2007.
(I am not responsible for the content of the linked pages)
Citation: Crone, S. F. and S., Pietsch, S.: A Naïve Support Vector Regression Benchmark for the NN3 Forecasting Competition, International Joint Conference on Neural Networks, Orlando (USA), 2007
Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings
Stefan Lessmann, Bart Baesens, Christophe Mues and Swantje Pietsch
Abstract: Software defect prediction strives to improve software quality and testing efficiency by constructing predictive classification models from code attributes to enable a timely identification of fault-prone modules. Several classification models have been evaluated for this task. However, due to inconsistent findings regarding the superiority of one classifier over another and the usefulness of metric-based classification in general, more research is needed to improve convergence across studies and further advance confidence in experimental results. We consider three potential sources for bias: comparing classifiers over one or a small number of proprietary datasets, relying on accuracy indicators that are conceptually inappropriate for software defect prediction and cross-study comparisons, and finally, limited use of statisti-cal testing procedures to secure empirical findings. To remedy these problems, a framework for comparative software defect prediction experiments is proposed and applied in a large-scale empirical comparison of 22 classifiers over ten public domain datasets from the NASA Metrics Data repository. Our results indicate that the importance of the particu-lar classification algorithm may have been overestimated in previous research since no significant performance differ-ences could be detected among the top-17 classifiers.
The complete paper is availbale at http://doi.ieeecomputersociety.org/10.1109/TSE.2008.35
(I am not responsible for the content of the linked pages)
Stefan Lessmann, Bart Baesens, Christophe Mues, Swantje Pietsch, "Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings," IEEE Transactions on Software Engineering, vol. 34, no. 4, pp. 485-496, Jul/Aug, 2008
|
