An Optimal Design Method for Artificial Neural Networks by Using the Design of Experiments

Eiichi Inohira; Hirokazu Yokoi

doi:10.20965/jaciii.2007.p0593

single-jc.php

« previous

JACIII Vol.11 No.6 pp. 593-599

doi: 10.20965/jaciii.2007.p0593

(2007)

Paper:

Views over last 60 days: 569

An Optimal Design Method for Artificial Neural Networks by Using the Design of Experiments

Eiichi Inohira and Hirokazu Yokoi

Department of Biological Functions and Engineering, Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology, 2-4 Hibikino, Kitakyushu, Fukuoka 808-0196, Japan

Received:

January 16, 2007

Accepted:

March 20, 2007

Published:

July 20, 2007

Keywords:

artificial neural network, optimal design, design of experiments, statistical analysis, multilayer neural network

Abstract

This paper presents a method to optimally design artificial neural networks with many design parameters using the Design of Experiment (DOE), whose features are efficient experiments using an orthogonal array and quantitative analysis by analysis of variance. Neural networks can approximate arbitrary nonlinear functions. The accuracy of a trained neural network at a certain number of learning cycles depends on both weights and biases and its structure and learning rate. Design methods such as trial-and-error, brute-force approaches, network construction, and pruning, cannot deal with many design parameters such as the number of elements in a layer and a learning rate. Our design method realizes efficient optimization using DOE, and obtains confidence of optimal design through statistical analysis even though trained neural networks very due to randomness in initial weights. We apply our design method three-layer and five-layer feedforward neural networks in a preliminary study and show that approximation accuracy of multilayer neural networks is increased by picking up many more parameters.

Cite this article as:

E. Inohira and H. Yokoi, “An Optimal Design Method for Artificial Neural Networks by Using the Design of Experiments,” J. Adv. Comput. Intell. Intell. Inform., Vol.11 No.6, pp. 593-599, 2007.

Data files:

References

[1] D. Tikk, L. T. Koczy, and T. D. Gedeon, “A Survey on Universal Approximation and its Limits in Soft Computing Techniques,” International Journal of Approximate Reasoning, pp. 185-202, 2003.
[2] F. Scarselli and A. C. Tsoi, “Universal Approximation Using Feedforward Neural Networks: A Survey of Some Existing Methods, and Some New Results,” Neural Networks, Vol.11, No.1, pp. 15-37, 1998.
[3] D. Rumelhart, G. Hinton, and R. Williams, Parallel Distributed Processing, MIT Press, 1986.
[4] A. P. Engelbrecht, “A New Pruning Heuristic Based on Variance Analysis of Sensitivity Information,” IEEE Transactions on Neural Networks, Vol.12, No.6, pp. 1386-1399, 2001.
[5] M. A. Arbib, “The handbook of brain theory and neural networks,” 1998.
[6] J. D. Paola and R. A. Schowengerdt, “A Review And Analysis of Backpropagation Neural Networks for Classification of Remotely-Sensed Multispectral Imagery,” International Journal of Remote Sensing, Vol.16, No.16, pp. 3033-3058, 1995.
[7] A. Dean and D. Voss, “Design and Analysis of Experiments,” 1999.
[8] E. Inohira and H. Yokoi, “Multilayer Neural Networks with Intermediate Elements Using a Distance,” Journal of Biomedical Fuzzy Systems Association, Vol.7, No.1, pp. 21-31, 2005 (in Japanese).
[9] G. E. Peterson, D. C. St. Clair, S. R. Aylward, and W. E. Bond, “Using Taguchi’s Method of Experimental Design to Control Errors in Layered Perceptrons,” IEEE Transactions on Neural Networks, Vol.6, No.4, pp. 949-961, 1995.
[10] M. S. Packianather, P. R. Drake, and H. Rowlands, “Optimizing the Parameters of Multilayered Feedforward Neural Networks Through Taguchi Design of Experiments,” Quality and Reliability Engineering International, Vol.16, pp. 461-473, 2000.
[11] W.-T. Chien and C.-S. Tsai, “The Investigation on the Prediction of Tool Wear and the Determination of Optimum Cutting Conditions in Machining 17-4PH Stainless Steel,” Journal of Materials Processing Technology, Vol.140, pp. 340-345, 2003.
[12] Y. S. Kim and B. J. Yum, “Robust Design of Multilayer Feedforward Neural Networks: an Experimental Approach,” Engineering Applications of Artificial Intelligence, Vol.17, pp. 249-263, 2004.
[13] W. Sukthomya and J. Tannock, “The Optimisation of Neural Network Parameters Using Taguchi’s Design of Experiments Approach: an Application in Manufacturing Process Modelling,” Neural Computing & Applications, Vol.14, No.4, pp. 337-344, 2005.
[14] F. H. F. Leung, H. K. Lam, S. H. Ling, and P. K. S. Tam, “Tuning of the Structure and Parameters of a Neural Network Using an Improved Genetic Algorithm,” IEEE Transactions on Neural Networks, Vol.4, No.1, pp. 79-88, 2003.
[15] P. A. Castillo, J. J. Merelo, A. Prieto, V. Rivas, and G. Romero, “GProp: Global Optimization of Multilayer Perceptrons Using GAs,” Neurocomputing, Vol.35, pp. 149-163, 2000.
[16] Y. A. Alsultanny and M. M. Aqel, “Pattern Recognition Using Multilayer Neural-Genetic Algorithm,” Neurocomputing, Vol.51, pp. 237-247, 2003.

This article is published under a Creative Commons Attribution-NoDerivatives 4.0 Internationa License.

[1] [1] D. Tikk, L. T. Koczy, and T. D. Gedeon, “A Survey on Universal Approximation and its Limits in Soft Computing Techniques,” International Journal of Approximate Reasoning, pp. 185-202, 2003.

[2] [2] F. Scarselli and A. C. Tsoi, “Universal Approximation Using Feedforward Neural Networks: A Survey of Some Existing Methods, and Some New Results,” Neural Networks, Vol.11, No.1, pp. 15-37, 1998.

[3] [3] D. Rumelhart, G. Hinton, and R. Williams, Parallel Distributed Processing, MIT Press, 1986.

[4] [4] A. P. Engelbrecht, “A New Pruning Heuristic Based on Variance Analysis of Sensitivity Information,” IEEE Transactions on Neural Networks, Vol.12, No.6, pp. 1386-1399, 2001.

[5] [5] M. A. Arbib, “The handbook of brain theory and neural networks,” 1998.

[6] [6] J. D. Paola and R. A. Schowengerdt, “A Review And Analysis of Backpropagation Neural Networks for Classification of Remotely-Sensed Multispectral Imagery,” International Journal of Remote Sensing, Vol.16, No.16, pp. 3033-3058, 1995.

[7] [7] A. Dean and D. Voss, “Design and Analysis of Experiments,” 1999.

[8] [8] E. Inohira and H. Yokoi, “Multilayer Neural Networks with Intermediate Elements Using a Distance,” Journal of Biomedical Fuzzy Systems Association, Vol.7, No.1, pp. 21-31, 2005 (in Japanese).

[9] [9] G. E. Peterson, D. C. St. Clair, S. R. Aylward, and W. E. Bond, “Using Taguchi’s Method of Experimental Design to Control Errors in Layered Perceptrons,” IEEE Transactions on Neural Networks, Vol.6, No.4, pp. 949-961, 1995.

[10] [10] M. S. Packianather, P. R. Drake, and H. Rowlands, “Optimizing the Parameters of Multilayered Feedforward Neural Networks Through Taguchi Design of Experiments,” Quality and Reliability Engineering International, Vol.16, pp. 461-473, 2000.

[11] [11] W.-T. Chien and C.-S. Tsai, “The Investigation on the Prediction of Tool Wear and the Determination of Optimum Cutting Conditions in Machining 17-4PH Stainless Steel,” Journal of Materials Processing Technology, Vol.140, pp. 340-345, 2003.

[12] [12] Y. S. Kim and B. J. Yum, “Robust Design of Multilayer Feedforward Neural Networks: an Experimental Approach,” Engineering Applications of Artificial Intelligence, Vol.17, pp. 249-263, 2004.

[13] [13] W. Sukthomya and J. Tannock, “The Optimisation of Neural Network Parameters Using Taguchi’s Design of Experiments Approach: an Application in Manufacturing Process Modelling,” Neural Computing & Applications, Vol.14, No.4, pp. 337-344, 2005.

[14] [14] F. H. F. Leung, H. K. Lam, S. H. Ling, and P. K. S. Tam, “Tuning of the Structure and Parameters of a Neural Network Using an Improved Genetic Algorithm,” IEEE Transactions on Neural Networks, Vol.4, No.1, pp. 79-88, 2003.

[15] [15] P. A. Castillo, J. J. Merelo, A. Prieto, V. Rivas, and G. Romero, “GProp: Global Optimization of Multilayer Perceptrons Using GAs,” Neurocomputing, Vol.35, pp. 149-163, 2000.

[16] [16] Y. A. Alsultanny and M. M. Aqel, “Pattern Recognition Using Multilayer Neural-Genetic Algorithm,” Neurocomputing, Vol.51, pp. 237-247, 2003.