Fuzzy logic, neural networks--Basic principles--part 2

Neural networks

Neural networks are collections of 'neurons' and 'synapses' that change their values in response from inputs from surrounding neurons and synapses. The neural net acts like a computer because it maps inputs to outputs. The neurons and synapses may be silicon components or software equations that simulate their behavior. A neuron sums all incoming signals from other neurons and then emits its own response in the form of a number. Signals travel across the synapses that have numerical values that weigh the flow of neuronic values.

When new input data 'fires' a network of neurons, the synaptic values can change slightly. A neural net 'learns' when it changes the value of its synapses.

The learning process:

+=+=+=+ a typical neural network and its synaptic connections. Each neuron, as illustrated, receives a number of inputs XI which are assigned weights, by the inter connecting synapse WI. From the weighted total input, the processing element computes a single output value Y. +=+=+=+ what occurs inside each neuron when it’s activated or processed.

Neuron inputs: Various signals (inputs of the neuron XI) are received from other neurons via synapses. Neuron input calculation A weighted sum of these input values is calculated.

==== Input layer Middle layer Output layer

+=+=+=+Typical neural network connections Inputs signals to a neuron Output signals to other neurons Summed inputs Add bias weight Transform

+=+=+=+ Processing steps inside a neuron

===

Neuron internal function transform: The sum of the input calculation is transformed by an internal function, which is normally, but not always, fixed for a neuron at the time the network is constructed.

Neuron output: The transformed result (outputs) are sent individually onto other neurons via the interconnecting synapse.

Neuron actions in 'learning':

Learning implies that the neuron changes its input to output behavior in response to the environment in which it exists. However, the neurons transform function is usually fixed, so the only way the input-to-output transform can be changed is by changing the bias weight as applied to the inputs.

So 'learning' is achieved by changing the weights on the inputs, and the internal model of the network is embodied in the set of all these weights. How are these 'weights' changed? One of the most common forms widely used is called 'back propagation networking', commonly used for chemical engineering.

Neural back propagation networking

These networks always consist of three neuron layers: input, middle and output layer.

The construction is such that a neuron in each layer is connected to every neuron in the next layer. The number of middle layer neurons varies, but has to be selected with care; too many result in unmanageable patterns, and too few will require an excessive number of iterations to take place before an acceptable output is obtained.

Forward output flow (neuron initialization):

The initial pattern of neuron weights is randomized and presented to the input layer, which in turn passes it on to the middle layer. Each neuron computes its output signal as ...

The output is determined by multiplying each input signal by the random weight value on the synoptical interconnection:

This weighted sum is transformed by a function, f (X) called the activated function of the neuron and it determines the activity generated in the neuron as a result of an input signal of a particular size. Neural sigmoidal functions:

For back propagation networks, and for most chemical engineering applications, the function described is a sigmoidal function. This function is:

• Continuous

• S shaped

• Monotonically increasing

• Asymptotically approaches fixed values as the input approaches . 8 ±

+=+=+=+ A sigmoidal function

Generally, the upper limit of a sigmoid is set to +1 and the lower limit to 0 or 1 - . The steepness of the curve and even the exact function used to compute it’s generally less important than the general 'S' shape.

The following sigmoidal curve expressed as a function of Ij, the weighted input to the neuron is widely used.

Where T is a simple threshold and X is the input. This transformed input signal becomes the total activation of the middle layer neurons, which is used for their outputs, and which in turn become the inputs to the output neuron layer, where a similar action takes place in the output neuron, using the sigmoidal function which produces a final output value from the neuron.

Backward error propagation (the delta rule):

The result of the output is compared with the desired output. The difference (or error)

becomes a bias value by which we modify the weights in the neuron connections. It usually takes several iterations to match the target output required.

The delta method is one of the most common methods used in backward propagation.

The delta rule iteratively minimizes the average squared error between the outputs of each neuron and the target value. The error gradient is then determined for the hidden (middle) layers by calculating the weighted error of that layer. Thus:

• The errors are propagated back one layer;

• The same procedure is applied recursively until the input layer is reached;

• This is backward error flow.

The calculated error gradients are then used to update the network weights. A momentum term can be introduced into this procedure to determine the effect of previous weight changes on present weight changes in weight space. This usually helps to improve convergence.

Thus back propagation is a gradient descent algorithm that tries to minimize the average squared error of the network by moving down the gradient of the error curve. In a simple system the error curve is bowl shaped (paraboloid) and the network eventually gets to the bottom of the bowl.

Training a neuron network

There are, in principle, seven standard techniques used to 'train' a network for a zero error value (resting at the bottom of the bowl-shaped error curve).

Re-initialize the weights:

If difficulty is found in trying to find a global minimum, the process can have a new set of random weights applied to its input, and the learning process repeated.

Add step changes to the existing weights:

It’s possible for the network to 'oscillate' around an error value due to the fact that any calculated weight change in the network does not improve (decrease) the error term.

All that is normally needed is for a 'slight push', to be given to the weighted factors. This can be achieved by randomly moving the weight values to new positions, but not too far from the point of origin.

Avoiding over-parameterization:

If there are too many neurons in the middle, or hidden, layer then over-parameterization occurs which in turn gives poor predictions. Reduction of neurons in this layer affects a cure to this problem. There is no rule of thumb here, and the number of neurons needed is best determined experimentally.

Changing the momentum term:

This is the easiest thing to do if the system is a software network, the momentum term a is implemented by adding a part of the last weighted term to the new one, changing this value, again best done experimentally, can assist with a cure.

Noise and repeated data:

Avoid repeated or less noisy data Repeated or noise-free inputs makes the network remember the pattern rather than generalizing their features. If the network never sees the same input values twice, this prevents it from remembering the pattern. Introducing noise can assist in preventing this from occurring.

Changing the learning tolerance:

The training of a network ceases once the error value for all cases is equal or less than the learning tolerance. If this tolerance is too small, the learning process never ceases.

Experiment with the tolerance level until a satisfactory point is reached where the weights cease changing their value in any significant way.

Increasing the middle (hidden) layer value:

This is the inverse of the problem described, 'avoiding over-parameterization' and is used if-all-else-fails. In other words we have too few neurons in the middle layer, where we had too many. In general an increase >10% shows improvements.

Conclusions and then the next step

Fuzzy logic has been used since the early 1980s and has been very successful in many applications, such as Hitachi's use of it in controlling subway trains, and it proving so accurate and reliable that its performance exceeds what a trained (no pun intended) driver can do. When its principles are used with neuron, or self-learning networks, we have a very formidable set of tools being made available to us. When this is applied to process control systems, it gives us a foreseeable future of control systems that are error free, and can cope with all the variances, including the operator, to such an extent that we could finally have a 'perfect' system. Meanwhile, we have two more issues to look at that may well be considered as the intermediate step from the past and current technology to the ultimate, self-learning and totally accurate control system. These two issues, discussed and described, conclude this guide on practical process control. They are statistical process control (SPC) and self-tuning controllers. SPC is used to see where the process is in error, a problem solved by neuron networks, and self-tuning controllers, a problem that can be solved by fuzzy logic.

NEXT: Self-tuning intelligent control + statistical process control
PREV: Fuzzy logic, neural networks--Basic principles--part 1

All related articles   Top of Page   Home