# Object Recognition Reference

- Eigen Objects (PCA) Functions
  - CalcCovarMatrixEx
  - CalcEigenObjects
  - CalcDecompCoeff
  - EigenDecomposite
  - EigenProjection
- Embedded Hidden Markov Models Functions
  - HMM
  - ImgObsInfo
  - Create2DHMM
  - Release2DHMM
  - CreateObsInfo
  - ReleaseObsInfo
  - ImgToObs_DCT
  - UniformImgSegm
  - InitMixSegm
  - EstimateHMMStateParams
  - EstimateTransProb
  - EstimateObsProb
  - EViterbi
  - MixSegmL2

## Eigen Objects (PCA) Functions

The functions described in this section do PCA analysis and compression for a set of 8-bit images that may not fit into memory all together. If your data fits into memory and the vectors are not 8-bit (or you want a simpler interface), use cvCalcCovarMatrix, cvSVD and cvGEMM to do PCA

### CalcCovarMatrixEx

Calculates covariance matrix for group of input objects

    void cvCalcCovarMatrixEx( int nObjects, void* input, int ioFlags,
                              int ioBufSize, uchar* buffer, void* userData,
                              IplImage* avg, float* covarMatrix );

nObjects
  Number of source objects.
input
  Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
ioFlags
  Input/output flags.
ioBufSize
  Input/output buffer size.
buffer
  Pointer to the input/output buffer.
userData
  Pointer to the structure that contains all necessary data for the
callback
  functions.
avg
  Averaged object.
covarMatrix
  Covariance matrix. An output parameter; must be allocated before the call.

The function cvCalcCovarMatrixEx calculates a covariance matrix of the input objects group using previously calculated averaged object. Depending on ioFlags parameter it may be used either in direct access or callback mode. If ioFlags is not CV_EIGOBJ_NO_CALLBACK, buffer must be allocated before calling the function.

### CalcEigenObjects

Calculates orthonormal eigen basis and averaged object for group of input objects

    void cvCalcEigenObjects( int nObjects, void* input, void* output, int ioFlags,
                             int ioBufSize, void* userData, CvTermCriteria* calcLimit,
                             IplImage* avg, float* eigVals );

nObjects
  Number of source objects.
input
  Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
output
  Pointer either to the array of eigen objects or to the write callback function according to the value of the parameter ioFlags .
ioFlags
  Input/output flags.
ioBufSize
  Input/output buffer size in bytes. The size is zero, if unknown.
userData
  Pointer to the structure that contains all necessary data for the callback functions.
calcLimit
  Criteria that determine when to stop calculation of eigen objects.
avg
  Averaged object.
eigVals
  Pointer to the eigenvalues array in the descending order; may be NULL .

The function cvCalcEigenObjects calculates orthonormal eigen basis and the averaged object for a group of the input objects. Depending on ioFlags parameter it may be used either in direct access or callback mode. Depending on the parameter calcLimit, calculations are finished either after first calcLimit.maxIters dominating eigen objects are retrieved or if the ratio of the current eigenvalue to the largest eigenvalue comes down to calcLimit.epsilon threshold. The value calcLimit -> type must be CV_TERMCRIT_NUMB, CV_TERMCRIT_EPS, or CV_TERMCRIT_NUMB | CV_TERMCRIT_EPS . The function returns the real values calcLimit -> maxIter and calcLimit -> epsilon .

The function also calculates the averaged object, which must be created previously. Calculated eigen objects are arranged according to the corresponding eigenvalues in the descending order.

The parameter eigVals may be equal to NULL, if eigenvalues are not needed.

The function cvCalcEigenObjects uses the function cvCalcCovarMatrixEx.

### CalcDecompCoeff

Calculates decomposition coefficient of input object

    double cvCalcDecompCoeff( IplImage* obj, IplImage* eigObj, IplImage* avg );

obj
  Input object.
eigObj
  Eigen object.
avg
  Averaged object.

The function cvCalcDecompCoeff calculates one decomposition coefficient of the input object using the previously calculated eigen object and the averaged object.

### EigenDecomposite

Calculates all decomposition coefficients for input object

    void cvEigenDecomposite( IplImage* obj, int nEigObjs, void* eigInput,
                             int ioFlags, void* userData, IplImage* avg, float* coeffs );

obj
  Input object.
nEigObjs
  Number of eigen objects.
eigInput
  Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
ioFlags
  Input/output flags.
userData
  Pointer to the structure that contains all necessary data for the callback functions.
avg
  Averaged object.
coeffs
  Calculated coefficients; an output parameter.

The function cvEigenDecomposite calculates all decomposition coefficients for the input object using the previously calculated eigen objects basis and the averaged object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

### EigenProjection

Calculates object projection to the eigen sub-space

    void cvEigenProjection( int nEigObjs, void* eigInput, int ioFlags,
                            void* userData, float* coeffs,
                            IplImage* avg, IplImage* proj );

nEigObjs
  Number of eigen objects.
eigInput
  Pointer either to the array of IplImage input objects or to the read callback function according to the value of the parameter ioFlags.
ioFlags
  Input/output flags.
userData
  Pointer to the structure that contains all necessary data for the callback functions.
coeffs
  Previously calculated decomposition coefficients.
avg
  Averaged object.
proj
  Decomposed object projection to the eigen sub-space.

The function cvEigenProjection calculates an object projection to the eigen sub-space or, in other words, restores an object using previously calculated eigen objects basis, averaged object, and decomposition coefficients of the restored object. Depending on ioFlags parameter it may be used either in direct access or callback mode.

The functions of the eigen objects group have been developed to be used for any number of objects, even if their total size exceeds free RAM size. So the functions may be used in two main modes.

Direct access mode is the best choice if the size of free RAM is sufficient for all input and eigen objects allocation. This mode is set if the parameter ioFlags is equal to CV_EIGOBJ_NO_CALLBACK . In this case input and output parameters are pointers to arrays of input/output objects of IplImage* type. The parameters ioBufSize and userData are not used.

## Embedded Hidden Markov Models Functions

In order to support embedded models the user must define structures to represent 1D HMM and 2D embedded HMM model.

### CvHMM

Embedded HMM Structure

        typedef struct _CvEHMM
        {
            int level;
            int num_states;
            float* transP;
            float** obsProb;
            union
            {
                CvEHMMState* state;
                struct _CvEHMM* ehmm;
            } u;
        } CvEHMM;

level
  Level of embedded HMM. If level ==0, HMM is most external. In 2D HMM there are two types of HMM: 1 external and several embedded. External HMM has level ==1, embedded HMMs have level ==0 .
num_states
  Number of states in 1D HMM.
transP
  State-to-state transition probability, square matrix (num_state×num_state ).
obsProb
  Observation probability matrix.
state
  Array of HMM states. For the last-level HMM, that is, an HMM without embedded HMMs, HMM states are real.
ehmm
  Array of embedded HMMs. If HMM is not last-level, then HMM states are not real and they are HMMs.

For representation of observations the following structure is defined:

### CvImgObsInfo

Image Observation Structure

        typedef struct CvImgObsInfo
        {
            int obs_x;
            int obs_y;
            int obs_size;
            float** obs;
            int* state;
            int* mix;
        } CvImgObsInfo;

obs_x
  Number of observations in the horizontal direction.
obs_y
  Number of observations in the vertical direction.
obs_size
  Length of every observation vector.
obs
  Pointer to observation vectors stored consequently. Number of vectors is obs_x*obs_y .
state
  Array of indices of states, assigned to every observation vector.
mix
  Index of mixture component, corresponding to the observation vector within an assigned state.

### Create2DHMM

Creates 2D embedded HMM

    CvEHMM* cvCreate2DHMM( int* stateNumber, int* numMix, int obsSize );

stateNumber
  Array, the first element of the which specifies the number of superstates in the HMM. All subsequent elements specify the number of states in every embedded HMM, corresponding to each superstate. So, the length of the array is stateNumber [0]+1 .
numMix
  Array with numbers of Gaussian mixture components per each internal state. The number of elements in the array is equal to number of internal states in the HMM, that is, superstates are not counted here.
obsSize
  Size of observation vectors to be used with created HMM.

The function cvCreate2DHMM returns the created structure of the type CvEHMM with specified parameters.

### Release2DHMM

Releases 2D embedded HMM

    void cvRelease2DHMM(CvEHMM** hmm );

hmm
  Address of pointer to HMM to be released.

The function cvRelease2DHMM frees all the memory used by HMM and clears the pointer to HMM.

### CreateObsInfo

Creates structure to store image observation vectors

    CvImgObsInfo* cvCreateObsInfo( CvSize numObs, int obsSize );

numObs
  Numbers of observations in the horizontal and vertical directions. For the given image and scheme of extracting observations the parameter can be computed via the macro CV_COUNT_OBS( roi, dctSize, delta, numObs ), where roi, dctSize, delta, numObs are the pointers to structures of the type CvSize . The pointer roi means size of roi of image observed, numObs is the output parameter of the macro.
obsSize
  Size of observation vectors to be stored in the structure.

The function cvCreateObsInfo creates new structures to store image observation vectors. For definitions of the parameters roi, dctSize, and delta see the specification of the function cvImgToObs_DCT.

### ReleaseObsInfo

Releases observation vectors structure

    void cvReleaseObsInfo( CvImgObsInfo** obsInfo );

obsInfo
  Address of the pointer to the structure CvImgObsInfo .

The function cvReleaseObsInfo frees all memory used by observations and clears pointer to the structure CvImgObsInfo .

### ImgToObs_DCT

Extracts observation vectors from image

    void cvImgToObs_DCT( IplImage* image, float* obs, CvSize dctSize,
                         CvSize obsSize, CvSize delta );

image
  Input image.
obs
  Pointer to consequently stored observation vectors.
dctSize
  Size of image blocks for which DCT (Discrete Cosine Transform) coefficients are to be computed.
obsSize
  Number of the lowest DCT coefficients in the horizontal and vertical directions to be put into the observation vector.
delta
  Shift in pixels between two consecutive image blocks in the horizontal and vertical directions.

The function cvImgToObs_DCT extracts observation vectors, that is, DCT coefficients, from the image. The user must pass obsInfo.obs as the parameter obs to use this function with other HMM functions and use the structure obsInfo of the CvImgObsInfo type.

Calculating Observations for HMM

        CvImgObsInfo* obs_info;

            ...

            cvImgToObs_DCT( image,obs_info->obs, //!!!

            dctSize, obsSize, delta );

### UniformImgSegm

Performs uniform segmentation of image observations by HMM states

    void cvUniformImgSegm( CvImgObsInfo* obsInfo, CvEHMM* hmm );

obsInfo
  Observations structure.
hmm
  HMM structure.

The function cvUniformImgSegm segments image observations by HMM states uniformly (see Initial Segmentation for 2D Embedded HMM for 2D embedded HMM with 5 superstates and 3, 6, 6, 6, 3 internal states of every corresponding superstate).

Initial Segmentation for 2D Embedded HMM

### InitMixSegm

Segments all observations within every internal state of HMM by state mixture components

    void cvInitMixSegm( CvImgObsInfo** obsInfoArray, int numImg, CvEHMM* hmm );

obsInfoArray
  Array of pointers to the observation structures.
numImg
  Length of above array.
hmm
  HMM.

The function cvInitMixSegm takes a group of observations from several training images already segmented by states and splits a set of observation vectors within every internal HMM state into as many clusters as the number of mixture components in the state.

### EstimateHMMStateParams

Estimates all parameters of every HMM state

    void cvEstimateHMMStateParams( CvImgObsInfo** obsInfoArray, int numImg, CvEHMM* hmm );

obsInfoArray
  Array of pointers to the observation structures.
numImg
  Length of the array.
hmm
  HMM.

The function cvEstimateHMMStateParams computes all inner parameters of every HMM state, including Gaussian means, variances, etc.

### EstimateTransProb

Computes transition probability matrices for embedded HMM

    void cvEstimateTransProb( CvImgObsInfo** obsInfoArray, int numImg, CvEHMM* hmm );

obsInfoArray
  Array of pointers to the observation structures.
numImg
  Length of the above array.
hmm
  HMM.

The function cvEstimateTransProb uses current segmentation of image observations to compute transition probability matrices for all embedded and external HMMs.

### EstimateObsProb

Computes probability of every observation of several images

    void cvEstimateObsProb( CvImgObsInfo* obsInfo, CvEHMM* hmm );

obsInfo
  Observation structure.
hmm
  HMM structure.

The function cvEstimateObsProb computes Gaussian probabilities of each observation to occur in each of the internal HMM states.

### EViterbi

Executes Viterbi algorithm for embedded HMM

    float cvEViterbi( CvImgObsInfo* obsInfo, CvEHMM* hmm );

obsInfo
  Observation structure.
hmm
  HMM structure.

The function cvEViterbi executes Viterbi algorithm for embedded HMM. Viterbi algorithm evaluates the likelihood of the best match between the given image observations and the given HMM and performs segmentation of image observations by HMM states. The segmentation is done on the basis of the match found.

### MixSegmL2

Segments observations from all training images by mixture components of newly assigned states

    void cvMixSegmL2( CvImgObsInfo** obsInfoArray, int numImg, CvEHMM* hmm );

obsInfoArray
  Array of pointers to the observation structures.
numImg
  Length of the array.
hmm
  HMM.

The function cvMixSegmL2 segments observations from all training images by mixture components of newly Viterbi algorithm-assigned states. The function uses Euclidean distance to group vectors around the existing mixtures centers.