March 2020

Preliminaries Committee Boosting The committee has an equal weight for every prediction from all models, and it gives little improvement than a single model. Then boosting was built for this problem. Boosting is a technique of combining multiple ‘base’ classifiers to produce a form of the committee that: performances better than any of the base classifiers and each base classifier has a different weight factor Adaboost Adaboost is short for adaptive boosting.

Ver mais

Preliminaries Basic machine learning concepts Probability Theory concepts expectation correlated random variable Analysis of Committees1 The committee is a native inspiration for how to combine several models(or we can say how to combine the outputs of several models). For example, we can combine all the models by: [ y_{COM}(X)=frac{1}{M}sum_{m=1}^My_m(X)tag{1} ] Then we want to find out whether this average prediction of models is better than every one of them.

Ver mais

Preliminaries Bayesian Theorem Bayesian Model Averaging(BMA)1 Bayesian model averaging(BMA) is another wildly used method that is very like a combining model. However, the difference between BMA and combining models is also significant. A Bayesian model averaging is a Bayesian formula in which the random variable are models(hypothesizes) (h=1,2,cdots,H) with prior probability (Pr(h)), then the marginal distribution over data (X) is: [ Pr(X)=sum_{h=1}^{H}Pr(X|h)Pr(h) ] And the MBA is used to select a model(hypothesis) that can model the data best […]

Ver mais

Preliminaries ‘Mixtures of Gaussians’ Basic machine learning concepts Combining Models1 The mixture of Gaussians had been discussed in the post ‘Mixtures of Gaussians’. It was used to introduce the ‘EM algorithm’ but it gave us the inspiration of improving model performance. All models we have studied, besides neural networks, are all single-distribution models. That is just like that, to solve a problem we invite an expert who is very good at this kind of problem, then we just […]

Ver mais

When working on Spark jobs (in Scala), we often sequentially write the code in a single class, giving more attention to the transformations we do and forgetting how our code is structured or even if it’s tested. Today I’ll be talking about how I personally like to structure and design my Spark jobs, such as they are highly maintainable and testable.

Ver mais

Preliminaries Gaussian distribution log-likelihood Calculus partial derivative Lagrange multiplier EM Algorithm for Gaussian Mixture1 Analysis Maximizing likelihood could not be used in the Gaussian mixture model directly, because of its severe defects which we have come across at ‘Maximum Likelihood of Gaussian Mixtures’. With the inspiration of K-means, a two-step algorithm was developed. The objective function is the log-likelihood function: [ begin{aligned} ln Pr(mathbf{x}|mathbf{pi},mathbf{mu},Sigma)&=ln (Pi_{n=1}^Nsum_{j=1}^{K}pi_kmathcal{N}(mathbf{x}|mathbf{mu}_k,Sigma_k))\ &=sum_{n=1}^{N}ln sum_{j=1}^{K}pi_jmathcal{N}(mathbf{x}_n|mathbf{mu}_j,Sigma_j)\ end{aligned}tag{1} ]

Ver mais

Preliminaries Probability Theory multiplication principle joint distribution the Bayesian theory Gaussian distribution log-likelihood function ‘Maximum Likelihood Estimation’ Maximum Likelihood1 Gaussian mixtures had been discussed in ‘Mixtures of Gaussians’. And once we have a training data set and a certain hypothesis, what we should do next is estimate the parameters of the model. Both kinds of parameters from a mixture of Gaussians (Pr(mathbf{x})= sum_{k=1}^{K}pi_kmathcal{N}(mathbf{x}|mathbf{mu}_k,Sigma_k)): – the parameters of Gaussian: (mathbf{mu}_k,Sigma_k) – and latent variables: (mathbf{z})

Ver mais