Abstract: Forgetting factor stochastic gradient algorithm (FG algorithm for short) is presented and its convergence is studied by using stochastic process theory. The analyses indicate that the FG algorithm can track the time-varying parameters and has the same properties as the forgetting factor least squares algorithms but takes less computational effort, and that the stationary data can improve the precision of the parameter estimates. The way of choosing the forgetting factor is stated so that the minimum upper bound of the parameter estimation error is obtained. For time invariant deterministic systems, the FG algorithm is exponentially convergent~ for time-varying or time invariant stochastic systems, the estimation error given by the FG algorithm consistently has the upper bound.