19 Dec 2015

1.1: Logistic Regression: Gradients

So today I was waiting for an experiment to finish and since it ususally take a couple of hours to finish one of these heafty simulations so I decided to revist some of the concepts from a couple of years ago. While browsing through my textbook I came across an excercise that I could easily do with my eyes closed during my taught master days but out of context today, I felt that it has become somewhat difficult to do these simple derivations.

As the title suggests, the exercise was to derive the expression of the gradient of a two class binary logistic regression model. So without any more blabbering, let

be the negative log-likelihood of the logistic regression, where is the weight, and .

For calculating the gradient of the NLL we will need the derivative of the sigmoid function.

Now the gradient of the NLL simply becomes,

And we know that


In case you were interested in the notebook version of this Click Here