Automatic facial expression analysis is an area of great research especially in the field of computer vision and robotics. In the work done so far, the facial expression analysis is done either by recognizing the facial expression directly or indirectly by first recognizing AUs and then applying this information for facial expression analysis. The various challenges in facial expression analysis are associated with face detection and tracking, facial feature extraction and the facial feature classification. The presented review gives a brief description of the time line view of the research work carried for AU detection/estimation in static and dynamic image sequences and possible solutions proposed by researchers in this field since 2002. In short, the paper will provide an impetus for various challenges and applications of AU detection, and new research topics, which will increase the productivity in this exciting and challenging field.