Recent research in video surveillance has shown an increasing focus in creating reliable systems for observing humans’ appearance, movements and activities, thus providing analytical information for advanced human behavior analysis. In order for the system to function, it requires robust method for detecting and tracking human from a given input of video streams. In this book, a human detection technique suitable for video surveillance is presented. The techniques proposed include adaptive frame differencing for background subtraction, contrast adjustment for shadow removal, and shape based approach for human detection. The tracking technique on the other hand uses correspondence approach. Event Based Video Retrieval (EBVR) system is proposed for efficient surveillance data management and automated human recognition. Proposed human detection and tracking are integrated with EBVR into a complete automated surveillance system which produces good result and real-time performance especially in non-crowded scene and handles automated human recognition with unique ID assignment accurately. This book could serve as an introduction for new researchers in this field.