This piece of code utilizes OpenCV libraries to optimize a series of time lapse photos. In applications where a time lapse intents to capture movements in a scene, a movement triggered camera is used. However, such camera is expensive. A more common setupÂ is to capture photos with a fixed time interval, this is supported by cameras like GoPro and many DSRL natively, or can be otherwise achieved with a cheapÂ intervalometer with other cameras.
The problem with fixed time interval capturing is that, it will generate a large percentage of no-movement photoÂ series. Culling these photos by human comparison is time consuming and unfeasible with large data set. For example a 5 second time lapse in 24 hours will generateÂ 17280 photos. Not to mention the inherit computer storage that consumes.
The solution to this problem is to use computer vision techniques to automatically compare the photos within the series. There are two sub-problem, first we need to decide on a method to visually compare two images which can conclude a numerical “Difference Value”, next we need to decide on a logic that will use to cull a set ofÂ images based on such 1 to 1 comparison.
The amount of differenceÂ between two images is a very subjective question. Human perception works in a very complicated manner, we recognize not only brightness and colour difference but also gradient, shapes, silhouettes.Â In my implementation, I have choose to compare the RGB values of the corresponding pixel in two images, then average all the values in the image to get one scalar value as percentage to represent the image. Three channels are compared separately, thus the value ranges from 0.0 to 3.0
Since I’m writing in VB.net with OpenCv via EmguÂ (Engu is a .net wrapper for OpenCV). My code contains such lines:
Dim previousImage As New Image(Of Bgr, Byte)(previousImageLocation) Dim thisImage As New Image(Of Bgr, Byte)(thisImageLocation) Dim imgAbs As Image(Of Bgr, Byte) imgAbs = thisImage.AbsDiff(previousImage) imgAbs = imgAbs.ThresholdBinary(New Bgr(30, 30, 30), New Bgr(255, 255, 255)) Dim result As MCvScalar = Emgu.CV.CvInvoke.cvAvg(imgAbs, Nothing) Dim difference As Float = Math.Round(result.v0 + result.v1 + result.v2, 2) previousImage.Dispose() thisImage.Dispose() imgAbs.Dispose()
The culling algorithm
- Start with first image from the “original list”, moveÂ that image toÂ the “culled list”,
- Compare the last image (the one you just placed in) in the “culled list” it with the next image in the “original list”.
- If the difference value is below a threshold,Â moveÂ that image from the “original list” toÂ the “culled list”
- If the difference value is aboveÂ or equal toÂ thatÂ threshold, Â discardÂ that image from the “original list”
- Jump to (instruction 2) unless theÂ “original list” is empty.
- “culled list” is the final list of Â images that contains motion.
Various points to note
- Camera is assumed to be non-moving. Comparing photos whenÂ the camera is also movingÂ requires a stabilizing process that matches the moved images. This can be done via featuring detection and matching and transforming one image onto the space of another. I have never written one, but more information can be found about Feature matching here.
- Exposure settings is assumed to be non changing.
- Gradual changes in the images (think about sunset) will cause the code to cull out a sparsed-out list of the original images. This is a caused by the comparator discarding the image below the threshold. This is a desirable effect in certain applications, the effect can be tuned by adjusting the threshold. Once the gradual changes accumulates above the threshold value (assuming the change is gradual and with trend)Â the comparator will pick that image into the “culled list”.
- Colour difference is a topic in colour science that deals with colour distance in a certain colour space. Some colour space are more related to perception of human eyes.