Automated Categorization of Drosophila Learning and Memory Behaviors using Video Analysis
For more information about this project, goto:
https://www.cs.drexel.edu/~david/Papers/Md_Reza_MS_Thesis.pdf
This repository contains all of the source codes used for the Fly Tracking project.
Scripts
You will find a few scripts to automate the processing of the data:
- ./process-video-beta
- ./batch-process
Tools
You will find the utilities in fly-tools. You will need to 'make' them which is easy to do. If you have trouble building, please make sure that you have gsl and ImageMagick++, MagickWand, OpenCV and CvBlob libraries and headers installed.
$ cd fly-data
$ make
The programs that will be built are located here:
- ./fly-tools/FilterFlyMask
- ./fly-tools/FlyTracking
- ./fly-tools/mask-generator
- ./fly-tools/standard-deviation
- ./fly-tools/derive-background
- ./fly-tools/filter-mask
Dependencies
Make sure you have these libraries and headers installed:
- OpenCv
- cvBlob
- MagickWand
- ImageMagick++
You will also need MPlayer to extract the PNGs.
Usage
Basically the fly tracking process takes several steps to complete. All of these steps can be automated with the process-video-beta script available in the scripts directory.
- Cropping and Extracting Cell Images
- Deriving Cell Backgrounds
- Generating Cell Masks
- Filtering Cell Masks
- Fly Tracking
Step 1: Cropping and Extracting Cell Images
Since each video has several fly cells at one time, we need to crop each cell into it's own set. To do so, extract one frame of the video and then use gimp to determine the x, y, height and width of each cell (the select tool is very handy for this). You should write each cell into a file in this format:
height:width:x:y
You should save this in a file (make sure to name it properly. For the purpose of this explaination I am going to be working on video "Group1"
Group1First10MinCropFile will look like this:
180:172:119:116
181:173:195:120
179:170:370:114
After you have completed that, you will want to run the extraction step of the process-video-beta script. To get a more detailed look at the arguments accepted by the processing script, run it with the -h flag.
process-video-beta -v First10MinGroup1.avi -l Group1First10MinCropFile -n 1 -t First10MinSet -d data/ -1
The -d flag indicates the folder you want to extract these files to, and will be used as the directory structure for the rest of the steps. The -1 flag tells it to run the first out of five steps.
This will take some time, and require a lot of CPU and disk space.
Step 2: Deriving Cell Backgrounds
The second step of the process is deriving the background image from the video. By doing this, we can subtract the background from the current frame to generate the mask. When subtracted the parts left over are the fly, while things in the background won't show up.
process-video-beta -v First10MinGroup1.avi -l Group1First10MinCropFile -n 1 -t First10MinSet -d data/ -2
This will derive the backgrounds for all of the Cells in the video seperately. After this tool is done running, it will output the Background.png into the mask folder. $data-root/$settype$setnumber/Background.png, for example it might be in data/First10MinSet2/Background.png.
Verify this image. Sometimes when the flies do not move the background might include the fly! If there are any left of flie parts, use GIMP to airbrush over that section with an appropriate color (guess what the background would be "close" too).
Step 3: Generating Cell Masks
The third step of the process is generating the Masks from the background and the cropped images.
process-video-beta -v First10MinGroup1.avi -l Group1First10MinCropFile -n 1 -t First10MinSet -d data/ -3
This step will output the masks in $data-root/$settype$setnumber/Masks/
Step 4: Filtering Cell Masks
The fourth step is the filtering step. It's important because it makes sure the masks do not have stray objects in them (which will mess up the final tracking sequence)
process-video-beta -v First10MinGroup1.avi -l Group1First10MinCropFile -n 1 -t First10MinSet -d data/ -4
The filtered masks are in $data-root/$settype$setnumber/Filtered/final/
Step 5: Fly Tracking
This is the final step of the image processing. It will generate results in the $data-root/$settype$setnumber/Final/. There will be PNGs showing the tracking results (refer to the paper to see what the polygons and lines refer too), a few files that indicate the standard deviation and mean of certain fly behaviors, and a "stat" file, which contains debug and statistical information as well.
This process willl take a long time beceause it iterates through each images in a set (about 18,000 images).
process-video-beta -v First10MinGroup1.avi -l Group1First10MinCropFile -n 1 -t First10MinSet -d data/ -4
After this is done running you should review the pngs to make sure that the fly tracking was performed correctlyAfter this is done running you should review the pngs to make sure that the fly tracking was performed correctly.
Into the Beyond
Now that we've done the hard work of image processing the rest is up to you! The data is all there, now you need to do manually generate the feature vectors, and you need to run our simple k-means clustering.
The K-Means clustering requires Java to run and our Least Squares Solution CCI calculator was written in Matlab and requires it to run.