Commit d247534c authored by Baja, Hilmy's avatar Baja, Hilmy
Browse files

Added PointTrack to 'Algorithms/PointTrack' along with simple instructions to...

Added PointTrack to 'Algorithms/PointTrack' along with simple instructions to train and test datasets on the algorithm. README.md file in branch master was slightly modified to point to the added changes.
parent b1dd01ec
......@@ -2,6 +2,8 @@
This is the code to our research 'Stepping towards real-time detection and tracking of apples using deep learning'. We provide scripts and tutorials for applying multi-object tracking and segmentation (MOTS) on an apple orchard by using the [TrackR-CNN][TrackR-CNN_MOTS] algorithm.
To use the **PointTrack** algorithm instead of **TrackR-CNN**, go to the [PointTrack](Algorithms/PointTrack/) folder for instructions.
## Video of results
![](visualizations/test_8_wearable_sensor.gif)
......
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# celery beat schedule file
celerybeat-schedule
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
# Realtime Apples Detection and Tracking using UAVs
This codebase implements [PointTrack](https://github.com/detectRecog/PointTrack), a highly effective framework for multi-object tracking and segmentation (MOTS) described in:
```
@inproceedings{xu2020Segment,
title={Segment as Points for Efficient Online Multi-Object Tracking and Segmentation},
author={Xu, Zhenbo and Zhang, Wei and Tan, Xiao and Yang, Wei and Huang, Huan and Wen, Shilei and Ding, Errui and Huang, Liusheng},
booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
year={2020}
}
```
**PointTrack presents a new learning strategy for pixel-wise feature learning on the 2D image plane, which has proven to be effective for instance association.**
The network architecture adopts [SpatialEmbedding](https://github.com/davyneven/SpatialEmbeddings) as the segmentation sub-network.
## Getting started
This codebase showcases the proposed framework named PointTrack for MOTS using the a MOTS annotated dataset.
### Prerequisites
Dependencies:
- Pytorch 1.3.1 (and others), please set up an virtual env and run:
```
$ pip install -r requirements.txt
```
- Python 3.6 (or higher)
- [KITTI Images](http://www.cvlibs.net/download.php?file=data_tracking_image_2.zip) + [Annotations](https://www.vision.rwth-aachen.de/media/resource_files/instances.zip)
Note that the scripts for evaluation is included in this repo. After images and instances (annotations) are downloaded, put them under **kittiRoot** and change the path in **repoRoot**/config.py accordingly.
The structure under **kittiRoot** should looks like:
```
kittiRoot
│ images -> training/image_02/
│ instances
│ │ 0000
│ │ 0001
│ │ ...
│ training
│ │ image_02
│ │ │ 0000
│ │ │ 0001
│ │ │ ...
│ testing
│ │ image_02
│ │ │ 0000
│ │ │ 0001
│ │ │ ...
```
##Config.py file
Change the file directories accordingly.
`kittiRoot` is the directory where the dataset is stored.
## Pretrained weights
Fine-tuned models on KITTI MOTS (for cars):
- SpatialEmbedding for cars.
- PointTrack for cars.
You can download them via [Baidu Disk](https://pan.baidu.com/s/1Mk9JWNcM1W08EAjhyq0yLA) or [Google Drive](https://drive.google.com/open?id=14Hn4ZztfjGUYEjVd-9FRNB5a-CtBkPXc).
- testset segs can be found in the folder 'testset_segs'.
For Apples, models will be uploaded soon.
The testset segs are found in the folder 'apple_testset_segs'
##Train a dataset from scratch
To train a dataset from scratch, follow instructions from the master branch of this repository on how to label a custom dataset.
Next, order the dataset folders in accordance to the `kittiRoot` structure above.
Next, create a new environment and install all the dependencies.
## Training of SpatialEmbedding
An older way of training SpatialEmbedding with KITTI MOTS needs KITTI object detection left color images as well as the KINS annotations.
However, that is not necessary anymore. It can be done by copying MOTS ground truth annotations to a folder named KINS inside the training and testing folder of kittiRoot.
A folder structure of kittiRoot needs to be structured in the following:
```
kittiRoot
│ images -> training/image_02/
│ instances
│ │ 0000
│ │ 0001
│ │ ...
│ training
│ │ image_02
│ │ │ 0000
│ │ │ 0001
│ │ │ ...
│ │ KINS
│ │ │ 0000
│ │ │ 0001
│ │ │ ...
│ testing
│ │ image_02
│ │ │ 0000
│ │ │ 0001
│ │ │ ...
│ │ KINS
│ │ │ 0000
│ │ │ 0001
│ │ │ ...
```
Each of the KINS subfolders should be filled with the instance PNGs of your dataset.
For the training of SpatialEmbedding, we follow the original training setting of [SpatialEmbedding](https://github.com/davyneven/SpatialEmbeddings).
1.This step selects valid frames that contains objects of the class; since there could be frames that contain no masks of the object.
```
$ python -u datasets/MOTSImageSelect.py
```
2.Before generating the crops, make sure that the crops sizes are correct in the file. Then do the following:
```
$ python -u utils/generate_crops.py
```
After this step, crops are saved under **kittiRoot**/crop_KINS.
3.Afterwards start training on crops:
```
$ python -u train_SE.py car_finetune_SE_crop
```
You can change desired parameters inside the config file.
4.Afterwards start finetuning on KITTI MOTS with BN fixed. SpatialEmbeddings trained the finetuning with a crop size 2x larger than the first training.
Create a file on the root with the name **resume_SE**, then copy the **checkpoint.pth** file into it.
Then run the following command:
```
$ python -u train_SE.py car_finetune_SE_mots
```
The finetuning will be saved to a different folder.
## Training of PointTrack
The training procedure of instance association is as follows.
1.To generate the segmentation result on the validation set as the instruction of the first step in Testing.
Make sure the file **datasets/KittiMOTSDataset.py** and seqmaps in **datasets/mots_tools/mots_eval/** has the correct dataset configuration.
Then, copy the checkpoint.pth file from the finetuning folder to the **tuned** folder in `rootDir`.
Finally run the segmentation result (forwarding):
```
$ python -u test_mots_se.py car_test_se_to_save
```
2.To generate the instance DB from videos:
```
$ python -u datasets/MOTSInstanceMaskPool.py
```
3.Change the line in **train_tracker_with_val.py** which loads weights to the default path as follows:
```
checkpoint_path='./tuned/checkpoint.pth'
```
4.Afterwards start training:
```
$ python -u train_tracker_with_val.py car_finetune_tracking
```
The best tracker on the validation set will be saved under the folder specified in **repoRoot**/config_mots/car_finetune_tracking.py.
## Testing
You can download pretrained models from the above links. Save these weight file under **repoRoot**/pointTrack_weights.
1.To generate the instance segmentation results:
```
$ python -u test_mots_se.py car_test_se_to_save
```
The segmentation result will be saved according to the config file **repoRoot**/config_mots/car_test_se_to_save.py.
2.To test PointTrack on the instance segmentation results:
```
$ python -u test_tracking.py car_test_tracking_val
```
## Cite us
We borrow some code from [SpatialEmbedding](https://github.com/davyneven/SpatialEmbeddings).
```
@inproceedings{xu2020Segment,
title={Segment as Points for Efficient Online Multi-Object Tracking and Segmentation},
author={Xu, Zhenbo and Zhang, Wei and Tan, Xiao and Yang, Wei and Huang, Huan and Wen, Shilei and Ding, Errui and Huang, Liusheng},
booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
year={2020}
}
@inproceedings{xupointtrack++,
title={PointTrack++ for Effective Online Multi-Object Tracking and Segmentation},
author={Xu, Zhenbo and Zhang, Wei and Tan, Xiao and Yang, Wei and Su, Xiangbo and Yuan, Yuchen and Zhang, Hongwu and Wen, Shilei and Ding, Errui and Huang, Liusheng},
booktitle={CVPR Workshops},
year={2020}
}
```
## Contact
If you find problems in the code, please open an issue, or contact Hilmy (hilmy.baja@wur.nl)
## License
This software is released under a creative commons license which allows for personal and research use only. For a commercial license please contact the authors. You can view a license summary [here](http://creativecommons.org/licenses/by-nc/4.0/).
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
"""
Author: Zhenbo Xu
Licensed under the CC BY-NC 4.0 license (https://creativecommons.org/licenses/by-nc/4.0/)
"""
import os
project_root = os.path.dirname(os.path.realpath(__file__))
category_embedding = [[0.9479751586914062, 0.4561353325843811, 0.16707628965377808], [0.1,-0.1,0.1], [0.5455077290534973, -0.6193588972091675, -2.629554510116577], [-0.1,0.1,-0.1]]
systemRoot = "/home/saidlab/"
kittiRoot = os.path.join(systemRoot, "Thesis-Hilmy/PT/data/APPLE_MOTS/train/")
rootDir = os.path.join(systemRoot, 'Thesis-Hilmy/PT')
pythonPath = os.path.join(systemRoot, "anaconda3/envs/pt_env/bin/python")
import config_mots.car_test_se_to_save
import config_mots.car_test_tracking_val
import config_mots.car_finetune_tracking
import config_mots.car_finetune_SE_crop
import config_mots.car_finetune_SE_mots
\ No newline at end of file
"""
Author: Zhenbo Xu
Licensed under the CC BY-NC 4.0 license (https://creativecommons.org/licenses/by-nc/4.0/)
"""
import copy
import os
from PIL import Image
import torch
from utils import transforms as my_transforms
from config import *
n_sigma=2
args = dict(
cuda=True,
display=False,
display_it=5,
save=True,
save_dir='./mots_finetune_car2',
#resume_path='./tuned/best_seed_model.pthCar',
# resume_path='./mots_finetune_car2/checkpoint.pth',
train_dataset = {
'name': 'mots_cars',
'kwargs': {
'root_dir': kittiRoot,
'type': 'crop',
'size': 7000,
'transform': my_transforms.get_transform([
{
'name': 'AdjustBrightness',
'opts': {}
},
{
'name': 'ToTensor',
'opts': {
'keys': ('image', 'instance','label'),
'type': (torch.FloatTensor, torch.LongTensor, torch.ByteTensor),
}
},
{
'name': 'Flip',
'opts': {
'keys': ('image', 'instance','label'),
}
},
]),
},
#'batch_size': 4,
#'workers': 1,
'batch_size': 64,
'workers': 32
},
val_dataset = {
'name': 'mots_cars',
'kwargs': {
'root_dir': kittiRoot,
'type': 'val',
# 'size': 500,
'transform': my_transforms.get_transform([
{
'name': 'RandomCrop',
'opts': {
'keys': ('image', 'instance', 'label'),
'size': (80, 80),
}
},
{
'name': 'ToTensor',
'opts': {
'keys': ('image', 'instance', 'label'),
'type': (torch.FloatTensor, torch.LongTensor, torch.ByteTensor),
}
},
]),
},
'batch_size': 16,
'workers': 32,
#'batch_size': 2,
#'workers': 1
},
model={
'name': 'branched_erfnet',
'kwargs': {
'num_classes': [2 + n_sigma, 1],
'input_channel': 3
}
},
lr=5e-4,
milestones=[100],
n_epochs=400,
start_epoch=1,
max_disparity=192.0,
# loss options
loss_opts={
'to_center': True,
'n_sigma': n_sigma,
'foreground_weight': 10,
},
loss_w={
'w_inst': 1,
'w_var': 10,
'w_seed': 1,
},
loss_type='MOTSSeg2Loss'
)
def get_args():
return copy.deepcopy(args)
"""
Author: Zhenbo Xu
Licensed under the CC BY-NC 4.0 license (https://creativecommons.org/licenses/by-nc/4.0/)
"""
import copy
import os
from PIL import Image
import torch
from utils import transforms as my_transforms
from config import *
n_sigma=2
args = dict(
cuda=True,
display=False,
display_it=5,
save=True,
fix_bn=True,
save_dir='./mots_finetune_car2_with_BN',
resume_path='./resume_SE/checkpoint.pth',
train_dataset = {
'name': 'mots_cars',
'kwargs': {
'root_dir': kittiRoot,
'type': 'train',
'kins': False,
'size': 1000,
'transform': my_transforms.get_transform([
{
'name': 'RandomCrop',
'opts': {
'keys': ('image', 'instance','label'),
'size': (160, 160)
}
},
{
'name': 'ToTensor',
'opts': {
'keys': ('image', 'instance','label'),
'type': (torch.FloatTensor, torch.LongTensor, torch.ByteTensor),
}
},
]),
},
#'batch_size': 2,
#'workers': 1,
'batch_size': 2,
'workers': 32
},
val_dataset = {
'name': 'mots_cars',
'kwargs': {
'root_dir': kittiRoot,
'type': 'val',
# 'size': 500,
'transform': my_transforms.get_transform([
{
'name': 'RandomCrop',
'opts': {
'keys': ('image', 'instance', 'label'),
'size': (160, 160),
}
},
{
'name': 'ToTensor',
'opts': {
'keys': ('image', 'instance', 'label'),
'type': (torch.FloatTensor, torch.LongTensor, torch.ByteTensor),
}
},
]),
},
'batch_size': 16,
'workers': 32,
#'batch_size': 2,
#'workers': 1,
},
model={
'name': 'branched_erfnet',
'kwargs': {
'num_classes': [2 + n_sigma, 1],
'input_channel': 3
}
},
lr=5e-6,
milestones=[100],
n_epochs=600,
start_epoch=1,
max_disparity=192.0,
# loss options
loss_opts={
'to_center': True,
'n_sigma': n_sigma,
'foreground_weight': 200,
},
loss_w={
'w_inst': 1,
'w_var': 10,
'w_seed': 1,
},
loss_type='MOTSSeg2Loss'
)
def get_args():
return copy.deepcopy(args)
"""
Author: Zhenbo Xu
Licensed under the CC BY-NC 4.0 license (https://creativecommons.org/licenses/by-nc/4.0/)
"""
import copy
import os
from PIL import Image
import torch
from utils import transforms as my_transforms
from config import *
args = dict(
cuda=True,
display=False,