Added PointTrack to 'Algorithms/PointTrack' along with simple instructions to...

Added PointTrack to 'Algorithms/PointTrack' along with simple instructions to train and test datasets on the algorithm. README.md file in branch master was slightly modified to point to the added changes.

README.md

@@ -2,6 +2,8 @@
 
 This is the code to our research 'Stepping towards real-time detection and tracking of apples using deep learning'. We provide scripts and tutorials for applying multi-object tracking and segmentation (MOTS) on an apple orchard by using the [TrackR-CNN][TrackR-CNN_MOTS] algorithm. 
 
+To use the **PointTrack** algorithm instead of **TrackR-CNN**, go to the [PointTrack](Algorithms/PointTrack/) folder for instructions.
+
 ## Video of results
 
 ![](visualizations/test_8_wearable_sensor.gif)

algorithms/PointTrack/.gitignore

+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/

algorithms/PointTrack/README.md

+# Realtime Apples Detection and Tracking using UAVs
+
+This codebase implements [PointTrack](https://github.com/detectRecog/PointTrack), a highly effective framework for multi-object tracking and segmentation (MOTS) described in: 
+
+```
+@inproceedings{xu2020Segment,
+  title={Segment as Points for Efficient Online Multi-Object Tracking and Segmentation},
+  author={Xu, Zhenbo and Zhang, Wei and Tan, Xiao and Yang, Wei and Huang, Huan and Wen, Shilei and Ding, Errui and Huang, Liusheng},
+  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
+  year={2020}
+}
+```
+
+
+
+**PointTrack presents a new learning strategy for pixel-wise feature learning on the 2D image plane, which has proven to be effective for instance association.**
+
+The network architecture adopts [SpatialEmbedding](https://github.com/davyneven/SpatialEmbeddings) as the segmentation sub-network.
+  
+
+## Getting started
+
+This codebase showcases the proposed framework named PointTrack for MOTS using the a MOTS annotated dataset. 
+
+### Prerequisites
+Dependencies: 
+- Pytorch 1.3.1 (and others), please set up an virtual env and run:
+```
+$ pip install -r requirements.txt
+```
+- Python 3.6 (or higher)
+- [KITTI Images](http://www.cvlibs.net/download.php?file=data_tracking_image_2.zip) + [Annotations](https://www.vision.rwth-aachen.de/media/resource_files/instances.zip)
+
+Note that the scripts for evaluation is included in this repo. After images and instances (annotations) are downloaded, put them under **kittiRoot** and change the path in **repoRoot**/config.py accordingly. 
+The structure under **kittiRoot** should looks like:
+
+```
+kittiRoot
+│   images -> training/image_02/ 
+│   instances
+│   │    0000
+│   │    0001
+│   │    ...
+│   training
+│   │   image_02
+│   │   │    0000
+│   │   │    0001
+│   │   │    ...  
+│   testing
+│   │   image_02
+│   │   │    0000
+│   │   │    0001
+│   │   │    ... 
+```
+
+##Config.py file
+
+Change the file directories accordingly.
+`kittiRoot` is the directory where the dataset is stored.
+
+## Pretrained weights
+Fine-tuned models on KITTI MOTS (for cars):
+- SpatialEmbedding for cars.
+- PointTrack for cars.
+You can download them via [Baidu Disk](https://pan.baidu.com/s/1Mk9JWNcM1W08EAjhyq0yLA) or [Google Drive](https://drive.google.com/open?id=14Hn4ZztfjGUYEjVd-9FRNB5a-CtBkPXc).
+- testset segs can be found in the folder 'testset_segs'.
+
+For Apples, models will be uploaded soon.
+The testset segs are found in the folder 'apple_testset_segs'
+
+##Train a dataset from scratch
+To train a dataset from scratch, follow instructions from the master branch of this repository on how to label a custom dataset.
+Next, order the dataset folders in accordance to the `kittiRoot` structure above.
+Next, create a new environment and install all the dependencies.
+
+
+## Training of SpatialEmbedding
+
+An older way of training SpatialEmbedding with KITTI MOTS needs KITTI object detection left color images as well as the KINS annotations.
+However, that is not necessary anymore. It can be done by copying MOTS ground truth annotations to a folder named KINS inside the training and testing folder of kittiRoot.
+
+A folder structure of kittiRoot needs to be structured in the following:
+
+```
+kittiRoot
+│   images -> training/image_02/ 
+│   instances
+│   │    0000
+│   │    0001
+│   │    ...
+│   training
+│   │   image_02
+│   │   │    0000
+│   │   │    0001
+│   │   │    ...  
+│   │   KINS
+│   │   │   0000
+│   │   │   0001
+│   │   │   ...
+│   testing
+│   │   image_02
+│   │   │    0000
+│   │   │    0001
+│   │   │    ... 
+│   │   KINS
+│   │   │   0000
+│   │   │   0001
+│   │   │   ...
+```
+
+Each of the KINS subfolders should be filled with the instance PNGs of your dataset.
+
+For the training of SpatialEmbedding, we follow the original training setting of [SpatialEmbedding](https://github.com/davyneven/SpatialEmbeddings). 
+
+1.This step selects valid frames that contains objects of the class; since there could be frames that contain no masks of the object.
+ ```
+$ python -u datasets/MOTSImageSelect.py
+ ```
+
+2.Before generating the crops, make sure that the crops sizes are correct in the file. Then do the following:
+```
+$ python -u utils/generate_crops.py
+```
+After this step, crops are saved under **kittiRoot**/crop_KINS.
+
+3.Afterwards start training on crops: 
+```
+$ python -u train_SE.py car_finetune_SE_crop
+```
+You can change desired parameters inside the config file.
+
+4.Afterwards start finetuning on KITTI MOTS with BN fixed. SpatialEmbeddings trained the finetuning with a crop size 2x larger than the first training.
+
+Create a file on the root with the name **resume_SE**, then copy the **checkpoint.pth** file into it.
+
+Then run the following command:
+```
+$ python -u train_SE.py car_finetune_SE_mots
+```
+The finetuning will be saved to a different folder.
+
+## Training of PointTrack
+The training procedure of instance association is as follows.
+
+1.To generate the segmentation result on the validation set as the instruction of the first step in Testing.
+Make sure the file **datasets/KittiMOTSDataset.py** and seqmaps in **datasets/mots_tools/mots_eval/** has the correct dataset configuration.
+
+Then, copy the checkpoint.pth file from the finetuning folder to the **tuned** folder in `rootDir`.
+
+Finally run the segmentation result (forwarding):
+
+```
+$ python -u test_mots_se.py car_test_se_to_save
+```
+
+2.To generate the instance DB from videos:
+```
+$ python -u datasets/MOTSInstanceMaskPool.py
+```
+
+3.Change the line in **train_tracker_with_val.py** which loads weights to the default path as follows:
+```
+checkpoint_path='./tuned/checkpoint.pth'
+```
+
+4.Afterwards start training:
+```
+$ python -u train_tracker_with_val.py car_finetune_tracking
+```
+The best tracker on the validation set will be saved under the folder specified in **repoRoot**/config_mots/car_finetune_tracking.py.
+
+
+## Testing
+
+You can download pretrained models from the above links. Save these weight file under **repoRoot**/pointTrack_weights.
+
+1.To generate the instance segmentation results:
+
+```
+$ python -u test_mots_se.py car_test_se_to_save
+```
+The segmentation result will be saved according to the config file **repoRoot**/config_mots/car_test_se_to_save.py.
+
+2.To test PointTrack on the instance segmentation results:
+```
+$ python -u test_tracking.py car_test_tracking_val
+```
+
+
+## Cite us
+We borrow some code from [SpatialEmbedding](https://github.com/davyneven/SpatialEmbeddings).
+```
+@inproceedings{xu2020Segment,
+  title={Segment as Points for Efficient Online Multi-Object Tracking and Segmentation},
+  author={Xu, Zhenbo and Zhang, Wei and Tan, Xiao and Yang, Wei and Huang, Huan and Wen, Shilei and Ding, Errui and Huang, Liusheng},
+  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
+  year={2020}
+}
+@inproceedings{xupointtrack++,
+  title={PointTrack++ for Effective Online Multi-Object Tracking and Segmentation},
+  author={Xu, Zhenbo and Zhang, Wei and Tan, Xiao and Yang, Wei and Su, Xiangbo and Yuan, Yuchen and Zhang, Hongwu and Wen, Shilei and Ding, Errui and Huang, Liusheng},
+  booktitle={CVPR Workshops},
+  year={2020}
+}
+```
+
+## Contact
+If you find problems in the code, please open an issue, or contact Hilmy (hilmy.baja@wur.nl)
+
+
+## License
+
+This software is released under a creative commons license which allows for personal and research use only. For a commercial license please contact the authors. You can view a license summary [here](http://creativecommons.org/licenses/by-nc/4.0/).
+
+
+
+
+
+

algorithms/PointTrack/config.py

+"""
+Author: Zhenbo Xu
+Licensed under the CC BY-NC 4.0 license (https://creativecommons.org/licenses/by-nc/4.0/)
+"""
+import os
+
+project_root = os.path.dirname(os.path.realpath(__file__))
+category_embedding = [[0.9479751586914062, 0.4561353325843811, 0.16707628965377808], [0.1,-0.1,0.1], [0.5455077290534973, -0.6193588972091675, -2.629554510116577], [-0.1,0.1,-0.1]]
+
+systemRoot = "/home/saidlab/"
+kittiRoot = os.path.join(systemRoot, "Thesis-Hilmy/PT/data/APPLE_MOTS/train/")
+rootDir = os.path.join(systemRoot, 'Thesis-Hilmy/PT')
+pythonPath = os.path.join(systemRoot, "anaconda3/envs/pt_env/bin/python")

algorithms/PointTrack/config_mots/__init__.py

+import config_mots.car_test_se_to_save
+import config_mots.car_test_tracking_val
+import config_mots.car_finetune_tracking
+import config_mots.car_finetune_SE_crop
+import config_mots.car_finetune_SE_mots
\ No newline at end of file

algorithms/PointTrack/config_mots/car_finetune_SE_crop.py

+"""
+Author: Zhenbo Xu
+Licensed under the CC BY-NC 4.0 license (https://creativecommons.org/licenses/by-nc/4.0/)
+"""
+import copy
+import os
+
+from PIL import Image
+
+import torch
+from utils import transforms as my_transforms
+from config import *
+
+n_sigma=2
+args = dict(
+
+    cuda=True,
+    display=False,
+    display_it=5,
+
+    save=True,
+    save_dir='./mots_finetune_car2',
+    #resume_path='./tuned/best_seed_model.pthCar',
+    # resume_path='./mots_finetune_car2/checkpoint.pth',
+
+    train_dataset = {
+        'name': 'mots_cars',
+        'kwargs': {
+            'root_dir': kittiRoot,
+            'type': 'crop',
+            'size': 7000,
+            'transform': my_transforms.get_transform([
+                {
+                    'name': 'AdjustBrightness',
+                    'opts': {}
+                },
+                {
+                    'name': 'ToTensor',
+                    'opts': {
+                        'keys': ('image', 'instance','label'),
+                        'type': (torch.FloatTensor, torch.LongTensor, torch.ByteTensor),
+                    }
+                },
+                {
+                    'name': 'Flip',
+                    'opts': {
+                        'keys': ('image', 'instance','label'),
+                    }
+                },
+            ]),
+        },
+        #'batch_size': 4,
+        #'workers': 1,
+         'batch_size': 64,
+         'workers': 32
+    },
+
+    val_dataset = {
+        'name': 'mots_cars',
+        'kwargs': {
+            'root_dir': kittiRoot,
+            'type': 'val',
+            # 'size': 500,
+            'transform': my_transforms.get_transform([
+                {
+                    'name': 'RandomCrop',
+                    'opts': {
+                        'keys': ('image', 'instance', 'label'),
+                        'size': (80, 80),
+                    }
+                },
+                {
+                    'name': 'ToTensor',
+                    'opts': {
+                        'keys': ('image', 'instance', 'label'),
+                        'type': (torch.FloatTensor, torch.LongTensor, torch.ByteTensor),
+                    }
+                },
+            ]),
+        },
+         'batch_size': 16,
+         'workers': 32,
+        #'batch_size': 2,
+        #'workers': 1
+    },
+
+    model={
+        'name': 'branched_erfnet',
+        'kwargs': {
+            'num_classes': [2 + n_sigma, 1],
+            'input_channel': 3
+        }
+    },
+
+    lr=5e-4,
+    milestones=[100],
+    n_epochs=400,
+    start_epoch=1,
+    max_disparity=192.0,
+
+    # loss options
+    loss_opts={
+        'to_center': True,
+        'n_sigma': n_sigma,
+        'foreground_weight': 10,
+    },
+    loss_w={
+        'w_inst': 1,
+        'w_var': 10,
+        'w_seed': 1,
+    },
+    loss_type='MOTSSeg2Loss'
+)
+
+
+def get_args():
+    return copy.deepcopy(args)

algorithms/PointTrack/config_mots/car_finetune_SE_mots.py

+"""
+Author: Zhenbo Xu
+Licensed under the CC BY-NC 4.0 license (https://creativecommons.org/licenses/by-nc/4.0/)
+"""
+import copy
+import os
+
+from PIL import Image
+
+import torch
+from utils import transforms as my_transforms
+from config import *
+
+n_sigma=2
+args = dict(
+
+    cuda=True,
+    display=False,
+    display_it=5,
+    save=True,
+    fix_bn=True,
+
+    save_dir='./mots_finetune_car2_with_BN',
+    resume_path='./resume_SE/checkpoint.pth',
+
+    train_dataset = {
+        'name': 'mots_cars',
+        'kwargs': {
+            'root_dir': kittiRoot,
+            'type': 'train',
+            'kins': False,
+            'size': 1000,
+            'transform': my_transforms.get_transform([
+                {
+                    'name': 'RandomCrop',
+                    'opts': {
+                        'keys': ('image', 'instance','label'),
+                        'size': (160, 160)
+                    }
+                },
+                {
+                    'name': 'ToTensor',
+                    'opts': {
+                        'keys': ('image', 'instance','label'),
+                        'type': (torch.FloatTensor, torch.LongTensor, torch.ByteTensor),
+                    }
+                },
+            ]),
+        },
+        #'batch_size': 2,
+        #'workers': 1,
+        'batch_size': 2,
+        'workers': 32
+    },
+
+    val_dataset = {
+        'name': 'mots_cars',
+        'kwargs': {
+            'root_dir': kittiRoot,
+            'type': 'val',
+            # 'size': 500,
+            'transform': my_transforms.get_transform([
+                {
+                    'name': 'RandomCrop',
+                    'opts': {
+                        'keys': ('image', 'instance', 'label'),
+                        'size': (160, 160),
+                    }
+                },
+                {
+                    'name': 'ToTensor',
+                    'opts': {
+                        'keys': ('image', 'instance', 'label'),
+                        'type': (torch.FloatTensor, torch.LongTensor, torch.ByteTensor),
+                    }
+                },
+            ]),
+        },
+        'batch_size': 16,
+        'workers': 32,
+        #'batch_size': 2,
+        #'workers': 1,
+    },
+
+    model={
+        'name': 'branched_erfnet',
+        'kwargs': {
+            'num_classes': [2 + n_sigma, 1],
+            'input_channel': 3
+        }
+    },
+
+    lr=5e-6,
+    milestones=[100],
+    n_epochs=600,
+    start_epoch=1,
+    max_disparity=192.0,
+
+    # loss options
+    loss_opts={
+        'to_center': True,
+        'n_sigma': n_sigma,
+        'foreground_weight': 200,
+    },
+    loss_w={
+        'w_inst': 1,
+        'w_var': 10,
+        'w_seed': 1,
+    },
+    loss_type='MOTSSeg2Loss'
+)
+
+
+def get_args():
+    return copy.deepcopy(args)

algorithms/PointTrack/config_mots/car_finetune_tracking.py

+"""
+Author: Zhenbo Xu
+Licensed under the CC BY-NC 4.0 license (https://creativecommons.org/licenses/by-nc/4.0/)
+"""
+import copy
+import os
+
+from PIL import Image
+
+import torch
+from utils import transforms as my_transforms
+from config import *
+
+args = dict(
+
+    cuda=True,
+    display=False,