Toolkit
The benchmark offers light-weighted and compile-free toolkit written in Python. You will find tutorials and examples in the corresponding repositories.
Dataset
To download VideoCube files, please click one of the following links and provide your email address in the new page. We will send you an email with an link to your download.
To make it easier for users to research with VideoCube, we have selected 150 representative sequences from the original version (500 sequences) to form VideoCube-Tiny. The original full version includes 500 sequences (1.4T), while the tiny version includes 150 sequences (344G). Furthermore, we have updated additional semantic information for the VideoCube-Tiny and proposed a new benchmark named MGIT. Please select the suitable version for you.
Full Version (500 Sequences)
- Full Version: Train Data (928G)
- Full Version: Validation Data (136G)
- Full Version: Test Data (286G)
- Full Version: Info
Tiny Version and Multi-modal Version (150 Sequences)
- Tiny Version and Multi-modal Version: Train Data (229G, 105 Sequences)
- Tiny Version and Multi-modal Version: Validation Data (37G, 15 Sequences)
- Tiny Version and Multi-modal Version: Test Data (78G, 30 Sequences)
- Tiny Version and Multi-modal Version: Info
DTVLT
Data File Structure
The VideoCube dataset includes 500 sequences, divided into three subsets (train/val/test). The content distribution in each subset still follows the 6D principle proposed in the GIT paper.
The dataset download and file organization process is as follows:
- Download three subsets (train/val/test) and the info data.
- Check the number of files in each subset and run the unzipping script. Before unzipping: (1) the train subset of full version should include 456 files (455 data files and an unzip_train bash); the train subset of tiny version should include 129 files (128 data files and an unzip_train bash); (2) the val subset of full version should include 69 files (68 data files and an unzip_val bash); the val subset of tiny version should include 22 files (21 data files and an unzip_val bash); (3) the test subset of full version should include 140 files (139 data files and an unzip_test bash); the test subset of tiny version should include 41 files (40 data files and an unzip_test bash).
- Run the unzipping script in each subset folder, and delete the script after decompression.
-
Taking val subset of full version as an example, the folder structure should follow:
|-- val/ | |-- 005/ | | |-- frame_005/ | | | |-- 000000.jpg/ | | | ...... | | | |-- 016891.jpg/ | |-- 008/ | | ...... | |-- 486/ | |-- 493/ - Unzip attribute.zip in info data. Attention that we only provide properties files for train and val subsets. For ground-truth files in the test subset, we only offer a small number of annotations for restart frames to support the essential function of the R-OPE mechanism. The annotations of other frames in the test subset have been set as zero. Please upload the final results to the server for evaluation.
-
Rename and organize folders as follows (this example is illustrated for full version, while tiny
version has the similar data structure):
|-- VideoCube/ | |-- data/ | | |-- train/ | | | |-- 002/ | | | | ...... | | | |-- 499/ | | |-- val/ | | | |-- 005/ | | | | ...... | | | |-- 493/ | | |-- test/ | | | |-- 001/ | | | | ...... | | | |-- 500/ | | |-- train_list.txt | | |-- val_list.txt | | |-- test_list.txt | |-- attribute/ | | |-- absent/ | | |-- color_constancy_tran/ | | | ...... | | |-- shotcut/
The MGIT dataset includes 150 sequences, divided into three subsets (train/val/test). The content distribution in each subset still follows the 6D principle proposed in the GIT paper.
The dataset download and file organization process is as follows:
- Download three subsets (train/val/test) and the info data.
- Check the number of files in each subset and run the unzipping script. Before unzipping: (1) the train subset of tiny version should include 129 files (128 data files and an unzip_train bash); (2) the val subset of tiny version should include 22 files (21 data files and an unzip_val bash); (3) the test subset of tiny version should include 41 files (40 data files and an unzip_test bash).
- Run the unzipping script in each subset folder, and delete the script after decompression.
-
Taking val subset of tiny version as an example, the folder structure should follow:
|-- val/ | |-- 005/ | | |-- frame_005/ | | | |-- 000000.jpg/ | | | ...... | | | |-- 016891.jpg/ | |-- 029/ | | ...... | |-- 350/ | |-- 362/ - Unzip attribute.zip in info data. Attention that we only provide properties files for train and val subsets. For ground-truth files in the test subset, we only offer a small number of annotations. The annotations of other frames in the test subset have been set as zero. Please upload the final results to the server for evaluation.
-
Rename and organize folders as follows:
|-- MGIT/ | |-- data/ | | |-- train/ | | | |-- 002/ | | | | ...... | | | |-- 480/ | | |-- val/ | | | |-- 005/ | | | | ...... | | | |-- 362/ | | |-- test/ | | | |-- 001/ | | | | ...... | | | |-- 498/ | | |-- train_list.txt | | |-- val_list.txt | | |-- test_list.txt | |-- attribute/ | | |-- absent/ | | |-- color_constancy_tran/ | | | ...... | | |-- shotcut/
Results
Download baseline tracking results and performance reports of 20 public entries on VideoCube from the official account page:
- Results on VideoCube (OPE Mechanism with Full Version)
- Results on VideoCube (R-OPE Mechanism with Full Version)
- Results on VideoCube (OPE Mechanism with Tiny Version)
- Results on VideoCube (R-OPE Mechanism with Tiny Version)
Download baseline tracking results and performance reports of 4 public entries on MGIT from the official account page:
- Results on MGIT (OPE Mechanism with Tiny Version, Action Granularity)
- Results on MGIT (OPE Mechanism with Tiny Version, Activity Granularity)
- Results on MGIT (OPE Mechanism with Tiny Version, Story Granularity)
License
The VideoCube and MGIT dataset is licensed under CC BY-NC-SA 4.0. You are free to use the dataset for research purpose. If you want to use it for commercial purpose, please contact us.
Citation
Please cite our IEEE TPAMI paper if VideoCube helps your research.
Please cite our NeurIPS paper if MGIT helps your research.