World Scientific
Skip main navigation

Cookies Notification

We use cookies on this site to enhance your user experience. By continuing to browse the site, you consent to the use of our cookies. Learn More
Our website is made possible by displaying certain online content using javascript.
In order to view the full content, please disable your ad blocker or whitelist our website

System Upgrade on Mon, Jun 21st, 2021 at 1am (EDT)

During this period, the E-commerce and registration of new users may not be available for up to 6 hours.
For online purchase, please visit us again. Contact us at [email protected] for any enquiries.

Influence of Data Representations and Deep Architectures in Image Time Series Classification

    This article is part of the issue:

    Image time series, such as Satellite Image Time Series (SITS) or MRI functional sequences in the medical domain, carry both spatial and temporal information. In many pattern recognition applications such as image classification, taking into account such rich information may be crucial and discriminative during the decision making stage. However, the extraction of spatio-temporal features from image time series is difficult to handle due to the complex representation of the data cube. In this paper, we present a strategy based on Random Walk to build a novel segment-based representation of the data, passing from a 2D+t dimension to a 2D one, more easily manipulable and without losing too much spatial information. Such new representation is then used to feed a classical Convolutional Neural Network (CNN) in order to learn spatio-temporal features with only 2D convolutions and to classify image time series data for a particular classification problem. The influence of the way the 2D+t data are represented, as well as the impact of the network architectures on the results, are carefully studied. The interest of this approach is highlighted on a remote sensing application for the classification of complex agricultural crops.


    • 1. L. Andres, W. Salas and D. Skole , Fourier analysis of multi-temporal AVHRR data applied to a land cover classification, Int. J. Remote Sens. 15(5) (1994) 1115–1121. Crossref, ISIGoogle Scholar
    • 2. A. Bagnall, J. Lines, A. Bostrom, J. Large and E. Keogh , The great time series classification bake off: A review and experimental evaluation of recent algorithmic advances, Data Min. Knowl. Discov. 31(3) (2017) 606–660. Crossref, ISIGoogle Scholar
    • 3. L. Bruzzone and D. Prieto , Automatic analysis of the difference image for unsupervised change detection, IEEE Trans. Geosci. Remote Sens. 38(3) (2000) 1171–1182. Crossref, ISIGoogle Scholar
    • 4. M. Chelali, C. Kurtz, A. Puissant and N. Vincent , Urban land cover analysis from satellite image time series based on temporal stability, in JURSE Proc. (2019), pp. 1–4. CrossrefGoogle Scholar
    • 5. P. Coppin, I. Jonckheere, K. Nackaerts, B. Muys and E. Lambin , Digital change detection methods in ecosystem monitoring: A review, Int. J. Remote Sens. 25(9) (2004) 1565–1596. Crossref, ISIGoogle Scholar
    • 6. N. Di Mauro, A. Vergari, T. M. A. Basile, F. G. Ventola and F. Esposito , End-to-end learning of deep spatio-temporal representations for satellite image time series classification, in [email protected]/ECML Proc. (2017), pp. 1–8. Google Scholar
    • 7. L. Grady , Multilabel random walker image segmentation using prior models, in CVPR Proc. (2005), pp. 763–770. CrossrefGoogle Scholar
    • 8. B. Huang, K. Lu, N. Audebert, A. Khalel, Y. Tarabalka, J. Malof and A. Boulch , Large-scale semantic classification: Outcome of the first year of inria aerial image labeling benchmark, in IGARSS Proc. (2018), pp. 6947–6950. CrossrefGoogle Scholar
    • 9. F. Iandola, M. Moskewicz, K. Ashraf, S. Han, W. Dally and K. Keutzer, Squeezenet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size, Comput. Res. Reposit., arXiv:abs/1602.07360. Google Scholar
    • 10. D. Ienco, R. Gaetano, C. Dupaquier and P. Maurel , Land cover classification via multitemporal spatial data by deep recurrent neural networks, IEEE Geosci. Remote Sens. Lett. 14(10) (2017) 1685–1689. Crossref, ISIGoogle Scholar
    • 11. J. Inglada, A. Vincent, M. Arias, B. Tardy, D. Morin and I. Rodes , Operational high resolution land cover map production at the country scale using satellite image time series, Remote Sens. 9(1) (2017) 95–108. Crossref, ISIGoogle Scholar
    • 12. H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar and P. Muller , Deep learning for time series classification: A review, Data Min. Knowl. Discov. 33(4) (2019) 917–963. Crossref, ISIGoogle Scholar
    • 13. J. R. Jensen , Urban change detection mapping using Landsat digital data, Cartogr. Geogr. Inf. Sci. 8(21) (1981) 127–147. CrossrefGoogle Scholar
    • 14. R. Johnson and E. Kasischke , Change vector analysis: A technique for the multispectral monitoring of land cover and condition, Int. J. Remote Sens. 19(16) (1998) 411–426. Crossref, ISIGoogle Scholar
    • 15. A. Krizhevsky, I. Sutskever and G. E. Hinton , ImageNet classification with deep convolutional neural networks, in NIPS Proc. (2012), pp. 1106–1114. Google Scholar
    • 16. C. Pelletier, G. Webb and F. Petitjean , Temporal convolutional neural network for the classification of satellite image time series, Remote Sens. 11(5) (2019) 523–534. Crossref, ISIGoogle Scholar
    • 17. F. Petitjean, J. Inglada and P. Gançarski , Satellite image time series analysis under time warping, IEEE Trans. Geosci. Remote Sens. 50(8) (2012) 3081–3095. Crossref, ISIGoogle Scholar
    • 18. F. Petitjean, C. Kurtz, N. Passat and P. Gançarski , Spatio-temporal reasoning for the classification of satellite image time series, Pattern Recognit. Lett. 33(13) (2012) 1805–1815. Crossref, ISIGoogle Scholar
    • 19. P. Ravikumar and V. S. Devi , Weighted feature-based classification of time series data, in CIDM Proc. (2014), pp. 222–228. CrossrefGoogle Scholar
    • 20. M. Russwurm and M. Korner , Temporal vegetation modelling using long short-term memory networks for crop identification from medium-resolution multi-spectral satellite images, in [email protected] Proc. (2017), pp. 1496–1504. Google Scholar
    • 21. C. Senf, P. Leitao, D. Pflugmacher, S. Van der Linden and P. Hostert , Mapping land cover in complex mediterranean landscapes using landsat: Improved classification accuracies from integrating multi-seasonal and synthetic imagery, Remote Sens. Environ. 156 (2015) 527–536. Crossref, ISIGoogle Scholar
    • 22. K. Simonyan and A. Zisserman , Very deep convolutional networks for large-scale image recognition, in ICLR Proc. (2015). Google Scholar
    • 23. D. Tran, L. Bourdev, R. Fergus, L. Torresani and M. Paluri , Learning spatiotemporal features with 3D convolutional networks, in ICCV Proc. (2015), pp. 4489–4497. CrossrefGoogle Scholar
    • 24. J. Verbesselt, R. Hyndman, G. Newnham and D. Culvenor , Detecting trend and seasonal changes in satellite image time series, Remote Sens. Environ. 114(1) (2010) 106–115. Crossref, ISIGoogle Scholar
    Published: 16 September 2021