User-aware Social Tagging Analysis

In typical image sharing websites, users are allowed to choose tags to describe the uploaded images for easy organization. Therefore, three types of interrelated entities exist in the social image tagging ecosystems, i.e., user, image and tag. In this work, we propose to introduce the user entity into the social image tag analysis. The observed raw tags can be seen as product of ternary interactions among the three entities, which construct the raw 3-order tensor. Combined with the homogeneous intro-entity priors, a rank-based optimization scheme is presented to reconstruct the tensor and obtain the final ternary interrelations. The improved ternary interrelations can be applied along three lines: (1) image-tag: tag enrichment; (2) user-tag: user modeling; (3) user-image: image recommendation; and (4) user-image-tag: personalized tag recommendation, personalized image retreival.      [project]
[1] User-aware Image Tag Refinement via Ternary Semantic Analysis. IEEE Transactions on Multimedia, vol 14, no.3-2, pp.883-895, 2012.
[2] Learn to Personalized Image Search from the Photo Sharing Websites. IEEE Transactions on Multimedia, vol 14, no.4-1, pp.963-974, 2012.
[3] Exploiting user information for image tag refinement. ACM Multimedia, short, 2011, pp.1129-1132.

Online Activity-based User Modeling

Users' online activities well indicate their interests, which can be used to infer their background and contruct user profile. Specifically, we consider two problems: user activity data sparisity, and user attribution relation. For the first problem, data expansion is performed to enrich user acitivity collection considering user and activity collaboration information. User-specific topic modeling is conducted on the expanded user collection to learn user preferences. For the second one, we exploit the relations between user attributes via a relational latent model. The derived attribute relation is utilized for accurate user attribute inference, as well as applied to structural attribute-based user retrieval.      [project]
[1] Learn to Personalized Image Search from the Photo Sharing Websites. IEEE Transactions on Multimedia, vol 14, no.4-1, pp.963-974, 2012.

Personalized Image Search and Video Recommendation

     [project]
[1]  Right Buddy Makes the Difference: an Early Exploration of Social Relation Analysis in Multimedia Applications. ACM Multimedia, oral, 2012, pp.19-28.
[2]  Personalized Celebrity Video Search Based on Cross-Space Mining. Pacific-rim Conference on Multimedia, 2012, pp.455-463.

Contextual and Personalized Mobile Recommendation

     [project]
[1]  Probabilistic Sequential POIs Recommendation via Check-in Data. ACM SIGSPATIAL GIS, 2012, pp.402-405.
[2]  Contextual and Personalized Mobile Recommendation Systems. Chapter in Tools for Mobile Multimedia Programming and Development, IGI Global Book, 2013. In Press.

Topic-level Social Relation Analysis

(1) Peer-to-peer influence Modeling      [project].
(2) Influence Expert Mining      [project]
[1]  Right Buddy Makes the Difference: an Early Exploration of Social Relation Analysis in Multimedia Applications. ACM Multimedia, oral, 2012, pp.19-28.

Cross-platform Information Aggregation and User Modeling

     [project]
[1]  Personlized Video Recommendation based on Cross-Platform User Modeling. IEEE International Conference on Multimedia & Expo(ICME) , 2013, accepted..

Cross-platform Collaborated Applications

     [project]
[1]  FriendTransfer: Cold-Start Friend Recommendation With Cross-Platform Transfer Learning of Social Knowledge. IEEE International Conference on Multimedia & Expo(ICME) , oral, 2013, accepted.

Cluster-based Retrieved Video Organization

The overwhelming amount of web videos makes effective search and browsing a very challenging task. Users have to painstakingly browse through the long rank list to judge whether the results match their requriements and locate the interesting ones. Instead of organizing the returned results via ranked list, in this work, we propose to group the videos into semantically consistent clusters, where users are provided with a visualized summary of the video collection to have a quick overview and locate desired videos efficiently. Based on data observation into the returned video collection, we found inherent hierachical structure that the specific sub-topics share one common general topic related to the query. Inspired by this, we extend the traditional hierarchical topic model (hLDA) by considering query-root topic relation and pairwise visual duplicate constraints. A novel user interface is designed to visualize the derived video clusters.      [project]
[1] Facet Subtopic Retrieval: Exploiting the Topic Hierarchy via a Multi-modal Framework. Journal of Multimedia, vol 7, no.1, pp.9-20, 2012.
[2] Browse by chunks: Topic Mining and organizing on Web-scale Social media. ACM Transactions on Multimedia Computing, Communications and Applications, vol 7s, no.1: article No. 30, 2011.

Semantic and Geographical Multimedia Organization

In one hand, with the development of positioning technologies and popularity of mobile devices, extensive geographical information is attached with human behaviors. In the other, social media provides a best platform to share these human behavior generated multimedias, which make large-scale geo-referenced multimedia publicly available for acadmia investigation and industrial application. One natural application is for geographical organization, which cluster and visualize multimedias on the map according to their GPS position. We emphasize in this work that, besides providing a sharing platform, social media produces remarkable metadata to describe the multimedias. We propose to exploit the metadata for semantic mining to organize multimedias semantically as well as geographically. Specifically, we are interested to visualize a city from multiple representative themes, where each theme is extracted by aggregating several lower-level Point-Of-Interest (POI).      [project][demo]
[1] Paint the City Colorfully: Location Visualization from Multiple Themes. International Conference on Multimedia Modeling(MMM) , 2013, pp.92-105.

Deep Structure-based Feature Learning

The quality of features has been proved crucial in computer vision tasks such as scene classification and object recognition. Instead of extracting data-driven features, current multimedia research works tend to use off-the-shelf visual features, like SIFT, HOG, Haarlet, etc. Feature learning is largely ignored, especially in complex problems. Under Web 2.0 circumstance, like the social tagging systems, medias are extensively associated, either homogeneously or hetergeneously. Feature learning, for both homogeneous and hetergeneous medias, is more important and challenging than ever before. Appropriate feature will alleviate the burden for higher-level model design and thus improve the overall performance. Deep learning is distinguished abstraction and representation ability. In this work, we propose to exploit the deep learning framework to pursue unified feature for the associated heterogeneous as well as homogeneous medias.      [project]
[1]  Tag-aware Image Classification via Nested Deep Belief Nets. IEEE International Conference on Multimedia & Expo(ICME) , 2013, accepted.

User Study and Interactive Design for Mobile Visual Search

Robust Name-Face Graph Matching based on Script-Video Alignment

Character-based Movie Summarization

[1]  Character-based movie summarization. ACM Multimedia , short, 2010, pp.855-858.