Cities have been studied as geo-social systems embedded with intricate and complicated spatial and social networks (e.g., transportation, telecommunication, and internet). In addition to the duo of spatial and social aspects, semantics, which study the logic aspects of meanings behind behaviours and phenomena, come underneath as the latent information (e.g., activity types of people) to enrich the geo-social models for spatial phenomena. For example, individual-level similarity of semantic trajectories for location-based social networks can be used to recommend potential friends or develop collaborative travels. Semantics infer the activity behind people's spatial choices and the functions of places, transform coordinates of trajectories/spatial flows into certain types of activities, and remark locations in space with meaningful labels of functions of cities. Although the interconnections of spatial, social, and semantic domains are widely observed, the deeper theoretical integration of geography, social network, and semantic spaces, as well as the corresponding research challenges, is not yet sufficient for a comprehensive understanding of urban systems. In order to address this gap, this work proposes a novel theoretical framework for the integration of semantic perspective into geographic and social network perspectives applied to understanding urban systems. We discuss the advantages and disadvantages in terms of available data sets in urban studies within the theoretical framework. We also discuss research challenges in terms of integrating heterogeneous data sources and creating innovative analytical approach based on the theoretical framework. We believe that the proposed theoretical framework can shed light on a wide range of urban related research fields and decision-making, such as transportation, public health, urban planning, and emergency management.