The paper presents an interdisciplinary project which is the first step towards a 3D Geographical Information System (GIS) dedicated to Cultural Heritage with a specific focus application on the Castle of Shawbak, also known as the "Crac de Montréal" in Jordan. Current 3D GIS already provide support for urban models on a city scale. Our project however focuses on a building scale encompassing its atomic elements such as ashlars blocks, cement, stratigraphic unit and architectonic elements. At this scale we need a full 3D interface in order to manage accurate measurements and a mainly heterogeneous archaeological documentation. The project is conducted by four laboratories: the MAP-GAMSAU located in the school of Architecture of Marseilles, France in charge of the photogrammetric survey phase; The LSIS laboratory, France, will be in charge of the knowledge based approach; SimVis from The Department of Computer Science, University of Hull, UK, for the virtual reality aspect and of course the "Dipartimento di Studi storici e Geografici" from the University of Florence, Italy, in charge of the archaeological part. To manage these archaeological data the project is divided into three phases: The survey phase: using a knowledge based photogrammetric tool, Arpenteur (http://www.arpenteur.net), the photogrammetric campaign ensures a survey founded on archaeological knowledge and directly linked with a database built by archaeologists. The objective here is to link an already existing archaeological database with a photogrammetric tool in order to simplify the photogrammetric process. Our goal here is to offer to the archaeology community a new tool for surveying where technical photogrammetric aspects are more or less hidden from the surveyor. The second phase is the use of the knowledge base to ensure data consistency through a complex and multi-user survey phase. Based on data fusion coming from different sources, this phase will ensure a reversible way to merge several partial surveys exploiting the complementarities between sources, solving different existing conflicts and reducing the possible redundancies. This fusion process deals with archaeological information as well as spatial information. Finally we need a high resolution interface between the final geometry and the archaeological database. Virtual reality using interactive immersive devices and specially designed software tools is an efficient method for revisiting the site and for analysing, updating and revising knowledge. This project described in this paper is work in progress. After three photogrammetric campaigns in Jordan the first results are available on the project web site: http://www.shawbak.net