The presented work explains the architecture and mechanism of the purposed building information integration concept, the coupling of user driven BIM data with mobile AR (as shown in Fig. 1), server-client communication using REST (Representational State Transfer) paradigms, user-driven annotation objects and services as well as embedded tracking technologies on smartphone platforms. In order to interlink different instances of collaborative workers, we describe in detail the new concepts for BIM exchange that enable user driven annotations for real-time content generation during on-site sessions.