Experiental Design Journal

5/29/18 Week 1-14
Kamal Afiq [0330643]
Experiential Design

---

App Proposal

DESIGNER DETAILS

Name: Kamal Afiq bin Kamarul Bahri

Student ID: 0330643

ABOUT THE PROJECT

Project Title: Pocket Places

Project Description: An AR experience that simulates historical locations through marked cards, given out as packs at museums to visitors as souvenirs and portable learning experiences.

Problem Statement: Children are able to visit museums to look at old items and antiques, but are unable to visit many historical places themselves.

Aims/Goals: Provide an interactive and immersive AR experience that will bring the historical world to the palm of a child’s hand, allowing them to visit a historical and culturally significant place all from home or anywhere.

RESEARCH & SOLUTION

Experience Description: A pack of cards (small enough to fit into a small bag) are imprinted with AR markers, which upon being scanned by the app Pocket Places, augments a 3D model of a historical place (e.g. museum, temple, Colosseum) on top of the card. Each card displays a different part of a single historical place, and if the user puts all the cards together, the entire building is reconditioned in real-time, interactive 3D. The user can tap on small icons that appear on different parts of the building to find out more about the location and to interact with media (audio recordings, pictures). On top of that small animations (moving characters, objects) make these historical places come alive, especially for younger users.

Key Features:

• AR marked cards that display 3D locations of historical places.
• Cards can be put together to simulate an entire building or place.
• A single card pack can be used to simulate a single, entire building or place.
• Interactive elements like interactive buttons, audio logs, videos and such allow the user to learn more about the place
• Small animated character and objects makes these places feel immersive

Target Demographic: Young children aged 6-12, visual learners who like trying out new ideas.

Personas: As the experience is targeted towards primary and middle-school children, the demographic is likely to be easily excited by new media experiences, have short attention spans and are very receptive of colorful designs and animations.

User Experience Map: The main feature of the experience is the 3D visualization of a historical building or monument; allowing the user to rotate, zoom in and explore the object. Smaller features like pop-up buttons that yield text descriptions, audio logs, links and galleries are also smaller interactions they can make while interacting with the 3D models.

USER JOURNEY PAIN POINTS
Journey Step	Problems
Open the app	Loading time. Responsiveness.
Selects ‘explore’ in the menu	Functionality of links. Responsiveness of buttons. Loading time.
Scanning marked card	Camera angle. Clarity of markings on card. Responsiveness.
Exploring the building	Camera angle and clarity of markings. Loading time of 3D model. Detail on model as it is zoomed in. How close the card can be zoomed in before model disappears
Interacting with prompts, buttons	Responsiveness. Loading time. Overlay and GUI. Clutter of screen.
Interacting with animations	Responsiveness. Animation quality and playback. Fluidity of movement
Returning to main menu	Functionality of links. Responsiveness of buttons. Loading time.

Potential challenges:

• Programming the cards to, when put together to render a single building
• Coding the interactive elements
• Researching layout and history of places
• Animating 3D models
• Testing the card

Visualization & Concept:

---

Existing App Research

STORY OF THE FOREST

An immersive AR Installation that turning 69 drawings from a collection of natural history drawings into three-dimensional animations. Visitors use an AR app to hunt for flora and fauna in the various mural installations. Like Pokemon Go, when nearby an animal you can add it to a collection. The app later collects the photo or animal and displays more information about it.

SKIN AND BONES

The Bone Hall in the Smithsonian National Museum's anatomy exhibit allows guests to use an app to overlay skin and movements onto bones. The app features games and video content to entertain and educate, in the museum and at home, by using triggering images that can be downloaded and printed out.

HEROES AND LEGENDS

An AR exhibition at the American Space program exhibit at the Kennedy Space Center. Interactive pieces and display give information about early astronauts and NASA legends. A hologram of Gene Cernan, a spacewalking astronaut  is superimposed over the actual historical space capsule, following him as he struggles to get back inside following a spacewalk gone wrong.

JINSHA SITE MUSEUM

Visitors can use an app to explore ancient religious relics from an ancient Chinese civilization known as the Shu in 3D, learn what they were used for. One exhibit includes a marked card, that when scanned produces a golden mask that floats and spins in front of the visitors face. A stationary AR camera over an actual burial site reconstructs the location in AR 3D.

ENGLAND'S HISTORIC CITIES

An AR APP that allows tourists to interact with heritage sites across England. Durnham Cathedral, Hadrian's Wall and Roman Baths are just examples of sites that work with the app. Historical figures serve as virtual guides and information is superimposed over walls and artifacts. For example, William Shakespeare leads visitors through a behind-the-scenes history of his life at his family home.

---

Site Research

In this section, I will explore potential candidates for buildings or sites which I can use for my Pocket Places AR experience.

LUXOR TEMPLE, Egypt

An ancient temple on the river Nile built in 1400 BC. It was used in the past to coronate or celebrate new kings. Predominantly made of sandstone, it is a complex that includes facades, closed buildings, giant columns and statues.

Advantages

• Large structure, plenty of different faces and structures
• Diverse number of locations, lots of potential points of interest for the App
• Well documented, open to the public.
• Relatively easy to model the main buildings, the construction is relatively simple
• Open buildings and structures makes the location very easy to look through

Challenges

• Quite a large structure, a large amount of 3D modelling to perform
• Statues may seem difficult to model, modelling has to potentially be low poly

TEMPLE OF JUPITER, Pompeii

Built in Ancient Rome in Pompeii, it was dedicated to worship of the god Jupiter. It was built in 150 BC, and stored sacrificial offerings as well as the treasury of the city. It was destroyed by an earthquake in 62 AD, and was buried along with the rest of Pompeii when Mount Vesuvius erupted in 79 AD. It has since been excavated.

Advantages

• Modestly sized structure, can be filled with lots of interior elements
• Well documented and virtually reconstructed
• Simple architecture will be relatively easy to model

Challenges

• Not many points of interest, kinda boring
• Large portion of the building is interior, ceiling needs to be sorted out
• Cutting the building into small segment for the card needs to be planned out

---

App Development

In this section of the app's development, I will document the process and steps I actually undertook to develop Pocket Places in Unity using the Vuforia AR plugin.

Before setting things up in Unity however, I first had to recreate the blueprints and map for the actual building I chose to make the subject of the app. I chose the Luxor temple because of its large scope, its various different sections and because it was able to be sliced up in a relatively consistent size. 

The first step was to build up photo and diagram references of the actual temple itself. The diagrams would help me recreate the blueprints for the building's foundation, while the photographs would help me model the specific pillars and construction details. The various images were taken from various tourist and historical websites around the internet. I made sure to take up images from various different sources so that I would build an overall accurate recreation of the actual temple itself. 

With references in hand, I began drawing out the blueprint of the temple using Adobe Illustrator. Thicker and outer walls had a thick stroke, while smaller rooms had thin strokes. Areas covered with flooring and roofs were marked out by grey sections of the blueprint.

Here I began to cut up the temple's structure using the marker card boundaries, and added further decorations and details so that the individual cards would be more visually appealing. I also created a simple logo for the brand of the app. 

Take note that while the actual marker cards have green borders and decorations, only the actual black and white blueprint was uploaded as image target's to Vuforia's database. 

With the image targets set up, I also began to create the UI elements of the app. I decided early on that green would be the brand color for the app, and would be used consistently throughout the app in the borders and text. Frames and scanning images were also created to further build upon the app's style and personality.

After having conducting all the graphic design work and setting up Vuforia and its databases within Unity, I began carrying out tests to ensure that the markers were tracking properly, and that the app was able to detect more than one image marker at a time. One of the features I planned to include was that if more than one marker card was being sighted by the camera, multiple parts of the building could be shown at once. If the user were to place all the cards side by side in the right order, the whole temple can be displayed at once.

Next was to reconstruct the actual temple itself within a 3D modelling program. I used Autodesk Maya to create all the structures, including the walls, pillars and roofs. I imported the marker cards as reference image plans into Maya, and then rebuilt the structures on top of that. Due to time constraints however I was unable to UV unwrap and texture the model as a whole.

The lower walls, upper walls, pillars and roofs were built as separate pieces that could be toggled on and off. I then exported the different structures as separate pieces depending on which marker card.

With modelling done, I imported the models into Unity and placed them under the various marker cards, so that when a certain marker is scanned only the corresponding building elements would appear.

Initially I wanted each of the structure parts to just pop in when tapped, but after doing it I realized that doing animations of the building actually coming together was more engaging and entertaining. For each separate piece of the structure I added both an idle and entry animation, and using scripting when a structure it tapped the next part appears, but only appears after beginning its entry animation.

Using this method I was able to get the different structure pieces to build themselves up as a whole building in an entertaining way. I also coded it so that when the roof was tapped, all the pieces would reset leaving behind only the foundation of the structure, so that the user can begin building up the temple again.

With the base functionality down, I began adding in the UI elements and creating pop up text within the canvas of the app to give it a better finish. I also used the Postprocessing stack available on the Unity store to make the visuals more engaging. I added depth of field, anti-aliasing, ambient occlusion and a subtle 3D distortion affect this way by hooking up a postprocessing profile with the main AR camera. This way only the actual camera is distorted, without affecting any of the UI elements.

Moving on to add further functionality and user interaction to the app, I added small interactive cameras that the user can tap to open up pop ups displaying various photos of the temple in real life shot at different angles. The camera models were also made by me, and I put them in an empty Gameobject and animated them to rotate and however so that they appear more engaging. 

Using code similar to how the structure pops up, I also made it so that when the pop up photographs appear, they also come in through entry animations.

Next was to add sound to the app. I wanted it so that when the building was being constructed, stone grinding sounds would be heard so that the interaction would have more feedback and come off as more realistic. I also used a camera shutter sound taken from online for the interactive floating cameras as well. 

Adding to this, I also recorded my own voice to play whenever the photographs appear on screen. All the sounds were edited for pitch, reverb and quality on Adobe Audition before being exported and imported as .wavs. 

 All the interactive objects were assigned audio sources that play predetermined sounds when they were tapped and interacted with. All these actions were referenced and executed using pre-existing code used to drive the interaction in the first place.

To add another touch of personality to the app, I modeled various cartoony characters for the temple, imported then and gave them looping animations of them talking and walking around in the temple to give the user a sense of scale as well as make the entire app appear more lively. 

Finally after all the main work was done, I began configuring the build and player settings to change the logo and splash screen, as well as to ensure the app itself ran and performed properly. I made sure that the screen and UI elements matched properly, and that everything appeared properly on my phone, a Samsung Note 8.

This marks the end of development for the app Pocket Places. It's here that all the main work was done, but I also carried out some minor tweaks and improvements to ensure that the app was made to a higher standard.

Given more time, I would have made it so that if the player taps the structure after it has been constructed, it would have deconstructed itself and animated in reverse. Also if I had more time I would also like to have made the small characters interactive, jumping or doing actions if the user tapped on them. Small 3D turnovers of certain pieces like statues and pillars were also a feature I would have liked to include if I had more time and resources to develop the app further.