Work Done

This was the last week to prepare for the exhibit, specifically: the portfolio, the demonstration video, finalizing Spud's build and doing user testing.

Portfolio

I planned 1 homepage and 4 other pages (Concept, Product, Features, Process) to display my work on Spud. Each section of the index page showed a short summary or introduction with links to the subsequent pages. The demonstration video I recorded is displayed in the Product page showing the detailed features and functions of Spud.

Build

The build of Spud is completed. It can now detect when someone is physically moving close in two distances and react accordingly and listen to voice commands for tricks. I also added two button with LEDs for the users to change between the two modes: friendly and alert. However, I had no time to implement the volume detection for loud noises in the alert mode and had to simulate it with a button due to limited time as I still have to finish my portfolio website.

User Testing

I conducted user testing with my housemates and friends again with the this last version of Spud. I first observed them taking turns experiencing Spud in both the user's perspective and the person approaching the user's perspective. I then interviewed them by asking related questions about the project objectives and success criteria that I had written down in the prototype document.

The results determined that Spud is better in the friendly mode than the alert mode. Spud is not intimidating enough to make people leave the user alone in the alert mode and instead looks cute when its angry. However, the scared expression worked better that the angry expression and the stop gesture as most of the participants felt bad that they made it scared. I am still happy with how Spud turned out to be as it is well liked even with its imperfections and had fun seeing my friends' interactions with Spud as everyone had interesting and amusing reactions.

Work to be done

I still have to complete my portfolio by adding more content, taking and adding more pictures and formatting it so it does not look like a document or a report.

I also have to clean up my messy table in my room (covered with scraps and unused parts for Spud) and set up Spud and test out the webcam with Discord before the exhibit.

I look forward to the exhibit on Wednesday. Hopefully I get everything done by that day and Spud works as intended without any mistakes.

Exhibition!

Uh oh.

Arduino IDE crashed on the day of the exhibition! There was an update that caused the app to crash as soon as it opened. It took hours to figure out that you had to delete some hidden files to resolve this issue. At this stage the exhibition was about to begin and the SassMobile was still in separate parts... Due to the ongoing power issues still we couldn't get all the components working together. So we managed to simulate the functionality of the robot by showing each part separately. I had to manually play a sound when appropriate, while the SassMobile roamed around and flashed its lights. he IR emitter didn't work and so we just had to explain what was meant to happen. Despite these setbacks, most of the presentations went really well.

I ran Steven (DECO lecturer) through the interation process and explained the concept. He found the idea quite interesting and especially enjoyed the audio :D His one query was "What if you're watching some important news and want the SassMobile to leave you alone?" The answer being that you should throw a towel over its eye to blind it and handicap its ability to hack. Although it can still yell at you...

Overall the exhibition was a success :) Seeing everyone's projects was really interesting. My favourites were the Sound Mixer Lab, Pose Elevator and the Memory Ball.

It was great to talk to Dimitri, Seamus, Alistair and Ansuan on discord. We had some laughs about our projects and a good catch up in general. I wish I saw more of them this semester :( Sad thing is that most of them graduate this semester. We definitely need to meet up outside of uni.

Exhibition

This Wednesday is the time of the online exhibition. We booked a library room with a projector as the display location in advance. We entered the site for one hour in advance to debug the project to ensure that everything worked properly. It was our advance inspection that found some unstable connections and made corresponding adjustments. The following pictures are the photos in the exhibition.

The entire exhibition lasted for three hours, during which we accepted the review of the classmates and the teaching team. We also communicated with other groups through discord during this period. On the whole, the online demo was smooth, and the idea was well communicated.

Live video demonstration

Overall reflection

Overall, I am satisfied with the final result of our group. Despite being affected by the epidemic, learning and life have become less convenient than usual, our team still completed the project as much as possible and carried out the final improvement before the final exhibition, and in the final online exhibition, the students and teachers who watched our products in our live broadcast room also gave positive comments. I will complete my reflection on the following aspects.

'Actual' to'Ideal'

Our team develops the project in the form of teamwork. Although we have realized the main functions, the actual product still deviates from the expected effect. Most of the deviations are mainly due to technical reasons, as well as the emergence of uncontrollable factors caused by the impact of the epidemic. Through three main iterations, we have designed a device that allows users to better interact with daily sounds, and also completed the realization of the main functions. Reviewing the overall design ideas, our products still have the following defects:

1. In our original plan, the dropper was used as a radio device. It is wire-free, and users can take it to record their favorite sounds anywhere. Although we have connected an extension cord, it can be used in a limited range, but it still cannot provide users with a more intelligent experience. Similarly, due to the installation of many sensors, there are still many wires connected to the rest of the project. Although we tried many ways to hide the wires, it was still unavoidable that the nakedness caused the visual unsightly. How to beautify products to a greater extent, as well as intelligent products will be the goal and improvement direction of future research.
2. Unity has not successfully connected with Arduino and python. In the original plan, we planned to create visual effects for the sound playback, but unfortunately, we have no way to connect it with python and Arduino. Although the generated visual effects are very cool, at present, it is only possible to display the visual effects by manually inputting the path. Overcoming this technical barrier is also the direction of future efforts.
3. I did not find a suitable sensor to make the sound deletion function. In our original plan, there was the concept of a trash can. As soon as the instrument is dumped over the trash can, the sound will be deleted. At present, this part does not think of a suitable technical implementation method but uses other methods to achieve the deletion effect of music.
4. The product does not completely get rid of the computer and achieves a highly intelligent effect. In our original plan to use the raspberry pi to replace the computer, and because of time constraints and limited materials on hand, we still use the computer to complete the relevant functions.

Relevance to course

The course theme is mainly about novel physical interactions. Our products are highly consistent with the course theme. Unlike traditional interaction methods (such as keyboard and mouse interaction), our products simulate the operation of the chemical laboratory, providing users with a variety of novel interaction forms such as squeezing, touching, dumping, shaking, etc. In order to give users a better operating experience.

Relevance to studio

Studio theme refers to an interesting, highly playable, innovative feature that can produce music for daily life. Our products meet the above requirements.

First, the combination of music synthesis and chemistry experiments is a very new concept. The user also showed his curiosity when he saw the device. The design idea is very attractive.

Second, our project is very close to life. In an ideal situation, people can take a dropper to collect various sounds in daily life, and take them home to interact with daily sounds. The place where the sound is mixed can also be more commonplace in daily life such as homes, companies, etc. The visual effects and auditory effects produced by playing sounds can play a good role in soothing emotions.

Third, the product is highly playable and interactive, and also gives users a high degree of freedom. Users can not only record any sound they like but also mix their favorite music clips. When using the device, the user can not only interact with the device but also enjoy a collaborative experience with friends.

Reflect on Human value

As an interactive device for daily sound processing, the "Sound lab" can make music more into people's lives. Users can alleviate the tension and anxiety of life through the interaction with daily sounds. The product is simple to operate and has a high degree of freedom. Users can "mix" their own DIY music according to their favorite, which adds a lot of fun to the daily boring life. From the feedback of user tests, they believe that the device can greatly reduce the stress of life, and they are more attentive to the voice of life.

Relect on User experience

In the stage of final product display, users can quickly know the concept of our project through the introduction, and can quickly get started. They think that the method of using a combination of visual effects and sound is very novel, and it is very unique to use daily sound as an entry point. Users think that in this way, everyone will pay more attention to the surrounding life and also play a good role in relieving stress. In addition, users can complete the mixing operation without instructions. They also appreciated the different visual effects we designed. I can think of our products as suitable for user experience.

Thanks

Due to the impact of the epidemic, this semester is not easy. Thanks to the teaching team for our entire project.

This is my last journal. If I graduate successfully, the completion of this project will also draw a perfect end to my student career. Although there are doubts, uncertainties, and stagnation in the completion process, thank you for my team members and everyone, we finally completed our original idea!

Exhibition

This Wednesday is the time of the online exhibition. We booked a library room with a projector as the display location in advance. We entered the site for one hour in advance to debug the project to ensure that everything worked properly. It was our advance inspection that found some unstable connections and made corresponding adjustments. The following pictures are the photos in the exhibition.

The entire exhibition lasted for three hours, during which we accepted the review of the classmates and the teaching team. We also communicated with other groups through discord during this period. On the whole, the online demo was smooth, and the idea was well communicated.

Live video demonstration

Overall reflection

Overall, I am satisfied with the final result of our group. Despite being affected by the epidemic, learning and life have become less convenient than usual, our team still completed the project as much as possible and carried out the final improvement before the final exhibition, and in the final online exhibition, the students and teachers who watched our products in our live broadcast room also gave positive comments. I will complete my reflection on the following aspects.

'Actual' to'Ideal'

Our team develops the project in the form of teamwork. Although we have realized the main functions, the actual product still deviates from the expected effect. Most of the deviations are mainly due to technical reasons, as well as the emergence of uncontrollable factors caused by the impact of the epidemic. Through three main iterations, we have designed a device that allows users to better interact with daily sounds, and also completed the realization of the main functions. Reviewing the overall design ideas, our products still have the following defects:

1. In our original plan, the dropper was used as a radio device. It is wire-free, and users can take it to record their favorite sounds anywhere. Although we have connected an extension cord, it can be used in a limited range, but it still cannot provide users with a more intelligent experience. Similarly, due to the installation of many sensors, there are still many wires connected to the rest of the project. Although we tried many ways to hide the wires, it was still unavoidable that the nakedness caused the visual unsightly. How to beautify products to a greater extent, as well as intelligent products will be the goal and improvement direction of future research.
2. Unity has not successfully connected with Arduino and python. In the original plan, we planned to create visual effects for the sound playback, but unfortunately, we have no way to connect it with python and Arduino. Although the generated visual effects are very cool, at present, it is only possible to display the visual effects by manually inputting the path. Overcoming this technical barrier is also the direction of future efforts.
3. I did not find a suitable sensor to make the sound deletion function. In our original plan, there was the concept of a trash can. As soon as the instrument is dumped over the trash can, the sound will be deleted. At present, this part does not think of a suitable technical implementation method but uses other methods to achieve the deletion effect of music.
4. The product does not completely get rid of the computer and achieves a highly intelligent effect. In our original plan to use the raspberry pi to replace the computer, and because of time constraints and limited materials on hand, we still use the computer to complete the relevant functions.

Relevance to course

The course theme is mainly about novel physical interactions. Our products are highly consistent with the course theme. Unlike traditional interaction methods (such as keyboard and mouse interaction), our products simulate the operation of the chemical laboratory, providing users with a variety of novel interaction forms such as squeezing, touching, dumping, shaking, etc. In order to give users a better operating experience.

Relevance to studio

Studio theme refers to an interesting, highly playable, innovative feature that can produce music for daily life. Our products meet the above requirements.

First, the combination of music synthesis and chemistry experiments is a very new concept. The user also showed his curiosity when he saw the device. The design idea is very attractive.

Second, our project is very close to life. In an ideal situation, people can take a dropper to collect various sounds in daily life, and take them home to interact with daily sounds. The place where the sound is mixed can also be more commonplace in daily life such as homes, companies, etc. The visual effects and auditory effects produced by playing sounds can play a good role in soothing emotions.

Third, the product is highly playable and interactive, and also gives users a high degree of freedom. Users can not only record any sound they like but also mix their favorite music clips. When using the device, the user can not only interact with the device but also enjoy a collaborative experience with friends.

Reflect on Human value

As an interactive device for daily sound processing, the "Sound lab" can make music more into people's lives. Users can alleviate the tension and anxiety of life through the interaction with daily sounds. The product is simple to operate and has a high degree of freedom. Users can "mix" their own DIY music according to their favorite, which adds a lot of fun to the daily boring life. From the feedback of user tests, they believe that the device can greatly reduce the stress of life, and they are more attentive to the voice of life.

Relect on User experience

In the stage of final product display, users can quickly know the concept of our project through the introduction, and can quickly get started. They think that the method of using a combination of visual effects and sound is very novel, and it is very unique to use daily sound as an entry point. Users think that in this way, everyone will pay more attention to the surrounding life and also play a good role in relieving stress. In addition, users can complete the mixing operation without instructions. They also appreciated the different visual effects we designed. I can think of our products as suitable for user experience.

Thanks

Due to the impact of the epidemic, this semester is not easy. Thanks to the teaching team for our entire project.

This is my last journal. If I graduate successfully, the completion of this project will also draw a perfect end to my student career. Although there are doubts, uncertainties, and stagnation in the completion process, thank you for my team members and everyone, we finally completed our original idea!

Exhibition

This Wednesday is the time of the online exhibition. We booked a library room with a projector as the display location in advance. We entered the site for one hour in advance to debug the project to ensure that everything worked properly. It was our advance inspection that found some unstable connections and made corresponding adjustments. The following pictures are the photos in the exhibition.

The entire exhibition lasted for three hours, during which we accepted the review of the classmates and the teaching team. We also communicated with other groups through discord during this period. On the whole, the online demo was smooth, and the idea was well communicated.

Live video demonstration

Overall reflection

Overall, I am satisfied with the final result of our group. Despite being affected by the epidemic, learning and life have become less convenient than usual, our team still completed the project as much as possible and carried out the final improvement before the final exhibition, and in the final online exhibition, the students and teachers who watched our products in our live broadcast room also gave positive comments. I will complete my reflection on the following aspects.

'Actual' to'Ideal'

Our team develops the project in the form of teamwork. Although we have realized the main functions, the actual product still deviates from the expected effect. Most of the deviations are mainly due to technical reasons, as well as the emergence of uncontrollable factors caused by the impact of the epidemic. Through three main iterations, we have designed a device that allows users to better interact with daily sounds, and also completed the realization of the main functions. Reviewing the overall design ideas, our products still have the following defects:

1. In our original plan, the dropper was used as a radio device. It is wire-free, and users can take it to record their favorite sounds anywhere. Although we have connected an extension cord, it can be used in a limited range, but it still cannot provide users with a more intelligent experience. Similarly, due to the installation of many sensors, there are still many wires connected to the rest of the project. Although we tried many ways to hide the wires, it was still unavoidable that the nakedness caused the visual unsightly. How to beautify products to a greater extent, as well as intelligent products will be the goal and improvement direction of future research.
2. Unity has not successfully connected with Arduino and python. In the original plan, we planned to create visual effects for the sound playback, but unfortunately, we have no way to connect it with python and Arduino. Although the generated visual effects are very cool, at present, it is only possible to display the visual effects by manually inputting the path. Overcoming this technical barrier is also the direction of future efforts.
3. I did not find a suitable sensor to make the sound deletion function. In our original plan, there was the concept of a trash can. As soon as the instrument is dumped over the trash can, the sound will be deleted. At present, this part does not think of a suitable technical implementation method but uses other methods to achieve the deletion effect of music.
4. The product does not completely get rid of the computer and achieves a highly intelligent effect. In our original plan to use the raspberry pi to replace the computer, and because of time constraints and limited materials on hand, we still use the computer to complete the relevant functions.

Relevance to course

The course theme is mainly about novel physical interactions. Our products are highly consistent with the course theme. Unlike traditional interaction methods (such as keyboard and mouse interaction), our products simulate the operation of the chemical laboratory, providing users with a variety of novel interaction forms such as squeezing, touching, dumping, shaking, etc. In order to give users a better operating experience.

Relevance to studio

Studio theme refers to an interesting, highly playable, innovative feature that can produce music for daily life. Our products meet the above requirements.

First, the combination of music synthesis and chemistry experiments is a very new concept. The user also showed his curiosity when he saw the device. The design idea is very attractive.

Second, our project is very close to life. In an ideal situation, people can take a dropper to collect various sounds in daily life, and take them home to interact with daily sounds. The place where the sound is mixed can also be more commonplace in daily life such as homes, companies, etc. The visual effects and auditory effects produced by playing sounds can play a good role in soothing emotions.

Third, the product is highly playable and interactive, and also gives users a high degree of freedom. Users can not only record any sound they like but also mix their favorite music clips. When using the device, the user can not only interact with the device but also enjoy a collaborative experience with friends.

Reflect on Human value

As an interactive device for daily sound processing, the "Sound lab" can make music more into people's lives. Users can alleviate the tension and anxiety of life through the interaction with daily sounds. The product is simple to operate and has a high degree of freedom. Users can "mix" their own DIY music according to their favorite, which adds a lot of fun to the daily boring life. From the feedback of user tests, they believe that the device can greatly reduce the stress of life, and they are more attentive to the voice of life.

Relect on User experience

In the stage of final product display, users can quickly know the concept of our project through the introduction, and can quickly get started. They think that the method of using a combination of visual effects and sound is very novel, and it is very unique to use daily sound as an entry point. Users think that in this way, everyone will pay more attention to the surrounding life and also play a good role in relieving stress. In addition, users can complete the mixing operation without instructions. They also appreciated the different visual effects we designed. I can think of our products as suitable for user experience.

Thanks

Due to the impact of the epidemic, this semester is not easy. Thanks to the teaching team for our entire project.

This is my last journal. If I graduate successfully, the completion of this project will also draw a perfect end to my student career. Although there are doubts, uncertainties, and stagnation in the completion process, thank you for my team members and everyone, we finally completed our original idea!

SassBot Construction

Great to see the boys again! It'd been about a couple months since seeing each other in person so it was nice to catch up and have a closer look at the progress of each component. Anshuman had been working on the main body of the robot. His eye's light sensor, ear vibration and mouth lights were all working well. His latest feature was a nose knob that could adjust the mouth lights. Tim showed off the IR emitter Tv controller which was cool. I showed the boys how specific sounds could be played and output settings adjusted such as volume. Combing each part proved to be harder than anticipated.

We started by adding my audio component to Anshuman's Arduino board. Despite wiring up perfectly, the board had problems registering the DFMiniPlayer. We spent a while reinstalling libraries and adjusting the wiring until we eventually got it working. We never pinned down what the issue was since the code would randomly throw errors. We decided that soldering these components together may be necessary to avoid faulty circuits. Nonetheless, during the time that the DFMiniplayer did work, I added some code to anshuman's program to make a happy sound play when the robot smiles, and a sassy comment when it's angry. Here's an example of this working:

Integrating Tim's component proved to be even harder since he ran his program using an ESP32 board. We managed to connect the two boards, but we couldn't get any sensors or components to work. After seeking the help of the tutors, we narrowed this down to not providing enough power to the boards. This way as far as we got on the Sassbot - solid progress.

To finish off the session, Tim showed us his progress on getting facial recognition working. The idea was to get the Roomba to move towards and face the user by connecting his phone camera to the Arduino using bluetooth. It occasionally worked... but it was fun playing around with it anyway.

Reflection

Spending the day with Anshuman and Tim was a lot of fun. It was good to get our hands dirty trying to integrate our parts. Debugging with friends is always a good time. We made some solid progress putting together the Sassbot, it's finally taking shape. In the coming weeks we'll aim to get everything working cohesively to begin designing some interaction plans.

studio session

This week is the last week of the studio session, and the time in this semester seems to be passing by very fast, and I have only one week left to work on our project and show it online. The last week of the studio focused more on the reflective of our own projects and plans for online exhibitions. In the report back session, I shared with my classmates about the recent changes in life caused by the relaxation of the ban. Due to the relaxation of the current social ban, my team members and I will go to the UQ innovate to do some group activities about soldering and project assembly.

In the report back stage, we also discussed how to change the project if I have magic. This can't help me to reflect on the whole project of our team. The most needed change in our group project is to make the entire device wire-free. The wire-free here not only refers to removing the messy jumper wire connecting each sensor, but the most important thing is also the design of the dropper part, our ideal is to use the dropper to record any sound users like, that is to say, users can take the dropper to everywhere for sound collection. Although we used extension cords in the design of the project to expand the range of recorded sounds, the sound recording will still be limited because the extension cords are fixed. If there is no wire restriction, the product will be more popular with users. Another thing I hope to solve with magic is the visual effects part of my design. Despite trying many methods, but still can not connect Arduino with python and unity. This caused that in the future exhibition, we need to manually enter the path to complete the display of the visual part.

Since the online exhibition is relatively unknown, our group conducted a group meeting on how to present the project after the report back. Our main concern is how to better communicate our design ideas through online platforms and how to allow users to interact with our products. For this reason, I also seek help from the teaching team in class, based on the discussion results, we decided to act as a ‘robot’ in the online exhibition, that is, we only give a brief introduction to the project, all other operations will be given by the user, maximize the distance between users and products. We all looking forward to next week's online exhibition...

project progress

This week is very busy because it is at the end of the semester, not only do I have to prepare for the exhibition of the next week, but also I will have 4000-word report to submit. This makes me feel overloaded. Fortunately, the main functions of our entire project have been basically completed. Our main work this week is to integrate various parts and think about how to achieve the visual-pleasing design.

Since the chemical equipment toys we bought for the project before are very small in size, we can't even have spaces for the installation of all sensors, especially vibration sensors, so we purchased large glass laboratory equipment instead. However, although the large glass-based experimental equipment can put all the sensors inside, because these types of equipment are transparent, the sensors installed in the equipment can be seen clearly, which is not visually beautiful, so I tried to use white acrylic to paint on the equipment. On the one hand, let all equipment reduce transparency to hide the sensors and wires that were originally exposed. On the other hand, make the lighting effect on the equipment more obvious. The colored equipment can well hide sensors and messy jumper wires and can better display the led inside, but we found that the colored equipment is a bit different from the chemical equipment. To this end, we use a marker to draw a symbolic mark, so that it is visually more like a set of laboratory equipment.

At the same time, I made a test tube rack according to the size of the test tube. I chose a larger cardboard box in order to hide the development board and breadboard in order to prevent users from seeing the messy wires to achieve a good visual effect. The first two pictures are the status when we have just integrated our respective parts together. Due to the installation of many sensors, the line is particularly messy. We use electric tape to wrap up the messy wires of different parts and hide the excess wires in the carton. The whole work becomes neat.

Major changes in my group members' respective parts

Iris is responsible for designing the dropper part in our whole project. The main change in her part of the final delivery project is the design of the dropper. In order to better hide the switch and achieve a good "squeeze" interactive effect, we are replacing the material of the dropper. The picture below is what the final dropper looks like.

The role of dropper: sound recording

The function currently implemented: squeeze the plastic head and place the dropper close to the sound source to record.

Rhea is responsible for designing the sound mixing part in the group. The main change in her part of the final delivery project is the design of the visual effect of the sound mixing as well as adding the function to interact with friends. The interactive device in the picture below is composed of two handles. The user can invite his friends to comment on the remix he made. Shaking the smiling face means like, the system will automatically play a clapping sound, shaking the crying face means dislike, the system will play a sad sound effect and then the mix will be deleted automatically. In the process of mixing, close the lid and playback will stop automatically....

Overall Product after integration

Portfolio

At present, I have completed the text part and updated the corresponding pictures and text to my portfolio. My portfolio can be found at UQ Interaction Design Exhibit 2020. URL is as follows:

Sophia Jin's portfolio link

prepare for the exhibition

Next Wednesday is the online exhibition, we will conduct another project integration and confirm the presentation form next week.

We are going to shoot live video and conduct a group test to confirm that the project can be carried out smoothly.

Prototype Preparation

I spent this week preparing my audio output component for the group's assembly of the full SassMobile next week. Since I have the parts available to build a second prototype, I made a replica prototype to give to Anshuman or Steve to permanently integrate the audio component into the SassMobile. I also had to reorganise the audio files on the SD card into folders for when specific sounds are targeted. I had issues doing this because there were hidden index files on the drive that caused the code to play a flat sound despite a sound being targeted.

Here is a code snippet of the sound organisation:

// phase 1

  myDFPlayer.volume(15);  // Set volume value. From 0 to 30

  

  myDFPlayer.playFolder(1, 1);  // how good is tv

  delay(5000); // 5s

  myDFPlayer.playFolder(1, 2);  // are you enjoying this show?

  delay(10000); // 10s

  myDFPlayer.playFolder(1, 3);  // I think you've watched enough now

  myDFPlayer.playFolder(1, 4);  // it's a beautiful day outside, not that you'd know

  

  myDFPlayer.volume(25);  // loud

  myDFPlayer.playFolder(1, 5);  // yaaaayyyyy

  

  // phase 2

  myDFPlayer.volume(15); 

  // 1. *Robot Changes channel*

  myDFPlayer.playFolder(2, 6);  //  i. "I'm bored of this channel"

  myDFPlayer.playFolder(2, 7);  //  ii. "Are you sitting on the remote?"

  

  // 2. *Robot Turns down/mutes volume*

  myDFPlayer.playFolder(2, 8);  //  i. "I have sensitive hearing"

  myDFPlayer.playFolder(2, 9);  // ii. "Was that a profanity?"

  

  myDFPlayer.volume(20); 

  // 3. *Robot changes language*

  myDFPlayer.playFolder(2, 10);  // i. "You know Mandarin right?"

  myDFPlayer.playFolder(2, 11);  // ii. "Now it's more educational"

  myDFPlayer.volume(25); 

  // 4. *User throws towel over robot* (light sensor)

  myDFPlayer.playFolder(2, 12);  // i. "Ahhh I'm blind!"

  myDFPlayer.playFolder(2, 13);  // ii. "this isn't funny"

  myDFPlayer.volume(20); 

  // 5. *User unblocks view* (light sensor)

  myDFPlayer.playFolder(2, 14);  // i. "Thanks a lot"

  myDFPlayer.playFolder(2, 15);  // ii. "I missed seeing your pretty face"

  

  //Phase 3

  myDFPlayer.volume(20); 

  //1. *Robot Turns off TV*

  myDFPlayer.playFolder(3, 16);  //i. "Time to go do something else" #3RR0R

  myDFPlayer.playFolder(3, 17);  //ii. "I think your TV is broken"

  myDFPlayer.volume(25); 

  //2. *User throws something at it* (vibration sensor)

  myDFPlayer.playFolder(3, 18);  //i. "Ahh!"

  myDFPlayer.playFolder(3, 19);  //i. "Ouch!"

  myDFPlayer.playFolder(3, 20);  //ii. "stop hitting me you loafer"

  myDFPlayer.volume(25); 

  //3. *Pull ears*

  myDFPlayer.playFolder(3, 21);  //i. "Ow stop it!"

  myDFPlayer.playFolder(3, 22);  //ii. "I have sensitive ears"

  

  //4. *Robot puts volume to max*

  myDFPlayer.playFolder(3, 23);  //i. "How do you like that?"

  myDFPlayer.playFolder(3, 24);  //ii. "Need subtitles now?"

  myDFPlayer.playFolder(3, 25);  //iii. "I lied about having sensitive hearing"

Reflection

It's been a couple of weeks since the any changes have been made to the concept. We've been focusing on preparation for building the SassMobile. I'm sure we'll run into troubles so it's important that I could do as much as possible on my component. My aim is to make the robot's voice and mannerisms as emotional as possible, so there are few thoughts I've had about how to do more. Making the volume dynamic in accordance to how angry the robot is ie. very loud when it's yelling. It's also occurred to me that the robot has no positive feedbacks that would be triggered when the robot is pleased with the user. I'll have to ask Steven to record some more audio... Looking forward to building the Sassbot next week!

This week I mainly improved my part of work (test-tube part) based on user feedback from last week combined with my research on sound visualization. At the same time, I set a framework for my portfolio and started to conceive the components of the portfolio.

Studio session

In the online studio stage, Lorna gave us a brief introduction to the requirements of the portfolio, this inspires me in terms of the design of the concept section of my portfolio in the future. During the studio session, I realized that there are only two weeks away from the exhibition. The time is relatively tight. I must arrange the time reasonably in the next two weeks in order to have a good user experience in the online exhibition.

At the moment I working on using unity to make visual effects of playing sounds. In the previous week of the exhibition, our team plan to work together to beautify the previous prototype, and reorganize the parts to ensure that each part can be used normally. Although the time is a bit tight, we still need to work hard to guarantee the project to be completed step by step as planned.

Due to the impact of the epidemic, this year's exhibition was changed to online, and we will use discord as a tool to interact with other groups and visitors online. In this week's studio, we are also familiar with the use of this software to prepare for the subsequent online exhibition.

Project progress

This week our team decided to improve our own parts, and we will carry out unified project integration next week.

This week, I referred to the tutorial video I found last week about using unity to visualize the sound and made a small program to increase the visual effect of the sound playback part of the group project.

Through this week's production, the unity program can generate different waveform diagrams according to different audios of different sounds.

The visual effect achieved so far is shown below:

Visual effects when no sound is playing:

Visual effects when playing low-frequency music:

Visual effects when playing high-frequency music:

This week I tried to connect this unity project with our overall project (that is link unity, Arduino and python together) to add this visual effect to the delivery of the entire project, but due to technical limitation, the complete connection has not been achieved. Through the discussion with the teammates, we decided to retain the main function of the overall project, consider whether it is possible to present this visual effect from other manual methods. As an alternative to displaying this part, I designed an input box on the interface for manually inputting the path of the sound to be played. If the technical barriers are not overcome in the next few weeks, we will present this visual effect separately by manually filling in the audio path.

I will also briefly describe the implementation process here:

This effect uses a plug-in: audiovisualizer, which allows users to produce different visual effects based on different sound files.

The input box of this interface is used to accept user input, store the input string as a path, click the button to call its bound event, play audio, and generate visual effects. The input detection uses the inoutfile component to monitor the editing events, and when the input is completed, the string is obtained and stored in the path.

The following figures are some screenshots for the code parts for the unity project.

Portfolio

Through this week's introduction in the studio session, I plan to start thinking about the components of my portfolio and the parts I want to cover in my portfolio. I plan to refer to excellent open-source portfolio templates and make changes according to the needs of my content.

My intention is to make a one-page responsive portfolio that can adjust the different ratios of content according to the different platforms visited. When I browsed through last year’s portfolio, I found that the pleasing portfolio is often using a combination of pictures, text, and video to introduce the product and convey the ideas. Although pure text can express the design ideas clearly, with the help of pictures, videos can make the content more intuitive and won't feel boring while. This week I confirmed the sections displayed in my portfolio and made a determination of the overall framework.

Next, I need to fill the content and finally make a small modification about the overall layout according to the content and can complete the production of the personal portfolio.

The logo

Future Plan

1. Work as a team to connect the entire project together
2. Focus on considerations related to aesthetic design
3. Start with the text part of the portfolio

This week was predominantly dedicated to last minute improvements of my prototype and portfolio completion. Over the weekend, I finished my portfolio development and wrote the majority of my content for the portfolio. I categorized the content according to the criteria with pages for the "product", "process", "reflection" and "outputs". So far I have detailed the 'actual' and 'ideal' versions of the prototype through text explanation and I aim to make videos to demonstrate the form, functionality and interaction later in the week. In the process section, I explained my process through the major milestones of the project and highlighted key user engagement and how user feedback guided the design.

Detailing the 'ideal' and 'actual' prototype forms in my portfolio prompted me to really reflect on the comparison between the two forms. I created a visual representation of the 'ideal' concept to include in my portfolio to make it easier for viewers to understand the intent of my concept.

I have felt that my prototype doesn't look very 'professional' or neat, especially when comparing my prototype to other peers work. After reflecting on the comparison, I actually feel that I did quite well in building a concept that embodied all of the relevant themes and brought about all the desired experience outcomes. My main focus of cultivating open-ended experience and creative freedom is definitely evident in the prototype design, with no real constraints to use at all.

Build:

With the build almost completed, this week I aimed to improve the prototype and solve any issues with the prototype. An issue was evident with the reading of the Arduino byte data in Unity where different values were being printed than expected and resulting in incorrect colours being displayed. After talking to Ben it became clear that the issue was related to double-digit messages being sent which could not be read using readByte in Unity. Therefore, each Serial.Write message in Arduino needed to be changed to a single byte. To do so, instead of double-digit numbers, single alphabetic characters were used. This revealed another issue regarding the bytes being translated to numbers based on the ASCII codes.

Therefore, I changed my ChangeColour function to instead read the translated numbers for each byte message. This significantly improved the functionality of my prototype which enabled me to better depict the intended concept and its use.

As discussed in the last journal post, for the selection feature, I utilised the OnMouseOver function to determine the "selectedGameObject" for the material colour changes to be applied to. Whilst talking to Steven in the workshop, he also recommended that I use raycasting to detect which element is hovered over and helped me to implement this.

These two changes were really beneficial to the general flow of user experience and decreased the amount of lag and glitches experienced when using the prototype. With these final changes, my prototype is now completed to the intended form/functionality for this deliverable. If I have any spare time after completing my portfolio I may try and make further improvements, however, the current form is able to bring about the experience outcomes as desired.

Testing:

Now that I have completed the prototype, I aim to test it so that I can make any necessary changes in the next few days. These changes, if any, should only be minor one e.g. a wrong colour entered for a colour pad or combination. As the previous prototype was tested extensively, Twisted shouldn’t require large changes from a conceptual perspective. I decided to ask my cousins to test the prototype as they are 4 and 6, ages very close to the target audience for the concept. With relaxed social distancing rules, they were actually able to come over and test the prototype in person which was great and allowed me to receive much better feedback and easily record observations.

I started by giving them a very brief explanation of the concept e.g. "you can use the mouse to draw and then when you want to colour in one of the shapes, you move the mouse over it and then step on the colour you want it to be. You can also make new colour by stepping on more than one of the colour pads at a time". I gave minimal instructions to observe how they would use Twisted and to see if they would 'teach themselves' through exploration. This approach produced the results I hoped, as my cousins eventually discovered every possible interaction and colour mixing. Aly, 4, said she wanted to make pink, and successfully did so by standing on pink and red at the same time. This showed to me that I successfully designed the prototype to draw upon users innate skills. Hopefully this would be the same for others in the target audience, however, more testers would be needed to see if these participants are actually representative of the broader audience.

Once my cousins had become familiar with how to use the prototype, I gave them a few specific tasks to complete:

1) Outline the circle in the top corner

2) Colour the circle in orange

3) Colour the tree in green

4) Draw your own image on the canvas

Both of the testers were able to complete the tasks without hesitation and actually completed them together collaboratively, also demonstrating that the objective of facilitating collaboration was achieved. To create the green and orange, they worked together by each stepping on a single colour pad. Whilst they didn’t get the correct combination for green straight away, they tested different combinations and finally came to the correct combination of yellow and blue. This also demonstrated that the objective of teaching colour theory (primary colours that mix to create other colours) was achieved. Overall, the testing session went really well and only one minor issue was found where the mixing of blue and black was set to the wrong colour.

Reflection:

Getting the prototype completed, tested and revised this week was a big relief as I still have until next Wednesday to finalise any elements. As I couldn’t get access to a large touchscreen for the prototype, I am having to use my computer screen. Although it doesn’t allow for touch interaction, my cousins who tested it still seemed to enjoy the interaction and I feel that it doesn’t negatively impact the form too much. I am pretty happy with the final prototype and look forward to presenting it at the exhibit next week.

It was also really great this week to be able to observe target users actually interacting with the prototype as it proved lots of the findings discovered through team and individual research. The testers showed active collaboration, engagement with the concept and clear communication when problem solving, all behaviours that were expected due to research and aided in the learning of colour theory. This not only supports the strong relation of my concept to our team domain, but to the broader studio context as well.

Over the next week, my main goals are to complete videos for my portfolio, finish my portfolio content, fix any development issues with my portfolio and get started on my section of the team report.

I use this week to build the prototype as well. I have visited the in- campus workshop to solve the problem that I have faced last week by consulting the tutor. The first problem that I have solved is that the servo keeps shaking and can be used without the flex sensor. The tutor said that the code is correct, the problem is that the system is very sensitive, that the servo change ever second base on the mini change of the data of the flex sensor and the ultrasonic sensor. To make it stable, I need to observe the live value when I bending the flex sensor to find the average value and try to set the value around the average value to see which one can make the system more stable. The code change from the left picture below to the right one.

The second problem is that the LED strip cannot turn off by the ultrasonic sensor then automatically turn on with different light effects in a few seconds later. To solve the problem, the count of the loop running should be recorded. The tutor has given me the suggestion that I can learn from the sample in the Arduino IDE, which called “state change detection”, it is a code that presents using the button to control the led base on the times of the button been pushed. It is similar to what I need.

Then, I have learned how to making the function that two buttons keep pressing to make the LCD work as a timer, if only one button keeps pressing, the LCD timer will stop. This function present when user put their phone into the box and sit on the seat the time will be counted as their study time, if the user takes out their phone or leave the seat, that means the user does not sit for study, so the time should not be count as study time, the timer will stop. The code here is similar to the sample code “state change detection” as well.

Week13

Exhibtion Days

Everything has been done and team CDI is ready for the exhibition day. To be honest, the first hearing doing an exhibition on discord this is weird. However, after introduced the prototype to first guest it just the same as a normal exhibition as another course.

Several feedbacks have been received about the demonstration physical interaction element are not enough as there is a better way to show to the guest. Beside of that the about greeting the guest I can do it much better.

An overall fun day to explore people works and doing a demonstration to guest

Week 12

Group Work / Exhibition Preapation

After the audio activation have been implementing into the prototype, I have been touch the prototype anymore. Most of the time of this week I have been working at the portfolio of the website. There are 5 topics going to discuss in the portfolio. Product ideal, showcase video, technical spec, design process & blog and the team members information. During the coding phase, I have also looked at other teams' portfolio website in UQ zone to reference them the style and information. This gives me reflection that am I adding enough content and proper style of my portfolio? Therefore it helps me to improve my website any time.

Prototype Broke:

Few days before the exhibition the post estimation features are completely broke and unusable. I have to spend a lot of time still I could not make it work. Therefore I have to look for another way to simulate how the posenet works. After the research, I have found that google learning machine can do this. TherefORE I quickly change my code into google learning machine standard and make it into image identification. Surprisluy it works smoothly than posenet and neural network.

After the changes, the prototype is functional I have to think about what I have done wrong during the development. I haven't considered what if the prototype is unusable at the exhibition day, as I don't have any backup plan for this kind of issues? Beside of that after the audio activation activates feature I haven't opened my prototype anymore. I should check my product every day to make sure there is no bug at all instead of leaving it behind. A huge mistake has been made luckily the issue has been discovered early.

Week 11

Group Work / Idea Changes

On this week my work is to keep going with the prototype building which makes the pose identification more stable to use. However I still struggle with it, There are few guests have been making as make be the neural network are not working or the train data are not implemented properly. Beside of the making the prototype more stable more feature have been considering after the critique, one of the comment said the post is quite easy to perform and there might be some user accidentally perform it and they don’t want to change the screen inside the elevator. Therefore I have been considering to add a feature as the audio activation. Such as when the user says to turn on in front of the camera then the post estimation will turn on after it. This makes the product easier to use.

After research, I have chosen p5.speech js as the model to implement javascript. It provides Web Speech (Synthesis and Recognition) API functionality. It consists of two object classes (p5.Speech and p5.SpeechRec) along with accessor functions to speak and listen for text, change parameters (synthesis voices, recognition models, etc.), and retrieve callbacks from the system.

Website Portfolio:

Have been started yet. but I have to go through deco1400 notes as the HTML CSS coding have to review once because I haven’t coded a website a long time. Therefore I will use bootstrap pre-write CSS to help me to build a reasonable website. Beside of review the basic code I have also watch website that made few semesters before to have a look what has to be include and the style I should follow.

Overall a quick review of coding and website will be starting next wee.

Week10

Group Work / Idea Changes

This week we have to critique other team works inside the course with 3 teams and watching their video introduce their works. Most of them are brilliant and I am very happy to see how other works working through the 10 weeks. Most of the video are well explained and well made. This reflects my video showcase as it is not well explained because there is not much live showcase most of it is words and background. This gives me inspiration on how to make a good showcase video. Besides I have received a lot of comment on more idea which most of them a valuable and adapt them to evaluate the idea. One of the most important questions is what is the poser represents and does it have any meaning? And this is the question I never thought this will have to improve before the exhibition.

Prototype Building and video showcase

My prototype mainly builds on posenet and neutral network. Posenet is used for post estimation and record the body xy asix data. The neutral network will use the data that record to train the data for post matching. This prototype mainly has 3 functions press key blind to record the body data. If you press “f” then the data record later will represent this variable. Another function is to train the data for post matching the last one is post matching with the data yo record . However the post matching system are not stable sometimes I can’t be used. This will improve in the next prototype.

Video Showcase lINK

https://www.youtube.com/watch?v=T_CNnzv4b3U&t=228s

Final tasks & outcomes

---

It was intense in the last two weeks, cause we still left a number of tasks. The map is a huge issues due to the limited material choice. Yet, we need our map to have the features that attract users' attention. Therefore, the ornament should be at least eye-catching. Based on this rule, we chose to full in the blank area with grass and add light color labels on it. It is hard to find real grass, so we used a green towel instead. Cover it over the hardboard turns up a satisfying result.

The irregular edge of the route is not the artifact issues, instead, it leaves the space for the robot to turn itself. And the circles represent one step (one forward command distance).

Final product

• All the detail information is demonstrated in the portfolio website: https://ixd-2020.uqcloud.net/jiang

• Some sketches of pixel-style figures for our project:

• Final concept & demonstration

Enjoy the peaceful moment

---

Cheer for the survival in the final exhibition and salute to all tutors and classmates, especially, to my great teammate Solomon who brings a great contribution to our project. It is a tough semester that every experience is unexpected but precious. Besides, it is definitely a pity that no one can actually interact with our project during the exhibition. Anyhow, for my personal perspective, our team accomplished the majority of the initial goals and successfully explore the robot-based interactions' effect on teaching programming. And it is time for us to enjoy a short enjoyable moment.

Week9

Group Work / Idea Changes

After the research phase of the target audience, the design of the individual project has been started and building the prototype. Before confirming and lock in the concept and I have discussed with my team member about my original idea which user finish a simple puzzle outside of the elevator and change the screen of the flat monitor inside of the elevator. However, during decision and a user acceptance test both feedbacks are the interaction are cause too long time even the puzzle are easy or not. For example, a situation if the user is doing the puzzle and the elevator has arrived on the floor, the user might just give up and go into the elevator. Therefore the concept might not work in the actual world. This reflects me that I did;t not considered enough the actual experience like time issue or social awareness, in this case, I only do what I like to do and this is not proper project development.

Therefore, I have changed my concept into the pose identification to change the screen inside the elevator as it uses less time to operate this interaction and there is common technology between other members works.

Prototype Building

After lockin the concept I have been looking for how can I code the pose identification feature to use in my prototype which I found out there is a pre-train mode call posenet that can be used to estimate the pose of a person in an image or video by estimating where key body joints are. By estimating the key body joints xy asix we can compare it and use it to pose matching to activate something.

This week still working on coding and hopefully, there will be a prototype next week which can use on testing to evaluate it.

Website of the tensor flow about the posenet https://www.tensorflow.org/lite/models/pose_estimation/overview

The final week is finally here and we all made it through ! The exhibition went smoothly and we all managed to display what we have achieve to our visitors. At the start of the exhibition, there were some technical difficulties on discord and we weren’t to communicate with the visitor but we managed to fix this in the end. There were positive feedbacks did boost the morale of the entire team, however, due to the short exhibition time prevented us from getting sufficient feedback from the visitors. Further elaboration of the feedback would also be beneficial for our team. Overall, We would like to thank the people who have taken the time to visit our discord channels and looked at what we have achieved through out the semester.

Week8

ClassWork / Group Work

During this week on the general report, we have report our concept is and how the concept develops between each individual project. My individual project is focused on how children or student user elevator and special changes around them for the idea. After the report back a team meeting has been conducting and discussed how team member going to explore the target audience. This week group work has given me hints or direction which formed have to use as during covid19 restriction. Sometimes doing group work can give you a clue where other tea members were doing and you can ask them how to do it. This method helps you to found the best way to fit your need for the interview or even other works.

User Interview

As above mentioned asking how other member did give me a direction to conduct a video chat interview for understanding what the target audience needs and confirm the user ages group. Because I have to know which age group of children are suitable to use the product. Therefore I have been conducting an interview with my cousin and their friend which ages from 5-10 and 3-5. There are few questions have been asked to them and parents 1: What are your normal activities inside the elevator? 2. What experience do you expect to ride an elevator?3: Do you care how other people see you doing weird action outside the elevator ( social awareness)

Overall the children are expecting an interesting and interactive experience no matter inside or outside the elevator and ages 5-10 children are more stable to use the product. The user interview gives me a clue what the target audience need and expect. This helps me to design my product more easily and accurately target their need. However, there are some I didn’t do well before the interview as I don’t have enough research about the audience first. Therefore the question is not detailed enough. Therefore it should have more preparation before the interview.

For this week, I focused on getting the portfolio done. The prototype that Fardeen and I have been working on for the past few weeks seems to be coming together nicely. We did testing again for our prototype to make sure that it is ready for exhibition day. An individual website portfolio was needed to be created for exhibition day to show case to the visitors all the work that we have achieved thus far. The main challenge was actually coding the website because I was not really good at coding. I decided to use a simple design for my portfolio. In the next week, I aim to get some feedback from the visitors as well as improving my annotated portfolio done by next week. I will also be have a zoom session on next week to discuss on what needs to be done for the team report.