Documentation & Reflection

Journal Week 12

Tianyi Liu - Mon 1 June 2020, 4:52 am

Since we didn't think we could get our larger FSR sensor delivered on time under current situation, last week ,we are focusing on finding alternative for the FSR sensor. The alternative have to meet 2 requirements:

  1. have a detect range larger than FSR(which is 0-6kg, far less than the weight of a human being) that allow our user to stand on it.
  2. the data is accurate and instant.

The tutor suggest us to use Wii Balance Board(WBB) which is a controller like thing for the wii console. We buy a second hand WBB last week and start to study how to make use of it. WBB is just like a weighing scale, it have 4 sensors inside the board, work just like a FSR, a better thing is, it could detect specific four parts of the board, on which the user is currently standing on.

However, we met some knotty problem in gaining the data from the WBB. The WBB was connected to PC through Bluetooth HID, there are a few software(wii-scale) that could read the data from the bluetoot port and display it on the screen. But to read the data from the port relies on several library on C++ which we are not familiar with, we are not able to get those beautiful data from the WBB. Therefore this approach meet a dead end.

So, the next week we will go back to mechenical solutions, to try to use mechenical design to match the detect range of the FSR and the weight of people, and I will also try to use some simple CV to try to read the data from wii-scale software.

Week 10 First prototype reflection and Future work

Tianyi Liu - Sun 17 May 2020, 7:30 pm
Modified: Sun 17 May 2020, 7:31 pm

First Prototype Reflection

Code Review

The code could be divided into 3 parts, codes in Arduino IDE, code to receive data from Arduino, and code to convert and play out as music.

The code in the Arduino UNO is just a simple could to print out the serials data of the FSR.

To output the music notes and chords, it's essential to define music note as a Class in the program:

noteDict = {}               #create a dictionary for music notes

Notes = ["c2","c2s","d2","d2s","e2","f2","f2s","g2","g2s","a2","a2s","b2","c3","c3s","d3","d3s","e3","f3","f3s","g3","g3s","a3","a3s","b3","c4","c4s","d4","d4s","e4","f4","f4s","g4","g4s","a4","a4s","b4","c5"]

startpoint = 48

for i in Notes:

    noteDict[i] = startpoint

    startpoint += 1

The midi signal is formed by several numbers, each music notes was matched with a certain integer, we have middle C(C3) match with 60, and we could easily know the other numbers.

Create class for music notes and chords:

class Note():                   # A class of the music notes


    def __init__(self, notename):

        self._notename = notename

    def get_note_number(self):

        return noteDict[self._notename]

    def play_notes(self, duration, velocity):

        return [self.get_note_number(), duration, velocity]


for i in Notes:             # create instance for each note

    locals()[i] = Note(i)

class Chord():              # A class of chords, which is fromed by a list of notes

    def __init__(self,notes):

        self.notes = notes

        self.base = notes[0]


    def note_included(self, note):

        included = False

        for i in self.notes:

            if i.get_note_number() == note.get_note_number():

                included = True

        return included


    def get_notes(self):

        return self.notes

# Pre defined common chords

chord_C = Chord([c2,g2,e3])

chord_Cmaj7 = Chord([c2,e2,g2,b2])

chord_D7 = Chord([d2,a2,c3])

chord_Dm7 = Chord([d2,f2,a2,c3])

chord_E7 = Chord([e2,g2s,b2,d3])

chord_F = Chord([f2,c3,f3])

chord_G = Chord([g2,b2,d3])

chord_Am = Chord([a2,c3,e3])

chord_Bdim = Chord([b2,d3,f3])

chord_Em = Chord([e2,g2,b2])

Use rtmidi to send virtual signal to music software.

class Player():

    '''A Class of the music player'''

    def __init__(self):

        self.midiout = rtmidi.MidiOut()

        self.available_ports = self.midiout.get_ports()


        if self.available_ports:



            self.midiout.open_virtual_port("My virtual output")

    def play_note_start(self,note, velocity):               #Start to send a midi signal that start a music note

        self.midiout.send_message([0x90, note.get_note_number(),velocity])

    def play_note_end(self, note):

        self.midiout.send_message([0x80, note.get_note_number(),0])

    def play_chordnote(self, target, duration, velocity):           

        # [(c3,chord_C)]

        if type(target) !=tuple:

            self.note = target

            # play single note

            self.play_note_start(self.note, velocity)





            #play chord and note

            self.note = target[0]

            self.chord = target[1]


            interval = (len(self.chord.get_notes())-1)*0.1

            for i in self.chord.get_notes():






            for i in self.chord.get_notes():


    def close(self):

        del self.midiout

A music note played with midi is based on following parameters, the certain note that is going to be played, velocity of the note, and duration. The duration is actualy not a independent parameter, the whole signal if formed by Start message and End message, the start message define the time start to play the note and the end will stop it.

Converting data into Drum Beats is based on following code:

when the read data is larger than 500, the program start to send a start message of a drum beats and if it drop under 500 send a message to stop it.

while 1:

    data = server.readline()                

    result = convertData(data)

    print(hit,result)               # Read data from Arduino UNO

    if result>500:

        if hit == False:

            if hit_count%2 == 1:




            hit = True

            hit_count += 1


    elif result < 500:

        if hit == True:

            if hit_count%2 == 1:




            hit = False




Code used to play chords progression:

We pre defined some chords progressions for testing, it should be able for user to change it.

Chords_Kanon = [chord_C, chord_G, chord_Am, chord_Em, chord_F, chord_C, chord_F, chord_G]

Chords_Blues = [chord_Cmaj7, chord_Dm7, chord_E7, chord_F, chord_G, chord_Am, chord_Bdim, chord_Cmaj7]

while 1:

    data = server.readline()

    result = convertData(data)

    last_chord = Chords_Kanon[hit_count%8-1]

    this_chord = Chords_Kanon[hit_count%8]


    if result ==0:

        if hit == True:


            hit = False

            hit_count += 1


        if hit == False:


            hit = True



            player1.play_chords_start(this_chord, int(result/6))



The velocity of the chords is based on the pressure use placed on the sensor. Each time the system detect a "Press" action, it will change the chord to play.

This Week

This week we were planning for our final prototype, we are going to combine our work together for the exhibition. Our final prototype will be an installation in a much larger scale than teh current one, so we will need some more materials.

We discussed the form our prototype will be. There should be a structure to contain all the sensors and wires inside, and should be stable enough to allow our user to stand on. It should be like a "Box", the FSR sensor placed on the top of the surface, and LED lights on the bottom side. The top surface should be transparent to allow the light comes out. So we considered Acrylic Board. Also it should be structed on its edge and corner a little bit higher than the ground to allow sensors and wires stuff.

Future plan

For the comming weeks, the focus of the work will be deciding "how we are going to allow our user to switch between different types of exercising". We could have a switch for our user to switch themselves, we could also have the system to detect it automactically, however there are still things need to be considered.

If we have system detect it automatically: there will be a lot of code work, and I am not sure whether we could carry it out, it may involve some work concern machine learning...

If we have our user to select themselves:We need to find a way to allow our user select the type of exercise, we don't have buttons, switch nor remote or touch screen, so it might be related with the sensor, but we still need some user research to support our idea.

Week 8 Journal

Tianyi Liu - Sun 3 May 2020, 6:12 pm

This is week I got a cold and spend most of the time resting in my room.

In order to discover the possiblity of using mobile devices as data collection and user input methods, I tried to learn the coremotion framework, which is a frame work in iOS platform, but I defined following challenges of using it:

  1. The framework provide easy access to get the user's moving states(walking, running, driving, riding a bike etc.),but it's not easy to get the detailed data(speed, accelerate).
  2. It will be hard to test and debug, because the to run the application on a mobile phone that is still developing will required a developer's account.

There are also advantages of using it: the data collected will be easier to exchange to other components and analysed as output.

Week 7 Journal

Tianyi Liu - Sun 26 April 2020, 9:57 pm

Reflective Questions

What is your concept / individual focus / individual responsibility for the project? Include a text-based description and imagery to support (sketches, photos etc).

My individual focus in our team project is, to design an implementation that could recognize what kind of exercise the user is doing and transform data collected from user to music output as feedback.

What is the ideal finished product? (not what you think you can implement, achieve, but what you would like to be if you had all the resources/skills you need?)

If I have all the resources I want.. I would want to build an exercising yards with different exercising instruments with sensors planted in. Those sensors could collect data from user and give those data to the software which would transform data into music and output to user.

Week 5 Proposal

Tianyi Liu - Sun 5 April 2020, 1:25 am

At the weekends of week 4, our group have a zoom meeting, discussed what we are going to do in week 5. We devided the writing work of our proposal, and what specific aspects each team member will focus on. We read the feedback from the week 4 team pitch try to think about what kind of changes we could make to our concept based on these feedbacks.

According to the current situation, its impossible to conduct any face to face user research, I tried to gather a few people in a group channel and have a free chat and tried to get some general ideas about our concept. However, unlike in the real world, it was really difficult to hold a discussion online, the focus point of the discussion soon changed to some very detailed questions(maybe because the participants are mostly college students in engineering). The talked about how to translate data into sounds. Because a clip sound wave have three variables: time, frequency and volume. While the data we got from the user only have two, time and pressure. Therefore there are two ways to implement the translation, either transform the pressure to volume or to frequency. The former will emphasize the rhythm of the music and the latter will emphasize the melody. Although the name of our concept is "Fitlody", however it might be more suitable to combine "rhythm" with "Fitness", instead of "melody."

The insights I got from the discussion.

  1. There will be a lot of noise in data and how to deal with these data would have large effect on user's experience.
  2. Some of the participants couldn't understand why we want to generate music from the data, they just suggest to identify what kind of exercise user is doing and provide different styles of music.
  3. Better find some sample data and have an early experiment on translating these data to sounds and could discover any severe issues that could help to decide a better way to implement the idea.

Week 3 Group Meeting

Tianyi Liu - Sun 15 March 2020, 10:48 pm

We have a group meeting in Friday and decided what we are going to do for the next several days. Because we are going to give a presentation and bodystorm next Wednesday, we planned to make a video to present our design concept.

The first 30min we discuessed the concept we are going to use. Basically its the design idea I presented in week 2, but we thought its a little bit simple and we want to add some interactions to make it more interesting. Although the idea is about "Music" and "Fitness", but its core is "Data visualization", so we are thinking to make more use of the data collected from the sensor, We want to make the data not only presented as sounds, but also showed as visuals, like the visual effects in the early Windows XP music players.

We spent the next 30min talked about some more details of the video we are going to make. We decide to meet at Sunday, bring some instruments(both musical instruments and instruments for exercise), and try to use bodystorm to show how the user expected to interact with the product.

Week 2 Soldering and Presentations

Tianyi Liu - Sat 7 March 2020, 9:24 pm


We participated in a soldering induction at the beginning of this week. It was a very interesting experience. I have took the fundamental of computer hardware basics courses, but I never think about how the very basic part of a computer is like. Although what we did in in Monday is quite simple---- a circut only include battery, switch, resistance and a LED light. But it gave me a total different experience, compare to using the A..d toolkit. It's like you are building up your own electronic from the very first step.



Week2 Presentation

Tianyi Liu - Tue 3 March 2020, 3:03 am


Exercise/Fitness /Music /Synesthesia


Inspired by a video (I forget the original resources) which transcript the Cosmos Microwave Background (CMB) to a sound clip. It sounds deep and mysterious. Every digital signal or waveform could be transcript into sound waves. This process transforms these things that we couldn’t sense into sounds we could hear. It’s a total new way to perceive our living world.

How it works:

The pressure transducer is an object that could sense the pressure on a surface and transfer it into digital signals. These digital signals could be transcript into sound waves through some algorithm, and output as relaxing

Why this is interesting:

Ring fit Adventure, a very successful game in the beginning of 2020

The most important idea I got from this product is that when people doing exercises, positive feedback could provide them strong motivation to persist.

Target User

Fitness lovers, Childs (they love anything that could provide feedback), any people who are interested.

Context of use

This could alternate or implemented with other fitness instruments in the outdoors. User could do exercises on the implementation after their works or studies.

Week 1

Tianyi Liu - Fri 28 February 2020, 5:57 pm

I'm Tianyi, 4th semester of postgraduate students in Interaction Design. Just managed to get back to Brisbane after a 14 days self isolation in Bangkok. I have some experience in group working and development in HCI program from the coursese in the last few semesters, and I have confident that these experiences might help me to be a good teammate in the future group work.

About Physical Computing

It's the last and most important course in our study plan, which I considered as a final application of all the methods and theories we have learned in the last 3 sems. Because of the timetable clash with some exams, I didn't attend the exhibition last year therefore I'm not sure what kind of achievements we are expected to done at the end of the course, but I know things won't be done without hard efforts.