Our objective is to design a fully functional Cooking Robot that will automatically dispense the ingredients needed for a user input recipe. This design saves users' time when cooking and guarantees the correct composition of the dispensed mix of ingredients. Our project focuses both on functionality and endurance of the machine, aiming to approximate a real-life appliance.
In this project, we build a cooking robot that takes in user input recipe via a user interface and automatically dispenses the available ingredients according to the user's specifications. Our initial design assumes the following six ingredients: salt, sugar, MSG, garlic powder, ginger powder, and chili powder. The process of dispensing is 100% automated, and we made sure that the design is robust enough to approximate a real-life kitchen appliance. Our robot holds six shaker cans that contain various ingredients, a rotatable plate that holds the bowl, and a Raspberry Pi - PiTFT set that serves as the robot's "brain". For the full report with accompanying photos, demo video and code, please visit: https://courses.cit.cornell.edu/ece5990/ECE5725_Spring2017_Projects/Final%20Report/index.html
DESIGN AND TESTING
The project involves both the hardware component and the software component. The hardware component includes the overall robot framework, the mechanical control for the shaker can cover rotation and the mechanical control for the bowl holder's rotation. The software part contains the user interface that utilizes rendering on-screen elements, touch screen event detection, GPIO event detection for physical buttons on the PiTFT and software PWM control for the servos. The design of user interface and software GPIO control uses a significant amount of functions provided in pygame and RPi.GPIO library.
Our project involves design and testing for 4 major parts: robot framework, shaker can cover rotation, software user interface and wiring. We adopted the modularization principle -- design and test each individual part separately and integrate the overall structure.
Robot framework refers to the supporting structure of our robot; it involves a top plate, 6 shaker cans attached to the top plate, 3 legs attached to the top and bottom plates, a bowl holder plate with wheels, and a bottom plate. The three plates are made of panel boards, and the three legs are made of furring strips.
For the top and bottom plates, we shaped them to be circular and Professor Skovira helped us with the sawing. Our base is smaller in diameter than the top cover since we need to account for the space occupied by the legs. Specifically, our top cover diameter is 14" and our base diameter is 12". The furring strip is 8' in length i.e. 96". We set the 3 legs of the robot to be 11" and attached them at the edge, 120 degrees apart, to the top cover with the help of Professor Skovira. We divided the top cover into 8 sectors with each spanning 45 degrees, marked out the radial lines and drilled 6 holes along 6 of these radial lines. We decided on 8 sectors and did not place shaker cans on two neighbouring radial lines so we can easily place and retrieve the bowl holder. The 6 holes are 10" from the center of the circle as we want the bowl's center to ultimately align with the center of the shaker can to prevent spillage during the dispensing process. Professor Skovira also helped us with drilling the holes both on the top cover as well as the bottom of the shaker can so we can screw the shaker cans to the top cover. We then attached the top plate to the 3 legs using screws. We finally attached the bottom plate to the legs in a similar fashion.
For the bowl holder, since there are 4 wheels -- two driving two idle, we decided they will be 90 degrees apart and marked out the radial lines on the circular bowl holder. Based on the size of the wheels and the servos, we decided the bracket will be placed 1.5" from the edge of the circle so the wheels are entirely below the bowl holder. We then measured the distance between the centers of the two holes on each bracket and marked out them out on the bowl holder so we know exactly where to drill the holes on the bowl holder for attaching the brackets. Two of the servos we are using will be driven by the Raspberry Pi and will be responsible for turning the bowl holder. However, two of the other servos are broken and is in place simply to help stabilize the bowl holder. We attached the wheels on these two servos loosely so they can spin along as the bowl holder rotates. We found the midpoint of the bowl holder plate, and used a screw to pin this mid-point to the base. The goal of this attachment is to pin the center of the bowl holder, so that it can rotate in a circle instead of wondering around. To do that, we also need the connection between the screw and the bottom plate to be loose enough so that there is space for the screw to rotate. Thus, we used a small chunk of wood and attached it firmly onto the bottom plate. The thickness of wood allows us to prevent the screw from pulling out and, at the same time, attach the screw loose enough for rotation.
After every individual component was setup, we inserted the bowl holder plate in between the shaker cans and the bottom plate to finish our robot framework.
To test the robustness of the framework, we focused on the reliability of the bowl holder plate rotation. We first took out the bowl holder plate and tested its rotation. We then inserted the bottom plate back into the overall framework to test if it is able to work properly.
- One of the idle wheels was not touching the ground
- The bottom plate of the framework is not flat enough to allow constant smooth rotation.
We fixed the first problem by loosening the problematic wheel. And for the second problem, we wrapped rubber bands around the two driving continuous servos, such that there is more friction and less instability when the plate is climbing uphill (very small slope due to the unevenness of the bottom plate).
Shaker Can Cover Rotation
In order to dispense steadily, we would like to come up with a way to rotate the shaker can cover reliably. Our final design uses the micro servo to control the rotating cover, as they are relatively easy to control and has a horn which can be attached to the rotating cover using superglue. To integrate the design into the framework, we added three more furring strips to the framework and secured the micro servos to the furring strips for effective rotation of the covers.
The center of the can cover is drilled out to attach the servo more easily. More specifically, the micro servo has three available positions, left, center, and right. To control the position, we output PWM signals to the GPIO to which the servo is connected. When the duty cycle is set to 1.5ms, the servo is in the middle. When the duty cycle is set to 2ms, the servo turns 45 degrees clockwise. When the duty cycle is set to 1ms, the servo turns 45 degrees counterclockwise. We programmed the Raspberry Pi so that it outputs the mentioned signals and rotates horn correctly.
After we have familiarized ourselves with the functionality of the servo, we superglued the horn to the cover of the shaker can, and use the provided screw to attach the servo body to the top part of the can. By adjusting the position and angle of the cover, we are able to program the micro servos so that they rotate steadily each time, dispensing 1/8 teaspoon per rotation.
In order to integrate this design into the overall framework, we need to come up with an idea to hold the servo body fixed, because if not, the servo body will rotate instead of the can cover. At the same time, we still want the attachment between the servo and the framework to be removable, because we would like to take off the cover for refilling purposes. After a long time of trials and errors, our final solution involves firm foam blocks and velcro. We first attach the micro servo to one end of the firm foam using velcro, and attach the furring strip to the other end of the firm foam in a similar fashion. In order to attach all servos, we added three more furring strip blocks besides the three legs to hold all six shaker cans' corresponding servos.
We had several failed attempts on designing this functionality, and we believe it is always beneficial to share our experience with more people to prevent others from going into the wrong direction.
- We tried to use DC motors with H-bridge to control the cover's rotation, but the DC motor is rotating too fast, and it is very vulnerable to resistance, i.e. a small friction will cause it to get stucked. Besides, the attachment between the DC motor and the cover is not as stable, since we mainly used foam tape to glue them together. Therefore, the overall performance of DC motor is not desirable.
- We tried to use slotted firm foam to hold the servos instead of using velcro, but the slot became too loose to hold the servo after several rounds of rotation.
We deployed a similar method to test the design of the can covers: first test each can individually and then test the integrated design. We realized two problems in the process:
- The ingredient gets stuck in between the cover and the top part of the can due to the slightly loose attachment and the holes on the can.
- When the servo first started, it jitters and rotates for a random degree if no signal was sent as an input. This rotation exposed the holes on the bottle, causing the ingredients to leak when not intended.
To solve the first problem, we used tape to cover most of the holes, which are not used for dispensing anyways. This fix allows almost no ingredient to leak through the hole when the can is upside down.
To solve the second problem, we programmed the Raspberry Pi such that it gives a signal to push the servo to the extreme left position, which keeps pushing the cover to the same safe position. Since the cover is already at the safe position, when the power is applied, the cover stays fixed at its original point, preventing the ingredients from leaking.
Software User Interface
For the software user interface, we wrote a python script that utilizes the pygame and the RPi GPIO library. The pygame library serves the purpose of controlling the aesthetics of the interface as well as gathering user input via the touchscreen to transition the program between different states. The RPi GPIO library is used to add event listeners and callback functions to detect the pressing of the physical buttons on the PiTFT to also transition the program between different states.
We will first start by explaining how the main functionalities our program leverages on before delving into the details of the code. The main functionalities we used are rendering elements on the screen, detecting touchscreen events for the touchscreen buttons, detecting GPIO events for the physical buttons and sending PWM signals to control the servos.
We composed the different rendering helper functions for displaying the background, the various touchscreen buttons and texts. We wanted to keep the code modular and helper functions in this case would be of great convenience since we want the user to move back and forth between the different pages and will need to render the background, buttons and texts more than once.
For the background of our program, we searched for a photo we like on the Internet, resized it using the Paint program such that the photo is 320x240 pixels in size i.e. the same size as the PiTFT screen, and slightly modified the color tone of the picture such that it is light enough to make the touchscreen buttons clearly visible. For our render_background helper function, which loads this background image onto the screen, we first cleared the screen by filling it with black, using the command screen.fill(black) where both screen and black have been previously declared as global variables. We then load the image using the pygame.image.load function and passed it the background image file name. We get the enclosing rectangle for the image using the get_rect function and combined the background with the enclosing rectangle with the workspace screen using the blit function. This only updates the workspace screen but not the actual screen. To update the actual screen with the workspace screen, we use the flip function.
Other elements displayed on the screen include texts and buttons. Buttons are simply texts that respond to touchscreen events and transition the program to another state but the rendering process itself is the same across all of them. For rendering a text, we first set the size of the text using pygame.font.Font function. Then we indicate the text we want, whether we want the characters to have smooth edges and the color of the text using the render function. This generates a surface and we can get the rectangle enclosing the surface using get_rect just like the background case, except this time, since the surface will not be 320x240 and fill up the entire screen, we pass into the function the position of the rectangle's center coordinates. This will place the text at that position on the screen. Finally, we update the workspace screen using the blit function. The rendering of all the touchscreen buttons follow the exact same logic.
For detecting touchscreen events, we used the pygame.event.get function and a for loop to retrieve the event that occurred. If the event type is MOUSEBUTTONDOWN which means the user pressed down on the touchscreen. At this point, we do not do any processing but waits until the user releases the touch. At this point, we get the position where the user pressed on the touchscreen using pygame.mouse.get_pos(). This returns the x,y coordinates where the touchscreen event occurred. To turn the text into a button, we experiment with the boundaries of text and set the condition such that when the x and y coordinate retrieved from the touchscreen event is within the boundary of the text, we transition to another state by rendering a different page, thus making it appear like the text is a button.
For detecting GPIO events, we have three options: polling, polling but without missing an event, interrupts. We did not want callbacks as we wanted to reuse some physical buttons for different functionalities depending on the state we are in. This means having to add the callback and removing it when the state changes which we found to be troublesome. We did not want to use polling as we want the user interface to respond to events quickly and miss as few events as possible. Thus, we went with an approach similar to polling except even if the physical button was pressed not at the time the poll happened, the event will still be kept and the system will still respond accordingly. Firstly, to listen for an event, we use the GPIO.add_event_detect functionality. We set it to listen for each of the physical button when there is a falling edge and set a bouncetime of 300 so if the user pressed the button for a bit longer, we will not treat it as two button presses. To check if the event has occurred, we used the function GPIO.event_detected which returns true if the event happened and false otherwise.
For controlling the servos, we used software PWM. This means first using GPIO.PWM on the pin where we want to send the PWM signal and passing into it the frequency of the signal. Then we simply call start then subsequently ChangeDutyCycle to set and modify the duty cycle of the PWM signal. The information for the frequency and duty cycle needed to control the micro servos and the two continuous rotation servos can be found in their respective datasheets. To stop a servo from turning, we can just modify the duty cycle to 0.
Having explained the major components used by our design, we will now explain in greater detail on what the various helper functions do and how the different states are connected together.
The script starts off with some initialization and setup of the PiTFT screen such that when the program launches, it will start up on the PiTFT screen instead of desktop. We also need to initialize pygame by calling the init() function and setting the mouse visibility to false.
We also setup the GPIO pins for input or output. We chose the pin numbering system such that they will be referred to using the "Broadcom SOC channel" number. Since we have 6 micro servos, 2 continuous rotation servos and 4 physical buttons on the PiTFT, we need to set up a total of 12 GPIO pins. Since we are using the Pi to send signals to control the servos, the corresponding pins need to be declared as outputs. Similarly, since we trying to detect whether the physical buttons have been pressed by reading the GPIO input value, the corresponding pins need to be declared as inputs with internal pull up resistors.
We initialized the PWM signals for the micro and continuous rotation servos. For the micro servos, we used software PWM to control their motions. We looked at the datasheets for the micro servos and the frequency of the PWM signal is 50Hz which we set using the GPIO.PWM function. One issue was that when the servos are first connected to the power supply and no signal has been supplied, they are in an undefined state and thus jitter a little. This is very problematic in our case since small movement of the micro servos can cause the rotating cover to turn and the ingredients to start spilling. To solve this problem, at the start of our program, we called start(5), which sends a PWM signal with duty cycle of 0.05, and this turns the servos to the extreme left position. This extreme left position corresponds to the position when the rotating cover is closed. Only when the program is running do we turn on the power supply for the micro servos. Since there is a signal already being output from the GPIO pins connected to the micro servos, the servos are now in a defined state and will not jitter randomly. For the continuous rotation servos, we also used software PWM but with different frequency and duty cycle compared to the micro servos. Specifically looking at the datasheet, the frequency for these servos are all around 46.5Hz. Initially we do not want the bowl holder to turn so we set the duty cycle for the PWM signal to 0. Although when the power supply for the servos are turned on, the continuous rotation servos also jitter a little, this is not a problem since they do not affect any functionality of our cooking robot and we designed the framework such that we can easily reset the position of the bowl holder even if the jitter caused the bowl to be slightly misaligned with the first shaker can.
We setup some global variables which are shared across states and functions. Since we have the amount page for the individual ingredient where the user can input in the amount but we also want this amount to be reflected in the menu page after the user pressed Ok, we had a global dictionary that gets updated to the amount entered by the user and whenever we need to read the amount for the individual ingredient, we index into the dictionary. After each dispense, we want our program to return to the home page and since we want the user to use the cooking robot more than once on each run of the program, we have a global variable new_start to keep track of if one round of dispensing has completed and the user can now start a new round. Other global variables include the size of the screen, setting up the screen workspace, and different colors using the RGB values.
The render_background function is as aforementioned which basically loads the background. We did not call the flip function as we have other buttons to render and only want to update the actual screen when all the elements have been written to the workspace screen. The render_quit, render_cancel, render_ok helper functions are used to render the various buttons and works the same way as how texts are rendered as explained earlier as well. The render_dispense function simply renders the text "DISPENSING" on the screen.
For rendering_menu function, rather than rendering each ingredient name text individually, we created a local dictionary that uses the names of the ingredient as keys and the position the button will be placed as the value. We then simply go through the dictionary, get the key and the corresponding value and pass it into the render and get_rect functions respectively. For rendering the amount, we need to retrieve the amount from the global dictionary since there are two cases where the menu page is displayed: (1) when the user pressed start from the home page (2) the user has just entered an amount for a particular ingredient and pressed the Ok button. For the second case, the amount for a particular ingredient might have no longer been 0 and hence, we cannot simply just render a fixed text but rather needs to get the amount the user actually entered. We created the global dictionary holding the ingredient names and amount such that its keys are the same as the local dictionary holding the ingredient names and positions. Thus, when going through the local dictionary, the key we retrieved can also be used to index into the global dictionary. This will return the amount for that particular ingredient and we simply convert the number to a string and pass it into the render function. We want the amount text to be aligned with the ingredient name text along the x-axis but lower. Hence, using the position we retrieved from the local dictionary, we further get the x-coordinate which is index 0 and add 30 to the y-coordinate which is index 1 to place the amount text lower than the ingredient name text.
The render_amount and render_ingredient helper functions work together. The helper function render_ingredient takes in the name of the ingredient and listens to see if the the physical +, - and Ok buttons have been pressed using the GPIO.event_detected functions as explained earlier. We left the rendering of the +, - and ok texts on the screen to the render_amount helper function which the render_ingredient calls. If either of the + or - buttons are pressed, we need to use the name passed into this function as key to index into the global dictionary to retrieve the current amount before adding 1 or subtracting 1 from it. We then simply call another function render_amount that also takes in the name of the ingredient and use it as key to index into the global dictionary, retrieve the current amount and render the text on the screen. It Ok is pressed, we simply return. We do not render anything here since we want the start_h helper function, which called the render_ingredient helper function, to do the rendering process.
The render_menu and render_ingredient helper functions are all called within the start_h helper function. This function essentially connects the menu page, amount page, dispensing page and return back to home page together. In start_h, we first set the duty cycle for all the micro servos to 0. Recall from previous explanation, the duty cycle was set to 0.05 previously to prevent the jitters but is now no longer required since we have already ensured when the micro servos are powered up, they are in a defined state. Start_h helper function is entered when the user presses the start button on the home page. On entering start_h, we first render the menu page. Then we need to both listen for if the user wants to dispense, return back to home page, or selected a particular ingredient and wants to set its amount.
If the user pressed a touchscreen button corresponding to an ingredient name, we simply call our helper functions for render_ingredient and render_amount. When the render_ingredient has returned, this means the user has finished selecting the amount and we simply render the menu page again.
If the user pressed the physical button for dispensing, we simply render the dispensing page which just displays a text to indicate dispensing. Then we need to perform the actual dispensing. Since we already know the PWM duty cycle needed to turn the micro servos clockwise or counterclockwise from the datasheet, we simply read the amount value from the global dictionary and using a for loop, open and close the micro servos for the corresponding number of times by sending the right PWM signal. However, we cannot continuously change the duty cycle of the PWM signal as that would cause the micro servos to simply jitter, we used the sleep function in between changing the duty cycle to allow time for the micro servo to be turned fully before turning it back.
When an ingredient is being dispensed, the bowl will be aligned below it. The bowl's position is controlled by the bowl holder which is in turn controlled by the continuous rotation servos. We want the bowl to be completely in place before starting the dispensing process and similarly for the dispensing process to completely finish before rotating the bowl. Hence, we asked the program to sleep for a second before and after the dispensing process to ensure little spillage. After a particular ingredient has finished dispensing, we need to now rotate the bowl holder so the bowl is now aligned below the next shaker can. To do this, we need to turn the two continuous rotation servos attached to the bowl holder. This is performed by the next_bottle helper function. Essentially this function sets the duty cycle for both the continuous rotation servos such that they turn in the same direction. The two servos need to turn in the same direction for the bowl holder to turn about a pivot point. When setting up the framework, we had ensured the shaker cans are placed 45 degrees apart. This helps us ensure that when moving the bowl between shaker cans, the duration the servos need to turn for should be the same. In this case, after experimenting, we found that asking the program to sleep for 0.48s is just right for the bowl to reach the next shaker can.
After all the ingredients have been dispensed, we want the bowl to return to the initial position to avoid the wires from being all tangled up and to make it easier for the user to retrieve the bowl. This is done by the helper function new_round. This helper function simply turns the servos in the opposite direction compared to the direction taken during the dispensing process. Since we have 6 shaker cans and the bowl is aligned below the first shaker can at first, this means the bowl moved 5 times with each time being 0.48s. Hence, to turn the bowl back to its initial position, we technically need to only let the program sleep for 0.48*5=2.4s. However, after experimenting, we found 2.7s to be more accurate. After returning the bowl to its initial position, we simply return from the start_h function. This gives control back to the main function which called start_h when the start button was pressed.
In our main function, we set up all the event detection for the GPIO pins. Then we render the home page which comprises the background, start and quit buttons. If the start button is pressed, we enter start_h. Once start_h returns, this means the current round of dispensing has completed and we render the home page again, reset the global dictionary so the amount are all 0 again, and set the global new_start variable to 1 to indicate ready for a new round. If a touch event is detected on the quit button, we clean up the GPIO and return from the main program.
To test the software design, we tested the touch sensitivity for all of the touch buttons, and tested the reaction time and effectiveness of the used physical buttons as well. We also tried various corner cases that could possibly affect how our robot behaves. We did the corner case test both incrementally for each subfunction, and for the entire program as a whole.
We also experimented with the various sleep times to ensure they are appropriately set so as to allow just the right time for dispensing the ingredients with little spillage.
Furthermore, during our testing process, we added a callback quit button that allows us to quit the program anytime in the event there is bug in the code that could potentially prevent us from returning to the home page where the touch screen quit button is located. This minimized the number of times we needed to unplug the Raspberry Pi.
Overall, the software component went fairly smoothly and we did not hit any major problems.
The wiring design for our project is not as complicated. The main concern for us was the fact that not all plates are rotating, i.e. the bowl holder plate is constantly rotating, but the other two plates are not. Therefore, we have to make sure that the wire will not tangle together after several rounds of rotation. We solved the problem in two ways:
- We always rotate the bowl holder plate back to where it was after a round of dispensing, so that the wires for the continuous servos are also back to the original positions.
- We drilled a hole in the middle of the top plate, and let the wires from the continuous servos go through the middle hole instead of wrapping around the whole frame. This allows much less tangling for the wires, since they need nearly no rotation when they are placed in the middle.
We wrapped the wires from the micro servos directly around the framework, as the cans are not moving, and there is no possibility for those wires to tangle.
The testing is fairly simple for the wiring design. We simply set it up, and tested whether the connections are robust and if there was any tangling. Fortunately, we did not find any problem with this wiring design.
CONCLUSION AND DEMO
We are able to implement a cooking robot that automatically dispenses ingredients according to the user's input. The rotation of the shaker bottle covers is steady and gives 1/8 teaspoon of ingredients for every open-close cycle; the rotation of the bottom bowl holder plate is also reliable, allowing the bowl to be at the correct position for each and every dispense, and in the end, it's able to return the bowl to the starting position; the user interface to robust and user friendly, giving no operational difficulties to the users.
For the entirety of this project, we met up to work on the project and completed all components together, from the initial brainstorming of the idea, acquiring the different parts for the project to assembling the framework and writing the software user interface.
FUTURE WORK AND ACKNOWLEDGEMENTS
Currently, one of the most significant drawbacks of our design is the fact that we have to perform a not-so-easy sequence of tasks in order to refill the shaker bottles: unwire the servos, take out the bottom plate, flip the entire frame over, refill the bottles with ingredients, replug the wires and covers, flip the frame upright, and insert in the bottom plate. This process is really tedious, and we would like to make it easier for the users for refilling the bottles. Our current solution to this would be drilling holes on the top plate above each bottle, so that we will be able to open up the holes directly from the top if we want to refill. And this also requires us to drill off the bottom of all of our shaker bottles as well. We believe this can be easily done since we already gained some experience while drilling the other components of the robot.
Another idea that we have in mind is to create a new option for saving copies of recipes. Since right now, we have to have the user inputting a new recipe every time he/she starts the program, and it is a little annoying if the user is using the same recipes over and over again, or maybe the situation that the user keeps entering wrong values. We would like to have an option called "save" besides "ok" and "X" on our menu page. The user then will have the option to save a recipe just entered before dispensing. We believe this can be done with some effort on investigating Raspberry Pi's file system and other available storage options available. Besides, we are sure that we do have a currently idle physical button available for the "save" option to map to.
A lot of thanks to Professor Joseph Skovira who helped us a lot on hardware constructions!
Also to 2017SP ECE5725 TAs: Jacob George, Brendon Jackson, and Steven Yue.