Fereshteh Shahmiri

PhD in Computer Science - School of Interactive Computing - Georgia Institute of Technology

 

Technology Square Research Building
85 Fifth Street NW
Atlanta, GA 30308

© 2017 by Fereshteh Shahmiri. All rights reserved.

MS - Computational Design Final Project

Carnegie Mellon University

 

Special Thanks to Supervisors:  Professor Ramesh Krishnamurti , Professor Jeremy Ficca and Professor Joshua Bard

 

 

 

TITLE:

A User Friendly MODEL for :

 

REAL-TIME WORKFLOW from

GRASSHOPPER ENVIRONMENT TO ROBOTIC ARM MOTIONS

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

pdf Version of thesis is available here.

ABSTRACT |

 

WHAT ?

This research study is situated in parametric design and robotic fabrication areas and attempts to provide continuous environment of both by operating a real-time workflow. The research explores the benefits and challenges existing in a real-time programming software environment and live control of robotic arm kinematics. Since the main challenge is handling the connection and compatibility of different interfaces, the research goal is to provide a simple and user-friendly Method that offers such required real-time process. This research claims that this method can works as a Model applicable in many modeling processes that are originated specifically from grasshopper environment.


WHY ?

Why Real-time Workflow?

Why Providing a General MODEL?

 

One part of the research is concentration on specifications of both real-time process and product coming out of this process. There are some claims that live control can lead to creative product with some unpredictable features. (Braumann and Brell-Cokcan, 2012) So, one of the reasons for choosing this subject is exploration of whether and how real-time process causes mentioned product features in real world. Furthermore, there are some technical and pedagogical benefits in adoption of real-time process rather than non-real-time one in the scope of architecture. Most important reasons are first, “possibility of error detection and correction inside the workflow” (Braumann and Brell-Cokcan, 2012), second, fast and strong feedback loop originating from design action and fabrication reaction (Fischer, Biswas, Ham, Naka and Huang, 2012) and third, programming simplification in some parts by omission of importing and exporting data. (Braumann and Brell- Cokcan, 2012)

 

HOW ?

The other part is about the way of approaching the objective and handling the challenges technically in software and hardware layers. Therefore, some methods including alternatives that theoretically made this research feasible was put forward and precisely codified, tested and ultimately ran in real world.

 

Hopefully, this research benefits architects in the field of robotic fabrication and also, architecture students fascinated to this research area.


Research Key Words:

Robotic Fabrication/ Socket Communication/ Real-time Workflow/ Parametric Design/ Dynamic Model

 

Real time workflow causes overlapping between process and fabricating the product

SCOPE AND RESEARCH BACKGROUND|

 

Reference to:

Daniel Parker, Andrew Payne, Thibault Schwartz, Braumann, Brell-Cokcan, Csokmai Szabolcs, Fischer and Biswas are researchers who each one by more emphasis on one of these areas, move toward the overlapping part.

 

 

DEFINITION OF RESEARCH PROBLEM

INVESTIGATION OF TECHNICAL SUBJECTS |

 

Knowledge acquisition about technical subjects has led to an Integrated Practical Model which illustrates how different interfaces from dynamic flow in grasshopper to robot controller should be designed, codified, interacted and at the end offered a real-time workflow both in software environment and between software and physically robotic fabricated model.


1. Messaging in Robot Language (RAPID) either real or non-real time

In this area underlying concepts such as RMQ messaging (in RAPID code) and Socket Programming as the basis of remote communication have been investigated.


2. Potential of grasshopper environment and specifically the concepts of physics based add-on plug-in like Kangaroo

 The process of socket communication between data generated as the output of this interface owing a dynamic behavior through RAPID code that has been a critical step in problem solving and offering the final model that has mentioned, too.


 3. Simplification and Integration of the all steps

This part has an important status in this research. Although some valuable visual robot programming interfaces has been created and packed in grasshopper components to facilitate the design-to fabrication process, they have some critical problems that cannot be ignored. One suggestion to remove existing challenges in them is using a middleware layer with an intermediary function between CAAD and robot necessities’ interfaces.

 

 

PROBLEM SOLVING:
RESEARCH METHODS

Illustration of Socket Communication (Szabolcz, 2012)

 

Existence of specific packed component named 'HAL to Controller' makes it possible to send the data to robot controller in a real-time process. All needed information about tool, toolpath, targets and external axis and so on should be provided as the inputs for IKSolver and then Post Processor (equivalent with RAPIDGenerator in old version of HAL) components. The output will be StreamingData of the RAPIDGenerator component as the input for Streaming part in HAL to Controller. Accordingly, in the last step, after identifying the Robot Controller, establishing the connection between HAL interface and Robot Controller, activating the ECM Generator for controlling the execution, theoretically the real-time flow should be achieved. An important point is that in this approach there is no need to provide the data for feeding the Task part. Furthermore, in TeachPendant the RMQ messaging control property of the task has to be modified to “Remote” instead of “None”

 

 

 

HAL to Controller Component

Alternative 2:

Step 1: Communication over TCP Protocol by using Socket Communication
Limitation 1: Grasshopper can NOT support Socket Communication over TCP Protocol


Based on knowledge about this way of communication in first attempt, the idea was a direct connection between dynamic flow generative component output in grasshopper and RAPID code in Robot Studio. The logic was using python socket library inside the grasshopper. However, after different tests illustrated in the following coding, it was figured out that grasshopper cannot support socket programming

 

 

 

try to make a connection over TCP/IP between Grasshopper component and RAPID code

Step 2:

Sending data out of grasshopper and saving in .txt file & Providing Socket connection out of grasshopper


This method in compare with using UDP is so slow and controlling large set of data is a bit harder. So, although this method is also one solution and in some specific context would be helpful, the preference here is something that is offered in next step.

 

 

 

 

Different methods to create socket inside grasshopper and sending the data out

RESEARCH METHODS:

 

Alternative 1: HAL Programming and Real-time Process (Scripting Logic: RMQ Messaging):

 

 

The RAPID code as the SERVER for all following steps is also the same as here.

Sending data out of grasshopper and saving in .txt file &Providing Socket connection out of grasshopper

Step 3: Sending the data over UDP Protocol


Limitation 3: RAPID Code can NOT support Socket Communication over UDP Protocol
Limitation 4: UDP is NOT reliable on Wireless connection

 

Step 4: Providing a Middleware layer for receiving the data over UDP/IP from Grasshopper environment and sending over TCP/IP to RAPID code


In this step, in fact two networks are created. In first one by using UDP/IP data is sent to python coding in Komodo that acts as a mediator and receives the data as the server. In the second one, that is in the role of the client and sends data over TCP/IP to RAPID code. Based on data transmission inside one computer and omission of wireless usage in first network UDP is reliable way and there is no problem about missing or order of data packages. Moreover, TCP/IP is completely safe for sending data to Controller. Consequently this method is considered as the final model in the research for programming a real-time workflow between dynamic modeling in grasshopper and robotic arm motions. Furthermore, other benefit of this method is the possibility of sending data to several servers. And it was something that tested in CMU dFAB by two robots. Accordingly by having just one set of data in part of the Client, several physical prototypes (based on the number of available robots) can be created. More importantly, this set of data is adjustable and scalable so that the physical outputs are homogeneous but in different dimensional proportion

 

 

 

 

Using a Middleware layer for receiving the data over UDP/IP and Send over TCP/IP to RAPID code

Final Model
Accordingly, An Integrated Practical Model which can illustrate how different interfaces from dynamic flow in grasshopper to Robot controller are connected to each other is offered here:

 

 

data transmission in a real-time process to two robots

CONCLUSION:

 

GENERATION OF A RELIABLE MODEL

BY SOLVING TECHNICAL PROBLEMS AND DEALING WITH VARIETY OF INTERFACES

 

By focusing on different methods and approaches to find the most convenient and reliable method of data transmission in real time process, it has been understood that the matter is not only concentrated on the identity of a live data transmission but also about discovering and solving the challenges related to specific features of each existing interface.

 

Second point was about the idea that whether the suggested method is comprehensive and easy to dealing with for other users who need to have a real time process between grasshopper and robotic kinematics or not.

As a result, the aforementioned method has been suggested and in the next step by implementation of this method a practical experience will be tested.

PRACTICAL EXPERIENCE:

 

1. Achieving an acceptable knowledge about particle systems, their characteristics, attributes and influential forces that cause their formation.

 

2. Preparing a general context for generation of particle systems in grasshopper  ( An important point here is that in all examples like fire work, water flow, smoke and etc, for generation of the particle systems there are required logics related to particles’ positions and velocities as well as consideration of imposed forces that are also similar in different systems.)

 

Holistic Goal:

The attempt is to design a general package considering mentioned ideas that could embody different examples of particle systems in grasshopper.


Following three distinct interfaces are the pivotal parts of the suggested model

 

- Interface 1: Requirements for Any Particles System Simulation in Grasshopper
- Interface 2: Using the Middleware Layer between grasshopper and RAPID code
- Interface 3: Using the sample code written for robotic arm motions in order to simulate the particles behavior

 

Distinguishing among different procedures

designed and codified in grasshopper as the first interface of suggested model

 

What is mainly focused here is the content of first interface that are three following procedures;


- Generation of the particles by Kangaroo Engine
- Data saving in an external memory
- Reading the data in grasshopper and sending out to next interface by using socket communication.

 

Important note is that these three distinct procedures inside grasshopper are just about virtual modeling and data transmission and nothing about how data should be processed for physical modeling. Successively, there are two more interfaces in connection with grasshopper interface to develop and embody the workflow in an integrated virtual and physical model simultaneously. As it has been mentioned previously the other two interfaces are first any kind of editor to run piece of python code and second RobotStudio software as the context for running the RAPID code in order to control robotic arm motions.

 

Preparation of a comprehensive context in order to generate different types of particles’ systems in Grasshopper


Based on a holistic attitude toward the general behaviors of particles systems, some ideas should be brought forth, investigated and considered in order to progress the research goal and operate three mentioned procedures. Some of them are following ideas:


- How an emitter generates the initial set of particles.
- How different types of forces such as unary (like gravity or wind) and binary (like springs) forces influence on particles’ attributes.
- How mentioned forces lead to formation of each specific system.
- How logics like Collision lead to formation of a dynamic system.
- How some necessary physics-based ideas such as Tolerance, Time Step, Drag, Restitution, Frictions and so on in each particle’s behavior and interaction between particles should be considered in generation of the systems?
- How calculation of physics rules and mathematical process should be done.

 

 

 

 

Procedure 1: kangaroo engine can do all calculation and generates the particle and geometry out

Particle generation in grasshopper by using Kangaroo Plug-in

 

Procedure 2:  Data saving in an external memory

(Calculation and storing the Particles’ Position and Velocity in each Time Step)

 

Regardless of specific type of particle system or variety of influential forces on particles formation,  first by adoption of a general method applicable for different types of systems, particles’ position and velocity be calculated. Secondly, it is required this huge amount of data be managed and saved in order to be used in next interfaces.

 

As there is no memory in grasshopper for saving data inside the grasshopper, the writing and reading needs to have access to an external memory like .txt file. In this thesis, these two steps which are independent to type of particle systems are designed in grasshopper with grasshopper components and Gh.Python scripting .

Calculation and storing the Particles’ Position and Velocity in each Time Step

Sifting the data: Relation between curves resolution and data set size storing in external file

 

One important point is about a great amount of data related to particles’ positions and velocities. As long as the goal for any particles’ system formation is just visualization in virtual space, data management won’t be problematic. The problems from different aspects will happen when their simulation happens in real world by using some methods like robotic motions. In this case, the best suggestion would be about sifting the data in Gh environment before sending them all out to next interface relevant to robotic fabrication. 

 

Timer, counter and part of scripting as shown in the below, can do the required sifting. In part of coding that is “if t% n ==0”, the variable n determines from how many set of data, one set should be sent for next step. The important note is about the resolution of the modeling that will be decreased by increasing the amount of the variable n.

Data filtering:

trade-off between resolution and speed

Procedure 3:   Reading the data in grasshopper and sending out to next interface by using socket communication

 

The last thing that is needed to be distinguished here is about how all data generated in grasshopper should be sent over UDP to second interface.

 

 

 

Reading the data in grasshopper and sending out to next interface by using socket communication

One Practical Example of Particle Systems:
Water flow simulation  and drawing the pattern by robotic arm motion

 

 

 

As it is illustrated here physical model and particles in grasshopper are drawing simultaneously. Any changes in grasshopper variables are imposed on model in real world and that is the reason that we claim the final pattern is not totally predictable

CONCLUSION

 

Although this research mostly concentrated on how particle systems will be generated and how real time process between grasshopper environment and robotic fabrication will be achieved, future plan is thinking more about other possible applications for such a real time prototyping.

 

One helpful application could be about how varied patterns can be dependent to each other in a way that one type of particle system in virtual space leads to variety of unpredicted but adjustable patterns in physical model. Variables like different shapes of freeform surfaces, scale of the whole model, density of the particles, particles movement patterns are some of them. Such differentiation between virtual and physical models can be created, controlled, exaggerated or moderated by using different types of sensors applied on design-fabrication process