LAB 3: Introduc on to Machine Learning algorithms with a focus on robo cs applica ons.

The intersec on of machine learning (ML) and robo cs is one of the most exci ng fron ers of modern
technology. At its core, machine learning is about teaching computers to learn from data, without being
explicitly programmed. When combined with robo cs, ML algorithms empower robots to learn from
their environment, adapt to uncertain es, and perform tasks that were once deemed too complex or
nuanced for automa on.In the world of robo cs, the applica on of machine learning is vast: from robots
learning to navigate unfamiliar terrains, to recognizing and manipula ng objects, to collabora ng with
humans safely. These capabili es are facilitated by a range of ML paradigms:

Supervised Learning, where robots can be trained using labeled data to perform tasks such as object
recogni on or trajectory predic on.

Unsupervised Learning, enabling robots to discern pa erns or structures in their environment without
any pre-labeled data, which can be invaluable for tasks like anomaly detec on or clustering sensory
inputs.

Reinforcement Learning, a paradigm where robots learn to make decisions by interac ng with their
environment, receiving feedback, and op mizing their ac ons to achieve specific goals.

LAB MATERIALS AND RESOURCES:

1. So ware:

Python Environment:

Python is a versa le programming language widely used in both the machine learning and robo cs
communi es due to its simplicity and vast library support.

Jupyter Notebooks:

A web-based interac ve compu ng environment that allows for dynamic code execu on, visualiza on,
and documenta on all in one place. It's especially useful for labs and tutorials because of its interac ve
nature and ability to mix code with explanatory text.

Installa on:

To set up JupyterLab (an evolved version of Jupyter Notebooks with a more flexible user interface), use
the command pip install jupyterlab.

Scikit-learn:

A popular Python library for data mining and data analysis. It offers simple and efficient tools for various
machine learning algorithms, including both supervised and unsupervised learning tasks.

Installa on:

You can integrate it into your Python environment using pip install scikit-learn.
TensorFlow and PyTorch:

These are two leading deep learning frameworks. They provide a set of flexible tools for building and
training complex neural network models.

TensorFlow:

Developed by Google Brain, TensorFlow is widely used for various machine learning applica ons,
especially deep neural networks.

PyTorch:

Developed by Facebook's AI Research lab, PyTorch is known for its dynamic computa onal graph, making
it par cularly useful for research and prototyping.

Installa on:

For TensorFlow, use the command pip install tensorflow. For PyTorch, installa on might vary based on
the system and whether GPU support is needed, so it's best to refer to the official guide.

Gym:

An open-source library developed by OpenAI for developing and comparing reinforcement learning
algorithms. It provides a variety of predefined environments, including some related to robo cs, which
makes it easier to train and test agents.

Installa on:

Integrate Gym into your Python setup using pip install gym.

2.Robo cs Simulators:

These are so ware pla orms that allow for the simula on of robo c systems. Simulators enable tes ng
and development of algorithms in a controlled environment before deploying them on real hardware.

Gazebo:

Widely used in the robo cs community, Gazebo offers a robust physics engine and 3D simula ons,
making it suitable for a wide range of robo c applica ons.
3. Datasets:

In order to prac cally understand the working of various machine learning algorithms, the right datasets
are crucial.

For Supervised Learning:

Iris Dataset: Consists of 3 classes of 50 instances each, where each class refers to a type of iris plant.
Useful for classifica on tasks.

Boston Housing Dataset: Pertains to housing values in suburbs of Boston. It contains 506 cases and is a
mix of categorical and con nuous variables, making it ideal for regression tasks.

For Unsupervised Learning:

Image Dataset: A collec on of unlabeled images. Robots might use such datasets to learn features or
detect pa erns.

Robot Sensor Readings: Data collected from sensors on robots, like Lidar or infrared sensors, without
predefined labels.

For Reinforcement Learning:

While it doesn't rely on tradi onal datasets, the learning environment, ini al condi ons, and the reward
structure play a role similar to that of a dataset.

Gym Environments: Pre-defined simulated environments that allow robots or agents to interact and
learn. For instance, CartPole-v1 or MountainCar-v0 can be used for various reinforcement learning tasks.

LAB 3. Prepara on

A primer on the concepts of machine learning and its transforma ve role in the domain of robo cs.
Discuss the evolu on of robo cs and how machine learning has opened avenues for advanced autonomy
and adaptability in robots.

Se ng up the Environment:

Python Setup:

Step 1: Downloading Python

Go to the official Python website at h ps://www.python.org/downloads/.

Download the latest version for your opera ng system.

Step 2: Installing Python

Run the installer.

Make sure to check the box that says "Add Python to PATH" during installa on. This makes it easier to
run Python from the command line or terminal.

Complete the installa on by following on-screen instruc ons.

Step 3: Verifying the Installa on

Open a terminal or command prompt.

Type python --version and press Enter. It should display the version of Python you installed.

2. Library Installa on:

Step 1: Se ng Up Pip

If you installed Python from the official site, pip (Python's package manager) should be included.

Verify its installa on by typing pip --version in your terminal.

Step 2: Installing Libraries

Install scikit-learn:

pip install scikit-learn

Install TensorFlow:

pip install tensorflow

Install Gym

pip install gym

Robo cs Interface

Step 1: Choosing a Simulator

Gazebo is recommended. It provides a virtual environment where robots and their sensors can be
simulated.

Step 2: Installa on

Gazebo:

Installa on varies depending on the OS. Detailed instruc ons can be found on the official website.

Step 3: Integra on with Python

Gazebo offers Python APIs. A er installing the simulator, you can control and interface with them using
Python scripts. And we use the gazebo-python package.
 Tutorials & Exercises:

Supervised Learning

1. Linear Regression:

Objec ve: To understand the rela onship between robot inputs (e.g., motor voltages) and outputs (e.g.,
speed).

Theory:

Linear regression aims to establish a linear rela onship between the input and output variables. This is
represented by the equa on:

Y=mX+c Where:

 Y is the dependent variable (output).

 X is the independent variable (input).
 m is the slope of the line.
 c is the y-intercept.
Exercise:

Predic ng Housing Prices using the Boston Housing dataset

Dataset Descrip on: The Boston Housing dataset comprises 13 features related to housing in Boston, like
crime rate, average number of rooms per dwelling, etc. The target variable is the median value of owner-
occupied homes in $1000's.

Implementa on Steps:

 Load the dataset.

 Split the dataset into training and tes ng sets.
 Use a linear regression model from scikit-learn to train on the training set.
 Predict the median value for the test set and evaluate the model's performance using metrics
like Mean Absolute Error.

2. Logis c Regression:

Objec ve:

Classify discrete outcomes based on input variables.

Theory:

Logis c Regression is used for binary classifica on tasks. It predicts the probability that a given instance
belongs to a par cular category. The output is a value between 0 and 1, which can be thresholded to
assign a class label.

Exercise:

Determine Robot Stability

Scenario: Imagine a bipedal robot that sends various sensor readings (like lt angle, accelera on, etc.).
Based on these readings, determine if the robot is in a stable posi on.

Implementa on Steps:

 Collect and preprocess sensor data: normalize values, handle missing data, etc.
 Split the data into training and tes ng sets.
 Train a logis c regression model using the training data.
 Evaluate the model using accuracy, precision, and recall metrics on the test set.

3. Support Vector Machines (SVM):

Objec ve:

Understand margin maximiza on for classifica on.

Theory:

SVM tries to find the op mal hyperplane that correctly classifies data into categories, maximizing the
margin between different classes.

Exercise:

Classifying Robot Encountered Objects

Scenario: A robot with a camera scans its environment. Using features extracted from images (like shape,
texture, and size), it should classify objects into predefined categories.

Implementa on Steps:

 Extract features from images.

 Split the data into training and test sets.
 Train an SVM model using the training set.
 Test the model's accuracy on the test set.

4. Ar ficial Neural Networks (ANN):

Objec ve:

Delve into complex pa ern recogni on using neural networks.

Theory:

ANNs are inspired by the neural networks present in the human brain. They consist of interconnected
nodes (neurons) that can learn pa erns from data.

Exercise:

Gesture Recogni on for Robot Commands

Scenario: A robot equipped with cameras can recognize specific human gestures. For example, a raised
hand might signal the robot to stop.

Implementa on Steps:

 Capture and preprocess images or video frames of various gestures.

 Design a neural network with input layers (for image pixels), hidden layers, and output layers (for
different gestures).
 Train the neural network using the processed images.
 Test the robot's real- me gesture recogni on ability.

Unsupervised Learning

Unsupervised Learning

1. K-means Clustering:

Objec ve: To understand data clustering based on inherent structures in the data.

Theory:

K-means is an itera ve clustering algorithm that aims to par on n observa ons into k clusters in which
each observa on belongs to the cluster with the nearest mean. The algorithm converges when the
assignments no longer change.

Exercise:

Grouping Robot Sensor Readings

Scenario: A robot constantly provides various sensor readings. By clustering these readings, one can
iden fy pa erns or anomalies which can be indica ve of robot behavior or health.

Implementa on Steps:

 Collect sensor readings: this can include temperature, pressure, lt angle, etc.
 Normalize the data.
 Apply the K-means algorithm and determine an op mal number of clusters.
 Visualize clusters to interpret pa erns or anomalies.

2. Principal Component Analysis (PCA):

Objec ve: To grasp dimensionality reduc on techniques to focus on the most significant data aspects.

Theory:

PCA transforms the original variables to a new set of variables (principal components) which are
orthogonal (uncorrelated), and which reflect the maximum variance in the data.

Exercise:

Dimensionality Reduc on of Robot Sensor Data

Scenario: Robots, especially those with mul ple sensors, generate a plethora of data. Reducing
dimensionality can aid in processing and understanding this data more effec vely.

Implementa on Steps:

 Gather extensive sensor data.

 Standardize the dataset.
 Apply PCA and select a number of principal components that capture a significant amount of
variance.
 Visualize data in reduced dimensions.

Reinforcement Learning

1. Q-Learning:

Objec ve:

To understand value-based reinforcement learning.

Exercise:

Program a robot in a simulated environment to navigate through a simple maze using Q-tables.

Implementa on Steps:

1. Setup Environment:
 Represent the maze as a grid. Each cell is a state.
 Ac ons are movements like up, down, le , right.
 Rewards are given for reaching the goal and penal es for hi ng walls.

maze = [['S', ' ', ' '],

[' ', 'W', ' '],

[' ', ' ', 'G']]

# S: start, W: wall, G: goal

2. Ini alize Q-table:

Rows represent states and columns represent ac ons.

import numpy as np
n_states = len(maze) * len(maze[0])
n_ac ons = 4 # up, down, le , right
Q = np.zeros((n_states, n_ac ons))

3.Q-learning Algorithm:
ALPHA = 0.1
GAMMA = 0.9
EPSILON = 0.1

for episode in range(num_episodes):

state = star ng_state
while not is_terminal(state):
if np.random.rand() < EPSILON:
ac on = np.random.choice(n_ac ons)
else:
ac on = np.argmax(Q[state])

next_state, reward = take_ac on(state, ac on)

Q[state, ac on] = Q[state, ac on] + ALPHA * (reward + GAMMA * np.max(Q[next_state]) -
Q[state, ac on])
state = next_state

2. Deep Q-Network (DQN):

2. Deep Q-Network (DQN):

Objec ve: To integrate deep learning with Q-learning.

Exercise:

Use a DQN to op mize robot naviga on in a more complex environment.

Implementa on Steps:

1. Neural Network Setup:

Approximate Q-values using a neural network.

import tensorflow as

model = .keras.Sequen al([

.keras.layers.Dense(24, ac va on='relu'),

.keras.layers.Dense(24, ac va on='relu'),

.keras.layers.Dense(n_ac ons)

])

model.compile(op mizer='adam', loss='mse')

2. DQN Algorithm:

With experience replay and target network for stability.

from collec ons import deque

replay_memory = deque(maxlen=2000)
TARGET_UPDATE_FREQ = 10
BATCH_SIZE = 64
target_model = .keras.models.clone_model(model)
target_model.set_weights(model.get_weights())
for episode in range(num_episodes):
state = star ng_state
while not is_terminal(state):
if np.random.rand() < EPSILON:
ac on = np.random.choice(n_ac ons)
else:
ac on = np.argmax(model.predict(state)
next_state, reward = take_ac on(state, ac on)
replay_memory.append((state, ac on, reward, next_state))
# Sample minibatch and update
minibatch = random.sample(replay_memory, BATCH_SIZE)
for s, a, r, ns in minibatch:
target = r + GAMMA * np.max(target_model.predict(ns))
current = model.predict(s)
current[0][a] = target
model.train_on_batch(s, current)
# Update target network
if episode % TARGET_UPDATE_FREQ == 0:
target_model.set_weights(model.get_weights())
state = next_state

3. Proximal Policy Op miza on (PPO):

Objec ve: to learn policy-based reinforcement learning.

Exercise:

Op mize robot movement strategy in a dynamic environment using PPO.

Implementa on Steps:

1. Policy Network:

Neural network to model the policy.

policy_model = .keras.Sequen al([

.keras.layers.Dense(24, ac va on='relu'),

.keras.layers.Dense(24, ac va on='relu'),

.keras.layers.Dense(n_ac ons, ac va on='so max')

])
PPO Algorithm:

CLIP_EPSILON = 0.2
EPOCHS = 10
for episode in range(num_episodes):
# Gather trajectories using current policy
trajectories = gather_trajectories(policy_model)
for _ in range(EPOCHS):
for trajectory in trajectories:
state, ac on_prob, reward, value = trajectory
old_prob = ac on_prob[ac on]
new_prob = policy_model.predict(state)[0][ac on]
ra o = new_prob / old_prob
# Clip the objec ve
surrogate_obj = .minimum(
ra o * advantage,
.clip_by_value(ra o, 1.0 - CLIP_EPSILON, 1.0 + CLIP_EPSILON) * advantage
)
loss = -surrogate_obj + vf_loss + entropy_bonus
# Backpropaga on to update policy_model
5. Evalua on and Feedback Tools:

1. Quizzes to test the understanding of the students.

2. Pla orms for students to submit their code and projects.