Skip to content

Commit b278fb0

Browse files
nickswalkernickswalker
committed
Track TwoJoint report
1 parent a965f23 commit b278fb0

File tree

1 file changed

+263
-0
lines changed

1 file changed

+263
-0
lines changed

‎TwoJoint/report/report.tex

Lines changed: 263 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,263 @@
1+
\documentclass{article}
2+
3+
\usepackage{graphicx} % Required for the inclusion of images
4+
\usepackage{natbib} % Required to change bibliography style to APA
5+
\usepackage{amsmath} % Required for some math elements
6+
\usepackage[final]{pdfpages}
7+
\usepackage[parfill]{parskip}
8+
\usepackage{bm}
9+
10+
\usepackage[utf8]{inputenc}
11+
\usepackage{mathpazo}
12+
\usepackage[T1]{fontenc}
13+
\usepackage{textcomp}
14+
\usepackage{gensymb}
15+
\usepackage{mathtools}
16+
\usepackage{algorithm}
17+
\usepackage{algpseudocode}
18+
\usepackage{fancyhdr}
19+
\usepackage{booktabs}
20+
21+
\usepackage{amsmath,amsfonts,amssymb}
22+
23+
\setlength\parindent{0pt} % Removes all indentation from paragraphs
24+
25+
\renewcommand{\labelenumi}{\alph{enumi}.} % Make numbering in the enumerate environment by letter rather than number (e.g. section 6)
26+
27+
%\usepackage{times} % Uncomment to use the Times New Roman font
28+
29+
%----------------------------------------------------------------------------------------
30+
% DOCUMENT INFORMATION
31+
%----------------------------------------------------------------------------------------
32+
33+
\title{Embedded Agent for Arm Control} % Title
34+
35+
\author{\textsc{Nick Walker}} % Author name
36+
37+
\date{CS394R Fall 2016} % Date for the report
38+
39+
\begin{document}
40+
41+
\maketitle % Insert the title, author and date
42+
43+
44+
%----------------------------------------------------------------------------------------
45+
% SECTION 1
46+
%----------------------------------------------------------------------------------------
47+
48+
\section{Motivation}
49+
50+
Embedded implementations of reinforcement learning agents will be key to enabling expressive interactions with everyday objects. These agents, which may not always have access to off-board computing resources, must learn from real-world interactions using limited memory and clock cycles. To explore the issues facing embedded agents, I built a robot and solved a simple control problem using reinforcement learning.
51+
52+
53+
%----------------------------------------------------------------------------------------
54+
% SECTION 2
55+
%----------------------------------------------------------------------------------------
56+
57+
\section{Introduction}
58+
59+
\subsection{Learning Platform}
60+
61+
I based the platform on the Atmel ATmega328p, a cheap and widely available 8-bit microcontroller. It has 32kb of program memory, 2kb of SRAM and operates at 16MHz. It is used by popular hobbyist electronics boards like the Arduino Uno, and for this project, I used an Arduino Pro Mini. It does not have a floating point unit, though the Arduino runtime provides a software implementation that mimics 32-bit IEEE float.
62+
63+
In order to provide a reasonably complex task for the agent, I decided to build a two degrees of freedom arm with an LED actuator. The arm's joints are SG90 micro-servos, each capable of 155$\degree$ of rotation. The base joint is fixed to a surface, and the elbow joint is mounted to the base with a pair of three-inch dowel rods. The elbow joint controls another rod which is tipped with an LED. The motors are connected such that the middle of their rotation ranges align.
64+
65+
66+
\begin{figure}
67+
\centering
68+
\includegraphics[width=10cm]{../photos/eagle_small.jpg}
69+
\includegraphics[width=10cm]{../photos/arm_detail_small.jpg}
70+
\caption{Top: The learning platform. The middle of the joints rotation range is the configuration that has both joints pointing north. Two photocells are pictured, but only the left one was used. Black tape marks the unused position for a third. Bottom: The arm in detail. Note that the LED has its sides covered to reduce the width of its light cone. }
71+
\label{fig:platform}
72+
\end{figure}
73+
74+
75+
\subsection{Problem}
76+
77+
The agent must point the LED at a photocell fixed to the surface in as few movements as possible, activating the LED as little as possible. It begins from a random initial configuration. The episode ends when the photocell reads above a defined threshold.
78+
79+
\[ r(s,a,s') = \left\{
80+
\begin{array}{ll}
81+
50 & \text{if photocell activated} \\
82+
-2 & \text{if LED activated and photocell not activated} \\
83+
-1 & \text{otherwise} \\
84+
\end{array}
85+
\right. \]
86+
87+
88+
\subsection{Learning Approaches}
89+
90+
\subsubsection{Tabular?}
91+
92+
The servo control library used for this project allows motor targets to be set with single-degree precision, so a single joint can have integer positions in $[0\degree, 155\degree]$. For each configuration, the LED can be either on or off, so there are a total of 48050 states. At a given time step, the agent may choose to keep a joint fixed, move it left, or move it right and it can choose to activate or deactivate the LED. Assuming we restrict the agent to movements of unit magnitude, this means there are eighteen actions.
93+
94+
A state-action value table based on this representation, assuming 4 byte floats, would occupy more than 4 megabytes of memory. Due to the spread of the light of the LED, it may be feasible to reduce the fidelity of the joint state representation and still achieve good performance, and because the optimal policy will likely always elect to move a joint, we may be able to remove actions which do not move a joint with little adverse effect. Even then, the microcontroller could only theoretically fit 10\% of all state action pairs (less without careful optimization, as the stack needs to live in memory as well). A tabular approach is not feasible due to the system's memory constraints.
95+
96+
\subsubsection{Function Approximation}
97+
98+
Approximation allows tight control of the amount of memory being used to represent the value function. Because of the complex update step however, only at most half of the memory can be used for storing weights. Even then, the update step must be implemented carefully to control the stack size. Consider the episodic semi-gradient one-step Sarsa update:
99+
100+
\begin{equation}\label{eqn:update}
101+
\bm{\theta}_{t+1} =
102+
\bm{\theta}_t +
103+
\alpha \Big[
104+
R_{t+1} +
105+
\gamma \hat{q}(S_{t+1}, A_{t+1}\, \bm{\theta}_t)
106+
- \hat{q}(S_t, A_t, \bm{\theta}_t)
107+
\Big]
108+
\Delta\hat{q}(S_t, A_t, \bm{\theta}_t)\tag{1}
109+
\end{equation}
110+
111+
And in the linear case:
112+
113+
\begin{equation}\label{eqn:linear_update}
114+
\bm{\theta}_{t+1} =
115+
\bm{\theta}_t +
116+
\alpha \Big[
117+
R_{t+1} + \gamma\, \bm{\theta}_t^\top\bm{\phi}_{t+1} \space
118+
- \bm{\theta}_t^\top\bm{\phi}_t
119+
\Big]\bm{\phi}_t\tag{2}
120+
\end{equation}
121+
122+
It is possible to implement the update using only $n$ additional space, where $n$ is the number of weights, but this is easy to do incorrectly. If the action selection step is placed after the memory allocation, the stack will consume $2n$ memory; maximizing the value function over possible next states requires an additional $n$ stack space.
123+
124+
\begin{algorithm}
125+
\caption{Memory-conscious Episodic Semi-gradient One-step Sarsa}
126+
\label{alg:update}
127+
\begin{algorithmic}[1] % The number tells where the line numbering should start
128+
\Procedure{Update}{$S_t$, $A_t$, $S_{t+1}$, \bm{$\theta$}}
129+
\State $A_{t+1} \gets $ choose action from $S_{t+1}$ according to policy
130+
\State Allocate \bm{$x$} to be a vector the size of \bm{$\theta$}, and floats $r$ and $a$
131+
\State $r\gets r(S_t$, $A_t$, $S_{t+1}$)
132+
\State $\bm{x} \gets \phi(S_{t+1}, A_{t+1})$ \Comment store $\phi_{t+1}$
133+
\State $a \gets \bm{\theta}^\top \bm{x}$ \Comment calculate $v(S_{t+1},A_{t+1})$ so we can discard $\phi_{t+1}$
134+
\State $\bm{x} \gets \phi(S_{t}, A_{t})$ \Comment store $\phi_t$
135+
\State $a \gets r + \gamma v - \bm{\theta}^\top \bm{x}$ \Comment $a$ is now the bracketed term in eq. $\ref{eqn:update}$
136+
\State $\bm{x} \gets (\alpha \space a)\bm{x}$ \Comment $\bm{x}$ is now the weight update
137+
\State \bm{$\theta} \gets \bm{\theta} + x$
138+
\
139+
\EndProcedure
140+
\end{algorithmic}
141+
\end{algorithm}
142+
143+
An approximation approach implemented on the microcontroller can use at most 1kb of RAM, or 250 features (less the incidental stack space required during the update step). It was not clear that this would be sufficient for good performance, but it was the only feasible approach. I implemented a semi-gradient one-step Sarsa agent using a linear function approximator.
144+
145+
I have not dwelt on time efficiency since, even with software floating point operations, 16MHz permits a fair amount of computation. For this project, I was satisfied as long as actions could be selected more quickly than the servos could execute them. Meeting this deadline, which was about 100ms, or 1.6 million cycles, was not an issue, even while the device was also streaming logging information over serial. If time performance requirements were tighter, special attention would need to be paid to the action selection process, which involves $|A|$ value function queries, each costing $n$ multiplications
146+
147+
148+
149+
%----------------------------------------------------------------------------------------
150+
% SECTION 3
151+
%----------------------------------------------------------------------------------------
152+
153+
\section{Experimental Setup}
154+
155+
156+
The agent learns for 50 episodes using $\alpha =$ 0.2, $\gamma =$ 0.99, following an $\epsilon$-greedy policy with $\epsilon=$ 0.1. Then, the agent operates in an evaluation mode for 50 episodes with $\alpha=\epsilon=0.0$. During this period, episodes are limited to 200 steps in case the agent executes a policy that never reaches the goal. A small delay is used between action execution and sensing to allow the arm to settle. The photocell threshold is calibrated before every session to prevent spurious activations.
157+
158+
\subsection{Features}
159+
160+
I used a simple discretization of the state space. The range of each joint was divided into 8\space\space20\degree\space sections. The two sets of section features along with a binary feature for the LED state were then gridded, resulting in 128 mutually exclusive binary features.
161+
162+
Because of the large number of actions, it was not feasible to maintain separate approximators per action, so I used three action features: two characterizing the direction of each joint's movement, taking values in $\{0, 1, -1\}$, and one binary feature describing whether or not the LED is activated by the action. Clearly, three features is insufficient to encode the differences in value for all actions across the state space, however there was not enough memory to grid these features with the state features. The total number of weights in the representation is still below the theoretical maximum number of weights for the microcontroller, so additional features could have been added with more time.
163+
164+
165+
\subsection{Actions}
166+
167+
At each time step, the agent selects one entry from each column to form its action. If the LED is already on and the agent chooses an action that includes turning the LED on, the light remains activated for the next timestep. The behavior is symmetric in the LED off case.
168+
169+
\begin{center}
170+
\begin{tabular}{ l l l}
171+
Joint 1 & Joint 2 & LED\\ \midrule
172+
Move left & Move left & Turn on\\
173+
Move right & Move right & Turn off\\
174+
No movement & No movement &
175+
176+
\end{tabular}
177+
\end{center}
178+
179+
To better support the state representation, I set the agent's movement increment to 20\degree. This ensures that the state resulting from any movement has a different feature vector than the previous state. This change does limit the granularity of the agent's movements, but without it, the agent must randomly escape the 20\degree\space sections by repeatedly making smaller movements, which makes learning take much longer.
180+
181+
%----------------------------------------------------------------------------------------
182+
% SECTION 4
183+
%----------------------------------------------------------------------------------------
184+
185+
186+
\section{Results}
187+
188+
189+
\begin{figure}[h]
190+
\begin{center}
191+
\includegraphics[width=\textwidth]{figure_0.pdf}
192+
\caption{The agent's performance. The first 50 episodes are learning time, the rest are evaluations. Due to the time expense of collecting data from the robot, I could only run ten trials. Still, within reasonable confidences, the agent's evaluation averages a reward of 44, demonstrating that it was able to quickly point the LED towards the photocell from arbitrary start positions.}
193+
\end{center}
194+
\end{figure}
195+
196+
\subsection{Discussion}
197+
198+
The agent learns and generalizes a fairly good policy within its first fifty steps. The evaluation period demonstrates that the learned policy performs well from arbitrary start positions.
199+
200+
Even though it points the LED at the photocell consistently, the agent does not learn the LED reward dynamics. As can be seen in the video\footnote{https://youtu.be/SCv1AomFDG0}, it opts to leave the light on at all times. This is not unexpected, since it lacks features that describe the interaction of the joint position with the value of actions that turn the LED on. The weight associated with the LED activation action-feature is forced to represent the value of LED actions from any state. Averaged across the entire state space, turning the light on has a higher value than turning it off, so it always leaves it on. Efficient LED activation is not as important as joint movement, so it seems reasonable to prioritize detailed state-space representation over features that would better capture the use of the light.
201+
202+
203+
%----------------------------------------------------------------------------------------
204+
% SECTION 5
205+
%----------------------------------------------------------------------------------------
206+
207+
\section{Conclusions}
208+
209+
I have demonstrated that it is possible to implement an embedded agent that achieves good performance on an arm control task, even with extraordinary memory constraints. Further work could investigate more sophisticated features, or explore the performance and memory characteristics of policy gradient methods in the same domain.
210+
211+
\clearpage
212+
213+
214+
\section{Appendix A: Images}
215+
216+
\begin{figure}[!htb]
217+
\centering
218+
\includegraphics[width=10cm]{../photos/long_shutter_small.jpg}
219+
\caption{A 30 second exposure taken early in the agent's learning.}
220+
\label{fig:long_shutter}
221+
\end{figure}
222+
223+
\begin{figure}[!htb]
224+
\centering
225+
\includegraphics[width=10cm]{../photos/breadboard_small.jpg}
226+
\caption{Detail of the breadboard. A piezo buzzer beeps whenever an episode ends. Voltage dividers for the photocells are visible. The board hanging off the side provides voltage regulation. At the top left, two capacitor banks smooth the high draw that occurs when the servos start. }
227+
\label{fig:breadboard}
228+
\end{figure}
229+
230+
\begin{figure}[!htb]
231+
\centering
232+
\includegraphics[width=10cm]{../photos/action_detail_small.jpg}
233+
\caption{The agent illuminates the photocell.}
234+
\label{fig:action}
235+
\end{figure}
236+
237+
238+
\section{Appendix B: Bill of Materials}
239+
240+
241+
\begin{center}
242+
\begin{tabular}{ l c c p{5cm} }
243+
\toprule
244+
Component & Quantity & Unit Price (\$) & Note \\ \midrule
245+
Arduino Pro Mini, 5v & 1 & 3.00 & \\
246+
Breadboard & 1 & 4.00 & \\
247+
SG90 micro-servo & 2 & 3.00 & Any servo with sufficient range of motion.\\
248+
Dowel rods & 3 & 2.00 & \\
249+
1500uF capacitor & 2 & 1.00 & \\
250+
5v 2.5A power supply & 1 & 8.00 & If variable supply unavailable. \\
251+
330\ohm\space resistor & 4 & 0.01 \\
252+
LED & 1 & 0.10 & The brighter the better.\\
253+
Photocell & 3 & 0.10 & \\
254+
Rubber bands & 8 & 0.10 & \\
255+
Assorted jumpers & & 3.00 & \\
256+
Adhesives, project surface & & 4.00 & \\
257+
\bottomrule
258+
259+
\end{tabular}
260+
\end{center}
261+
262+
263+
\end{document}

0 commit comments

Comments
 (0)