[SOLVED] EE5904-Project 1

Description

Task 1

Implement the Q-learning algorithm, using the reward function as given in task1.mat and with the 𝜀𝜀-greedy exploration algorithm by setting 𝜀𝜀_𝑘𝑘, 𝜕𝜕_𝑘𝑘 and 𝛾𝛾.

It is required to run 10 times and record the running time. The maximum number of trails in each run is set to 3000. The final output is an optimal policy and the associated reward.

In order to avoid that the learning rate is too small, I set a threshold for learning rate to early stop the program when less than 0.005. The difference of Q-values can help us find it is converged or not. Here, I set 0.005 as the threshold of two maximum Q-value difference. If it is less than the threshold, we can regard the this run has converged and then terminate the trails to process the next run.

Obtained Results:

Optimal Trajectory Output:

The final reward is 1861.9157

Figure 1. Robot Trajectory

Comments:

1. From the table, only two conditions can reach the goal within 10 runs. The parameter of discount rate is 0.9 in these two satisfied situations. Therefore, it means, given this reward matrix, we have to set up a larger discount rate which takes into account future rewards more strongly.
2. Compared the learning rate/exploration rate curve from two satisfied situations with the other two, the larger rate can help reach the optimal goal. As you can see from the following curve figures (Figure 2), the rate calculated from 100/ (100 + k) and (1 + 5log(k))/k are larger than the rest.

Even for far future steps (Figure 3), these two can still generate larger learning rate than the others and update the Q-value.

1. Here I implemented the 𝜀𝜀-greedy exploration. For those feasible actions with maximal Q-values in a given state, I assigned the probability of (1 – exportation rate) to them. For the rest, I assigned the probability of (exploration rate/ count_number). However, when there are more than two maximal Qvalues, we have to randomly select one rather than selecting the first appeared one.

In conclusion, for the task 1 with this given reward matrix, we should select the rate function that can generate larger and more stable values. The same as the discount factor since we care more about the future steps in this situation.

Figure 2. Curve comparison within 100 iterations

Figure 3. Curve comparison within 3000 iterations

Task 2

Own parameter setting for unknown evaluation test.

Parameters tuning:

Here I tuned the parameter of discount rate, learning rate functions, and exploration rate to find the optimal policy with least running time.

1. Learning rate functions.

Based on this results, I select exp(-0.001k) as the learning rate function.  b. Discount rate.

Based on this discount rate table, I found 0.75 can have better performance.

1. Various fixed exploration rate with exp(-0.001k) as learning rate function.

Here I found when exploration rate is fixed as 0.5, it has least execution time in random created reward matrix. But it doesn’t for Task 1. However, when exploration rate is set as 0.7, task 1 can have least running time and time of random created mat is not bad. They all run less time than (a) where the parameter is changing for each iteration (around 0.7).  Thus, I select 0.7 as the fixed exploration probability.

In conclusion, my parameter setting is as below.

Learning rate function: exp(-0.001k), where k is iteration when attempting to reach end state.

Discount rate: 0.75

Fixed exploration probability: 0.7