it predicts r + gamma*r' + gamma^2*r'' when being in state s. Indirect Reward: A model that predicts future rewards, i.e.It predicts the reward value using a softmax over 100 bins.

The reward is bound to the range -100 to +100. for (s, a, r), (s', a', r') it predicts r when being in s'.

Direct Reward: A model that predicts the direct reward, i.e.

The steering wheel position is approximated using a separate CNN.) Not merging this component with Embedder 1 allows to theoretically keep the weights from pretraining fixed. The current gear state and the speed is read out from the route advisor.

(These are: (1) Previous actions, (2) whether the gear is in reverse mode, (3) steering wheel position, (4) previous and current speeds. Embedder 2: Takes the results of Embedder 1 and converts them into a vector.

the mirrors are always at roughly the same positions, so it is logical to detect them partially by their position.) Instance Normalization was used, because Batch Normalization regularly broke, resulting in zero-only vectors during test/eval (seemed like a bug in the framework, would usually go away when using batch sizes >= 2 or staying in training mode). The two gradient inputs (see image) are just gradients from 1 to 0 which are supposed to give positional information.

Embedder 1: A CNN that is pretrained in semi-supervised fashion.

positions of cars and lanes in example images)Īs well as some automatically generated ones (e.g. That training uses some manually created annotations (e.g. To make training faster, a semi-supervised pretraining is applied to the first stage of the whole model (similar to Loss is its own Reward: Self-Supervision for Reinforcement Learning, though here only applied once at the start).

(While that paper uses only predictions for the next timestep, here predictions for the next T timesteps are generated via an LSTM.) The basic training method follows the standard reinforcement learning approach from the original Atari paper.Īdditionally, a separation of Q-values in V (value) and A (advantage) - as described in Dueling Network Architectures for Deep Reinforcement Learning - is used.įurther, the model tries to predict future states and rewards, similar to the description in Deep Successor Reinforcement Learning. it can not directly set the steering wheel angle). It is trained (mostly) via reinforcement learning and only has access to the buttons W, A, S and D The resulting AI will automatically steer, accelerate and brake. This repository contains code to train and run a self-driving truck in Euro Truck Simulator 2.

Euro truck simulator 1 how to upsift