java-reinforcement-learning

Package provides java implementation of reinforcement learning algorithms as described in the book "Reinforcement Learning: An Introduction" by Sutton

Features

The following reinforcement learning are implemented:

• R-Learn
• Q-Learn
• Q-Learn with eligibility trace
• SARSA
• SARSA with eligibility trace
• Actor-Critic
• Actor-Critic with eligibility trace

The package also support a number of action-selection strategy:

• soft-max
• epsilon-greedy
• greedy
• Gibbs-soft-max

Install

Add the following dependency to your POM file:

<dependency>
<groupId>com.github.chen0040</groupId>
<artifactId>java-reinforcement-learning</artifactId>
<version>1.0.5</version>
</dependency>

Application Samples

The application sample of this library can be found in the following repositories:

• java-reinforcement-learning-tic-tac-toe
• java-reinforcement-learning-flappy-bird

Usage

Create Agent

An reinforcement agent, say, Q-Learn agent, can be created by the following java code:

importcom.github.chen0040.rl.learning.qlearn.QAgent;
intstateCount = 100;
intactionCount = 10;
QAgentagent = newQAgent(stateCount, actionCount);

The agent created has a state map of 100 states, and 10 different actions for its selection.

For Q-Learn and SARSA, the eligibility trace lambda can be enabled by calling:

agent.enableEligibilityTrace(lambda)

Select Action

At each time step, a action can be selected by the agent, by calling:

intactionId = agent.selectAction().getIndex();

If you want to limits the number of possible action at each states (say the problem restrict the actions avaliable at different state), then call:

Set<Integer> actionsAvailableAtCurrentState = world.getActionsAvailable(agent);
intactionTaken = agent.selectAction(actionsAvailableAtCurrentState).getIndex();

The agent can also change to a different action-selection policy available in com.github.chen0040.rl.actionselection package, for example, the following code switch the action selection policy to soft-max:

agent.getLearner().setActionSelection(SoftMaxActionSelectionStrategy.class.getCanonicalName());

State-Action Update

Once the world state has been updated due to the agent's selected action, its internal state-action Q matrix will be updated by calling:

intnewStateId = world.update(agent, actionTaken);
doublereward = world.reward(agent);
agent.update(actionTaken, newStateId, reward);

Sample code

Sample code for R-Learn

importcom.github.chen0040.rl.learning.rlearn.RAgent;
intstateCount = 100;
intactionCount = 10;
RAgentagent = newRAgent(stateCount, actionCount);
Randomrandom = newRandom();
agent.start(random.nextInt(stateCount));
for(inttime=0; time < 1000; ++time){
intactionId = agent.selectAction().getIndex();
System.out.println("Agent does action-"+actionId);
intnewStateId = world.update(agent, actionId);
doublereward = world.reward(agent);
System.out.println("Now the new state is " + newStateId);
System.out.println("Agent receives Reward = "+reward);
agent.update(actionId, newStateId, reward);
}

Alternatively, you can use RLearner if you want to learning after the episode:

classMove {
intoldState;
intnewState;
intaction;
doublereward;
publicMove(intoldState, intaction, intnewState, doublereward) {
this.oldState = oldState;
this.newState = newState;
this.reward = reward;
this.action = action;
}
}
intstateCount = 100;
intactionCount = 10;
RLearneragent = newRLearner(stateCount, actionCount);
Randomrandom = newRandom();
intcurrentState = random.nextInt(stateCount));
List<TupleThree<Integer, Integer, Double>> moves = newArrayList<>();
for(inttime=0; time < 1000; ++time){
intactionId = agent.selectAction(currentState).getIndex();
System.out.println("Agent does action-"+actionId);
intnewStateId = world.update(agent, actionId);
doublereward = world.reward(agent);
System.out.println("Now the new state is " + newStateId);
System.out.println("Agent receives Reward = "+reward);
intoldStateId = currentState;
moves.add(newMove(oldStateId, actionId, newStateId, reward));
currentState = newStateId;
}
for(inti=moves.size()-1; i >= 0; --i){
Movemove = moves.get(i);
agent.update(move.oldState, move.action, move.newState, world.getActionsAvailableAtState(nextStateId), move.reward);
}

Sample code for Q-Learn

importcom.github.chen0040.rl.learning.qlearn.QAgent;
intstateCount = 100;
intactionCount = 10;
QAgentagent = newQAgent(stateCount, actionCount);
Randomrandom = newRandom();
agent.start(random.nextInt(stateCount));
for(inttime=0; time < 1000; ++time){
intactionId = agent.selectAction().getIndex();
System.out.println("Agent does action-"+actionId);
intnewStateId = world.update(agent, actionId);
doublereward = world.reward(agent);
System.out.println("Now the new state is " + newStateId);
System.out.println("Agent receives Reward = "+reward);
agent.update(actionId, newStateId, reward);
}

Alternatively, you can use QLearner if you want to learning after the episode:

classMove {
intoldState;
intnewState;
intaction;
doublereward;
publicMove(intoldState, intaction, intnewState, doublereward) {
this.oldState = oldState;
this.newState = newState;
this.reward = reward;
this.action = action;
}
}
intstateCount = 100;
intactionCount = 10;
QLearneragent = newQLearner(stateCount, actionCount);
Randomrandom = newRandom();
intcurrentState = random.nextInt(stateCount));
List<TupleThree<Integer, Integer, Double>> moves = newArrayList<>();
for(inttime=0; time < 1000; ++time){
intactionId = agent.selectAction(currentState).getIndex();
System.out.println("Agent does action-"+actionId);
intnewStateId = world.update(agent, actionId);
doublereward = world.reward(agent);
System.out.println("Now the new state is " + newStateId);
System.out.println("Agent receives Reward = "+reward);
intoldStateId = currentState;
moves.add(newMove(oldStateId, actionId, newStateId, reward));
currentState = newStateId;
}
for(inti=moves.size()-1; i >= 0; --i){
Movemove = moves.get(i);
agent.update(move.oldState, move.action, move.newState, move.reward);
}

Sample code for SARSA

importcom.github.chen0040.rl.learning.sarsa.SarsaAgent;
intstateCount = 100;
intactionCount = 10;
SarsaAgentagent = newSarsaAgent(stateCount, actionCount);
Randomrandom = newRandom();
agent.start(random.nextInt(stateCount));
for(inttime=0; time < 1000; ++time){
intactionId = agent.selectAction().getIndex();
System.out.println("Agent does action-"+actionId);
intnewStateId = world.update(agent, actionId);
doublereward = world.reward(agent);
System.out.println("Now the new state is " + newStateId);
System.out.println("Agent receives Reward = "+reward);
agent.update(actionId, newStateId, reward);
}

Alternatively, you can use SarsaLearner if you want to learning after the episode:

classMove {
intoldState;
intnewState;
intaction;
doublereward;
publicMove(intoldState, intaction, intnewState, doublereward) {
this.oldState = oldState;
this.newState = newState;
this.reward = reward;
this.action = action;
}
}
intstateCount = 100;
intactionCount = 10;
SarsaLearneragent = newSarsaLearner(stateCount, actionCount);
Randomrandom = newRandom();
intcurrentState = random.nextInt(stateCount));
List<TupleThree<Integer, Integer, Double>> moves = newArrayList<>();
for(inttime=0; time < 1000; ++time){
intactionId = agent.selectAction(currentState).getIndex();
System.out.println("Agent does action-"+actionId);
intnewStateId = world.update(agent, actionId);
doublereward = world.reward(agent);
System.out.println("Now the new state is " + newStateId);
System.out.println("Agent receives Reward = "+reward);
intoldStateId = currentState;
moves.add(newMove(oldStateId, actionId, newStateId, reward));
currentState = newStateId;
}
for(inti=moves.size()-1; i >= 0; --i){
Movenext_move = moves.get(i);
if(i != moves.size()-1) {
next_move = moves.get(i+1);
}
Movecurrent_move = moves.get(i);
agent.update(current_move.oldState, current_move.action, current_move.newState, next_move.action, current_move.reward);
}

Sample code for Actor Critic Model

importcom.github.chen0040.rl.learning.actorcritic.ActorCriticAgent;
importcom.github.chen0040.rl.utils.Vec;
intstateCount = 100;
intactionCount = 10;
ActorCriticAgentagent = newActorCriticAgent(stateCount, actionCount);
VecstateValues = newVec(stateCount);
Randomrandom = newRandom();
agent.start(random.nextInt(stateCount));
for(inttime=0; time < 1000; ++time){
intactionId = agent.selectAction().getIndex();
System.out.println("Agent does action-"+actionId);
intnewStateId = world.update(agent, actionId);
doublereward = world.reward(agent);
System.out.println("Now the new state is " + newStateId);
System.out.println("Agent receives Reward = "+reward);
System.out.println("World state values changed ...");
for(intstateId = 0; stateId < stateCount; ++stateId){
stateValues.set(stateId, random.nextDouble());
}
agent.update(actionId, newStateId, reward, stateValues);
}

Alternatively, you can use ActorCriticLearner if you want to learning after the episode:

classMove {
intoldState;
intnewState;
intaction;
doublereward;
publicMove(intoldState, intaction, intnewState, doublereward) {
this.oldState = oldState;
this.newState = newState;
this.reward = reward;
this.action = action;
}
}
intstateCount = 100;
intactionCount = 10;
SarsaLearneragent = newSarsaLearner(stateCount, actionCount);
Randomrandom = newRandom();
intcurrentState = random.nextInt(stateCount));
List<TupleThree<Integer, Integer, Double>> moves = newArrayList<>();
for(inttime=0; time < 1000; ++time){
intactionId = agent.selectAction(currentState).getIndex();
System.out.println("Agent does action-"+actionId);
intnewStateId = world.update(agent, actionId);
doublereward = world.reward(agent);
System.out.println("Now the new state is " + newStateId);
System.out.println("Agent receives Reward = "+reward);
intoldStateId = currentState;
moves.add(newMove(oldStateId, actionId, newStateId, reward));
currentState = newStateId;
}
for(inti=moves.size()-1; i >= 0; --i){
Movenext_move = moves.get(i);
if(i != moves.size()-1) {
next_move = moves.get(i+1);
}
Movecurrent_move = moves.get(i);
agent.update(current_move.oldState, current_move.action, current_move.newState, next_move.action, current_move.reward);
}

Save and Load RL models

To save the trained RL model (say QLeanrer):

QLearnerlearner = newQLearner(stateCount, actionCount);
train(learner);
Stringjson = learner.toJson();

To load the trained RL model from json:

QLearnerlearner = QLearn.fromJson(json);