I have been playing Red Dead Redemption 2, immersing myself in the Old West as I did with the first game. It’s an incredibly impressive game and there are many side activities that can keep you entertained in the world such as playing Poker in the saloon, Five Finger Fillet and Domino’s. I was disappointed to find out that Liars Dice is not in RDR2 unlike the first game, at least I haven’t found it yet. Instead I decided to code it up in R and play some outlaws off against each other.

The rules

The rules of Liars Dice are relatively straight forward but there are also many variants. For this project I’ll begin with the simpler variants. The object of the game is to be the last person standing with at least one die. The game is played in the following steps:

1. The pool of dice are equally distributed among the players
2. Each player rolls their dice and conceals their rolls from the other players
3. The starting player begins by placing a bid on how many dice of a particular value are on the table e.g. three 5’s
4. The next player then either raises the bid or calls if they do not think there are that many dice on the table.
   • If they raise the player can
     • Choose a value equal to or greater than the current value
     • Must increase the quantity by at least 1
   • If the player calls the dice are revealed and if the total number of dice value is less than or equal to the bid the player who calls loses and if it is greater the person who placed the bid loses.
5. The losing player places 1 die back in the box removing it from the game.
6. Steps 2-5 are repeated until there is only 1 player left

Other rules:

• When the dice value has reached 6 the next bid can be either for 6 or 1 and the quantity raised.

A popular variant on the game, and the one I tend to play is 1’s are considered wild and are included in the bid quantity meaning on reveal the 1’s and whatever the bid value was are counted. This makes the program a little more complicated so for this post I won’t include 1’s as wilds.

There are plenty more variants listed on the wiki page.

The simulation

Play one full round

The first step is to code a function to play one full round of the game from initial bid to one person losing their die. It begins by each player rolling their dice depending on how many they have left. An initial bid is placed by the agent in control and sets the dice value and dice quantity. Control is handed to the next player and given the dice value, dice quantity and the dice they rolled, make a decision on whether to call or raise the bid (more on this later). If they raise the bid, control moves to the next player and so on until an agent calls and the dice are revealed. Whoever loses gets penalty of -1 (i.e. loses 1 die). It is the penalty list which is returned by this function.

This has been set up to run on autopilot for simulation or manually against bots for fun (point of debate, is playing manually or running the simulations more fun?). If manual, the program will ask the user for input. Code can be found on Github.

# play a round of liars dice
liars.dice.round(players, control, player.dice.count,   agents, a = 1, verbose = 1)

Play the full game

The next step is to play as many rounds as needed until there is a winner. The inputs to this function are,

• The number of players
• The number of dice for each player (when this reaches 0 that player is out of the game)
• Autopilot
• verbose output toggle
• Agents list

The function begins by initialising the number of dice for each player and storing in a list. Control is randomly given to a player (there is actually an advantage for the starting player given the number of players, dice and how the agents make decisions which isn’t very sophisticated mind you). Now the above function is run until there is only 1 player left. The output of this function is the number of the winning player.

# play the game of liars dice
play.liars.dice(players = 4, num.dice = 6, auto = FALSE, verbose = 1, agents)

The agent

Now that the game is set up we get to the fun part which is how the agents make decisions. For this first post the agents are going to be very basic but in later posts will become more sophisticated.

Similar to Poker, players try to infer what dice the others have rolled based on their bid (dice value and quantity) and body langauge. Arguably picking up on the ques of someones body language is the most important part of the game to know if they’re bluffing or have a good hand, but also impossible to replicate here. Instead the agent will make a decision solely on the bid and the dice in their hand. How an agent can make a decision get’s complicated quickly so we’ll start small.

The good thing about this game is the probabilities are easy to calculate and therefore it’s easy to play the numbers game and choose the most likely scenario. The probability of the dice quantity is given by a binomial distribution. The probability that there is at least dice on the table with value is given by

For example the probability there are at least five 3’s in a pool of 20 dice is

Given their hand, the agent calculates this probability and randomly chooses whether to call or raise based on this probability. For example if there are 25 dice in play, 5 players, the bid is seven 3’s and the player has 2 in hand, the player is only really concerned with the probabilty there is at least another five 3’s in the 20 dice they can’t see. For the case above there is a 77% chance the agent will ‘call’ since it is more likely there is less than five 3’s. We could set a threshold for which the agent will ‘call’ if the probability is below 0.2 for example, but using the probability to either ‘call’ or ‘raise’ is suitable and adds some realism.

If the agent chooses to raise, it then decides if they are going to bluff which is decided randomly. This can be changed for each agent for example an agent may choose to bluff 100% of the time or 50% of the time or not at all. By bluffing the agent randomly selects a die value and increases it by 1 disregarding the dice they have rolled. If the agent chooses not to bluff they select the value which they have the most of in their hand and raises it by 1.

Ideally the agent would use the information from the previous players bids to better decide the next bid. For example a player that always raises to the same number is probably selecting the dice value they have most in their hand in which case sheds light on the other dice on the table. At the end of each round when the dice are revealed each player (if they have a good memory) can determine who was bluffing and who was playing the numbers. This information can be used as prior knowledge and incorporated into the probability calculations.

Assume we know player never bluffs, we know for sure that there is at least 1 die of that value in their hand therefore the probability that there is at least the bid quantity will be slightly higher. For example in the case, above since we know that there is at least 1 that is confirmed the problem then becomes, what is the probability there is at least four 3’s in 19 dice?

If a player always bluffs, then the bid is effectively random. This addition to the probability calculation isn’t too difficult but for now we’ll keep it simple. In the future we could allow the agents to learn the bluffing parameters of each player and refine their decisions.

To summarise, if a player chooses to bluff the dice value is chosen at random and quantity raise by 1. If they choose not to bluff they choose the value which they have the most of in their hand and raise the quantity by 1.

The aesthetics

Perhaps the most important part of this project are the aesthetics, a title and randomised dice blocks. Rather than just outputting numbers between 1 and 6 this function will build a blank dice block and then convert it into the value of the dice that is rolled. It’s best to see it in action.

liars.dice.title()

## 
## 
##      __       _______    __       ____    _____  
##     / /|     /__  __/|  /  |     / _  \  /  __/|
##    / / /     |_/ /|_|/ /   |    / /_| |  \ \__|/ 
##   / / /       / / /   / /| |   / _   /|   \ \     
##  / /_/_   __ / /_/   / __  |  / / | |/ __ / /|   
## /_____/| /_______/| /_/|_|_| /_/ /|_| /____/ /   
## |_____|/ |_______|/ |_|/ |_| |_|/ |_| |____|/    
## 
##      ____      _______    _____    ______
##     / _  \    /__  __/|  / ___/|  / ____/|
##    / / | |    |_/ /|_|/ / /|__|/ / /___ |/
##   / / / /|     / / /   / / /    / ____/|
##  / /_/ / / __ / /_/   / /_/_   / /____|/
## /_____/ / /_______/|  |____/| /______/|
## |_____|/  |_______|/  |____|/ |______|/

Dice(sample(1:6, 5, replace = TRUE))

##   _________    _________    _________    _________    _________  
##  /        /|  /        /|  /        /|  /        /|  /        /| 
## /________/ | /________/ | /________/ | /________/ | /________/ | 
## | o      | | | o      | | | o      | | | o      | | | o      | | 
## |        | | |        | | |   o    | | |   o    | | |        | | 
## |     o  | / |     o  | / |     o  | / |     o  | / |     o  | / 
## |________|/  |________|/  |________|/  |________|/  |________|/

Much better than numbers!

Play the game

To play a game of liars dice simply input the parameters into the following function.

# set the agent
# even if playing a manual game input the same number of agents as there are players
# the human player will overide one of them
agent1 <- build.agent(c(0.5,0.5))
agent2 <- build.agent(c(0.5,0.5))
agent3 <- build.agent(c(0.5,0.5))
agent4 <- build.agent(c(0.5,0.5))
agents <- list(agent1, agent2, agents3, agents4)

# play the game
play.liars.dice(auto = FALSE, players = 4, num.dice = 6, verbose = 1, agents = agents)

The game starts and player one has control. Sets the bid at four 5’s. Player 2 calls, the dice are reveals and since there are four 5’s player 2 loses a die. Player 2 now has control and starts the bidding.

Player 2 bids three 3’s. Player 3 raises the bid to four 5’s. Player 4 calls and player 3 loses a die since there are only three 5’s on the table. This continues until for a few more rounds until there is a winner.

And there you have it, you can now play Liar’s Dice just like in Red Dead Redemption, just worse graphics. The agent could definitely use some more brains, but still probably better than the NPC’s in Red Dead Redemption to be fair. At some point I may turn this into a shiny app just for fun.

Bluff or play the numbers

As mentioned, for this early version bluffing essentially means play random. For validation we can simulate many games and ensure that the numbers strategy defeats the random strategy. The game will be simulated 10000 times with 2 agents where one bluffs all the time and the other plays the numbers. We expect the “numbers guy” to win more than half the games, even if only slightly.

# 2 agents exactly the same
# 1 bluffs all the time nd the other plays the numbers
agent1 <- build.agent(c(1,0))
agent2 <- build.agent(c(0,1))
agents <- list(agent1, agent2)


# parallelise compute
library(parallel)
strt <- Sys.time()
n.cores <- detectCores()
clust <- makeCluster(n.cores)
set.seed(20181120)
clusterExport(clust, c("play.liars.dice", "liars.dice.round", "agents", "set.dice.value", "liars.dice.title", "agents"))
a <- parSapply(clust, 1:1e4, function(x) play.liars.dice(verbose = 0, auto = TRUE, players = 2, num.dice = 6, agents = agents))
stopCluster(clust)
end <- Sys.time()
end-strt

## Time difference of 37.1997 secs

# win results
table(a)

## a
##    1    2 
## 4934 5066

wins <- table(a)[2]
ggplot(data.frame(z = rbeta(1e5, wins, 1e4-wins)), aes(x = z)) + geom_histogram(fill = "darkturquoise", col = "grey20")

## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

To be fair, for any given game both agents are almost as equally likely to win. Not too surprising since they aren’t very sophisticated and the high degree of randomness in the game. However, playing the numbers will win just over 50% of the matches in the long run (that’s a long long run though!).

The simulation will be run again for 4 agents with different probabilities of bluffing.

agent1 <- build.agent(c(1, 0))
agent2 <- build.agent(c(0.75, 0.25))
agent3 <- build.agent(c(0.25, 0.75))
agent4 <- build.agent(c(0, 1))
agents <- list(agent1, agent2, agent3, agent4)

strt <- Sys.time()
clust <- makeCluster(n.cores)
set.seed(20181120)
clusterExport(clust, c("play.liars.dice", "liars.dice.round", "agents", "set.dice.value", "liars.dice.title", "agents"))
a <- parSapply(clust, 1:1e4, function(x) play.liars.dice(verbose = 0, auto = TRUE, players = 4, num.dice = 6, agents = agents))
stopCluster(clust)
end <- Sys.time()
end-strt

## Time difference of 1.164514 mins

# win results
table(a)

## a
##    1    2    3    4 
## 2379 2465 2540 2616

It’s nice to see the numbers in general going up as the bluffing probability goes to 0.

Bots need brains

This project was more about simulating the game rather than a high performing agent. But now that we have this we can start to give the agent more brains and play around with different learning methods. Given the random and probabilistic nature of the game it becomes an interesting and challenging RL problem. In a game like Tic Tac Toe all states of the game are known by the player, but in Liar’s Dice the opponents hands are unknown therefore the player doesn’t know for sure which state they are in. The challenge is to reduce the problem down to something more manageable. The results we have seen above are essentially the baseline that we can now improve on with more advanced methods.

The post Liar’s Dice in R appeared first on Daniel Oehm | Gradient Descending.