A couple of years ago, when I was learning Java programming, I thought of testing myself and my programming skills by writing a game in Java. Now, I'm not going to call it 'game programming', since game programming is way more than what I did. In fact, what I did was just a test for me. So I decided to write a game I was playing on my old Nokia E50 phone, a Slot Game.
This slot game I was playing on my phone was really simple. It had only 3 slots with different items in each. You had to push the Spin button in order to spin the slots, and you would win a small amount of coins if two or three slots were alike. Of course, 3 slots were better than 2. It is not really hard to make a game like this, but for a beginner it is good to start with. As I remember, this was probably the first program that I could tell others: 'Look at what I just did!'
So I started working on it (I remember using NetBeans at that time), firstly as console-only, and then using GUI. The first thing I did, was deciding what kind of images (actually their names, not the images themselves) I would use. I wrote this line of code:
- This slot game I was playing on my phone was really simple. It had only 3 slots with different items in each. You had to push the Spin button in order to spin the slots, and you would win a small amount of coins if two or three slots were alike. Of course, 3 slots were better than 2.
- You can deploy to staging environments, known as slots, and swap your new release to production with one click. Finally, you can split traffic between your slots to do A/B testing, or validate your new release with production traffic before swapping.
I also decided what would be the amount of 'money' that the user would win if he matched two or three symbols:
Digital Slot Inspection - jcsmileyjr.github.io. In their essence, scoped slots are just a more powerful superset of 'regular' slots. But since introducing them needed a slightly different API, Vue 2 had to distinguish between $slots and $scopedSlots. Vue 3 took the chance and unified both APIs. The less powerful old slots are gone and the old scoped slots are now just called 'slots'.
After that, I went on writing the code that was suposed to randomly choose one of the elements in the symbols array. This could be done by using the Math.random() method, or by calling the nextInt() method at a Random instance, or by the wrong way I used to do at the beginning:
Of course, I soon switched to calling Math.random(), and in order to get a number that I could use as an index for my array, I wrote this block of code:
So the variable choice was the random index that I could use to get a random item from the array (keep in mind that Math.random() returns a double between 0.0 and 1.0)
So after choosing 3 random items, I just printed them out at the console, saying whether there were none, two or three matches, and calculating the amount of money won, if there was any, with the given coefficient. It was a good start; I only had to think of the UI, and I suck at UI design. But for this one, all that I needed was a really simple design which I managed to code as I was planning.
Github Slots Free
For the items to show at slots, I just googled them and found 12 of them in a single sprite. I downloaded the sprite and started my old photo editing software which sometimes can really be magical; Microsoft Paint! I started cropping images from the sprite, paying attention to their dimensions that should be the same, 122px by 114px. Why these magical values? Just because!
What was left to do, was the UI. I could use the really-helpful drag-and-drop UI builder that ships with NetBeans, but I wanted to do it myself. I had a really hard time figuring out which kind of layout to use, since the only one I really knew was GridLayout. Anyway beside that, I managed to use FlowLayout and BorderLayout. There is a difference between them, but I'm not really capable of pointing that out, so you can check the online JavaDoc for them.
I managed to build the game, and started to play it. I figured out that the coefficients for multiplying the bet were too damn high, but I didn't care as long as I knew that the game worked.
My bad practices
As you can see, the source code of this simple game is in only one file. This is something that I don't like to do anymore. A better way to do it is by making the code as modular as it can be. By building small modular elements, dividing the UI from the logic of the application, you help yourself during the testing and debugging phase. So the first thing that I would like to change, is dividing the whole class Slots extends JFrame from the class that calls it.
This is done by firstly creating a file called Slots.java that will contain only the code for the UI. Then, creating an ActionListener that will listen to different button clicks (there are 5 different buttons). Finally, creating a class called App.java that will only create a Slots instance and make it run.
Basically, the App.java would look like this:
As I remember, SwingUtilities.invokeLater() is used to divide the UI thread from other threads, so if any UI changes are needed, they won't stall the application.
The Listener class, which might be called something like SlotButtonListener, might be something like this:
And finally, in the Slots class there would be only the code for defining the UI of the game. All the buttons would have SlotButtonListener as action listener.
Anyone who wants to change the code following these advices is free to do it. You can fork it anytime you want.
Do you have any Java programming advice for me? Feel free to comment below
Github Slots Game
Multi-armed banditry in Python with slots
Github Slatejs
Roy Keyes
22 Aug 2016 - This is a post on my blog.
I recently released slots, a Python library that implements multi-armed bandit strategies. If that sounds like something that won't put you to sleep, then please pip install slots and read on.
Github Sloth
Multi-armed bandits
The multi-armed bandit (MAB) problem is a classic problem of trying to make the best choice, while having limited resources to gain information. The classic formulation is the gambler faced with a number of slot machines (a.k.a. one-armed bandits). How can the gambler maximize their payout while spending as little money as possible determining which are the 'hot' slot machines and which are the 'cold' ones? In more generic, idealized terms, you are faced with n choices, each with an associated payout probability p_i, which are unknown to you. Your goal is to run as few trials, or pulls, as possible to establish the the choice with the highest payout probability. There are many variations on this basic problem.
Getting the best bang for your buck
Most of us do not spend our time strategizing about real slot machines, but we do see similar real-world problems, such as A/B testing or resource allocation. Because of that, strategies to solve the multi-armed bandit problem are of both practical and intellectual interest to the data scientist-types out there.
There are several strategies for solving the MAB problem. All of them attempt to strike a balance between exploration, searching for the best choice, and exploitation, using the current best choice. Because of these competing goals, determining if you are making optimal choices is not trivial. Instead of simply looking at your average payout from your trials, a quantity called regret is calculated. Intuitively, regret is the payout value lost by making the sequence of choices you have made relative to the payout you would have received having known the best choice from the start. Regret can thus be used as a stopping criterion for making a 'final', best choice.
Example strategy: epsilon greedy
To understand how you might approach the multi-armed bandit problem, consider the simplest reasonable strategy, epsilon greedy:
- You spend the fraction e of your trials randomly trying different choices, i.e. exploration.
- For the rest of the time, 1 - e, you always choose the choice with the current highest reward rate, i.e. exploitation.
Too little time exploring the options might lead you to stay with a sub-optimal choice. Too much time spent exploring might lead you to spend unnecessary money on options that you already know are sub-optimal.
slots
I wrote slots with two goals in mind, investigating the performance of different MAB strategies for educational purposes and creating a usable implementation of those strategies for real world scenarios. For both of these goals, a simple API with reasonable default values was desirable.
So what does slots do? Right now, as of version 0.3.0, it has implementations of a few basic MAB strategies and allows you to run those on test scenarios and with real, live data. Currently, those strategies include epsilon greedy, softmax, upper confidence bound (UCB1), and Bayesian bandits implementations.
For 'real world' (online) usage, test results can be sequentially fed into an MAB object. After each result is fed into the algorithm the next recommended choice is returned, as well as whether your stopping criterion is met.
What slots looks like:
Using slots to determine the best of 3 variations on a live website.
Make the first choice randomly, record the response, and input reward (arm 2 was chosen here). Run online_trial (input most recent result) until the test criteria is met.
The response of mab.online_trial() is a dict of the form:
Where:
- If the criterion is met,
new_trial=False. choiceis the choice of the next arm to try.bestis the current best estimate of the highest payout arm.
For testing and understanding MAB strategies, you can also assign probabilities and payouts to the different arms and observe the results. For example, to compare the regret value and max arm payout probability as more trials are performed with various strategies (in this case epsilon greedy, softmax, upper credibility bound (UCB1), and Bayesian bandits):
The resulting regret evolution:
The estimated payout probability of the 'best' arm after each trial. In this case, the actual payout probability of the best arm is 0.85.
Making it happen
Example slot signal definition. slots is on PyPI, so you can simply install with pip install slots. Currently, slots works with both Python 2.7+ and 3.4+ and the only dependency is Numpy.
The future
Github Slots Games
slots is open source (BSD license) and I welcome outside contributions. My desire is to make slots easy-to-use, robust, and featureful. If you are interested in slots or the multi-armed bandit problem in general, please check out the References and further reading section below.
This was a very brief overview of the rich subject of dealing with the MAB problem and the slots library. Please checkout slots and send me feedback!
Follow me on twitter!
As I remember, SwingUtilities.invokeLater() is used to divide the UI thread from other threads, so if any UI changes are needed, they won't stall the application.
The Listener class, which might be called something like SlotButtonListener, might be something like this:
And finally, in the Slots class there would be only the code for defining the UI of the game. All the buttons would have SlotButtonListener as action listener.
Anyone who wants to change the code following these advices is free to do it. You can fork it anytime you want.
Do you have any Java programming advice for me? Feel free to comment below
Github Slots Game
Multi-armed banditry in Python with slots
Github Slatejs
Roy Keyes
22 Aug 2016 - This is a post on my blog.
I recently released slots, a Python library that implements multi-armed bandit strategies. If that sounds like something that won't put you to sleep, then please pip install slots and read on.
Github Sloth
Multi-armed bandits
The multi-armed bandit (MAB) problem is a classic problem of trying to make the best choice, while having limited resources to gain information. The classic formulation is the gambler faced with a number of slot machines (a.k.a. one-armed bandits). How can the gambler maximize their payout while spending as little money as possible determining which are the 'hot' slot machines and which are the 'cold' ones? In more generic, idealized terms, you are faced with n choices, each with an associated payout probability p_i, which are unknown to you. Your goal is to run as few trials, or pulls, as possible to establish the the choice with the highest payout probability. There are many variations on this basic problem.
Getting the best bang for your buck
Most of us do not spend our time strategizing about real slot machines, but we do see similar real-world problems, such as A/B testing or resource allocation. Because of that, strategies to solve the multi-armed bandit problem are of both practical and intellectual interest to the data scientist-types out there.
There are several strategies for solving the MAB problem. All of them attempt to strike a balance between exploration, searching for the best choice, and exploitation, using the current best choice. Because of these competing goals, determining if you are making optimal choices is not trivial. Instead of simply looking at your average payout from your trials, a quantity called regret is calculated. Intuitively, regret is the payout value lost by making the sequence of choices you have made relative to the payout you would have received having known the best choice from the start. Regret can thus be used as a stopping criterion for making a 'final', best choice.
Example strategy: epsilon greedy
To understand how you might approach the multi-armed bandit problem, consider the simplest reasonable strategy, epsilon greedy:
- You spend the fraction e of your trials randomly trying different choices, i.e. exploration.
- For the rest of the time, 1 - e, you always choose the choice with the current highest reward rate, i.e. exploitation.
Too little time exploring the options might lead you to stay with a sub-optimal choice. Too much time spent exploring might lead you to spend unnecessary money on options that you already know are sub-optimal.
slots
I wrote slots with two goals in mind, investigating the performance of different MAB strategies for educational purposes and creating a usable implementation of those strategies for real world scenarios. For both of these goals, a simple API with reasonable default values was desirable.
So what does slots do? Right now, as of version 0.3.0, it has implementations of a few basic MAB strategies and allows you to run those on test scenarios and with real, live data. Currently, those strategies include epsilon greedy, softmax, upper confidence bound (UCB1), and Bayesian bandits implementations.
For 'real world' (online) usage, test results can be sequentially fed into an MAB object. After each result is fed into the algorithm the next recommended choice is returned, as well as whether your stopping criterion is met.
What slots looks like:
Using slots to determine the best of 3 variations on a live website.
Make the first choice randomly, record the response, and input reward (arm 2 was chosen here). Run online_trial (input most recent result) until the test criteria is met.
The response of mab.online_trial() is a dict of the form:
Where:
- If the criterion is met,
new_trial=False. choiceis the choice of the next arm to try.bestis the current best estimate of the highest payout arm.
For testing and understanding MAB strategies, you can also assign probabilities and payouts to the different arms and observe the results. For example, to compare the regret value and max arm payout probability as more trials are performed with various strategies (in this case epsilon greedy, softmax, upper credibility bound (UCB1), and Bayesian bandits):
The resulting regret evolution:
The estimated payout probability of the 'best' arm after each trial. In this case, the actual payout probability of the best arm is 0.85.
Making it happen
Example slot signal definition. slots is on PyPI, so you can simply install with pip install slots. Currently, slots works with both Python 2.7+ and 3.4+ and the only dependency is Numpy.
The future
Github Slots Games
slots is open source (BSD license) and I welcome outside contributions. My desire is to make slots easy-to-use, robust, and featureful. If you are interested in slots or the multi-armed bandit problem in general, please check out the References and further reading section below.
This was a very brief overview of the rich subject of dealing with the MAB problem and the slots library. Please checkout slots and send me feedback!
Follow me on twitter!
References and further reading
Github Slots App
- Multi-armed bandit (Wikipedia)
- Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems (Bubeck and Cesa-Bianchi)
- Multi-Armed Bandit Algorithms and EmpiricalEvaluation (Vermorel and Mohri)
- Upper confidence bound (Auer et al)
