Shingirayi Mandebvu's Blog

Q Learning Previously we were looking at the value of the state. Q Learning now moves to calculate the value of an action. We now move based on our actions as opposed to the value of a state.  People tend to think that its called q as shorthand for quality learning.  Now let's move to determine the equation for q-learning. Deriving the equation Remember the stochastic Markov equation?  V(s) = maxₐ (R(s,a)+  ɣ∑s' P(s, a, s') V(s')) Value of an Action equals Value of a state. ie V(s) = Q(s,a) ↡ Q(s,a)  = R(s,a)+  ɣ∑s' P(s, a, s') V(s') No max as we are not considering all of the alternative actions but just one action. We need to wean ourselves from V so we need to replace V(s'). V(s') represents all possible states. It is also worthy to note that V(s') is also max(Q(s', a')). With that it becomes  ↡ Q(s,a)  = R(s,a)+  ɣ∑s' P(s, a, s')  maxₐ  (Q(s', a')) Why max? Well, we still want to get all the po

Bellman equation  It's named after Richard Bellman and is defined as a necessary condition for optimality. It is associated with the mathematical optimization method known as dynamic programming. The deterministic equation is listed below:  V(s) = maxₐ  (R(s,a) + ɣ(V(s')) maxₐ : represents all the possible actions that we can take. R(s, a): The reward of taking an action at a particular state. ɣ: Discount. Works like the time value of money. You can see the dynamic programming aspect as we call the same method on s'. So it will recursively operate to solve the problem. Deterministic vs non-deterministic : Deterministic is definite, there is no randomness whereas non-deterministic is stochastic. So above we were not adding any randomness (deterministic) but nothing in this world is truly predictable, let's add randomness. Whereby each step is not so certain to be done (adding probability). It makes our agent more natural (being drunk! lol) so

What is reinforcement learning? Basically, it is learning from interaction and giving our agent a reward for achieving a goal. This field is essentially a class of problems with a class of solutions and a study of these classes. Of all the forms of Machine Learning, reinforcement learning is closest to how animals and humans learn. Elements of reinforcement learning a) Policy - defines the learning agents' behavior at any given time. Basically mapping States to action. b) Reward Signal - the goal of reinforcement learning. The aim is to maximize it. c) Value Function - while reward specifies what's good in the short run, value function specifies what's good in the long run. It is an accumulation. Rewards are primary whereas Value function is secondary. d) Model - Optional. Mimics the environment. Used for planning. Methods with models are called model-based whereas the opposite is called model free. Reinforcement versus Supervised and Unsupervised Learnin

Well why not httpsforfree? Well frankly the fact that they have access to my private keys freaks me out and letsencypt can have you up and running in seconds. Here are the steps 1. SSH into your VPS 2. Add the repo(if its not added already) add-apt-repository ppa:certbot/certbot 3. Install certbot $ apt-get update $ apt-get install python-certbot-nginx   4. Obtain certificate     $ sudo certbot --nginx -d example.com -d www.example.com     Follow the prompts and make sure you put your proper email!   (Optional) Setup a cron job to auto renew. I find the nano option easier and below I will show you my config.   $ crontab -e   Then 0 12 * * * /usr/bin/certbot renew --quiet     For subdomain i use   $ sudo certbot --nginx -d example.com -d  subdomain.example.com  

The perceptron Single node perceptron Perceptrons form the basis for ANNs. Perceptron takes input and produces output as below: Input ➡️ Activation Function ➡️ Output Input If the weight is 0- input remains unaltered coming into the perceptron. Below is what happens to the input ∑ w i z i  ≽ t then y = 1. Else y = 0.  i t is the threshold which is set by the outgoing part. So the key to output is based on the weighted sum and the threshold. Activation function This is the processing part of the neuron and this determines output. So most commonly the ones used are the Sigmoid function and the hyperbolic tangent. QUESTION: Which activation function should I use? I am going to talk of three key activation functions a) Sigmoid The Sigmoid returns 0 or 1 and in code can be written as return 1.0/(1.0+Math.exp(-x) A Sigmoid is a mathematical curve having the characteristic S shape. DISADVANTAGE: Descending Gradient b) Tanh DISADVANTAGE:

So basically this topic is a mix of three theories namely: Graph Theory Probability Theory Bayes Theory 1. Graph Theory  Graph theory is a mathematical field for the study of Graphs. Graph = Nodes (or vertices) + Arcs (or lines, or edges!) So a node can be anything that you want, a company, profits or a band. The thing is- you can have many nodes- but they have to have some form of relation between them. Graphs can be directed and undirected as shown below. 2. Probability Theory Probability is a measure of the likelihood of something happening. And is always a value between 0 and 1. The more likely, the closer the value is to 1 and vs-versa. Probability = (Particular Event)➗(All Possible Events) An example is coin flip. The probability of getting heads is 0.50 - and this brings us to our next idea, conditional probability . Instead of one event at a time, what is the the probability of getting a heads twice? Which then becomes P(Heads|Heads) The ' | ' sim

Here's a way you can tackle this annoying issue. Click on the Gradle side pane, select Tasks->android->androidDependencies. Running it will generate a list of the dependencies and the versions and a quick scroll should help you! Pic below should help

So I have been reading up on my cryptography and I figured I should give out a brief lesson on these three amazing concepts What are they ? a) PRG (Pseudo Random Generator) You probably know the difference between stream and block cipher. One of the main differences between them is key size. Stream ciphers require the key to be of equal length of greater than the plaintext ,   whereas Block Ciphers take a key smaller than the PT and is then expanded. This is the PRG The PRG expands the seed Considerations: Stream Ciphers base on Perfect Secrecy whereas Block Ciphers base on Semantic Security b) PRF (Pseudo Random Function) Lets share a secret- imagine something- you want to authenticate yourself with me by proving that you know a secret that we both share. Here's a possible option i) Possible Option 1:  PRNGs We both seed a PRNG with the shared secret, I pick and then send you some random number i.  You   then have to prove that you know the secret by respon

1. Try put startup type to delayed and restart the machine 2. If error code is 2146893007 this means that SSL3.0 and/or TLS1.0 are currently disabled. They will need to be enabled via registry. They need to be set to 1. If you regedit you will see them in these paths HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL3.0\Server\Enabled HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\SSL3.0\Server\Enabled 3. Go on Log Tab of the service and select log on as local account 4. Check the master files  First we need to find out the current location of the files. Once files are identified, either put them into the location where SQL wants or you will have to ALTER the database to point to the correct location and that will give you less headaches. If the issue lies with models or tempdb database you will need to start SQL Server using trace flag 3608 as explained in 

So I was starting off the Arduino and I did not know a thing about it! All I knew is that you could be Iron Man from this tiny thing. My skill was coding and none of the hardware stuff so I knew I had to learn. This post serves to document my journey into this Arduino world. What I know so far... Arduino is open source, the hardware AND software(Cool huh?) Arduino needs an IDE  You can connect it to your pi STEP ONE: Download the IDE from  this link. Depending on the OS. I'm on Ubuntu 16.04. I downloaded- did the ./install.sh and voila! done. What the challenge now was to navigate the board properly but after a bit of reading I learnt that pin 13 has an inbuilt resistor  and putting in the LED was now easy as cathode gets into the GND and  our friendly pin 13 has our back! Now my challenge became the permission error then I realised how to set permissions with this Linux code sudo chmod a+rw /dev/ttyACM0 And you ready to go! Seconds later- I had my blinking

So after a number of lines of code, brilliance and dreaming. Your next dream is for the world to see. Of course you can walk around with your computer and doing a 'manage.py runserver' But cumon guys, lets embrace the cloud. Not like this guy though! I choose to deploy on  PythonAnywhere . So you ask why? 1) Free amazing support - You actually talk to a live human ! 2) Easy - Very easy 3) Affordable - As you scale up, it gets way better! So by now I assume you are already on a version control system (So I will not waste much energy on that one). Maybe Ill someday write on my two favs  Github  and  Bitbucket . STEP 1: Create an account on pythonanywhere. Kindly note that your username will be included in your apps url. So it will be like : " yourUsername .pythonanywhere.com" STEP 2: Select other and set a bash console. STEP 3: Push your code from version control This will push from (in my example) github to your pythonanywhere. You

Requirements 1) Django (obviously) 2) Pyrebase (pip install pyrebase, you know the drill) So to give a better appreciation- I will first show you the HTML way then I'll proceed to show you how its done in Python. METHOD 1 : The HTML way Then you need to go to the firebase console. To setup a new project. After that you will see a web setup and you select that. Once selected you will see the config. It should be similar to this : Now that you have configured firebase, we now need to select the components which will be needed. depending on what you want for your app- you can now select the modules that you need. For us today, we will do the login authentication. Make sure you include firebase app first, so this is my screen: METHOD 2: Enter Python Open your dev environment and create a file named  pyrebase_settings within your django app folder. In it, you will have the following: Now, lets go to views.py!

Python Version: 3.6.1 Django Version : 1.10.2 So recently I migrated to the Bluehost VPS. Why Bluehost you may ask ? The price and they are django friendly.  They don't provide up to date versions of Python though so you have to install it yourself in your home directory.  Apart from that they are about as "Django Friendly" as any other shared host. One thing I can say is you really get good value for money with them and a huge amount of storage. Excellent decision but what made it a little hectic was now configuring so I decided to write this post  to help out on setting up django. One friendly reminder- make sure you have your versions on lockdown. I am compiling the current latest version 1. SSH Although the whole world seems to support PUTTY. I feel Teraterm does the job for me (Maybe its me but PUTTY doesn't support SSH2?) Anyways, if you are to use windows. Linux and Mac are straight forward.  So without further ado, login into your VPS server. 2.

So today I decided to be a little vain, I made an app for me ;) Feel free to check it out.. http://bit.ly/1KSqiek

So nostalgic, as I see the first program I made. It was in my room at Africa University. Ladies and Gentlemen. I present to you 'assassin antivirus'. They say you never forget your first, well here's mine.. Click here to download !!! For those who don't want to download but just want to see. Here are screenshots :