From f2f0f11ca6fc5da6be8a70aeb44b00ec36193372 Mon Sep 17 00:00:00 2001 From: Siddharths8212376 <40669232+Siddharths8212376@users.noreply.github.com> Date: Wed, 5 Jun 2019 23:28:57 +0530 Subject: [PATCH 1/2] Delete support_vector_machine_lesson.ipynb --- support_vector_machine_lesson.ipynb | 313 ---------------------------- 1 file changed, 313 deletions(-) delete mode 100644 support_vector_machine_lesson.ipynb diff --git a/support_vector_machine_lesson.ipynb b/support_vector_machine_lesson.ipynb deleted file mode 100644 index 817490d..0000000 --- a/support_vector_machine_lesson.ipynb +++ /dev/null @@ -1,313 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# The Math of Intelligence (Week 1) - Support Vector Machine Classification\n", - "\n", - "\n", - "## What will we do?\n", - "\n", - "We will build a Support Vector Machine that will find the optimal hyperplane that maximizes the margin between two toy data classes using gradient descent. \n", - "\n", - "![alt text](http://opticalengineering.spiedigitallibrary.org/data/journals/optice/24850/oe_52_2_027003_f005.png \"Logo Title Text 1\")\n", - "\n", - "\n", - "## What are some use cases for SVMs?\n", - "\n", - "-Classification, regression (time series prediction, etc) , outlier detection, clustering\n", - "\n", - "## Other Examples\n", - "\n", - "- Learning to use Scikit-learn's SVM function to classify images https://github.com/ksopyla/svm_mnist_digit_classification\n", - "- Pulse classification, more useful dataset \n", - "https://github.com/akasantony/pulse-classification-svm\n", - "\n", - "## How does an SVM compare to other ML algorithms?\n", - "\n", - "![alt text](https://image.slidesharecdn.com/mscpresentation-140722065852-phpapp01/95/msc-presentation-bioinformatics-7-638.jpg?cb=1406012610 \"Logo Title Text 1\")\n", - "\n", - "- As a rule of thumb, SVMs are great for relatively small data sets with fewer outliers. \n", - "- Other algorithms (Random forests, deep neural networks, etc.) require more data but almost always come up with very robust models.\n", - "- The decision of which classifier to use depends on your dataset and the general complexity of the problem.\n", - "- \"Premature optimization is the root of all evil (or at least most of it) in programming.\" - Donald Knuth, CS Professor (Turing award speech 1974) \n", - "\n", - "\n", - "## What is a Support Vector Machine?\n", - "\n", - "It's a supervised machine learning algorithm which can be used for both classification or regression problems. But it's usually used for classification. Given 2 or more labeled classes of data, it acts as a discriminative classifier, formally defined by an optimal hyperplane that seperates all the classes. New examples that are then mapped into that same space can then be categorized based on on which side of the gap they fall.\n", - "\n", - "## What are Support Vectors?\n", - "\n", - "![alt text](https://www.dtreg.com/uploaded/pageimg/SvmMargin2.jpg \"Logo Title Text 1\")\n", - " \n", - "Support vectors are the data points nearest to the hyperplane, the points of a data set that, if removed, would alter the position of the dividing hyperplane. Because of this, they can be considered the critical elements of a data set, they are what help us build our SVM. \n", - "\n", - "## Whats a hyperplane?\n", - "\n", - "![alt text](http://slideplayer.com/slide/1579281/5/images/32/Hyperplanes+as+decision+surfaces.jpg \"Logo Title Text 1\")\n", - "\n", - "Geometry tells us that a hyperplane is a subspace of one dimension less than its ambient space. For instance, a hyperplane of an n-dimensional space is a flat subset with dimension n − 1. By its nature, it separates the space into two half spaces.\n", - "\n", - "## Linear vs nonlinear classification?\n", - "\n", - "Sometimes our data is linearly seperable. That means for N classes with M features, we can learn a mapping that is a linear combination. (like y = mx + b). Or even a multidimensional hyperplane (y = x + z + b + q). No matter how many dimensions/features a set of classes have, we can represent the mapping using a linear function.\n", - "\n", - "But sometimes its not. Like if there was a quadratic mapping. Luckily for us SVMs can can efficiently perform a non-linear classification using what is called the kernel trick. I'll talk about that lter on in the course. \n", - "\n", - "![alt text](https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAuSAAAAJDlhYzcwMzhlLTA0MjYtNDEyYS1hMWM4LTE3Zjk5NDlhNzVkMQ.png \"Logo Title Text 1\")\n", - "\n", - "Alright let's get to building!" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": { - "collapsed": false - }, - "outputs": [ - { - "data": { - "text/plain": [ - "[]" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VHX+xfH3Z1IIBAi9d2ki3dAhsdBBUMSGgmJBBaS5\nq+tafuoW2xqKKIgoNmygIIKUoGxCx4QmVSmiICWKUkT69/dHEpd1QQbI5M5Mzut58pjEy8x5EA93\n7sycmHMOEREJHT6vA4iIyLlRcYuIhBgVt4hIiFFxi4iEGBW3iEiIUXGLiIQYFbeISIhRcYuIhBgV\nt4hIiIkMxI2WKFHCValSJRA3LSISltLT039wzpX059iAFHeVKlVIS0sLxE2LiIQlM9vm77G6VCIi\nEmJU3CIiIUbFLSISYlTcIiIhRsUtIhJiVNwiIiFGxS05qu+svvSd1dfrGCJhTcUtIhJi/CpuMyti\nZpPNbIOZrTezFoEOJiIip+fvOydHArOccz3NLBooEMBMIiLyB85a3GYWByQAtwE4544CRwMRZte+\nwyQlb+TBjrUpXjBfIO5Cctjvr2en7U477fcndJyQa5lEwp0/l0qqAhnABDNbYWbjzSz29weZWT8z\nSzOztIyMjPMKk7ZtL1NW7KBtUgofr9yBc+68bkdEJJzZ2crRzOKBJUAr59xSMxsJ7HfOPXqmXxMf\nH+/Od2Rq464DPPDhalZ99zNX1i7F36+pS9m4/Od1W5L7ss+0dYYtcm7MLN05F+/Psf6ccW8Htjvn\nlmZ9PRlofL7hzqZWmUJ8dG9LHulyMQs3/0C7pFQmLt3GyZM6+xYRAT+K2zm3C/jOzGplfetKYF0g\nQ0X4jDvbVGP2kATqlY/j4Slr6DV+Cd/88Esg71ZEJCT4+zru+4CJZrYaaAj8M3CR/qNy8VjeuasZ\nT/eox9od++kwIpVxqZs5fuJkbty9iEhQOus17vNxIde4z2TXvsM8MvVL5q7fQ4MKcTzTsz61yxTO\n0fsQEfFKTl/jDgpl4mJ4pU88L9zUiO0//UrXUQtISv6KI8dPeB1NRCRXhUxxA5gZVzUoR/KwRLrW\nL8uoz77mqhcWsOLbn7yOJiKSa0KquLMVi41mxI2NeO22eA4cPk6PMYv42/R1HDp63OtoIiIBF5LF\nne2K2qWZMzSBXk0r8eqCrXQcMZ9Fm37wOpaISECFdHEDFIqJ4h/X1OO9fs3xGfQav5S/fLiafb8e\n8zqaiEhAhHxxZ2terTgzBydwd0I1Pkj7jvbDU0het9vrWCIiOS5sihsgf3QED3W+mKkDWlG0QDR3\nvZnGwHeW88PBI15HExHJMWFV3NnqVyjCtIGtGdauJrPX7qJdUgpTV2i0SkTCQ1gWN0B0pI9BV9Zg\nxqA2VC4ey5D3V3LHG2l8//OvXkcTEbkgYVvc2WqWLsSH97bk0a51WLz5R9oPT+XtJRqtEpHQFfbF\nDZmjVXe0rsrsIQk0qBjHI1PXcNMrS9iq0SoRCUF5orizVSpegLfvaMYz19Zj3c79dByRysspGq0S\nkdCSp4obMt82f0OTSswdlkhCzZI8NXMDPcYsYv3O/V5HExHxS54r7mylC8cwrveljO7ViB0//cpV\nLywgac5GjVaJSNDLs8UNmWffXeuXY+6wRK5qUI5Rn2+i66gFLNdolYgEsTxd3NmKxkYz/IaGTLit\nCQePHOfaMYt48hONVolIcFJxn+Ly2qWYMzSBm5tV4rWFW+kwIpWFGq0SkSCj4v6dQjFR/P3qerzf\nrzmRPh83j1/Kg5M1WiUiwUPFfQbNqhVn5uA23J1YjUnp39EuKYU5a3d5HUtERMX9R2KiInioU+Zo\nVbHYaPq9lc6Ad5aTcUCjVSLiHRW3H+pXKMIn97Xm/nY1SV67m3bDU5iyYrtGq0TEEypuP0VF+Ljv\nyhrMGNSaqiViGfr+Kvq+/gU7NFolIrlMxX2OapQuxOR7WvJY1zos3bKX9kkpvKXRKhHJRSru8xDh\nM25vXZU5QxNoVKkoj05dw43jlrAl46DX0UQkD1BxX4CKxQrw1h1Nefba+qzftZ9OI+czVqNVIhJg\nKu4LZGZc36Qic4clklizJE/P3MDVLy1k3fcarRKRwPCruM3sGzP70sxWmllaoEOFotKFY3i596W8\n2Ksxu/YdptvoBTyv0SoRCYBzOeO+3DnX0DkXH7A0Ic7M6FK/LMlDE+nWsBwvfL6JLqMWkL5No1Ui\nknN0qSQAisZGk3R9Qyb0bcKhI8fpOXYRT3yyll+OaLRKRC6cv8XtgLlmlm5m/QIZKJxcXqsUc4Yl\n0rt5ZSYs/IYOI1KZ/3WG17FEJMT5W9ytnXMNgU7AADNL+P0BZtbPzNLMLC0jQ+WUrWC+SJ7sXpcP\n7m5BVISP3q8u44HJq9h3SKNVInJ+7Fzftm1mjwMHnXP/OtMx8fHxLi1Nz2H+3uFjJxj52deMS91C\nsdho/ta9Lh3rlvE6logEATNL9/c5xLOecZtZrJkVyv4caA+subCIeVNMVAQPdqzN1P6tKFEwH/e8\nnc6AiRqtEpFz48+lktLAAjNbBSwDZjjnZgU2VnirVyGOaQNb8ecOtUhelzla9dFyjVaJiH/O+VKJ\nP3SpxH+b9hzggcmrWf7tz1xWqyT/uKYe5Yvk9zqWiOSyHL1UIoFVvVQhJt3TksevqsOyrVmjVYu/\n0WiViJyRijsIRPiM21pVZfaQBBpXLsqjH6/VaJWInJGKO4hULFaAN29vynM967Nh1346jpzPmH9r\ntEpE/puKO8iYGdfFZ45WXV6rJM/M0miViPw3FXeQKlU4hpd7xzPm5sbs2neEbqMX8K/ZGzl8TKNV\nInmdijvIdapXlrnDEujesDyj522iy6j5pG/b63UsEfGQijsEFCkQzfPXN+CN25ty+NhJeo5dzOPT\nNFolklepuENIYs2SzB6aQJ/mlXl9kUarRPIqFXeIKZgvkie612XSPS2IjswcrfrzJI1WieQlKu4Q\n1aRKMT4d1Ib+l13ERyt20HZ4CrPW7PI6lojkAhV3CIuJiuCBjrX5eEArSmaNVvWfmM6eA4e9jiYi\nAaTiDgN1y8fxcdZo1dz1e2iXlMqH6RqtEglXKu4wERXhY8Dl1fl0UBuqlyrI/ZNWceuEL9j+0yGv\no4lIDlNxh5nqpQoy6e4WPNHtEtK+2UuH4am8uVijVSLhRMUdhnw+49aWVX4brXrs47XcMG4xmzVa\nJRIWVNxhLHu06l/XNeCr3QfpNHI+L/17E8c0WiUS0lTcYc7M6HlpBZKHJXBl7VI8O2sjV7+4kDU7\n9nkdTUTOk4o7jyhVKIYxt1zKmJsbs3v/Ebq/uJDnZm/QaJVICFJx5zHZo1XXNCrPi/M203nUfNK+\n0WiVSChRcedBRQpE86/rGvDm7U05cuwk172s0SqRUKLizsMSapZkztAEbm1RhTcWf0P74amkfqXR\nKpFgp+LO42LzRfJ4t0uYdHcL8kX56PPaMv40aRU/HzrqdTQROQMVtwAQnzVaNeDyi5iyYgdtk1KZ\n+eVOr2OJyGmouOU3MVER/LlDbaYNbEXpwvm4d+Jy7n1bo1UiwUbFLf/jknJxTB3Qigc61uKzDZmj\nVZPSvtNolUiQUHHLaUVF+Oh/WXVmDm5DzdIF+fPk1fR5bRnf7Q290aptvfuwrXcfr2OI5Bi/i9vM\nIsxshZlND2QgCS4XlSzI+/1a8GT3S1i+7Sc6jEjl9YVbNVol4qFzOeMeDKwPVBAJXj6f0adFFWYP\nTSC+SjEe/2Qd17+8mE17NFol4gW/itvMKgBdgPGBjSPBrELRArzRtwnPX9eAr/ccpPPI+bw4T6NV\nIrkt0s/jRgAPAIVy8s5veHmxX8e9f3eLnLxbuQBmxrWXViChZkn+b9oanpu9kRmrd/Jsz/rULR/n\ndTyA/7mefeiLL077/cpvvZlrmURy0lnPuM2sK7DHOZd+luP6mVmamaVlZOjdd+GuZKF8vHTzpYy9\npTEZBzNHq56ZpdEqkdxgZ3uJl5k9BfQGjgMxQGHgI+fcLWf6NfHx8S4tLS0nc0oQ23foGH+fsY5J\n6dupViKWZ3rWp0mVYl7H+k32mbbOsCWYmVm6cy7en2PPesbtnHvIOVfBOVcFuBH4/I9KW/KeuAJR\nPHddA966oylHT5zkurGLeezjNRzUaJVIQOh13JJj2tQoyewhCdzWsgpvLdlGh+GppGi0SiTHnVNx\nO+f+7ZzrGqgwEvqyR6sm39OCmCgft762jGEfrNRolUgOOus17vOha9wCcPjYCUZ/vomxKZspUiCK\nJ7vXpXO9sl7HEglKOXqNW+R8xURF8KcOtfh4YCvKxMXQf+Jy7nkrnT37NVolciFU3BJwl5SLY2r/\nVjzYsTafb9xD26QUPtBolch5U3FLroiM8HHvZRcxa3AbapcpzAMhPFol4jUVt+SqaiUL8l6/5vzt\nlNGqCQu3ckKjVSJ+U3FLrvP5jN4tqjBnWCJNqhTjid9Gqw54HU0kJKi4xTPli+Tn9b5NSLq+AZsz\nDtJ55AJGf/61RqtEzkLFLZ4yM3o0rkDy0ETaXVKaf835im6jF7Jmxz6vo4kELRW3BIWShfLxYq/G\nvNz7Un7IGq16eqZGq0ROR8UtQaXDJWWYOzSRno0rMDZlM51HzmfZ1r1exxIJKipuCTpxBaJ4pmd9\n3r6jGUdPnOT6lxfz6FSNVolkU3FL0GpdowRzhiZwe6uqvL10G+2TUpi3cY/XsUQ8p+KWoFYgOpLH\nrqrD5HtaUiBfJH0nfMGw91fy0y8arZK8S8UtIeHSykWZMag1g66ozrRV39NueAozVu/U2+YlT1Jx\nS8jIFxnBsPa1mDawNWXj8jPgneXcrdEqyYNU3BJy6pQrzJT+LXmoU21SvsrgyqQUPvhCo1WSd6i4\nJSRFRvi4O/EiZg5uw8VlC/PAh6vp/apGqyRvUHFLSKtWsiDv3dWcv19dl5Xf/Uz74am8tkCjVRLe\nVNwS8nw+45bmlZkzNIFm1Yrx5PR1XDd2EV/v1miVhCcVt4SNckXyM+G2Joy4oSFbf/iFLqMW8MJn\nGq2S8KPilrBiZlzdqDzJwxJpf0lpnk/+iqteWMCX2zVaJeFDxS1hqUTBfIzu1ZhxvS9l7y9H6f7i\nAp6auV6jVRIWVNwS1tpfUobkYYlcH1+Rl1O20GnkfJZu+dHrWCIXRMUtYS8ufxRPX1ufiXc24/jJ\nk9wwbgmPTP2SA4ePeR1N5LyouCXPaFW9BLOHJHBH66pMXPotHYanMm+DRqsk9Ki4JU8pEB3Jo13r\n8OG9LYnNF0nf179g6Psr2avRKgkhKm7JkxpXKsr0Qa0ZdGUNPln1Pe2SUpi++nu9bV5CwlmL28xi\nzGyZma0ys7Vm9kRuBBMJtHyREQxrV5NP7mtN+aL5GfjOCvq9lc5ujVZJkPPnjPsIcIVzrgHQEOho\nZs0DG0sk91xctjAf3duSv3auTepXGbRNSuH9L77V2bcErbMWt8t0MOvLqKwP/YmWsBIZ4aNfwkXM\nHpJAnbKFefDDL7l5/FK+/VGjVRJ8/LrGbWYRZrYS2AMkO+eWnuaYfmaWZmZpGRkZOZ1TJFdUKRHL\nu3c15x/X1GX19n10GJHKqxqtkiBj5/Jw0MyKAFOA+5xza850XHx8vEtLS8uBeCLe2bnvVx6esobP\nN+yhYcUiPNuzPjVLF/I6loQpM0t3zsX7c+w5varEOfczMA/oeD7BREJJ2bj8vHprPCNvbMi2H3+h\ny6j5jPrsa44e12iVeMufV5WUzDrTxszyA+2ADYEOJhIMzIzuDcszd1giHeuWJSn5K7qNXsCq7372\nOprkYf6ccZcF5pnZauALMq9xTw9sLJHgUrxgPl64qRGv9Innp0NHuealhTz16Xp+ParRKsl9kWc7\nwDm3GmiUC1lEgl67OqVpVq0YT326npdTtzB77S6evrY+zasV9zqa5CF656TIOSocE8VTPerzzp3N\nOOngxnFLeHiKRqsk96i4Rc5Ty6zRqjtbV+XdZd/Sfngqn2/Y7XUsyQNU3CIXIH90BI9kjVYVionk\n9tfTGPLeCo1WSUCpuEVyQKNKRZl+XxsGX1mDGV/upG1SCtNWabRKAkPFLZJDoiN9DM0arapYND+D\n3l3BXW+ms2ufRqskZ6m4RXJY7TKF+ah/Kx7ufDELNmXQLimFd5dptEpyjopbJAAifMZdCdWYNTiB\nS8oX5qGPvqTXK0vZ9uMvXkeTMKDiFgmgKiVieefO5vzzmnqs2ZE5WjV+/haNVskFUXGLBJjPZ/Rq\nVok5wxJodVEJ/j5jPdeOWcRXuw94HU1ClIpbJJeUjcvP+KzRqm/3HqLLqPmMnKvRKjl3Km6RXJQ9\nWpU8NIHO9coyfK5Gq+TcqbhFPFC8YD5G3tiI8X3i+fnQMa55aSH/1GiV+EnFLeKhtnVKM2dYAjc2\nrcS41C10GpnK4s0/eh1LgpyKW8RjhWOi+Oc19XjnrmY44KZXlvDXKV+yX6NVcgYqbpEg0fKiEswa\nnMBdbary3rJvaZ+k0So5PRW3SBDJHx3Bw13q8FH/VsTlj+L219MY/N4Kfjx4xOtoEkRU3CJBqGHF\nInxyX2uGtK3Bp1/upN3wVI1WyW9U3CJBKjrSx5C2NZl+XxsqFiuQNVqVptEqUXGLBLtaZQrx0b0t\neaTLxSzY9INGq0TFLRIKInzGnW2qMXtIAnXLx2m0Ko9TcYuEkMrFY3nnrmY81UOjVXmZilskxJgZ\nNzWtRPKwRFpXzxyt6jFmERt3abQqr1Bxi4SoMnExvNInnlE3NeK7vYfo+sJ8Rsz9SqNVeYCKWySE\nmRndGpRj7rBEOtcry4i5X3PVCwtYqdGqsKbiFgkDxWKjGXljI169NZ59vx6jx0sL+ceMdRqtClMq\nbpEwcuXF/xmtemX+VjqMSGXR5h+8jhWWpjy/nCnPL/fkvs9a3GZW0czmmdk6M1trZoNzI5iInJ/s\n0ap372qOGfR6ZSkPfaTRqnDizxn3ceB+51wdoDkwwMzqBDaWiFyoFhcVZ9bgBPolVOP9L76lXVIK\nc9dptCocnLW4nXM7nXPLsz4/AKwHygc6mIhcuPzREfy188VM6d+KogWiufPNNAa9q9GqUGfn8rZZ\nM6sCpAJ1nXP7z3RcfHy8S0tLu+BwIpJzjh4/yZh/b2b0vK8pmC+Sx7tdQrcG5TAzr6OFhN9fz/7+\n68xX7pSrUeS/vn/N/Y3P6/bNLN05F+/PsX4/OWlmBYEPgSGnK20z62dmaWaWlpGR4X9aEckV0ZE+\nBretwYxBbahcPJbB763kzjfS2LnvV6+jyTny64zbzKKA6cBs51zS2Y7XGbdIcDtx0jFh4Vb+NWcj\nkT4fD3WuzU1NKuHz6ezbX9ln4Od7hv17OXrGbZmPo14F1vtT2iIS/LJHq+YMSaR+hTgenrKGXuOX\n8M0PGq0KBf5cKmkF9AauMLOVWR+dA5xLRHJBpeIFmHhnM57uUY+1O/bTYUQq41I3c/yE3jYfzCLP\ndoBzbgGgx08iYcrMuLFpJS6rVYpHpq7hn59uYMbqnTzTsz61yxT2Op6cxjm9qsRfusYtEpqcc0xf\nvZPHp61l36/H6H95dQZcfhH5IiO8jhb2AvKqEhEJf2bGVQ3KkTwskasalGPUZ5mjVSu+/cnraHIK\nFbeI/I9isdEMv6Ehr90Wz4HDx+kxZhF/m76OQ0ePex1NUHGLyB+4onZp5gxN4OZmlXh1wVY6jpjP\nok0arfKailtE/lChmCj+fnU93uvXHJ9Br/FL+cuHq9n3q0arvKLiFhG/NK9WnFlDErg7sRofpH1H\n++EpJGu0yhMqbhHxW0xUBA91upipAzJHq+56M42B7yznB41W5SoVt4ics/oVijBtYGvub1eTOWt3\n0y4phakrdhCIlxfL/1Jxi8h5iY70cd+VNZgxqDVVSsQy5P2V3PFGGt//rNGqQFNxi8gFqVG6EJPv\nacljXeuwePOPtB+eyttLtnHypM6+A0XFLSIXLMJn3N66KrOHJNCgYhyPTF3DTa8sYatGqwJCxS0i\nOaZS8QK8fUcznr22Put27qfjiFReTtFoVU5TcYtIjjIzrm9SkbnDEkmoWZKnZm6gx5hFrN95xh+a\nJedIxS0iAVG6cAzjel/Ki70a8/3Pv3LVCwtImrORI8dPeB0t5Km4RSRgzIwu9cuSPDSRbg3KMerz\nTXQdtYDlGq26ICpuEQm4orHRJN3QkAl9m/DLkeNcO2YRT36i0arzpeIWkVxzea1SzB6awC3NKvPa\nwq10GJHKQo1WnTMVt4jkqkIxUfzt6rq83685kT4fN49fyoOTNVp1LlTcIuKJZtWKM3NwG+5JvIjJ\ny7fTLimFOWt3eR0rJKi4RcQzMVER/KVTbab2b0Xxgvno91Y6A95ZTsYBjVb9ERW3iHiuXoU4pg1s\nxZ/a1yR57W7aDU9hyortGq06AxW3iASFqAgfA6+owaeDW1OtRCxD319F39e/YIdGq/6HiltEgkr1\nUoWYdE9L/u+qOizdspf2SSm8pdGq/6LiFpGgE+Ez+raqypyhCTSqVJRHp67hxnFL2JJx0OtoQUHF\nLSJBq2KxArx1R1Oe7VmfDbv202nkfMZqtErFLSLBzcy4Pj5ztOqyWiV5euYGrn5pIeu+z7ujVSpu\nEQkJpQrHMPaWS3np5sbs2neYbqMX8HweHa06a3Gb2WtmtsfM1uRGIBGRMzEzOtfLGq1qWI4XPt9E\nl1ELSN+Wt0ar/Dnjfh3oGOAcIiJ+KxobTdL1DXm9bxN+PXqCnmMX8cQna/nlSN4YrYo82wHOuVQz\nqxL4KCKS6yZ08e+4vjMCm+M8XZY1WvXsrA1MWPgNyet281SPerSpUdLraAGVY9e4zayfmaWZWVpG\nRkZO3ayIyB8qmC+SJ7vX5YO7WxAd4aP3q8t4YPIq9h0K39Eq8+ctpVln3NOdc3X9udH4+HiXlpZ2\nYclERM7R4WMnGPnZ14xL3UKx2Gj+1r0uHeuW8TqWX8ws3TkX78+xelWJiISNmKgIHuxYm48HtKJk\nwXzc83Y6AyaG32iViltEwk7d8nF8PLAVf+5Qi+R1u2mblMKH6eEzWuXPywHfBRYDtcxsu5ndEfhY\nIiIXJirCx4DLq/Pp4DZUL1WQ+yet4rYJ4TFa5dc17nOla9wiEkxOnnS8ufgbnp29EQMe7FSbW5pV\nxuczr6P9Rte4RURO4fMZt7WqyuwhCTSuXJTHPl7LDeMWszlER6tU3CKSZ1QsVoA3b2/Kcz3rs3HX\nATqNnM9L/94UcqNVKm4RyVPMjOviKzL3/kSuqFWKZ2dt5OqXFrL2+31eR/ObiltE8qRShWIY2/tS\nxtzcmF37jtBt9EKem72Bw8eCf7RKxS0ieVqnemWZOyyBqxuW58V5m+kyaj7p2/Z6HesPqbhFJM8r\nUiCa569vwBu3N+XwsZP0HLuYx6cF72iViltEJEtizZLMHppAn+aVeWPxN7QfnkrqV8G3vaTiFhE5\nRcF8kTyRNVqVL8pHn9eW8adJwTVapeIWETmNJlWK8emgNvS/7CKmrNhB2+EpzFqz0+tYgIpbROSM\nYqIieOC/RquWc+/b6ew5cNjTXCpuEZGzOHW06rMNe2iXlMpkD0erVNwiIn74bbRqUBtqlCrIn7JG\nq44ez/13XZ71R5eJiMh/VC9VkA/ubsHbS7exJeMXoiNz//xXxS0ico58PqNPiyre3b9n9ywiIudF\nxS0iEmJU3CIiIUbFLSISYlTcIiIhRsUtIhJiVNwiIiFGxS0iEmIsEO+1N7MMYNt5/vISwA85GCen\nBGOuYMwEwZkrGDNBcOYKxkwQnLlyMlNl51xJfw4MSHFfCDNLc87Fe53j94IxVzBmguDMFYyZIDhz\nBWMmCM5cXmXSpRIRkRCj4hYRCTHBWNzjvA5wBsGYKxgzQXDmCsZMEJy5gjETBGcuTzIF3TVuERH5\nY8F4xi0iIn8gKIvbzJ4zsw1mttrMpphZEa8zAZjZdWa21sxOmpmnz26bWUcz22hmm8zsL15myWZm\nr5nZHjNb43WWbGZW0czmmdm6rP92g4MgU4yZLTOzVVmZnvA6UzYzizCzFWY23ess2czsGzP70sxW\nmlma13mymVkRM5uc1VXrzaxFbt13UBY3kAzUdc7VB74CHvI4T7Y1QA8g1csQZhYBvAh0AuoAN5lZ\nHS8zZXkd6Oh1iN85DtzvnKsDNAcGBMHv1RHgCudcA6Ah0NHMmnucKdtgYL3XIU7jcudcwyB7OeBI\nYJZzrjbQgFz8fQvK4nbOzXHOHc/6cglQwcs82Zxz651zG73OATQFNjnntjjnjgLvAd09zoRzLhXY\n63WOUznndjrnlmd9foDM/7nKe5zJOecOZn0ZlfXh+ZNNZlYB6AKM9zpLsDOzOCABeBXAOXfUOfdz\nbt1/UBb379wOzPQ6RJApD3x3ytfb8biMQoGZVQEaAUu9TfLbJYmVwB4g2TnneSZgBPAAkPs//faP\nOWCumaWbWT+vw2SpCmQAE7IuLY03s9jcunPPitvM5prZmtN8dD/lmIfJfKg7MZhySegxs4LAh8AQ\n59x+r/M450445xqS+WiyqZnV9TKPmXUF9jjn0r3McQats36vOpF5qSvB60Bk/rzexsAY51wj4Bcg\n155r8uyHBTvn2v7Rvzez24CuwJUuF1+zeLZcQWIHUPGUrytkfU9Ow8yiyCztic65j7zOcyrn3M9m\nNo/M5wa8fFK3FdDNzDoDMUBhM3vbOXeLh5kAcM7tyPrnHjObQualQk+fZyLzUe72Ux4pTSYXizso\nL5WYWUcyH7J1c84d8jpPEPoCqGFmVc0sGrgRmOZxpqBkZkbmdcj1zrkkr/MAmFnJ7FdKmVl+oB2w\nwctMzrmHnHMVnHNVyPzz9HkwlLaZxZpZoezPgfZ4+xccAM65XcB3ZlYr61tXAuty6/6DsriB0UAh\nIDnrJUBjvQ4EYGbXmNl2oAUww8xme5Ej64nbgcBsMp9s+8A5t9aLLKcys3eBxUAtM9tuZnd4nYnM\nM8newBVZf5ZWZp1VeqksMM/MVpP5l3Cycy5oXn4XZEoDC8xsFbAMmOGcm+Vxpmz3AROz/js2BP6Z\nW3esd07CU699AAAAOElEQVSKiISYYD3jFhGRM1Bxi4iEGBW3iEiIUXGLiIQYFbeISIhRcYuIhBgV\nt4hIiFFxi4iEmP8HX67tlYF/BHAAAAAASUVORK5CYII=\n", - "text/plain": [ - "" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "#To help us perform math operations\n", - "import numpy as np\n", - "#to plot our data and model visually\n", - "from matplotlib import pyplot as plt\n", - "%matplotlib inline\n", - "\n", - "#Step 1 - Define our data\n", - "\n", - "#Input data - Of the form [X value, Y value, Bias term]\n", - "X = np.array([\n", - " [-2,4,-1],\n", - " [4,1,-1],\n", - " [1, 6, -1],\n", - " [2, 4, -1],\n", - " [6, 2, -1],\n", - "])\n", - "\n", - "#Associated output labels - First 2 examples are labeled '-1' and last 3 are labeled '+1'\n", - "y = np.array([-1,-1,1,1,1])\n", - "\n", - "#lets plot these examples on a 2D graph!\n", - "#for each example\n", - "for d, sample in enumerate(X):\n", - " # Plot the negative samples (the first 2)\n", - " if d < 2:\n", - " plt.scatter(sample[0], sample[1], s=120, marker='_', linewidths=2)\n", - " # Plot the positive samples (the last 3)\n", - " else:\n", - " plt.scatter(sample[0], sample[1], s=120, marker='+', linewidths=2)\n", - "\n", - "# Print a possible hyperplane, that is seperating the two classes.\n", - "#we'll two points and draw the line between them (naive guess)\n", - "plt.plot([-2,6],[6,0.5])" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Let's define our loss function (what to minimize) and our objective function (what to optimize)\n", - "\n", - "#### Loss function\n", - "\n", - "We'll use the Hinge loss. This is a loss function used for training classifiers. The hinge loss is used for \"maximum-margin\" classification, most notably for support vector machines (SVMs).\n", - "\n", - "![alt text](http://i.imgur.com/OzCwzyN.png \"Logo Title Text 1\")\n", - "\n", - "\n", - "c is the loss function, x the sample, y is the true label, f(x) the predicted label.\n", - "\n", - "![alt text](http://i.imgur.com/FZ7JcG3.png \"Logo Title Text 1\")\n", - "\n", - " \n", - "#### Objective Function\n", - "\n", - "![alt text](http://i.imgur.com/I5NNu44.png \"Logo Title Text 1\")\n", - "\n", - "As you can see, our objective of a SVM consists of two terms. The first term is a regularizer, the heart of the SVM, the second term the loss. The regularizer balances between margin maximization and loss. We want to find the decision surface that is maximally far away from any data points.\n", - "\n", - "How do we minimize our loss/optimize for our objective (i.e learn)?\n", - "\n", - "We have to derive our objective function to get the gradients! Gradient descent ftw. As we have two terms, we will derive them seperately using the sum rule in differentiation.\n", - "\n", - "\n", - "![alt text](http://i.imgur.com/6uK3BnH.png \"Logo Title Text 1\")\n", - "\n", - "This means, if we have a misclassified sample, we update the weight vector w using the gradients of both terms, else if classified correctly,we just update w by the gradient of the regularizer.\n", - "\n", - "\n", - "\n", - "Misclassification condition \n", - "\n", - "![alt text](http://i.imgur.com/g9QLAyn.png \"Logo Title Text 1\")\n", - "\n", - "Update rule for our weights (misclassified)\n", - "\n", - "![alt text](http://i.imgur.com/rkdPpTZ.png \"Logo Title Text 1\")\n", - "\n", - "including the learning rate η and the regularizer λ\n", - "The learning rate is the length of the steps the algorithm makes down the gradient on the error curve.\n", - "- Learning rate too high? The algorithm might overshoot the optimal point.\n", - "- Learning rate too low? Could take too long to converge. Or never converge.\n", - "\n", - "The regularizer controls the trade off between the achieving a low training error and a low testing error that is the ability to generalize your classifier to unseen data. As a regulizing parameter we choose 1/epochs, so this parameter will decrease, as the number of epochs increases.\n", - "- Regularizer too high? overfit (large testing error) \n", - "- Regularizer too low? underfit (large training error) \n", - "\n", - "Update rule for our weights (correctly classified)\n", - "\n", - "![alt text](http://i.imgur.com/xTKbvZ6.png \"Logo Title Text 1\")\n" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "#lets perform stochastic gradient descent to learn the seperating hyperplane between both classes\n", - "\n", - "def svm_sgd_plot(X, Y):\n", - " #Initialize our SVMs weight vector with zeros (3 values)\n", - " w = np.zeros(len(X[0]))\n", - " #The learning rate\n", - " eta = 1\n", - " #how many iterations to train for\n", - " epochs = 100000\n", - " #store misclassifications so we can plot how they change over time\n", - " errors = []\n", - "\n", - " #training part, gradient descent part\n", - " for epoch in range(1,epochs):\n", - " error = 0\n", - " for i, x in enumerate(X):\n", - " #misclassification\n", - " if (Y[i]*np.dot(X[i], w)) < 1:\n", - " #misclassified update for ours weights\n", - " w = w + eta * ( (X[i] * Y[i]) + (-2 *(1/epoch)* w) )\n", - " error = 1\n", - " else:\n", - " #correct classification, update our weights\n", - " w = w + eta * (-2 *(1/epoch)* w)\n", - " errors.append(error)\n", - " \n", - "\n", - " #lets plot the rate of classification errors during training for our SVM\n", - " plt.plot(errors, '|')\n", - " plt.ylim(0.5,1.5)\n", - " plt.axes().set_yticklabels([])\n", - " plt.xlabel('Epoch')\n", - " plt.ylabel('Misclassified')\n", - " plt.show()\n", - " \n", - " return w" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": false - }, - "outputs": [], - "source": [ - "for d, sample in enumerate(X):\n", - " # Plot the negative samples\n", - " if d < 2:\n", - " plt.scatter(sample[0], sample[1], s=120, marker='_', linewidths=2)\n", - " # Plot the positive samples\n", - " else:\n", - " plt.scatter(sample[0], sample[1], s=120, marker='+', linewidths=2)\n", - "\n", - "# Add our test samples\n", - "plt.scatter(2,2, s=120, marker='_', linewidths=2, color='yellow')\n", - "plt.scatter(4,3, s=120, marker='+', linewidths=2, color='blue')\n", - "\n", - "# Print the hyperplane calculated by svm_sgd()\n", - "x2=[w[0],w[1],-w[1],w[0]]\n", - "x3=[w[0],w[1],w[1],-w[0]]\n", - "\n", - "x2x3 =np.array([x2,x3])\n", - "X,Y,U,V = zip(*x2x3)\n", - "ax = plt.gca()\n", - "ax.quiver(X,Y,U,V,scale=1, color='blue')" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "w = svm_sgd_plot(X,y)\n", - "#they decrease over time! Our SVM is learning the optimal hyperplane" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.6.0" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} From d1c710ff6319024cd17b1c32a4e3e394ece292be Mon Sep 17 00:00:00 2001 From: Siddharths8212376 <40669232+Siddharths8212376@users.noreply.github.com> Date: Wed, 5 Jun 2019 23:29:50 +0530 Subject: [PATCH 2/2] Added a small change in the order of execution... --- support_vector_machine_lesson.ipynb | 355 ++++++++++++++++++++++++++++ 1 file changed, 355 insertions(+) create mode 100644 support_vector_machine_lesson.ipynb diff --git a/support_vector_machine_lesson.ipynb b/support_vector_machine_lesson.ipynb new file mode 100644 index 0000000..03af691 --- /dev/null +++ b/support_vector_machine_lesson.ipynb @@ -0,0 +1,355 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# The Math of Intelligence (Week 1) - Support Vector Machine Classification\n", + "\n", + "\n", + "## What will we do?\n", + "\n", + "We will build a Support Vector Machine that will find the optimal hyperplane that maximizes the margin between two toy data classes using gradient descent. \n", + "\n", + "![alt text](http://opticalengineering.spiedigitallibrary.org/data/journals/optice/24850/oe_52_2_027003_f005.png \"Logo Title Text 1\")\n", + "\n", + "\n", + "## What are some use cases for SVMs?\n", + "\n", + "-Classification, regression (time series prediction, etc) , outlier detection, clustering\n", + "\n", + "## Other Examples\n", + "\n", + "- Learning to use Scikit-learn's SVM function to classify images https://github.com/ksopyla/svm_mnist_digit_classification\n", + "- Pulse classification, more useful dataset \n", + "https://github.com/akasantony/pulse-classification-svm\n", + "\n", + "## How does an SVM compare to other ML algorithms?\n", + "\n", + "![alt text](https://image.slidesharecdn.com/mscpresentation-140722065852-phpapp01/95/msc-presentation-bioinformatics-7-638.jpg?cb=1406012610 \"Logo Title Text 1\")\n", + "\n", + "- As a rule of thumb, SVMs are great for relatively small data sets with fewer outliers. \n", + "- Other algorithms (Random forests, deep neural networks, etc.) require more data but almost always come up with very robust models.\n", + "- The decision of which classifier to use depends on your dataset and the general complexity of the problem.\n", + "- \"Premature optimization is the root of all evil (or at least most of it) in programming.\" - Donald Knuth, CS Professor (Turing award speech 1974) \n", + "\n", + "\n", + "## What is a Support Vector Machine?\n", + "\n", + "It's a supervised machine learning algorithm which can be used for both classification or regression problems. But it's usually used for classification. Given 2 or more labeled classes of data, it acts as a discriminative classifier, formally defined by an optimal hyperplane that seperates all the classes. New examples that are then mapped into that same space can then be categorized based on on which side of the gap they fall.\n", + "\n", + "## What are Support Vectors?\n", + "\n", + "![alt text](https://www.dtreg.com/uploaded/pageimg/SvmMargin2.jpg \"Logo Title Text 1\")\n", + " \n", + "Support vectors are the data points nearest to the hyperplane, the points of a data set that, if removed, would alter the position of the dividing hyperplane. Because of this, they can be considered the critical elements of a data set, they are what help us build our SVM. \n", + "\n", + "## Whats a hyperplane?\n", + "\n", + "![alt text](http://slideplayer.com/slide/1579281/5/images/32/Hyperplanes+as+decision+surfaces.jpg \"Logo Title Text 1\")\n", + "\n", + "Geometry tells us that a hyperplane is a subspace of one dimension less than its ambient space. For instance, a hyperplane of an n-dimensional space is a flat subset with dimension n − 1. By its nature, it separates the space into two half spaces.\n", + "\n", + "## Linear vs nonlinear classification?\n", + "\n", + "Sometimes our data is linearly seperable. That means for N classes with M features, we can learn a mapping that is a linear combination. (like y = mx + b). Or even a multidimensional hyperplane (y = x + z + b + q). No matter how many dimensions/features a set of classes have, we can represent the mapping using a linear function.\n", + "\n", + "But sometimes its not. Like if there was a quadratic mapping. Luckily for us SVMs can can efficiently perform a non-linear classification using what is called the kernel trick. I'll talk about that lter on in the course. \n", + "\n", + "![alt text](https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAAuSAAAAJDlhYzcwMzhlLTA0MjYtNDEyYS1hMWM4LTE3Zjk5NDlhNzVkMQ.png \"Logo Title Text 1\")\n", + "\n", + "Alright let's get to building!" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[]" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd4VHX+xfH3Z1IIhN47iDSRbqiBxEIHQREbCooFFZDmrq6r7k/dYltDEQURxYYNFESQEiwJoZrQpCpFFKREUYpI//7+SOKyLsgAmdyZyXk9Tx5JvGTOQznc3LlzYs45REQkdPi8DiAiImdHxS0iEmJU3CIiIUbFLSISYlTcIiIhRsUtIhJiVNwiIiFGxS0iEmJU3CIiISYyEJ+0dOnSrnr16oH41CIiYSkjI+MH51wZf44NSHFXr16d9PT0QHxqEZGwZGZb/T1Wl0pEREKMiltEJMSouEVEQoyKW0QkxKi4RURCjIpbRCTEqLglV/Wb3Y9+s/t5HUMkrKm4RURCjF/FbWbFzWyKma03s3Vm1irQwURE5NT8feXkKGC2c66XmUUDhQKYSURE/sAZi9vMigIJwK0AzrkjwJFAhNm59xBJyRt4oFNdShUuEIiHkFz2++vZ6bvST/nxiZ0m5lkmkXDnz6WSGkAmMNHMlpvZBDOL/f1BZtbfzNLNLD0zM/OcwqRv3cPU5dtpl5TChyu245w7p88jIhLO7EzlaGZxwGIg3jm3xMxGAfucc4+c7ufExcW5cx2Z2rBzP/e/v4qV3/3MFXXL8o+r61OhWMFz+lyS93LOtHWGLXJ2zCzDORfnz7H+nHFvA7Y555Zkvz8FaHqu4c6kTvkifHBPax7uehELNv1A+6RUJi3ZyokTOvsWEQE/its5txP4zszqZH/oCmBtIENF+Iw72tZgztAEGlQqxkNTV9N7wmK++eGXQD6siEhI8Pc+7nuBSWa2CmgM/Ctwkf6jWqlY3rqzBU/2bMCa7fvoODKV8ambOHb8RF48vIhIUDrjNe5zcT7XuE9n595DPDztS+at202jysV4qldD6pYvmquPISLildy+xh0UyheL4aW+cTx3YxO2/fQr3UankZT8FYePHfc6mohIngqZ4gYwM65sVJHk4Yl0a1iB0Z98zZXPpbH825+8jiYikmdCqrhzlIyNZuQNTXjl1jj2HzpGz7EL+fuMtRw8cszraCIiAReSxZ3j8rrlmDssgd7Nq/Jy2hY6jZzPwo0/eB1LRCSgQrq4AYrERPHPqxvwTv+W+Ax6T1jCX95fxd5fj3odTUQkIEK+uHO0rFGKWUMSuCuhBu+lf0eHESkkr93ldSwRkVwXNsUNUDA6gge7XMS0gfGUKBTNna+nM+itZfxw4LDX0UREck1YFXeOhpWLM31QG4a3r82cNTtpn5TCtOUarRKR8BCWxQ0QHelj8BW1mDm4LdVKxTL03RXc/lo63//8q9fRRETOS9gWd47a5Yrw/j2teaRbPRZt+pEOI1J5c7FGq0QkdIV9cUPWaNXtbS5gztAEGlUpxsPTVnPjS4vZotEqEQlB+aK4c1QtVYg3b2/BU9c0YO2OfXQamcqLKRqtEpHQkq+KG7JeNn99s6rMG55IQu0yPDFrPT3HLmTdjn1eRxMR8Uu+K+4c5YrGML7PJYzp3YTtP/3Klc+lkTR3g0arRCTo5dvihqyz724NKzJveCJXNqrI6E830m10Gss0WiUiQSxfF3eOErHRjLi+MRNvbcaBw8e4ZuxCHv9Io1UiEpxU3Ce5rG5Z5g5L4KYWVXllwRY6jkxlgUarRCTIqLh/p0hMFP+4qgHv9m9JpM/HTROW8MAUjVaJSPBQcZ9GixqlmDWkLXcl1mByxne0T0ph7pqdXscSEVFx/5GYqAge7Jw1WlUyNpr+b2Qw8K1lZO7XaJWIeEfF7YeGlYvz0b1tuK99bZLX7KL9iBSmLt+m0SoR8YSK209RET7uvaIWMwe34YLSsQx7dyX9Xv2C7RqtEpE8puI+S7XKFWHK3a35W7d6LNm8hw5JKbyh0SoRyUMq7nMQ4TNua3MBc4cl0KRqCR6Ztpobxi9mc+YBr6OJSD6g4j4PVUoW4o3bm/P0NQ1Zt3MfnUfNZ5xGq0QkwFTc58nMuK5ZFeYNTySxdhmenLWeq15YwNrvNVolIoHhV3Gb2Tdm9qWZrTCz9ECHCkXlisbwYp9LeL53U3buPUT3MWk8q9EqEQmAsznjvsw519g5FxewNCHOzOjasALJwxLp3rgiz326ka6j08jYqtEqEck9ulQSACVio0m6rjET+zXj4OFj9Bq3kMc+WsMvhzVaJSLnz9/idsBcM8sws/6BDBROLqtTlrnDE+nTshoTF3xDx5GpzP860+tYIhLi/C3ueOdcU6AzMNDMEn5/gJn1N7N0M0vPzFQ55ShcIJLHe9TnvbtaERXho8/LS7l/ykr2HtRolYicGzvbl22b2aPAAefcv093TFxcnEtP13OYv3fo6HFGffI141M3UzI2mr/3qE+n+uW9jiUiQcDMMvx9DvGMZ9xmFmtmRXJ+DHQAVp9fxPwpJiqCBzrVZdqAeEoXLsDdb2YwcJJGq0Tk7PhzqaQckGZmK4GlwEzn3OzAxgpvDSoXY/qgeP7csQ7Ja7NGqz5YptEqEfHPWV8q8Yculfhv4+793D9lFcu+/ZlL65Thn1c3oFLxgl7HEpE8lquXSiSwapYtwuS7W/PolfVYuiV7tGrRNxqtEpHTUnEHgQifcWv8BcwZmkDTaiV45MM1Gq0SkdNScQeRKiUL8fptzXmmV0PW79xHp1HzGfu5RqtE5L+puIOMmXFtXNZo1WV1yvDUbI1Wich/U3EHqbJFY3ixTxxjb2rKzr2H6T4mjX/P2cChoxqtEsnvVNxBrnODCswbnkCPxpUY89lGuo6eT8bWPV7HEhEPqbhDQPFC0Tx7XSNeu605h46eoNe4RTw6XaNVIvmVijuEJNYuw5xhCfRtWY1XF2q0SiS/UnGHmMIFInmsR30m392K6Mis0ao/T9ZolUh+ouIOUc2ql+TjwW0ZcOmFfLB8O+1GpDB79U6vY4lIHlBxh7CYqAju71SXDwfGUyZ7tGrApAx27z/kdTQRCSAVdxioX6kYH2aPVs1bt5v2Sam8n6HRKpFwpeIOE1ERPgZeVpOPB7elZtnC3Dd5JbdM/IJtPx30OpqI5DIVd5ipWbYwk+9qxWPdLyb9mz10HJHK64s0WiUSTlTcYcjnM25pXf230aq/fbiG68cvYpNGq0TCgoo7jOWMVv372kZ8tesAnUfN54XPN3JUo1UiIU3FHebMjF6XVCZ5eAJX1C3L07M3cNXzC1i9fa/X0UTkHKm484myRWIYe/MljL2pKbv2HabH8wt4Zs56jVaJhCAVdz6TM1p1dZNKPP/ZJrqMnk/6NxqtEgklKu58qHihaP59bSNev605h4+e4NoXNVolEkpU3PlYQu0yzB2WwC2tqvPaom/oMCKV1K80WiUS7FTc+VxsgUge7X4xk+9qRYEoH31fWcqfJq/k54NHvI4mIqeh4hYA4rJHqwZediFTl2+nXVIqs77c4XUsETkFFbf8JiYqgj93rMv0QfGUK1qAeyYt4543NVolEmxU3PI/Lq5YjGkD47m/Ux0+WZ81WjU5/TuNVokECRW3nFJUhI8Bl9Zk1pC21C5XmD9PWUXfV5by3Z7QG63a2qcvW/v09TqGSK7xu7jNLMLMlpvZjEAGkuByYZnCvNu/FY/3uJhlW3+i48hUXl2wRaNVIh46mzPuIcC6QAWR4OXzGX1bVWfOsATiqpfk0Y/Wct2Li9i4W6NVIl7wq7jNrDLQFZgQ2DgSzCqXKMRr/Zrx7LWN+Hr3AbqMms/zn2m0SiSvRfp53EjgfqBIbj749S8u8uu4d+9qlZsPK+fBzLjmksok1C7D/01fzTNzNjBz1Q6e7tWQ+pWKeR0P4H+uZx/84otTfrzaG6/nWSaR3HTGM24z6wbsds5lnOG4/maWbmbpmZl69V24K1OkAC/cdAnjbm5K5oGs0aqnZmu0SiQv2Jlu8TKzJ4A+wDEgBigKfOCcu/l0PycuLs6lp6fnZk4JYnsPHuUfM9cyOWMbNUrH8lSvhjSrXtLrWL/JOdPWGbYEMzPLcM7F+XPsGc+4nXMPOucqO+eqAzcAn/5RaUv+U6xQFM9c24g3bm/OkeMnuHbcIv724WoOaLRKJCB0H7fkmra1yjBnaAK3tq7OG4u30nFEKikarRLJdWdV3M65z51z3QIVRkJfzmjVlLtbERPl45ZXljL8vRUarRLJRWe8xn0udI1bAA4dPc6YTzcyLmUTxQtF8XiP+nRpUMHrWCJBKVevcYucq5ioCP7UsQ4fDoqnfLEYBkxaxt1vZLB7n0arRM6HilsC7uKKxZg2IJ4HOtXl0w27aZeUwnsarRI5ZypuyRORET7uufRCZg9pS93yRbk/hEerRLym4pY8VaNMYd7p35K/nzRaNXHBFo5rtErEbypuyXM+n9GnVXXmDk+kWfWSPPbbaNV+r6OJhAQVt3imUvGCvNqvGUnXNWJT5gG6jEpjzKdfa7RK5AxU3OIpM6Nn08okD0uk/cXl+Pfcr+g+ZgGrt+/1OppI0FJxS1AoU6QAz/duyot9LuGH7NGqJ2dptErkVFTcElQ6XlyeecMS6dW0MuNSNtFl1HyWbtnjdSyRoKLilqBTrFAUT/VqyJu3t+DI8RNc9+IiHpmm0SqRHCpuCVptapVm7rAEbou/gDeXbKVDUgqfbdjtdSwRz6m4JagVio7kb1fWY8rdrSlUIJJ+E79g+Lsr+OkXjVZJ/qXilpBwSbUSzBzchsGX12T6yu9pPyKFmat26GXzki+puCVkFIiMYHiHOkwf1IYKxQoy8K1l3KXRKsmHVNwScupVLMrUAa15sHNdUr7K5IqkFN77QqNVkn+ouCUkRUb4uCvxQmYNactFFYpy//ur6POyRqskf1BxS0irUaYw79zZkn9cVZ8V3/1MhxGpvJKm0SoJbypuCXk+n3Fzy2rMHZZAixoleXzGWq4dt5Cvd2m0SsKTilvCRsXiBZl4azNGXt+YLT/8QtfRaTz3iUarJPyouCWsmBlXNalE8vBEOlxcjmeTv+LK59L4cptGqyR8qLglLJUuXIAxvZsyvs8l7PnlCD2eT+OJWes0WiVhQcUtYa3DxeVJHp7IdXFVeDFlM51HzWfJ5h+9jiVyXlTcEvaKFYziyWsaMumOFhw7cYLrxy/m4Wlfsv/QUa+jiZwTFbfkG/E1SzNnaAK3t7mASUu+peOIVD5br9EqCT0qbslXCkVH8ki3erx/T2tiC0TS79UvGPbuCvZotEpCiIpb8qWmVUswY3AbBl9Ri49Wfk/7pBRmrPpeL5uXkHDG4jazGDNbamYrzWyNmT2WF8FEAq1AZATD29fmo3vbUKlEQQa9tZz+b2SwS6NVEuT8OeM+DFzunGsENAY6mVnLwMYSyTsXVSjKB/e05q9d6pL6VSbtklJ494tvdfYtQeuMxe2yHMh+Nyr7TX+iJaxERvjon3Ahc4YmUK9CUR54/0tumrCEb3/UaJUEH7+ucZtZhJmtAHYDyc65Jac4pr+ZpZtZemZmZm7nFMkT1UvH8vadLfnn1fVZtW0vHUem8rJGqyTI2Nl8OWhmxYGpwL3OudWnOy4uLs6lp6fnQjwR7+zY+ysPTV3Np+t307hKcZ7u1ZDa5Yp4HUvClJllOOfi/Dn2rO4qcc79DHwOdDqHXCIhpUKxgrx8SxyjbmjM1h9/oevo+Yz+5GuOHNNolXjLn7tKymSfaWNmBYF2wPpABxMJBmZGj8aVmDc8kU71K5CU/BXdx6Sx8rufvY4m+Zg/Z9wVgM/MbBXwBVnXuGcENpZIcClVuADP3diEl/rG8dPBI1z9wgKe+Hgdvx7RaJXkvcgzHeCcWwU0yYMsIkGvfb1ytKhRkic+XseLqZuZs2YnT17TkJY1SnkdTfIRvXJS5CwVjYniiZ4NeeuOFpxwcMP4xTw0VaNVkndU3CLnqHX2aNUdbS7g7aXf0mFEKp+u3+V1LMkHVNwi56FgdAQPZ49WFYmJ5LZX0xn6znKNVklAqbhFckGTqiWYcW9bhlxRi5lf7qBdUgrTV2q0SgJDxS2SS6IjfQzLHq2qUqIgg99ezp2vZ7Bzr0arJHepuEVyWd3yRflgQDwPdbmItI2ZtE9K4e2lGq2S3KPiFgmACJ9xZ0INZg9J4OJKRXnwgy/p/dIStv74i9fRJAyouEUCqHrpWN66oyX/uroBq7dnjVZNmL9Zo1VyXlTcIgHm8xm9W1Rl7vAE4i8szT9mruOasQv5atd+r6NJiFJxi+SRCsUKMiF7tOrbPQfpOno+o+ZptErOnopbJA/ljFYlD0ugS4MKjJin0So5eypuEQ+UKlyAUTc0YULfOH4+eJSrX1jAvzRaJX5ScYt4qF29cswdnsANzasyPnUznUelsmjTj17HkiCn4hbxWNGYKP51dQPeurMFDrjxpcX8deqX7NNolZyGilskSLS+sDSzhyRwZ9sLeGfpt3RI0miVnJqKWySIFIyO4KGu9fhgQDzFCkZx26vpDHlnOT8eOOx1NAkiKm6RINS4SnE+urcNQ9vV4uMvd9B+RKpGq+Q3Km6RIBUd6WNou9rMuLctVUoWyh6tStdolai4RYJdnfJF+OCe1jzc9SLSNv6g0SpRcYuEggifcUfbGswZmkD9SsU0WpXPqbhFQki1UrG8dWcLnuip0ar8TMUtEmLMjBubVyV5eCJtamaNVvUcu5ANOzValV+ouEVCVPliMbzUN47RNzbhuz0H6fbcfEbO+0qjVfmAilskhJkZ3RtVZN7wRLo0qMDIeV9z5XNprNBoVVhTcYuEgZKx0Yy6oQkv3xLH3l+P0vOFBfxz5lqNVoUpFbdIGLniov+MVr00fwsdR6aycNMPXscKS1OfXcbUZ5d58thnLG4zq2Jmn5nZOjNbY2ZD8iKYiJybnNGqt+9siRn0fmkJD36g0apw4s8Z9zHgPufcRUBLYKCZ1QtsLBE5X60uLMXsIQn0T6jBu198S/ukFOat1WhVODhjcTvndjjnlmX/eD+wDqgU6GAicv4KRkfw1y4XMXVAPCUKRXPH6+kMflujVaHOzuZls2ZWHUgF6jvn9p3uuLi4OJeenn7e4UQk9xw5doKxn29izGdfU7hAJI92v5jujSpiZl5HCwm/v579/ddZd+5UrFX8vz5+9X1Nz+nzm1mGcy7On2P9fnLSzAoD7wNDT1XaZtbfzNLNLD0zM9P/tCKSJ6IjfQxpV4uZg9tSrVQsQ95ZwR2vpbNj769eR5Oz5NcZt5lFATOAOc65pDMdrzNukeB2/IRj4oIt/HvuBiJ9Ph7sUpcbm1XF59PZt79yzsDP9Qz793L1jNuyvo56GVjnT2mLSPDLGa2aOzSRhpWL8dDU1fSesJhvftBoVSjw51JJPNAHuNzMVmS/dQlwLhHJA1VLFWLSHS14smcD1mzfR8eRqYxP3cSx43rZfDCLPNMBzrk0QF8/iYQpM+OG5lW5tE5ZHp62mn99vJ6Zq3bwVK+G1C1f1Ot4cgpndVeJv3SNWyQ0OeeYsWoHj05fw95fjzLgspoMvOxCCkRGeB0t7AXkrhIRCX9mxpWNKpI8PJErG1Vk9CdZo1XLv/3J62hyEhW3iPyPkrHRjLi+Ma/cGsf+Q8foOXYhf5+xloNHjnkdTVBxi8gfuLxuOeYOS+CmFlV5OW0LnUbOZ+FGjVZ5TcUtIn+oSEwU/7iqAe/0b4nPoPeEJfzl/VXs/VWjVV5RcYuIX1rWKMXsoQnclViD99K/o8OIFJI1WuUJFbeI+C0mKoIHO1/EtIFZo1V3vp7OoLeW8YNGq/KUiltEzlrDysWZPqgN97Wvzdw1u2iflMK05dsJxO3F8r9U3CJyTqIjfdx7RS1mDm5D9dKxDH13Bbe/ls73P2u0KtBU3CJyXmqVK8KUu1vzt271WLTpRzqMSOXNxVs5cUJn34Gi4haR8xbhM25rcwFzhibQqEoxHp62mhtfWswWjVYFhIpbRHJN1VKFePP2Fjx9TUPW7thHp5GpvJii0arcpuIWkVxlZlzXrArzhieSULsMT8xaT8+xC1m347TfNEvOkopbRAKiXNEYxve5hOd7N+X7n3/lyufSSJq7gcPHjnsdLeSpuEUkYMyMrg0rkDwske6NKjL60410G53GMo1WnRcVt4gEXInYaJKub8zEfs345fAxrhm7kMc/0mjVuVJxi0ieuaxOWeYMS+DmFtV4ZcEWOo5MZYFGq86ailtE8lSRmCj+flV93u3fkkifj5smLOGBKRqtOhsqbhHxRIsapZg1pC13J17IlGXbaJ+Uwtw1O72OFRJU3CLimZioCP7SuS7TBsRTqnAB+r+RwcC3lpG5X6NVf0TFLSKea1C5GNMHxfOnDrVJXrOL9iNSmLp8m0arTkPFLSJBISrCx6DLa/HxkDbUKB3LsHdX0u/VL9iu0ar/oeIWkaBSs2wRJt/dmv+7sh5LNu+hQ1IKb2i06r+ouEUk6ET4jH7xFzB3WAJNqpbgkWmruWH8YjZnHvA6WlBQcYtI0KpSshBv3N6cp3s1ZP3OfXQeNZ9xGq1ScYtIcDMzrovLGq26tE4Znpy1nqteWMDa7/PvaJWKW0RCQtmiMYy7+RJeuKkpO/ceovuYNJ7Np6NVZyxuM3vFzHab2eq8CCQicjpmRpcG2aNVjSvy3Kcb6To6jYyt+Wu0yp8z7leBTgHOISLitxKx0SRd15hX+zXj1yPH6TVuIY99tIZfDueP0arIMx3gnEs1s+qBjyIieW5iV/+O6zczsDnO0aXZo1VPz17PxAXfkLx2F0/0bEDbWmW8jhZQuXaN28z6m1m6maVnZmbm1qcVEflDhQtE8niP+rx3VyuiI3z0eXkp909Zyd6D4TtaZf68pDT7jHuGc66+P580Li7Opaenn18yEZGzdOjocUZ98jXjUzdTMjaav/eoT6f65b2O5Rczy3DOxflzrO4qEZGwERMVwQOd6vLhwHjKFC7A3W9mMHBS+I1WqbhFJOzUr1SMDwfF8+eOdUheu4t2SSm8nxE+o1X+3A74NrAIqGNm28zs9sDHEhE5P1ERPgZeVpOPh7SlZtnC3Dd5JbdODI/RKr+ucZ8tXeMWkWBy4oTj9UXf8PScDRjwQOe63NyiGj6feR3tN7rGLSJyEp/PuDX+AuYMTaBptRL87cM1XD9+EZtCdLRKxS0i+UaVkoV4/bbmPNOrIRt27qfzqPm88PnGkButUnGLSL5iZlwbV4V59yVyeZ2yPD17A1e9sIA13+/1OprfVNwiki+VLRLDuD6XMPampuzce5juYxbwzJz1HDoa/KNVKm4Rydc6N6jAvOEJXNW4Es9/tomuo+eTsXWP17H+kIpbRPK94oWiefa6Rrx2W3MOHT1Br3GLeHR68I5WqbhFRLIl1i7DnGEJ9G1ZjdcWfUOHEamkfhV820sqbhGRkxQuEMlj2aNVBaJ89H1lKX+aHFyjVSpuEZFTaFa9JB8PbsuASy9k6vLttBuRwuzVO7yOBai4RUROKyYqgvv/a7RqGfe8mcHu/Yc8zaXiFhE5g5NHqz5Zv5v2SalM8XC0SsUtIuKH30arBrelVtnC/Cl7tOrIsbx/1eUZv3WZiIj8R82yhXnvrla8uWQrmzN/IToy789/VdwiImfJ5zP6tqru3eN79sgiInJOVNwiIiFGxS0iEmJU3CIiIUbFLSISYlTcIiIhRsUtIhJiVNwiIiHGAvFaezPLBLae408vDfyQi3FySzDmCsZMEJy5gjETBGeuYMwEwZkrNzNVc86V8efAgBT3+TCzdOdcnNc5fi8YcwVjJgjOXMGYCYIzVzBmguDM5VUmXSoREQkxKm4RkRATjMU93usApxGMuYIxEwRnrmDMBMGZKxgzQXDm8iRT0F3jFhGRPxaMZ9wiIvIHgrK4zewZM1tvZqvMbKqZFfc6E4CZXWtma8zshJl5+uy2mXUysw1mttHM/uJllhxm9oqZ7Taz1V5nyWFmVczsMzNbl/17NyQIMsWY2VIzW5md6TGvM+UwswgzW25mM7zOksPMvjGzL81shZmle50nh5kVN7Mp2V21zsxa5dVjB2VxA8lAfedcQ+Ar4EGP8+RYDfQEUr0MYWYRwPNAZ6AecKOZ1fMyU7ZXgU5eh/idY8B9zrmLgJbAwCD4tToMXO6cawQ0BjqZWUuPM+UYAqzzOsQpXOacaxxktwOOAmY75+oCjcjDX7egLG7n3Fzn3LHsdxcDlb3Mk8M5t845t8HrHEBzYKNzbrNz7gjwDtDD40w451KBPV7nOJlzbodzbln2j/eT9ZerkseZnHPuQPa7Udlvnj/ZZGaVga7ABK+zBDszKwokAC8DOOeOOOd+zqvHD8ri/p3bgFlehwgylYDvTnp/Gx6XUSgws+pAE2CJt0l+uySxAtgNJDvnPM8EjATuB/L+u9/+MQfMNbMMM+vvdZhsNYBMYGL2paUJZhabVw/uWXGb2TwzW32Ktx4nHfMQWV/qTgqmXEHATvExz8/YgpmZFQbeB4Y65/Z5ncc5d9w515isryabm1l9L/OYWTdgt3Muw8scpxHvnGtK1qXBgWaW4HUgsr5fb1NgrHOuCfALkGfPNXn2zYKdc+3+6P+b2S1AN+AKl4f3LJ4pV5DYBlQ56f3KwPceZQl6ZhZFVmlPcs594HWekznnfjazz8l6bsDLJ3Xjge5m1gWIAYqa2ZvOuZs9zASAc+777P/uNrOpZF0q9PR5JrL+Dm476SulKeRhcQflpRIz6wQ8AHR3zh30Ok8Q+gKoZWYXmFk0cAMw3eNMQcnMjKzrkOucc0le5wEwszI5d0qZWUGgHbDey0zOuQedc5Wdc9XJ+vP0aTCUtpnFmlmRnB8DHfD2HzgAnHM7ge/MrE72h64A1ubV4wdlcQNjgCJAcvYtQOO8DgRgZleb2TagFTDTzOZ4kSP7idtBwByynmx7zzm3xossJzOzt4FFQB0z22Zmt3udiawzyT7A5dl/llZkn1V6qQLwmZmtIusf4WTnXNDcfhdkygFpZrYSWArMdM7N9jhTjnuBSdn3a8DgAAAAQ0lEQVS/j42Bf+XVA+uVkyIiISZYz7hFROQ0VNwiIiFGxS0iEmJU3CIiIUbFLSISYlTcIiIhRsUtIhJiVNwiIiHm/wGVde6UqmWnmQAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "#To help us perform math operations\n", + "import numpy as np\n", + "#to plot our data and model visually\n", + "from matplotlib import pyplot as plt\n", + "%matplotlib inline\n", + "\n", + "#Step 1 - Define our data\n", + "\n", + "#Input data - Of the form [X value, Y value, Bias term]\n", + "X = np.array([\n", + " [-2,4,-1],\n", + " [4,1,-1],\n", + " [1, 6, -1],\n", + " [2, 4, -1],\n", + " [6, 2, -1],\n", + "])\n", + "\n", + "#Associated output labels - First 2 examples are labeled '-1' and last 3 are labeled '+1'\n", + "y = np.array([-1,-1,1,1,1])\n", + "\n", + "#lets plot these examples on a 2D graph!\n", + "#for each example\n", + "for d, sample in enumerate(X):\n", + " # Plot the negative samples (the first 2)\n", + " if d < 2:\n", + " plt.scatter(sample[0], sample[1], s=120, marker='_', linewidths=2)\n", + " # Plot the positive samples (the last 3)\n", + " else:\n", + " plt.scatter(sample[0], sample[1], s=120, marker='+', linewidths=4)\n", + "\n", + "# Print a possible hyperplane, that is seperating the two classes.\n", + "#we'll two points and draw the line between them (naive guess)\n", + "plt.plot([-2,6],[6,0.5])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Let's define our loss function (what to minimize) and our objective function (what to optimize)\n", + "\n", + "#### Loss function\n", + "\n", + "We'll use the Hinge loss. This is a loss function used for training classifiers. The hinge loss is used for \"maximum-margin\" classification, most notably for support vector machines (SVMs).\n", + "\n", + "![alt text](http://i.imgur.com/OzCwzyN.png \"Logo Title Text 1\")\n", + "\n", + "\n", + "c is the loss function, x the sample, y is the true label, f(x) the predicted label.\n", + "\n", + "![alt text](http://i.imgur.com/FZ7JcG3.png \"Logo Title Text 1\")\n", + "\n", + " \n", + "#### Objective Function\n", + "\n", + "![alt text](http://i.imgur.com/I5NNu44.png \"Logo Title Text 1\")\n", + "\n", + "As you can see, our objective of a SVM consists of two terms. The first term is a regularizer, the heart of the SVM, the second term the loss. The regularizer balances between margin maximization and loss. We want to find the decision surface that is maximally far away from any data points.\n", + "\n", + "How do we minimize our loss/optimize for our objective (i.e learn)?\n", + "\n", + "We have to derive our objective function to get the gradients! Gradient descent ftw. As we have two terms, we will derive them seperately using the sum rule in differentiation.\n", + "\n", + "\n", + "![alt text](http://i.imgur.com/6uK3BnH.png \"Logo Title Text 1\")\n", + "\n", + "This means, if we have a misclassified sample, we update the weight vector w using the gradients of both terms, else if classified correctly,we just update w by the gradient of the regularizer.\n", + "\n", + "\n", + "\n", + "Misclassification condition \n", + "\n", + "![alt text](http://i.imgur.com/g9QLAyn.png \"Logo Title Text 1\")\n", + "\n", + "Update rule for our weights (misclassified)\n", + "\n", + "![alt text](http://i.imgur.com/rkdPpTZ.png \"Logo Title Text 1\")\n", + "\n", + "including the learning rate η and the regularizer λ\n", + "The learning rate is the length of the steps the algorithm makes down the gradient on the error curve.\n", + "- Learning rate too high? The algorithm might overshoot the optimal point.\n", + "- Learning rate too low? Could take too long to converge. Or never converge.\n", + "\n", + "The regularizer controls the trade off between the achieving a low training error and a low testing error that is the ability to generalize your classifier to unseen data. As a regulizing parameter we choose 1/epochs, so this parameter will decrease, as the number of epochs increases.\n", + "- Regularizer too high? overfit (large testing error) \n", + "- Regularizer too low? underfit (large training error) \n", + "\n", + "Update rule for our weights (correctly classified)\n", + "\n", + "![alt text](http://i.imgur.com/xTKbvZ6.png \"Logo Title Text 1\")\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "#lets perform stochastic gradient descent to learn the seperating hyperplane between both classes\n", + "\n", + "def svm_sgd_plot(X, Y):\n", + " #Initialize our SVMs weight vector with zeros (3 values)\n", + " w = np.zeros(len(X[0]))\n", + " #The learning rate\n", + " eta = 1\n", + " #how many iterations to train for\n", + " epochs = 100000\n", + " #store misclassifications so we can plot how they change over time\n", + " errors = []\n", + "\n", + " #training part, gradient descent part\n", + " for epoch in range(1,epochs):\n", + " error = 0\n", + " for i, x in enumerate(X):\n", + " #misclassification\n", + " if (Y[i]*np.dot(X[i], w)) < 1:\n", + " #misclassified update for ours weights\n", + " w = w + eta * ( (X[i] * Y[i]) + (-2 *(1/epoch)* w) )\n", + " error = 1\n", + " else:\n", + " #correct classification, update our weights\n", + " w = w + eta * (-2 *(1/epoch)* w)\n", + " errors.append(error)\n", + " \n", + "\n", + " #lets plot the rate of classification errors during training for our SVM\n", + " plt.plot(errors, '|')\n", + " plt.ylim(0.5,1.5)\n", + " plt.axes().set_yticklabels([])\n", + " plt.xlabel('Epoch')\n", + " plt.ylabel('Misclassified')\n", + " plt.show()\n", + " \n", + " return w" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFPFJREFUeJzt3XeUXAXZx/HfT4pUD0cINUBEkJZAwCUkBAIECAFCU1C6YIkFERQFsbzgCwgWqgIWEFCaCoKaQEiAhNDDLCWkUCIJwkFk9T0eUVSUPO8fd8bdbHZ27m5m5t478/2csyfZy83MQ/vmyZ27O44IAQCK4x1ZDwAAGBjCDQAFQ7gBoGAINwAUDOEGgIIh3ABQMIQbAAqGcANAwRBuACiYlRvxoOutt14MGzasEQ8NAC2ps7PzTxExJM25DQn3sGHDVCqVGvHQANCSbL+U9lwulQBAwRBuACgYwg0ABUO4AaBgCDcAFAzhBoCCIdyoq5OmnaSTpp2U9RhASyPcAFAwqcJtex3bt9p+1vZC22MaPRgAoG9pv3LyMknTIuII26tKWqOBMwEA+lEz3LbfJWmcpBMlKSLekvRWY8dCUfS+nl36Y6nP49dOvLZpMwGtLs2lki0kdUm61vaTtq+2vWbvk2xPtl2yXerq6qr7oACAhCOi/xPsDkmPShobEY/ZvkzSXyPi69V+TUdHR/BNptpTZdNmwwYGxnZnRHSkOTfNxv2KpFci4rHy57dK2nmwwwEAVkzNcEfEa5Jetr11+dA+khY0dCoAQFVp7yo5RdKN5TtKXpTEV1gAQEZShTsinpKU6toL2hvXtoHG4ysnAaBgCDcAFAzhBoCCIdwAUDCEGwAKhnADQMEQbgAoGMINAAVDuAGgYAg3ABQM4QaAgiHcAFAwhBsACoZwA0DBEG4AKBjCDQAFQ7gBoGAINwAUDOEGgIIh3ABQMIQbAAqGcANAwRBuACgYwg0ABUO4AaBgCDcAFAzhBoCCIdwAUDANCffChdKUKVJEIx4dANpbQ8L95pvSwQdLu+xCwAGg3lKF2/YS28/Yfsp2qeaDlh+1s5OAA0C9DWTj3jsiRkZER60TR4yQvvY1ae21k88rAR81Spo6lYADwIpoyKWSlVeWzj1XWrJk2YCXStKkSQQcAFZE2nCHpOm2O21P7usE25Ntl2yXurq6JEnvfncS8MWLpa9+lYADQD04UlTT9sYR8art9SXNkHRKRMyudn5HR0eUSstfCv/zn6VLLpEuv1x6442e50vnnCMdeKBkD+LvAgAKznZnmkvRUsqNOyJeLf/4uqTbJY0azGDrriudd173Br7WWsnxyga+667SnXeygQNAf2qG2/aatteu/FzSBEnzVuRJKwFfsmTZgD/+uHTQQQQcAPqTZuPeQNKDtp+WNEfS1IiYVo8nJ+AAMHA1wx0RL0bEjuWP7SPi/HoP0TPgX/kKAQeA/uTqe5Wsu650/vnVAz56tHTXXQQcQHvLVbgrqgV8zpzkzhMCDqCd5TLcFQQcAJaX63BXVAK+eLF01lkEHEB7K0S4K9ZbT/rmNwk4gPZWqHBX1Ar4mDHStGkEHEBrKmS4K3oHfM01k+OPPSYdcAABB9CaCh3uikrAlyyRvvxlAg6gtbVEuCvWW0+64AICDqC1tVS4K2oFfLfdpLvvJuAAiqklw11RLeCPPipNnEjAARRTS4e7ohLwxYulM88k4ACKrS3CXTFkiHThhQQcQLG1VbgragV87Fhp+nQCDiCf2jLcFdUC/sgj0v77E3AA+dTW4a4g4ACKhHD30DPgZ5whrbFGcpyAA8gTwt2HIUOkb30ruY2wr4Dvvrs0YwYBB5ANwt2PagF/+GFpwgQCDiAbhDsFAg4gTwj3AFQCvnix9KUvEXAA2SDcg7D++tK3v03AAWSDcK+AWgHfYw/pnnsIOID6Itx1UC3gDz0k7bcfAQdQX4S7jnoG/ItflFZfPTlOwAHUE+FugPXXl77zneQuFAIOoN4IdwNVAl5tAx83Trr3XgIOYGAIdxNssEHfAX/wQWnffQk4gIEh3E1EwLPx0vEn6KXjT8h6DKBuCHcGegb89NMJOICBSR1u2yvZftL2lEYO1E422ED67nerB3zPPaX77iPgAJY1kI37VEkLGzVIO6sW8AcekPbZh4ADWJYjRQ1sD5V0vaTzJX0hIib1d35HR0eUSqWaj/vhHz6Sasiff3JMqvNaxR//mFxKufJK6R//6D6+xx7SOedIe+8t2ZmNl3u9r2e/+fjjkqQ1dtllmeOb/+ynTZsJqMV2Z0R0pDk37cZ9qaQzJC3t50kn2y7ZLnV1daV8WPSlsoG/+KL0hS+wgQNYVs2N2/YkSQdGxGds7yXpi/XauJHOa68lG/hVV7GBD0ZlA2fDRp7Ve+MeK+kQ20sk3SJpvO0bVmA+DNCGG0oXXVR9A99rL2nmTDZwoF3UDHdEnBURQyNimKSjJN0XEcc1fDIsp3fAV1stOT57tjR+PAEH2gX3cRdQJeCLF0uf/zwBB9pNqrtKBopr3M312mvJdyW86irpn//sPj5uXPc1cAD51oi7SpBjG24oXXxx/xv4rFlZTgigngh3C6kW8PvvT7ZuAg60BsLdgioBf/FF6bTTCDjQagh3C9toI+mSSwg40GoIdxuoFfC99ybgQJEQ7jZSLeCzZnUH/P77Mx0RQAqEuw31F/C99iLgQN4R7jbWM+CnnkrAgaIg3NBGG0mXXtod8He+MzlOwIF8Itz4r1oBHz+egAN5QLixnI037jvgM2d2B3z27ExHBNoa4UZVPQP+uc8tG/A99yTgQFYIN2raeGPpsssIOJAXhBup1Qr4PvsQcKAZCDcGrFrA77uvO+APPJDtjEArI9wYtErAf/c76ZRTlg34uHEEHGgUwo0Vtskm0uWXE3CgWQg36qZWwPfdl4AD9UC4UXfVAn7vvd0Bf/DBbGcEioxwo2F6Bvyzn5VWXTU5fu+90h57EHBgsAg3Gm6TTaTvfS+5C4WAAyuOcKNpagV8v/0IOJAG4UbTVQLe+xLKPfcQcCANwo3MDB1aO+APPZTtjEAeEW5krmfATz552YDvvjsBB3oj3MiNoUOl73+/esBPPJGAAxLhRg71FfDhw6Xrr08CPmECAUd7I9zIrZ4BP+yw7g18xozugD/8cLYzAlkg3Mi9oUOlc8+VFi2SPvOZZQM+diwBr6e99ko+kG81w217NdtzbD9te77tbzRjMKC3TTeVrriCgANpNu5/SRofETtKGilpou3RjR0LqK5WwPffn4CjtdUMdyT+Vv50lfJHNHQqIIVqAZ8+vTvgjzyS7YxAIziidoNtrySpU9KWkq6IiDP7O7+joyNKpVJ9JgRSevll6YILpKuvlv797+7jEyZI55wjjRmT2Wi51ft69v33Jz/uueeyx2fNasY07c12Z0R0pDk31YuTEfF2RIyUNFTSKNvD+3jSybZLtktdXV0Dmxiog003la68MrkL5dOfllZZJTk+fbq0225s4GgdqTbuZX6Bfbakv0fEd6udw8aNPPj976ULL1x+A99/f+nss9nA+1LZwNmwm6+uG7ftIbbXKf98dUn7Snp2xUYEGm+zzZINfNGiZTfwu+9ONvCJE9nAUUxpLpVsJGmm7bmSHpc0IyKmNHYsoH56BvxTnyLgKL40d5XMjYidImKHiBgeEf/bjMGAettsM+mqq/oP+KOPZjsjkAZfOYm201/Ax4yRDjigfQM+axbXt4uAcKNtVQv4tGkEHPlGuNH2KgF/4QXpk58k4Mg/wg2Ubb659IMf9B/wxx7LdkZAItzAcvoL+OjR0oEHEnBki3ADVVQL+F13EXBki3ADNfQM+OTJ0sorJ8cJOLJCuIGUNt9c+uEPk7tQqgV8zpxsZ0R7INzAAFUC3tcGvuuu0kEHEXA0FuEGBmnYsL4DfuedBByNRbiBFUTA0WyEG6iTngH/xCcIOBqHcAN1NmyY9KMfVQ/4pEnS449nOiIKjnADDVIt4FOnSqNGEXAMHuEGGoyAo94IN9AklYA//7z08Y8TcAwe4Qaa7D3vkX784+oBP/hgibdsRX8IN5CRagGfMkXaZRcCjuoIN5AxAo6BItxATlQC/txz0sc+RsBRHeEGcmaLLaSrr+4O+EorJccrAT/kEKmzM9sZkS3CDeRUJeDPP79swH/7W6mjg4C3M8IN5BwBR2+EGyiIngH/6EcJeDsj3EDBbLGFdM011QN+6KHSE09kOyMai3ADBVUt4L/5jfT+9xPwVka4gYIj4O2HcAMtohLw556TTjqJgLcywg20mPe+V/rJTwh4KyPcQIuqFfDDDpOefDLbGYvs9oue0O0XZfM7YM1w297U9kzbC23Pt31qMwYDUB/VAv7rX0s770zAiyjNxv0fSadHxLaSRks62fZ2jR0LQL1VAv7ss9KJJxLwIqsZ7oj4Q0Q8Uf75G5IWStqk0YMBaIwtt5SuvZaAF5kjIv3J9jBJsyUNj4i/Vjuvo6MjSnwbM2Rmr5TnzWrgDMWxaJF0/vnSz34mvf129/HDDpP+53+knXbKbrY86X09+9UX/iJJ2nirdZY5fvjpOw/q8W13RkRHmnNTvzhpey1Jt0k6ra9o255su2S71NXVlX5aAJmqtoHfcUeygR9+uPTUU5mOiF5Sbdy2V5E0RdLdEXFxrfPZuIHiWrRIOu886YYblt/Azz5bGjkyu9nypLKBD3bD7q2uG7dtS7pG0sI00QZQbFtuKV13XbKBf+Qj0jvKlbjjjuSyCRt49tJcKhkr6XhJ420/Vf44sMFzAchYrYB/4AMEPCtp7ip5MCIcETtExMjyx53NGA5A9rbaqu+A3347Ac/KgO4qSYtr3EDreuGF7mvgS5d2Hz/88OQuFK6BD05D7ioBACnZwK+/PtnATzih7w386aeznbHVEW4Ag9JfwEeOlD74QQLeKIQbwAqpFvBf/YqANwrhBlAXlYAvXCgdfzwBbyTCDaCu3vc+6ac/7T/gc+dmO2PREW4ADdFfwHfcUTriCAI+WIQbQENVC/httxHwwSLcAJqiEvAFC6TjjiPgK4JwA2iqrbdOvoUsAR88wg0gE7UCfuSR0jPPZDtjXhFuAJmqFvBbb5V22IGA94VwA8gFAp4e4QaQK5WAz58vHXssAe8L4QaQS9tsk3wHwmoB/9CHpHnzsp0xK4QbQK5VC/gvfymNGNGeASfcAAqhd8Dt5Hg7BpxwAyiUSsAXLJCOOaY9A064ARTSNttIN97Yd8B32EH68IdbN+CEG0ChVQI+f353wCOkX/yiO+Dz52c9ZX0RbgAtYdttqwd8xIjWCjjhBtBSegb86KNbM+CEG0BL2nZb6aabqgf8qKOKG3DCDaClVQv4z39e3IATbgBtoZUCTrgBtJVKwOfNS2LdV8AXLMh6yv4RbgBtabvtpJtv7jvgw4cnW3leA064AbS1agG/5Zb8BpxwA4CKFXDCDQA9VAL+zDPJPd95DHjNcNv+ie3XbbfoV/0DwPK23z6JdbWAH3OMtHBhNrOl2bivkzSxwXMAQC5VC/jNNyd/LYuA1wx3RMyW9H9NmAUAcitPAXdE1D7JHiZpSkQMT/OgHR0dUSqVVmwyAI137UHpzjtpamPnKKB586Rzz02+jWwlo3bywubXv57cLz4QtjsjoiPNuXV7cdL2ZNsl26Wurq56PSwA5NLw4ck933PnJm/e0HsDP/ZY6dlnG/PcbNwAUAfVNvCjj0428G226f/XZ7JxA0A7672BS0nAb7opucWwnht4mtsBb5b0iKStbb9i+2P1eWoAaD2VgD/zjHTkkcmxegc8zV0lR0fERhGxSkQMjYhrVuwpAaD1DR+efO/vRgScSyUA0ECVgM+du3zAt99eOu64gQeccANAE4wYsXzAly5N3mZt++0H9liEGwCaqGfAjzgiObZ06cAeg3ADQAZGjEhuHewZ8LQINwBkqBLwgUj1BTgDZbtL0kuD/OXrSfpTHceplzzOlceZpHzOlceZpHzOlceZpHzOVc+ZNo+IIWlObEi4V4TtUtqvHmqmPM6Vx5mkfM6Vx5mkfM6Vx5mkfM6V1UxcKgGAgiHcAFAweQz3j7IeoIo8zpXHmaR8zpXHmaR8zpXHmaR8zpXJTLm7xg0A6F8eN24AQD9yGW7b37H9rO25tm+3vU7WM0mS7SNtz7e91Hamr27bnmj7OduLbH85y1kq8vjG0rY3tT3T9sLyv7tTczDTarbn2H66PNM3sp6pwvZKtp+0PSXrWSpsL7H9jO2nbOfmG/3bXsf2reVWLbQ9plnPnctwS5ohaXhE7CDpeUlnZTxPxTxJH5A0O8shbK8k6QpJB0jaTtLRtrfLcqay65S/N5b+j6TTI2JbSaMlnZyDf1b/kjQ+InaUNFLSRNujM56p4lRJGb13eb/2joiRObsd8DJJ0yJiG0k7qon/3HIZ7oiYHhH/KX/6qKShWc5TERELI+K5rOeQNErSooh4MSLeknSLpEMznimXbywdEX+IiCfKP39Dyf9cm2Q8U0TE38qfrlL+yPzFJttDJR0k6eqsZ8k72++SNE7SNZIUEW9FxF+a9fy5DHcvH5V0V9ZD5Mwmkl7u8fkryjhGRVB+C76dJD2W7ST/vSTxlKTXJc2IiMxnknSppDMkDfBbHjVcSJpuu9P25KyHKdtCUpeka8uXlq62vWaznjyzcNu+x/a8Pj4O7XHOV5X8UffGPM2VA+7jWOYbW57ZXkvSbZJOi4i/Zj1PRLwdESOV/GlylO1U7+faKLYnSXo9IjqznKOKsRGxs5JLgyfbHpf1QJJWlrSzpKsiYidJf5fUtNeaVm7WE/UWEfv299dtf0TSJEn7RBPvWaw1V068ImnTHp8PlfRqRrPknu1VlET7xoj4Vdbz9BQRf7E9S8lrA1m+qDtW0iG2D5S0mqR32b4hIo7LcCZJUkS8Wv7xddu3K7lUmOnrTEr+H3ylx5+UblUTw53LSyW2J0o6U9IhEfFm1vPk0OOStrL9HturSjpK0m8ynimXbFvJdciFEXFx1vNIku0hlTulbK8uaV9JdXob2cGJiLPKb004TMl/T/flIdq217S9duXnkiYo29/gJEkR8Zqkl21vXT60j6QFzXr+XIZb0vclrS1pRvkWoB9kPZAk2T7c9iuSxkiaavvuLOYov3D7WUl3K3mx7RcRMT+LWXrK6RtLj5V0vKTx5f+WnipvlVnaSNJM23OV/CY8IyJyc/tdzmwg6UHbT0uaI2lqREzLeKaKUyTdWP73OFLSN5v1xHzlJAAUTF43bgBAFYQbAAqGcANAwRBuACgYwg0ABUO4AaBgCDcAFAzhBoCC+X8gZzsMWajoHgAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "for d, sample in enumerate(X):\n", + " # Plot the negative samples\n", + " if d < 2:\n", + " plt.scatter(sample[0], sample[1], s=120, marker='_', linewidths=2)\n", + " # Plot the positive samples\n", + " else:\n", + " plt.scatter(sample[0], sample[1], s=120, marker='+', linewidths=2)\n", + "\n", + "# Add our test samples\n", + "plt.scatter(2,2, s=120, marker='_', linewidths=2, color='yellow')\n", + "plt.scatter(4,3, s=120, marker='+', linewidths=2, color='blue')\n", + "\n", + "\n", + "\n", + "x2x3 =np.array([x2,x3])\n", + "X,Y,U,V = zip(*x2x3)\n", + "ax = plt.gca()\n", + "ax.quiver(X,Y,U,V,scale=1, color='blue')" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEKCAYAAADticXcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAD6dJREFUeJzt3X+MZXV5x/H3I8svtQjIVLeCDGuxhjb+WCYNi42xiFiQYAJEpbaitW4LbUFNUIh/GK3/SJAQqimuVqKUUoXSFrCCZAs0pi0wawGhQFkRCxXKbC1SjQWEp3/c79jLsjP3x8y9d+aZ9yu5ued8z7n3PGe+dz458z1nzo3MRJJUw/MmXYAkafkY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYWsG/cGDzjggJyenh73ZiVpVdu2bduOzJzqtd7YQ316eprZ2dlxb1aSVrWI+F4/6zn8IkmFGOqSVIihLkmFGOqSVIihLkmFGOqSVIihLkmFGOqSVIihLkmFGOqSVIihLkmFGOqSVIihLkmFGOqSVIihLkmFGOqSVIihLkmFGOqSVIihLkmFGOqSVMhYQj0iNkfEbETMzs3NjWOTkrQmjSXUM3NLZs5k5szU1NQ4NilJa5LDL5JUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYUY6pJUiKEuSYWMJdQjYnNEzEbE7Nzc3Dg2KUlr0lhCPTO3ZOZMZs5MTU2NY5OStCY5/CJJhRjqklSIoS5JhRjqklSIoS5JhRjqklSIoS5JhRjqklSIoS5JhRjqklSIoS5JhRjqklTIusUWRsTVQC60PDNPWPaKJElDWzTUgfPa84nAS4E/b/OnAA+MqCZJ0pAWDfXMvAkgIv44M9/QtejqiPiHkVYmSRpYv2PqUxGxYX4mIg4BvDG6JK0wvYZf5n0QuDEi7m/z08DvjaQiSdLQ+gr1zLw2Ig4FXtWa7snMJ0ZXliRpGH0Nv0TE84GzgD/MzNuBl0fE8SOtTJI0sH7H1C8GngQ2tfmHgE+OpCJJ0tD6DfVXZOa5wFMAmfkTIEZWlSRpKP2G+pMRsTftH5Ei4hWAY+qStML0e/XLx4BrgYMi4lLg9cB7RlWUJGk4/V79cn1EfAs4gs6wy5mZuWOklUmSBrbo8EtEvKo9bwQOBh4Gvk/n6peNoy9PkjSIXkfqHwI2A5/exbIEjlr2iiRJQ+sV6te35/dl5v2LrilJmrheV7+c056vGHUhkqSl63Wk/l8RcQNwSERctfNC76cuSStLr1B/K7ARuIRdj6tLklaQXvdTfxL454g4MjPnxlSTJGlIvb7O7oLM/ADwxYh4ztfaOfwiSStLr+GXS9rzeYuuJUlaEXoNv2xrzzfNt0XEfsBBmXnHiGuTJA2o3/up3xgR+0TE/sDtwMURcf5oS5MkDarfuzS+KDMfB04ELs7Mw4GjR1eWJGkY/Yb6uohYD7wduGaE9UiSlqDfUP8EcB2wPTNvjYgNwH2jK0uSNIy+Qj0zL8/MV2fm6W3+/sw8abSlPdf02V9b8mPn9+men59eqG3n5Qst63d6OeaX2rac7b2WLcfy5VxnkPVGue4w60/qdUt97XK8frneYxTvNYr3G1S/J0rPbSdKd4+IrRGxIyJ+a9TFSZIG0+/wyzHtROnxdL50+pXAWSOrSpI0lH5Dfff2fBxwWWb+YJCNRMTmiJiNiNm5Oe82IEmj0m+oXx0R9wAzwNaImAL+t9+NZOaWzJzJzJmpqalh6pQk9aHfE6VnA5uAmcx8Cvgx8LZRFiZJGlxfXzzdvAx4c0Ts1dX25WWuR5K0BJH5nJsvPneliI8BbwQOA/4OOBb4ZmaePOgGZ2ZmcnZ2dtCXSdKaFhHbMnOm13r9jqmfDLwJeCQz3wu8BthzCfVJkkag31D/SWY+A/w0IvYBHgU2jK4sSdIw+h1Tn42IfYHPA9uAHwG3jKwqSdJQ+gr1+dsDABdFxLXAPt5PXZJWnl5fZ7dxsWWZ+a3lL0mSNKxeR+qfXmRZAkctYy2SpCXq9XV2vz6uQiRJS9fvXRr/oJ0onZ/fLyJOX+w1kqTx6/eSxvdn5mPzM5n538D7R1OSJGlY/Yb68yIi5mciYjdgj9GUJEkaVr/XqV8HfDUiLqJzgvT3gWtHVpUkaSj9hvpHgM3AaUAA3wC+MKqiJEnD6fefj54BLqLzz0f7Awdm5tMjrUySNLB+r365sX1H6f7AbcDFEXH+aEuTJA2q3xOlL2rfUXoicHFmHg4cPbqyJEnD6DfU10XEeuDtwDUjrEeStAT9hvon6FwBsz0zb42IDcB9oytLkjSMfk+UXg5c3jV/P3DSqIqSJA2n110aP5yZ50bEn9C5Pv1ZMvOMkVUmSRpYryP1u9uzXyoqSatAr7s0Xt2evzSeciRJS9Fr+OWqxZZn5gnLW44kaSl6Db9sAh4ELgNupnOLAEnSCtUr1F8KvBk4BfhN4GvAZZl516gLkyQNbtHr1DPz6cy8NjNPBY4AtgM3RsQfjaU6SdJAel6nHhF7Am+lc7Q+DVwIXDnasiRJw+h1ovRLwK8AXwc+npl3jqUqSdJQeh2p/zbwY+CVwBndX34EZGbuM8LaJEkD6nWder/3hpEkrQCGtiQVYqhLUiGGuiQVYqhLUiGGuiQVYqhLUiGGuiQVYqhLUiGGuiQVYqhLUiFjCfWI2BwRsxExOzc3N45NStKaNJZQz8wtmTmTmTNTU1Pj2KQkrUkOv0hSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIYa6JBViqEtSIWMJ9YjYHBGzETE7Nzc3jk1K0po0llDPzC2ZOZOZM1NTU+PYpCStSQ6/SFIhhrokFWKoS1IhhrokFWKoS1IhhrokFWKoS1IhhrokFWKoS1IhhrokFWKoS1IhhrokFWKoS1IhhrokFWKoS1IhhrokFWKoS1IhhrokFWKoS1IhhrokFRKZOd4NRswB3xvy5QcAO5axnNXAfV4b3Oe1YSn7fHBmTvVaaeyhvhQRMZuZM5OuY5zc57XBfV4bxrHPDr9IUiGGuiQVstpCfcukC5gA93ltcJ/XhpHv86oaU5ckLW61HalLkhaxakI9In4jIu6NiO0Rcfak6xlERBwUETdExN0RcVdEnNna94+I6yPivva8X2uPiLiw7esdEbGx671ObevfFxGndrUfHhHfbq+5MCJi/Hv6XBGxW0T8S0Rc0+YPiYibW/1fiYg9WvuebX57Wz7d9R7ntPZ7I+ItXe0r7jMREftGxBURcU/r703V+zkiPtg+13dGxGURsVe1fo6IL0bEoxFxZ1fbyPt1oW0sKjNX/APYDfgOsAHYA7gdOGzSdQ1Q/3pgY5v+OeDfgMOAc4GzW/vZwKfa9HHA14EAjgBubu37A/e35/3a9H5t2S3ApvaarwPHTnq/W10fAv4CuKbNfxV4Z5u+CDitTZ8OXNSm3wl8pU0f1vp7T+CQ9jnYbaV+JoAvAb/bpvcA9q3cz8DLgO8Ce3f173uq9TPwBmAjcGdX28j7daFtLFrrpH8J+vyBbgKu65o/Bzhn0nUtYX/+FngzcC+wvrWtB+5t058DTula/962/BTgc13tn2tt64F7utqftd4E9/NAYCtwFHBN+8DuANbt3K/AdcCmNr2urRc79/X8eivxMwHs0wIudmov2890Qv3BFlTrWj+/pWI/A9M8O9RH3q8LbWOxx2oZfpn/4Mx7qLWtOu3PzdcBNwMvycyHAdrzz7fVFtrfxdof2kX7pF0AfBh4ps2/GHgsM3/a5rvr/Nm+teU/bOsP+rOYpA3AHHBxG3L6QkS8gML9nJn/AZwH/DvwMJ1+20btfp43jn5daBsLWi2hvqtxw1V32U5EvBD4K+ADmfn4Yqvuoi2HaJ+YiDgeeDQzt3U372LV7LFs1ewznSPPjcCfZubrgB/T+ZN5Iat+n9sY79voDJn8AvAC4NhdrFqpn3uZ6D6ullB/CDioa/5A4PsTqmUoEbE7nUC/NDOvbM3/GRHr2/L1wKOtfaH9Xaz9wF20T9LrgRMi4gHgL+kMwVwA7BsR69o63XX+bN/a8hcBP2Dwn8UkPQQ8lJk3t/kr6IR85X4+GvhuZs5l5lPAlcCR1O7neePo14W2saDVEuq3Aoe2M+p70DnBctWEa+pbO5P9Z8DdmXl+16KrgPkz4KfSGWufb393O4t+BPDD9qfXdcAxEbFfO0I6hs5448PA/0TEEW1b7+56r4nIzHMy88DMnKbTX3+fme8CbgBObqvtvM/zP4uT2/rZ2t/Zrpo4BDiUzkmlFfeZyMxHgAcj4pda05uAf6VwP9MZdjkiIp7faprf57L93GUc/brQNhY2yZMsA56kOI7OVSPfAT466XoGrP3X6Pw5dQdwW3scR2cscStwX3vev60fwGfbvn4bmOl6r98BtrfHe7vaZ4A722s+w04n6ya8/2/k/69+2UDnl3U7cDmwZ2vfq81vb8s3dL3+o22/7qXrao+V+JkAXgvMtr7+GzpXOZTuZ+DjwD2trkvoXMFSqp+By+icM3iKzpH1+8bRrwttY7GH/1EqSYWsluEXSVIfDHVJKsRQl6RCDHVJKsRQl6RCDHWVExFPR8RtXY9lu7NfREx336lPWmnW9V5FWnV+kpmvnXQR0iR4pK41IyIeiIhPRcQt7fGLrf3giNja7n29NSJe3tpfEhF/HRG3t8eR7a12i4jPR+ce4t+IiL0ntlPSTgx1VbT3TsMv7+ha9nhm/iqd/9q7oLV9BvhyZr4auBS4sLVfCNyUma+hcw+Xu1r7ocBnM/OXgceAk0a8P1Lf/I9SlRMRP8rMF+6i/QHgqMy8v91g7ZHMfHFE7KBzz+qnWvvDmXlARMwBB2bmE13vMQ1cn5mHtvmPALtn5idHv2dSbx6pa63JBaYXWmdXnuiafhrPTWkFMdS11ryj6/mf2vQ/0rn7H8C7gG+26a3AafCz71rdZ1xFSsPyCEMV7R0Rt3XNX5uZ85c17hkRN9M5oDmltZ0BfDEizqLzzUXvbe1nAlsi4n10jshPo3OnPmnFckxda0YbU5/JzB2TrkUaFYdfJKkQj9QlqRCP1CWpEENdkgox1CWpEENdkgox1CWpEENdkgr5P55wlHbsOt2EAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "w = svm_sgd_plot(X,y)\n", + "#they decrease over time! Our SVM is learning the optimal hyperplane\n", + "\n", + "# Print the hyperplane calculated by svm_sgd()\n", + "x2=[w[0],w[1],-w[1],w[0]]\n", + "x3=[w[0],w[1],w[1],-w[0]]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +}