2023-10-05 11:29:30 -04:00
{
"cells": [
{
"cell_type": "markdown",
"id": "d6186669",
"metadata": {},
"source": [
"# Polynomial fitting"
]
},
{
"cell_type": "code",
2023-10-05 14:52:28 -04:00
"execution_count": 1,
2023-10-05 11:29:30 -04:00
"id": "c9b00a3e",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "markdown",
"id": "f0c00302",
"metadata": {},
"source": [
"Let's consider the example of fitting a polynomial to some measurements of $y_i(x_i)$ (ie. $y$ measured at a set of points $x_i$). An obvious way to solve the normal equations\n",
"\n",
"$$\\mathbf{A^T} \\mathbf{d} = \\mathbf{A^T}\\mathbf{A}\\mathbf{a}$$ \n",
"\n",
"is to calculate the inverse of $\\mathbf{A^T}\\mathbf{A}$ and write\n",
"\n",
"$$\\mathbf{a} = (\\mathbf{A^T}\\mathbf{A})^{-1} \\mathbf{A^T} \\mathbf{d}.$$\n",
"\n",
"For simplicity, let's take a constant error $\\sigma_i$ for each point, so the $\\sigma_i$'s drop out of the equations and we'll just ignore them. "
]
},
{
"cell_type": "code",
2023-10-05 14:52:28 -04:00
"execution_count": 2,
2023-10-05 11:29:30 -04:00
"id": "5473e41a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2023-10-07 09:22:28 -04:00
"Polynomial coeffs = [-0.06810457 -0.91610815 0.89798733 -0.29154606 0.26086863 0.93971567\n",
" 0.24332917 -0.44333463 -0.39585528 0.974724 0.94526284 0.89132267\n",
" 0.58953405 0.61203039 -0.01513055 0.06471341 -0.17141171 -0.9525837\n",
" 0.97697325 0.870457 0.30180934 -0.3201125 -0.95648576 -0.84369759\n",
" -0.30176928 0.95358891 0.25481124 -0.17775 -0.42285418 -0.83911108\n",
" 0.15120805]\n",
"Fitted coeffs = [-4.32457335e+09 -1.67263774e+09 5.02037577e+09 1.62012140e+09\n",
" -2.39298194e+09 -5.89549081e+08 5.35570374e+08 7.41558693e+07\n",
" -2.05309226e+06 2.19871111e+06 -3.29462755e+07 3.11737081e+06\n",
" 1.03325380e+07 -2.25757037e+06 -2.39449501e+06 6.85293308e+05\n",
" 6.23022355e+05 -2.52531237e+05 -1.55019220e+05 8.31031818e+04\n",
" 2.93902694e+04 -1.08301980e+04 -4.34265523e+03 -2.27619395e+03\n",
" 7.54748993e+02 1.06871481e+03 -1.52381191e+02 -1.53604104e+02\n",
" 1.92676516e+01 7.22158292e+00 -8.90406676e-01]\n",
"frac error = [ 6.34990204e+10 1.82580816e+09 5.59069779e+09 -5.55699986e+09\n",
" -9.17313022e+09 -6.27369641e+08 2.20101180e+09 -1.67268390e+08\n",
" 5.18647078e+06 2.25572585e+06 -3.48540902e+07 3.49746511e+06\n",
" 1.75266168e+07 -3.68865829e+06 1.58255669e+08 1.05896643e+07\n",
" -3.63465547e+06 2.65100362e+05 -1.58673941e+05 9.54697496e+04\n",
" 9.73792518e+04 3.38314747e+04 4.53921941e+03 2.69687893e+03\n",
" -2.50207958e+03 1.11972907e+03 -5.99015965e+02 8.63158088e+02\n",
" -4.65657119e+01 -9.60622992e+00 -6.88861959e+00]\n",
"rms deviation / max(y) = 4.58505\n"
2023-10-05 11:29:30 -04:00
]
},
{
"data": {
2023-10-07 09:22:28 -04:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAGsCAYAAADg5swfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFKklEQVR4nO3dd3wUdeLG8c9ueoCEkgABQglSlN4NKsJZwI4ooqLAidjv9M4G3u/O84pY8PQO8cRTwYoCgh0UQUClSIsUCdJ7AqEkgYS0nd8fA4EIhGzI7nd293m/XvvaYZndfYaQ7JOZ73zHZVmWhYiIiIgBbtMBREREJHSpiIiIiIgxKiIiIiJijIqIiIiIGKMiIiIiIsaoiIiIiIgxKiIiIiJijIqIiIiIGKMiIiIiIsaoiIiIiIgxAVNE5s+fzzXXXEODBg1wuVx8/PHHXr/G5MmT6dixI7GxsTRp0oTnn3++6oOKiIhIhQVMETl8+DAdOnRg3LhxlXr+jBkzGDx4MPfccw+rV6/mlVde4cUXX+Tll1+u4qQiIiJSUa5AvOidy+Vi+vTp9O/fv/SxgoIC/vSnPzFp0iQOHjxI27ZtefbZZ+nduzcAt956K0VFRUyZMqX0OWPHjuW5555j27ZtuFwuP2+FiIiIBMwekTN54IEHWLhwIR988AErV65k4MCB9OvXj/Xr1wN2UYmOji7znJiYGHbs2MHWrVtNRBYREQl5QVFEtm3bxoQJE5gyZQoXXXQRzZs355FHHuHCCy9kwoQJAPTt25dp06Yxe/ZsPB4Pv/zyCy+88AIAu3fvNhlfREQkZIWbDlAVVq1aRUlJCS1btizzeEFBAXXq1AFgxIgRbNy4kauvvpqioiLi4uJ48MEH+etf/4rbHRR9TEREJOAERRE5dOgQYWFhLFu2jLCwsDJ/V716dcAeV/Lss8/y9NNPk5GRQWJiIrNnzwYgJSXF75lFREQkSIpIp06dKCkpYc+ePVx00UXlrhsWFkbDhg0BmDRpEqmpqSQmJvojpoiIiPxKwBSRQ4cOsWHDhtI/b968mbS0NGrXrk3Lli0ZPHgwQ4YM4YUXXqBTp07s3buX2bNn0759e6666iqysrKYOnUqvXv35siRI6VjSubNm2dwq0REREJbwJy+O3fuXPr06XPS40OHDmXixIkUFRXxj3/8g7fffpudO3eSkJDA+eefz1NPPUW7du3IysrimmuuYdWqVViWRWpqKv/85z/p0aOHga0RERERCKAiIiIiIsFHp4uIiIiIMSoiIiIiYoyjB6t6PB527dpFjRo1NAW7iIhIgLAsi9zcXBo0aHDGubocXUR27dpFcnKy6RgiIiJSCdu3b6dRo0blruPoIlKjRg3A3pC4uDjDaURERKQicnJySE5OLv0cL4+ji8ixwzFxcXEqIiIiIgGmIsMqNFhVREREjFEREREREWNURERERMQYFRERERExRkVEREREjFEREREREWNURERERMQYFRERERExRkVEREREjFEREREREWNURERERMQYFRERERExJmSLyO7sfBZszGJ3dr7pKCIiIiHL0Vff9ZXpP6xi3YxxJLOH20qGM3pAOwZ1a2w6loiISMgJuT0iu7Pz+dvnPzMyfBKDw2dT08rhiWmrtWdERETEgJArIpuzDnPAqs4vnoYAdHH/QollsSUrz3AyERGR0BNyRaRZQjXcLljqaQVAV/c6wlwumibEGk4mIiISekKuiCTFxzB6QDuWWa0B6Ob+hacHtCUpPsZwMhERkdATckUEYFC3xjx+91AAOoVvYVCHBMOJREREQlNIFhGAusmtoEYDXJ4i2L7IdBwREZGQFLJFBJcLzvkNhMfAga2m04iIiISkkJxHpNQlf4UrX4CIaNNJREREQlJoF5HqiaYTiIiIhLTQPTTzawWHTCcQEREJOSoiWxfCuB4w+XbTSUREREJOaB+aAaheF/amQ9Z6OJwF1XQqr4iIiL9oj0id5pDUAawSWPup6TQiIiIhRUUEoM0AALKXfKCL34mIiPiRigjwmScVj+UiPnMxg599jw+XbDMdSUREJCSEfBHZnZ3PgzOy+NbTEYBb3LN5Ytpq7RkRERHxg5AvIpuzDuOx4N2SSwEYGDaPcKuALVl5hpOJiIgEv5A/a6ZZQjXcLpjn6cC0kgv5oLgPxa4omibEmo4mIiIS9EJ+j0hSfAyjB7TD5Qrjj0X3sYzzeHpAW5LiY0xHExERCXohv0cEYFC3xvRqmciWrDyaJsTaJaTwMERWMx1NREQkqIX8HpFjkuJjSG1eh6RqYTDnH/BSO8jZZTqWiIhIUFMR+TV3OGyaC3n7YMZjYFmmE4mIiAQtFZFfc7vh6hfBFQZrP2PjzLE6lVdERMRHVEROpX470lo9CECjRU/x0LPjNMmZiIiID6iInMLu7HwG/NSFmSXdiHIV83rEGCZPn649IyIiIlVMReQU7EnOXDxYdD8LSs6jhiufiRGj2bFzp+loIiIiQUVF5BSOTXJWQCR3Fj3CNyWdeLb4ViJr1GHBxiztGREREakiLsty7mkhOTk5xMfHk52dTVxcnF/f+8Ml23hi2mpKLItwl8V1nRoxfcVOPBZcEfYjTySvIbnX7dD8NxBVo2re1LKg+AgU5oHLZZ/Bc+wWFmE/JiIi4nDefH6riJRjd3Y+W7LyiI10c/0rC/BYABZfRD5BG/fWo2u5ILEV1G4OcQ2gRn3o9cjxF1k6AfZtOF4wCg9BUZ49YZo7HIZ9fnzdt66BzfNPHcblhj9lQHiU/ecf/wd70yE+GWo2hlpNIbE1RGpqehERMcubz2/NrFqOpPgYkuJjWLAx62gJAXDxcNG9XB/2HUNrriL60Da7EOxNt/86pnbZIrJqKmz9/tRvEB5d9s8R5ZSIsKjjJQTgl69gw6yy67jcdiGq3xaue0WlREREHE9FpAKOjRk5VkbSrcY8V3IbdG/FGzMWcZ5rM8muLG5pHcZ5SXHszs5nc9ZhmiVUI6lNf2jY2S4dkdXK3iKq2Ydjjh1yuX48uMMg/Oh1bjzF9s0qgaIjZUN1vh2SOsDBbZC9HbLWQ14W7Dt6H3HCtXLmPw8eD7Tsaz9Hh3hERMQh/HJoZty4cTz//PNkZGTQoUMHxo4dS/fu3c/4PNOHZk504piRMJeLx/q14tmZ6SfsKcF+/IpWPDvDftztgtED2tGrZeLxYnL0YnplykpVXWAvNxMyV0H+QWh3o/2YZcG/zoPco9PV12hgF5K2A6DJhfYEbiIiIlXIUWNEPvzwQ4YMGcKrr75Kjx49eOmll5gyZQrr1q2jbt265T7XSUUEjo8ZaZoQy+asw9z6v8UnreNylZ0V3nX0sROLCcCoaau8LiunKi9nXMdTwsEf3qDkl6+ptft73MXHz/gpiK1PQZe7iLvk4cq9tg/XMf3+yuisdUy/vzIqoz9fOxg4qoj06NGDbt268fLLLwPg8XhITk7md7/7HSNHjiz3uU4rIifanZ3PBc/MKbNHxA14zvA8N8AJh3mO/vGMZeX6Tg1Lz9qp7DrRrkLGX5hHo4zZJG6bQZwrj9eKr2Jd+8eOPs8iwuXhHwM6+uT9K7qOL19bGZ2VMVi2w/T7K2NwZPTrHnQfc0wRKSwsJDY2lqlTp9K/f//Sx4cOHcrBgwf55JNPyqxfUFBAQUFB6Z9zcnJITk52ZBGBih2uqYxTlZWqWufYYxFWIX3caaRbyWyxkgBIda/h+YjxTCzux/ueS8izok7xqv7L6IvXVkZnZazIOqbfXxmV0V8ZXVR+D7rTOOasmaysLEpKSqhXr16Zx+vVq0d6evpJ648ePZqnnnrKl5Gq1KBujenVMrH0cE1SfAw1YyNKy4kbsI7ejqnIf1APv3pSFa5z7LECIpnpKTtOZ2DYPBq5svi/iHe52/qU14qv5t2SS8nnV2f3+CmjL167qtYx/f4VWcf0+1fVOqbfvyLrmH7/iqxj+v0rso7p96/IOr58bQv70H4X1zrqug7y3fRFWC4X3V012EUddlt1GPnRqpPKyqBujct/M4fz6R6RXbt20bBhQxYsWEBqamrp44899hjz5s1j8eKyYyw
2023-10-05 11:29:30 -04:00
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# first generate some data from a random polynomial\n",
"\n",
2023-10-07 09:22:28 -04:00
"k = 30 # order of polynomial\n",
"n = 100 # number of data points\n",
2023-10-05 11:29:30 -04:00
"\n",
"x = np.linspace(-2,2,n)\n",
"a0 = -1.0 + 2*np.random.rand(k+1)\n",
"print(\"Polynomial coeffs = \", a0)\n",
"poly = np.polynomial.Polynomial(a0)\n",
"y = poly(x) + np.random.normal(scale=0.0, size=n)\n",
"plt.plot(x,y,'.')\n",
"\n",
"# Compute the design matrix\n",
2023-10-05 14:52:28 -04:00
"# A_{ij} = f_j(x_i)\n",
2023-10-05 11:29:30 -04:00
"A = np.zeros((n, k+1))\n",
"for kk in range(k+1):\n",
" A[:, kk] = x**kk\n",
"\n",
"# Now do the linear algebra part \n",
"rhs = np.transpose(A)@y\n",
"lhs = np.transpose(A)@A\n",
"a = np.linalg.inv(lhs)@rhs\n",
"\n",
"print(\"Fitted coeffs = \", a)\n",
"print(\"frac error = \", (a-a0)/a0)\n",
"\n",
"xx = np.linspace(-2,2,1000)\n",
"poly = np.polynomial.Polynomial(a)\n",
"yy = poly(xx)\n",
"plt.plot(xx,yy,'--')\n",
"\n",
2023-10-07 09:22:28 -04:00
"dev = np.sqrt(np.mean((y-poly(x))**2))\n",
"print('rms deviation / max(y) = %g' % (dev/np.max(np.abs(y)),))\n",
"\n",
2023-10-05 11:29:30 -04:00
"plt.show()\n"
]
},
{
"cell_type": "markdown",
"id": "13fd394a",
"metadata": {},
"source": [
"If you run this with no noise (`scale=0` on line 10), you should see that for low polynomial order the coefficients are fit to machine precision, but the errors become large pretty quickly as you increase the polynomial order. Something is going horribly wrong with the matrix inversion as the polynomial order increases!"
]
},
{
"cell_type": "markdown",
"id": "1aaa27d6",
"metadata": {},
"source": [
"## Using SVD"
]
},
{
"cell_type": "markdown",
"id": "cac127b3",
"metadata": {},
"source": [
"Singular value decomposition (SVD) is extremely useful in these situations where you are dealing with a close-to-singular matrix. The ratio of the largest to smallest singular values is known as the **condition number** of the matrix, and measures how close to singular it is."
]
},
{
"cell_type": "code",
2023-10-05 14:52:28 -04:00
"execution_count": 3,
2023-10-05 11:29:30 -04:00
"id": "e8d9067e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2023-10-07 09:22:28 -04:00
"Range of singular values = 1.16147e+14\n",
"np.linalg.cond gives 1.16147e+14\n"
2023-10-05 11:29:30 -04:00
]
}
],
"source": [
"U, Sdiag, VT = np.linalg.svd(A,0)\n",
"\n",
"print(\"Range of singular values = %g\" % (max(abs(Sdiag))/min(abs(Sdiag))))\n",
"\n",
"print(\"np.linalg.cond gives %g\" % (np.linalg.cond(A)))"
]
},
{
"cell_type": "markdown",
"id": "762f72b0",
"metadata": {},
"source": [
"Let's use SVD to rewrite the normal equations:\n",
"\n",
"$$\\mathbf{A^T} \\mathbf{d} = \\mathbf{A^T}\\mathbf{A}\\mathbf{a}$$ \n",
"\n",
"$$ \\mathbf{VSU^T} \\mathbf{d} = \\mathbf{VSU^T}\\mathbf{USV^T} \\mathbf{a}$$ \n",
"\n",
"$$ \\mathbf{VSU^T} \\mathbf{d} = \\mathbf{VS^2V^T} \\mathbf{a}$$ \n",
"\n",
"$$ \\mathbf{V^TVS}\\mathbf{U^T} \\mathbf{d} = \\mathbf{V^T}\\mathbf{VS^2V^T} \\mathbf{a}$$ \n",
"\n",
"$$ \\mathbf{SU^T} \\mathbf{d} = \\mathbf{S^2V^T} \\mathbf{a}$$ \n",
"\n",
"$$ \\mathbf{U^T} \\mathbf{d} = \\mathbf{SV^T} \\mathbf{a}$$ \n",
"\n",
"$$\\Rightarrow \\mathbf{VS^{-1}}\\mathbf{U^T} \\mathbf{d} = \\mathbf{a}$$ \n",
"\n",
"The matrix $\\mathbf{VS^{-1}}\\mathbf{U^T}$ is called the **pseudo-inverse**. We can use it to map from the data vector $\\mathbf{d}$ to the model parameters $\\mathbf{a}$.\n"
]
},
{
"cell_type": "code",
2023-10-05 14:52:28 -04:00
"execution_count": 4,
2023-10-05 11:29:30 -04:00
"id": "1595e75e",
2023-10-05 14:52:28 -04:00
"metadata": {
2023-10-07 09:22:28 -04:00
"scrolled": false
2023-10-05 14:52:28 -04:00
},
2023-10-05 11:29:30 -04:00
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
2023-10-07 09:22:28 -04:00
"Fitted coeffs = [-0.06810442 -0.91610793 0.89798456 -0.29154781 0.2609005 0.93970728\n",
" 0.24313354 -0.44324493 -0.3951416 0.97437286 0.94372559 0.89199829\n",
" 0.59155273 0.61116028 -0.01757812 0.06556702 -0.17004395 -0.95316315\n",
" 0.97619629 0.87065506 0.30215454 -0.32018661 -0.95658875 -0.84368467\n",
" -0.30175591 0.95358771 0.25480914 -0.17774992 -0.42285404 -0.83911109\n",
" 0.15120804]\n",
"frac error = [-2.11980533e-06 -2.44679418e-07 -3.07735337e-06 5.99586571e-06\n",
" 1.22147670e-04 -8.92834498e-06 -8.03952611e-04 -2.02320286e-04\n",
" -1.80288818e-03 -3.60245783e-04 -1.62627498e-03 7.58003564e-04\n",
" 3.42420505e-03 -1.42168454e-03 1.61763938e-01 1.31906354e-02\n",
" -7.97943866e-03 6.08285240e-04 -7.95278498e-04 2.27540718e-04\n",
" 1.14377625e-03 2.31537019e-04 1.07666287e-04 -1.53112627e-05\n",
" -4.43319898e-05 -1.25424394e-06 -8.24456806e-06 -4.50115938e-07\n",
" -3.31363628e-07 4.82297004e-09 -4.12189271e-08]\n",
"rms deviation / max(y) = 3.81606e-08\n"
2023-10-05 11:29:30 -04:00
]
},
{
"data": {
2023-10-07 09:22:28 -04:00
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAGsCAYAAADg5swfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA420lEQVR4nO3deXxU1f3/8fedyb4NSwIkrAEEVJBNwKBFqVRQqmD4KrV8K/qluPzAShcVaJViW4Pot9pavtXWiq21YrUB6la1IlgVEVmUoKAgyJKwBGESQjJJZs7vjyFjAtll5mZmXs/HYxzm5ty5n5OZybw9995zLWOMEQAAgA0cdhcAAACiF0EEAADYhiACAABsQxABAAC2IYgAAADbEEQAAIBtCCIAAMA2BBEAAGAbgggAALANQQQAANgmbILIW2+9pSuvvFJZWVmyLEsrVqxo8XO8+uqruuCCC5SamqqMjAxNmTJFu3fvPuO1AgCA5gmbIFJWVqbBgwdryZIlrVp/165dmjRpkr75zW9q8+bNevXVV1VcXKzc3NwzXCkAAGguKxwvemdZlpYvX67JkycHlnk8Hv30pz/VM888o2PHjmngwIG6//77dckll0iSnn/+eV133XXyeDxyOPz564UXXtCkSZPk8XgUGxtrQ08AAIhuYTMi0pTZs2dr7dq1WrZsmT766CNdc801mjBhgj777DNJ0vDhw+VwOLR06VJ5vV653W499dRTGjduHCEEAACbRMSIyJ49e9S7d2/t2bNHWVlZgXbjxo3TyJEjdd9990mS1qxZo2uvvVZHjhyR1+tVTk6OXn75ZbVr186GXgAAgIgYEdmyZYu8Xq/69eunlJSUwG3NmjXauXOnJOnAgQOaOXOmpk+frvXr12vNmjWKi4vTf/3XfykMsxgAABEhxu4CzoTjx4/L6XRqw4YNcjqddX6WkpIiSVqyZIlcLpcWL14c+Nlf//pXde/eXevWrdMFF1wQ0poBAECEBJGhQ4fK6/Xq0KFD+sY3vlFvmxMnTgQOUq1RE1p8Pl/QawQAAKcLm10zx48f1+bNm7V582ZJ/tNxN2/erD179qhfv36aNm2arr/+euXn52vXrl16//33lZeXp5deekmSNHHiRK1fv1733nuvPvvsM23cuFE33nijevbsqaFDh9rYMwAAolfYHKy6evVqjR079rTl06dP15NPPqmqqir98pe/1F/+8hft379f6enpuuCCC7Rw4UINGjRIkrRs2TItXrxYn376qZKSkpSTk6P7779fAwYMCHV3AACAwiiIAACAyBM2u2YAAEDkIYgAAADbtOmzZnw+nwoLC5WamirLsuwuBwAANIMxRqWlpcrKyjrtjNVTtekgUlhYqO7du9tdBgAAaIW9e/eqW7dujbZp00EkNTVVkr8jaWlpNlcDAACao6SkRN27dw98jzemTQeRmt0xaWlpBBEAAMJMcw6r4GBVAABgG4IIAACwDUEEAADYhiACAABsQxABAAC2IYgAAADbEEQAAIBtCCIAAMA2BBEAAGAbgggAALBN1AaRIne53t1ZrCJ3ud2lAAAQtdr0tWaC5dn1ezQvf4t8RnJYUl7uIE0d0cPusgAAiDpRNyJS5C4PhBBJ8hlpfn4BIyMAANgg6oLIruIyna3d+knMs7rW+aYkyWuMdhefsLkyAACiT9QFkez0ZPVz7NPsmJW60rFWkuS0LPVKT7K5MgAAok/UBZFMV6KmXNBPkpRkeeS0LN2XO1CZrkSbKwMAIPpE5cGqF53dQ9ogDejo1Ns3jCWEAABgk6gMIopLkSQly6NkQggAALaJul0zkqS4k8eDVJbZWwcAAFEuOoNI7MkgUsWZMgAA2Ck6g0hcsv++skwyxt5aAACIYtF5jEhSunTTaik22e5KAACIatEZRJwxUtZQu6sAACDqReeuGQAA0CZE54iIJK17TCo7LI2YKaV2trsaAACiUvQGkXd/J7n3SP0uJ4gAAGCT6N01UzOXSBVziQAAYJfoDSI1c4lUMpcIAAB2id4gUjOXCCMiAADYJnqDSCzTvAMAYLfoDSJx7JoBAMBuURxE2DUDAIDdovf03Yt+JJ3/P5Kru92VAAAQtaI3iHTsY3cFAABEvejdNQMAAGwXvSMiB7ZIO1dJ7XtJ50yyuxoAAKJS9I6I7PtAev0e6cNldlcCAEDUit4gUnPWDPOIAABgG4JIFfOIAABgl+gNIsysCgCA7aI3iLBrBgAA20VvEKkZEWHXDAAAtoneIMK1ZgAAsF30ziOSmiVd/08pLsXuSgAAiFrRG0RiE6TeF9tdBQAAUS16d80AAADbRXcQ2fw36Z3fSmVH7K4EAICoFL27ZiTpjV9IpYVS9jek5I52VwMAQNSJ7hGR+JMHqnqO21sHAABRKsqDSKr/vpIgAgCAHUIWRBYtWiTLsjRnzpxQbbJpNafuekrtrQMAgCgVkiCyfv16PfbYYzrvvPNCsbnmqxkRIYgAAGCLoAeR48ePa9q0afrjH/+o9u3bB3tzLcOuGQAAbBX0IDJr1ixNnDhR48aNa7Ktx+NRSUlJnVtQxXGwKgAAdgrq6bvLli3Txo0btX79+ma1z8vL08KFC4NZUl0jZkgDrpA69AndNgEAQEDQRkT27t2r22+/XU8//bQSEhKatc68efPkdrsDt7179warPL9OZ0t9vim17xnc7QAAgHpZxhgTjCdesWKFrr76ajmdzsAyr9cry7LkcDjk8Xjq/Kw+JSUlcrlccrvdSktLC0aZAADgDGvJ93fQds1ceuml2rJlS51lN954owYMGKC77rqryRASEkd3S5+vlhLbS+dMsrsaAACiTtCCSGpqqgYOHFhnWXJysjp27Hjactsc2CK9cLvUbSRBBAAAG0T3zKo1Z81w+i4AALYI6UXvVq9eHcrNNS0woRlBBAAAO0T3iEhgQjNmVgUAwA7RHURqX2smOCcPAQCARkR3EIk/GUR81VK1x95aAACIQtEdRGpGRCQOWAUAwAYhPVi1zXE4palPS7GJdUMJAAAIiegOIpJ09rftrgAAgKgV3btmAACArRgR2blKOrZXyh4jdci2uxoAAKIKIyL/+bX0wg+k/RvsrgQAgKhDEAnMrsqkZgAAhBpBJMHlv/eU2FsHAABRiCASn+a/r3DbWwcAAFGIIFIzIkIQAQAg5AgiBBEAAGxDEElg1wwAAHZhHpHsMdI1f5badbe7EgAAog5BpH0v/w0AAIQcu2YAAIBtCCJV5dLWFdKmv9pdCQAAUYddM1Xl0nPT/f8+7zuSk18JAAChwohIzYRmErOrAgAQYgQRZ4wUl+L/d8UxW0sBACDaEEQkJjUDAMAmBBGJ680AAGATgohUa0SEY0QAAAglgojErhkAAGzCuaqSNHq2NOS7UtdhdlcCAEBUIYhI/uvNAACAkGPXDAAAsA0jIpJ09AupcJOU1FHK/obd1QAAEDUYEZGkz9/0T/O+dondlQAAEFUIIlKts2aO2VoGAADRhiAiSYnt/fcnvrS3DgAAogxBRJISO/jvywkiAACEEkFEkpJqgshRyRh7awEAIIoQRKSvRkR81Tpw+LC9tQAAEEUIIpKe/bBYFSZWknTtQy/p2fV7bK4IAIDoEPXziBS5yzUvf4ves76vCsXpiEnV/PwCjemXoUxXot3lAQAQ0aI+iOwqLpPPSMtNrYnMjNHu4hMEEQAAgizqd81kpyfLYdVd5rQs9UpPsqcgAACiSNQHkUxXovJyB+lsx15d4XhPZzv26b7cgYyGAAAQAlG/a0aSpo7ooW/vKVDylqdUesFPlDqih90lAQAQFaJ+RKRGsitDkpTqK7W5EgAAogdBpEYSs6sCABBqBJEaibVmVwUAACFBEKlRMyLChe8AAAgZgkiNmivwsmsGAICQIYjUqNk1c4JdMwAAhAqn79ZIy5K+/bCU1NHuSgAAiBoEkRrxKdL5N9pdBQAAUYVdMwAAwDYEkdr2b5AK8iX3PrsrAQAgKhBEanvtHun5G6U979ldCQAAUYEgUluKf5p3lR22tw4AAKIEQaS25E7+++OH7K0DAIAoQRCpLZkREQAAQokgUlt
2023-10-05 11:29:30 -04:00
"text/plain": [
"<Figure size 640x480 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"aSVD = VT.T @ np.diag(1/Sdiag) @ U.T @ y\n",
"\n",
"print(\"Fitted coeffs = \", aSVD)\n",
"print(\"frac error = \", (aSVD-a0)/a0)\n",
"\n",
"plt.plot(x,y,'.')\n",
"\n",
"xx = np.linspace(-2,2,1000)\n",
"poly = np.polynomial.Polynomial(aSVD)\n",
"yy = poly(xx)\n",
"plt.plot(xx,yy,'--')\n",
"\n",
2023-10-07 09:22:28 -04:00
"dev = np.sqrt(np.mean((y-poly(x))**2))\n",
"print('rms deviation / max(y) = %g' % (dev/np.max(np.abs(y)),))\n",
"\n",
2023-10-05 11:29:30 -04:00
"plt.show()"
]
2023-10-05 14:52:28 -04:00
},
{
"cell_type": "markdown",
"id": "4fab5983",
"metadata": {},
"source": [
"## Orthogonal polynomials"
]
},
{
"cell_type": "markdown",
"id": "7a3ee4b8",
"metadata": {},
"source": [
"SVD does a lot better than straight inversion of the normal equations, but still fails for large enough polynomial order. One way that we can do better is to choose a set of basis functions that are orthogonal in the domain that we are interested in. For example, the Legendre polynomials are orthogonal on the domain $x=(-1,1)$ (which you can rescale to by rescaling your $x$ variable). "
]
},
{
"cell_type": "markdown",
"id": "b10f5bf9",
"metadata": {},
"source": [
2023-10-07 09:35:15 -04:00
"> **Exercise**: Try changing the basis functions to Legendre polynomials and see how that improves the fit, both for the inversion and SVD methods. How do the results change when you change the number of data points and the order of the polynomial? Check also how the condition number of the design matrix changes as a function of the polynomial degree when you change the polynomials you are using to fit. Hint: You can use [`numpy.polynomial.legendre.legvander`](https://numpy.org/devdocs/reference/generated/numpy.polynomial.legendre.legvander.html) to generate the design matrix $\\mathbf{A}$ for you."
2023-10-05 14:52:28 -04:00
]
2023-10-05 11:29:30 -04:00
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
2023-10-07 09:22:28 -04:00
"version": "3.11.5"
2023-10-05 11:29:30 -04:00
}
},
"nbformat": 4,
"nbformat_minor": 5
}
