### Archive

Archive for the ‘python’ Category

## Taylor method with automatic differentiation – 2

In the previous post, we have talked about Taylor method to solve ordinary differential equation (ODE) numerically. All the derivatives in the Taylor method are computed using automatic differentiation. However, the drawback of this method is when we change our ODE, we have to write another code. In this post, we are going to present a way such that when we want to compute another ODE, we don’t need to change the majority of our code.

Continuing our analysis in the last post, we are going to need operations on polynomials. We are going to define operations such as:

1. Addition between a constant and a polynomial
2. Multiplication between  a constant and a polynomial
4. Multiplication between polynomials
5. Dividing a constant by a polynomial
6. Dividing a polynomial by a polynomial
7. Logarithm of a polynomial
8. Exponentiation of a polynomial
9. Power of a polynomial

Operation 1 and 2 are trivial. Suppose $x(h)$ is a polynomial, then $x(h)=\sum x_i h^i$ for some real constant $x_i$. Suppose $\alpha$ is a real constant, then the addition between $\alpha$ and $x(h)$ is a polynomial $y(h)$ where $y_0 = x_0 + \alpha$ and $y_i = x_i$ for all $i \geq 1$. The multiplication between $\alpha$ and $x(h)$ is a polynomial $z(h)$ where $z_i = \alpha x_i$ for all $i$.

Operation 3 and 4 are not difficult. Suppose $x(h), y(h)$ are polynomials. Then the addition between $x(h)$ and $y(h)$ is a polynomial $z(h)$ where $z_i = x_i + y_i$ for all $i$. The multiplication between $x(h)$ and $y(h)$ is a polynomial $w(h)$ where $w_i = \sum_{j=0}^{i} x_j y_{i-j}$.

Operation 5 and 6 are a little bit tricky. Suppose $x(h)$ is a polynomial and $\alpha$ is a constant. We want to compute $y(h)$ where $y(h) = \alpha / x(h)$. Notice that $x(h)y(h)=\alpha$. Therefore, we are going to take advantage of this equality to find the coefficient of $y(h)$. Thus we shall have $y_0 = \alpha / x_0$, and $y_i = - (\sum_{j=1}^{i} x_j y_{i-j})/x_0$. For operation 6, let’s suppose $z(h) = y(h) / x(h)$. As before, notice that $x(h)z(h) = y(h)$, thus we can compute the coefficient of $z(h)$ using this equality. Thus, $z_0 = y_0 / x_0$ and $z_i = (y_i - \sum_{j=1}^{i} x_j z_{i-j} )/x_0$.

Now, let’s consider the next operation, which is a logarithm of a polynomial. Suppose $x(h)$ is a polynomial and we are going to find $y(h)=\log(x(h))$. Let’s differentiate both side with respect to $h$, then we have $y'(h) = x'(h) / x(h)$. Thus by exploiting a division operation above, we have $y_0 = \log x_0$ and $y_i = (ix_i - (\sum_{j=1}^{i-1} x_j ((i-j)y_{i-j})))/(ix_0)$.

Now, let’s consider the exponential operator, which is an exponentiation of a polynomial. Suppose $x(h)$ is a polynomial and we are going to find $y(h)=\exp(x(h))$. Let’s take logarithm on both side and then differentiate it, we then have $y'(h)/y(h) = x'(h)$, or $y'(h)=y(h)x'(h)$. Therefore, by exploiting a multiplication operation, we shall have $y_0 = \exp x_0$ and $y_i = (\sum_{j=0}^{i-1} y_j ((i-j)x_{i-j}))/i$.

For the last operation of this post, we shall consider the power of a polynomial. Suppose $x(h)$ is a polynomial, we are going to find $y(h)=(x(h))^\alpha$ for any real number $\alpha$. First, we need to take logarithm on both sides and then differentiate the result with respect to $h$, $y'(h)/y(h) = \alpha x'(h)/x(h)$ or $y'(h) = \alpha x'(h)y(h)/x(h)$. By exploiting multiplication and division, we will get the coefficient of $y(h)$.

To facilitate these operation, we build a Python module that will handle these operations. We will also write another code to integrate an ODE. This integration code will work in general without any dependencies on the system or on the order of the system.

def integrate(F,y_init,t_init = 0.0 ,tStop = 1.0 ,h = 0.05 ):
dim = len(y_init)
y_sol = []
for i in range(0,dim):
y_sol.append([])
y_sol[i].append(y_init[i])
t_sol = []
t_sol.append(t_init)
t = t_init

while t &lt; tStop:
y = []
for i in range(0,dim):
y.append([])
y[i].append(y_sol[i][-1])

for k in range(0,20):
dy = F(y)
for i in range(0,dim):
y[i].append(dy[i][k]/float(k+1))
n = len(y[0])
for i in range(0,dim):
temp = 0.0
for j in range(0,n):
temp = temp + y[i][j]*h**j
y_sol[i].append(temp)
t_sol.append(t+h)
t = t + h

return np.array(t_sol) ,np.transpose(np.array(y_sol))


All the operations above and the last code will be written in a python file so that if we need to use this Taylor method, we can just import it. Let’s take an example to integrate one ODE. We will create another code that specifies the ODE and also imports all the operations above and then numerically integrate it.

from mytaylor import *
import matplotlib.pyplot as plt
import numpy as np
import math as m
def fun(y):
z = subtractpoli(y[0],multiplypoli(y[0],y[0]))
return [z]
y_init = [0.5]
t_init = 0
h = 0.1
tStop = 5.0
#integrating ODE
T, Y = integrate(fun,y_init,t_init,tStop,h)
#real solution computation
real_sol = []
for t in T:
temp = m.exp(t)/(m.exp(t)+1.0)
real_sol.append(temp)
#plotting the error versus time
real_sol = np.array(real_sol)
error = Y[:,0]-real_sol
plt.plot(T[:],error[:])
plt.show()


In the above code, we try to numerically integrate $y' = y(1-y)$ with initial condition $y(0)=0.5$. This is the same system we try to integrate in the previous post. I hope you can see the difference between the last code and the code we have shown in the previous post. We really don’t need to consider about Taylor method at all, we just need to write our system in the above code if we want to compute the numerical solution of our code. Notice that we have written all the operations and the integrating code in a file named mytaylor.py which is imported in the first line of the above code. In the fifth line, we can write any ODE we want to integrate, however, the way we write our ODE will be slightly different. Due to polynomial operations that we have defined in this post, $x+y$ cannot be written as just $x+y$ but addpoly($x,y$). Once we define our ODE, we can just integrate it like in line 15.

Another thing we would like to do is to extend the operation of polynomial. We have not included trigonometry operations on polynomials and et cetera. We would like also to test this Taylor method to a stiff system. Please find the following two files (mytaylor.py and test_case.py) for interested readers if you would like to test yourselves.

Categories: python, Teachings

## Taylor method with automatic differentiation

June 14, 2016 1 comment

I teach numerical analysis course and in the one of the chapters, I teach my students about the numerical solution of ordinary differential equations (ODEs). One of numerical solution methods I teach is the so-called Taylor method. This method works by applying the Taylor series to the solution of the differential equation. Suppose we have one-dimensional differential equation as follow:

$\frac{dy}{dt} = f(y)$ and $y(t) = y_0$.

Let’s assume that we only consider an autonomous ODE first.  To find a solution at the next step, $y(t+h)$, we need to apply Taylor expansion, as follow:

$y(t+h) = y(t) + y'(t) h + y''(t)\frac{h^2}{2} +y'''(t)\frac{h^3}{6}+\dots$.

Therefore, depending on how accurate we want our solution is, all we need is to compute the derivatives of $y(t)$. Unfortunately, computing all the derivatives is not an easy task for a computer. We have to derive all the derivatives by ourselves and then input them to the computer.

In this note, I will introduce an automatic differentiation as a tool to compute the derivatives of $y(t+h)$ so that we can compute $y(t+h)$ as accurately as possible. Let us consider the series expansion of $y(t+h)$ as above, but I will use constant $a_i$ to replace the $i-$th derivative of $y(t)$,

$y(t+h) = a_0 + a_1 h + a_2 h^2 + \dots$.

Let us derive the above series with respect to $h$.

$y'(t+h) = a_1 + 2 a_2 h + 3a_3 h^2 + \dots$.

Notice that the derivative at the left hand side of the above equation is actually equal to $f(y(t+h))$ since we know that $y(t+h)$ satisfies the ordinary differential equation. Then we have,

$f(y(t+h)) = a_1 + 2 a_2 h + 3a_3 h^2 + \dots$.

Since $y(t+h)$ is a series of $h$ then $f(y(t+h))$ is also a series of $h$. Then both sides of the above equation are series of $h$, thus we can equate each of the coefficient of $h^j$ to get $a_1$$a_2$ and so on. The most difficult task in this problem is to find the series of $f(y(t+h))$ since the function $f(y)$ can be anything.

Let’s take an example. Suppose we want to integrate the following differential equation,

$\frac{dy}{dt} = y(1-y)$ and $y(0)=0.5$.

Let us choose $h=0.1$ and compute $y(0.1)$. As before, we assume that $y(0.1)$ is a series of $h$ as follows,

$y(0.1) = a_0 + a_1 h + a_2 h^2 + \dots$.

Differentiating both sides of the above equation, we get

$y'(0.1) = a_1 + 2 a_2 h + 3 a_3 h^2 + \dots$.

The right hand side of the above is equal to

$y'(0.1) = y(0.1)(1-y(0.1)) = (a_0 + a_1 h + \dots)(1 - (a_0 + a_1 h + \dots)) = b_0 + b_1 h + b_2 h^2 + \dots$.

Therefore, we get,

$a_1 = b_0$$a_2 = b_1/2$,$a_3 = b_2/3$,  and so on.

The real question is how to find $b_0$, $b_1$ and so on. If the function $f(y)$ only involves operations such as addition, subtraction and multiplication, it would be easy to find the $b$‘s. But if the function $f(y)$ has operations such as division, exponential, logarithm and trigonometric functions then we have to think a bit harder on how to find $b$‘s. We go back to the previous example. Since the operations in this example are only multiplication and subtraction, we can easily compute $b_0$, $b_1$ and so on. Using algebraic modification, we get the following,

$b_0 = a_0 \times (1-a_0)$, thus $a_1 =a_0 \times (1-a_0)$
$b_1 = (a_0 \times -a_1 ) + (a_1 \times (1-a_0))$, thus $a_2 =((a_0 \times -a_1 ) + (a_1 \times (1-a_0)))/2$
$b_2 = (a_0 \times -a_2 ) + (a_1 \times (-a_1))+(a_2 \times (1-a_0))$, thus $a_3 =((a_0 \times -a_2 ) + (a_1 \times (-a_1))+(a_2 \times (1-a_0)))/3$

and so on.

The more $a$‘s we compute, the more accurate our solution is. Using Taylor method this way, we can easily apply this method in a program. It should be clear why this method is called automatic differentiation because we don’t need to manually derive all the derivatives.

Next, I tried to numerically integrate the previous example. Below is the graphic of the error of Taylor method with automatic differentiation versus time. As we can see from the figure, that the error of our numerical integrator is so small and it is almost exact. The number of $a$ I used in this graph is 10.

I use python to numerically integrate this example. You can find the code to produce the above figure in the end of this note. One of the drawbacks of this method is when you change your ODE, you have to change your code as well. In other numerical integrators such as Runge Kutta, we don’t need to change the whole code, we just need to change the ODE and we should get the solution. Together with my student, I am trying to tackle this problem and I hope it will be posted in here.

import math as m
import numpy as np
import pylab as plt

#preliminary constants
order = 10
h = 0.1
t = 0.0
t_end = 5
y0 = 0.5
#defining solution
sol = []
sol.append(y0)
time = []
time.append(t)
#start of the program
while t<t_end:
a = []
a.append(sol[-1])
for k in range(1,order):
if k == 1:
a.append(a[k-1]*(1-a[k-1]))
else:
sum = 0.0
for j in range(0,k-1):
sum = sum + a[j]*(-a[k-1-j])
a.append((sum+a[k-1]*(1.0-a[0]))/k)
sum = 0.0
for i in range(0,order):
sum = sum + a[i]*h**i
sol.append(sum)
t = t + h
time.append(t)
#end of the program

#real solution computation
real_sol = []
for t in time:
temp = m.exp(t)/(m.exp(t)+1.0)
real_sol.append(temp)

#plotting the error versus time
sol = np.array(sol)
real_sol = np.array(real_sol)
time = np.array(time)
error = sol-real_sol
plt.plot(time[:],error[:])
plt.show()

Categories: Justincase, python

## How to fold a paper into thirds, a real analysis perspective

Folding a paper into third is useful when you want to mail someone using an envelope. I search around the internet and there are so many ways to do this. So far, I didn’t see the method I am about to explain. I found this method by accident while I am teaching Real Analysis this semester. I am going to prove that this method actually works. Other methods that I found always use Geometry in the proof but I am going to use Real Analysis instead.

The procedure

Please have a look at the following figure.

The above figure is the first step of our method. Take a sheet of paper and let’s call the left side, side A and the right side, side B. First, you need to fold the paper side A as in the second image of the figure above. It doesn’t have to divide the paper equally, you can just fold it anywhere. Once you do that, you undo the fold and you will have a fold mark, which we call it $x_0$. Next, you need to fold again, but now you need to fold the right side (side B) along the fold mark $x_0$ as in the second row of the above figure. Now you will have two fold marks, the latter is called $y_0$. Now take a look at the following figure.

Now that we have two fold marks ($x_0$ and $y_0$), we need to fold side A along the fold mark $y_0$, as in the first row of the above figure. Undo the fold, you will have the third fold mark (we call it $x_1$). Next, we fold the side B until it touch the fold mark $x_1$ and we will have four fold marks, $x_0, x_1$ and $y_0, y_1$. Continue the step over and over again and at one stage, at the n-step, $x_n$ will be close to one third of the paper and $y_n$ will be very close to two third of the paper.

Simulation

Let’s take an example. Suppose in the beginning, let’s fold side A such that we divide the paper equally. Assume the length of a paper is 1. This means $x_0=0.5$. If we simulate this case, we will get the following table.

 n 1 2 3 4 5 $x_n$ 0.5 0.375 0.3475 0.3359375 0.333984375 $y_n$ 0.75 0.6875 0.671875 0.66796875 0.666992188 6 7 8 0.333496094 0.333374023 0.333343506 0.666748047 0.666687012 0.666671753

As we see, that it took approximately 4-8 steps until $x_n$ is closed to one third of a paper or $y_n$ is closed enough to two third of the paper. However, what happens if we choose different initial condition. We simulate this using a number of initial condition. Suppose we use an A4 paper which has length of 29.7 cm. We choose a number of different initial conditions. We can see the results in the following figures. The x-axis from the above figure is the initial condition and the y-axis is the number of iteration needed until it is closed enough to one third of paper. Since we use an A4 paper, the x-axis ranges from 0 to 29.7 cm.  The tolerance we use in the above figure is 1 mm. In conclusion, we only need to do only maximum four steps until the last fold mark is only 0.1 cm away from one third of the A4 paper.

We relax the tolerance in the above figure. We use 0.5 cm as the tolerance and the number of iterations is slightly improved. It can be seen that if our first trial is close enough to one third of A4 paper, we only need to do one step.

Mathematical proof

Finally, it comes to prove that this methods should work. mathematically not numerically. First of all, we need to mathematically model the above procedures. Let’s assume that the length of the paper we use is 1. Let’s use the same notation as used in the above procedures. Given $x_0$ we will get $y_0 = (1+x_0)/2$. Recursively, we could compute $x_n$ and $y_n$ as follow:

$x_n = y_{n-1}/2$ and $y_n=(1+x_n)/2$, as n=1,2, … .

Even though there are two sequences $(x_n)$ and $(y_n)$, we only need to consider $(x_n)$. We could simplify the sequence $(x_n)$ as follow.

$x_0$ is given and $x_n=y_{n-1}/2=((1+x_{n-1})/2)/2=(1+x_{n-1})/4$ for $n=1,2,\dots$.

Lemma 1
When $0 (as $1/3), the sequence $(x_n)$ is bounded above (bounded below).

Lemma 2
When $0 (as $1/3), the sequence $(x_n)$ is monotonically increasing (decreasing).

Theorem 3
Given $0, the sequence $(x_n)$ converges to 1/3.

Corollary 4
Given $0, the sequence $(y_n)$ converges to 2/3.

The proofs of the above results are given to my students as exercises.

This is the end of this note and the following is the code to produce the above figures.

import math
import numpy as np
import pylab as plt

def f(x):
return (29.7+x)/2.0

def g(y):
return y/2.0

def number_of_iter(x,tol,max_iter):
iter = 0
er = abs(x-29.7/3)
if er <= tol: return 1
while er > tol and iter < max_iter:
iter = iter + 1
y = f(x)
x = g(y)
er = min(abs(29.7/3-x),abs(2*29.7/3-y))

return iter

#main code
x = np.arange(0,29.7,0.1)
y = []
tol = 0.5
max_iter = 100

for t in x:
result = number_of_iter(t,tol,max_iter)
y.append(result)

y = np.array(y)
ax = plt.gca()
ax.set_ylim([0,5])
plt.plot(x,y)
plt.show()

Categories: python, Teachings

## My first Coursera course

In the middle of this year, I received a Fulbright Visiting Scholar that allowed me to go to Athens, OH, US. I spent four months in Ohio University starting in September, which means that I took a leave in the university I am working. One of my activities while I was in Ohio University was I took a course from Coursera. At that time I was eager to learn and to know python programming language therefore I decided to take the course, An Introduction to Interactive Programming in Python, offered by Rice University taught by the following professors: Joe Warren, Scott Rixner, John Griener, and Stephen Wong.

My first coursera was nothing but really, really good. The class could not definitely be better. The intructors have put so much effort in preparing the class and at the same time they were really enjoyable and fun. I really learned a lot from this class and I recommend everyone to take this class. After nine weeks, I finished the course and proudly said that I got 96.9% with distinction mark.

But what I wanted to tell here is the content of the course. From knowing nothing of Python I now know that the word ‘python’ is originally from the Month Python’s Flying Circus not the reptilia. I also know how useful this new programming language is. Python can be used to build an interactive game, to conduct some scientific computations, to analyse data science and many more. There are already many communities that use Python and they are (still) growing. One more good thing is the fact that it is free. Python works on either Windows, Linux or Mac and the installation on each device is not that difficult. I am using both Windows and Ubuntu. For Windows, I use mainly Python(x,y) and for Ubuntu, I don’t have to do anything as it is already inside. However, in this course, I did not have to use all of these as one of the instructors has built a nice Python web application that is called Codeskulptor. It will allows us to make a python scripts on your web browser so actually you don’t have to install python in your computer.

From the course, I learned to make interactive games. Even though the games I made were not perfect but I think it is playable and fun enough. I learned to make games like rock-paper-scissors-spock-lizzard, number guessing game, stopwatch game, pong, memory, and asteroids. The following links are the python scripts for all of the above games I made during the course. The link will bring you to a codeskulptor website, in which I had already written a python script. You just need to press the play button on the top left of your screen.

1. RPSLS  This game is called rock-paper-scissors-lizzard-spock game. It is an extention of rock-paper-scissors and first appeared in The Big Bang Theory series.

2. Number guessing game This game, as you might guess, is a number guessing game. The computer hide a number between a small number and a big number and you have finite tries to guess what number the computer is hiding.

3. Stopwatch This game will train your reflexes. It will show a stopwatch and you have to press the pause button every whole second.

4. Pong Oh, please tell me you know this game.

5. Memory There are 16 cards facing down, and all of them come in pair. Figure out the pair with the smallest number of tries.

6. Blackjack The blackjack game, yes, it is the blackjack game that you always know.

7. Asteroids Asteroids is an arcade space shooter game and I think I played this game when I was a kid. I played this game using an Atari device.

Other than those games, I made two another python scripts just for fun. The first is a fan and the second is the illustration of Hilbert’s curve. Hope you like it.

Categories: Justincase, python, Teachings