Here we are: another semester comes to an end. If, three weeks ago, you were wondering if you could survive, now you don't have to worry any more. Live or die, you will be relived soon.
To be honest, SBR600 is a tough course. If you take it with OPS535 (I know quite a few of you do.), it's really a challenge. I, for one, failed in time management for the first time since I started Seneca. I submitted my OPS535 assignments late for both occasions. Right now, I still have two assignments not started yet (due next week). One for SEC520, another for NDD430. But I'm very glad I can wrap up release 0.3 and get a big weight out of my back. I try to make a summary of the whole project. I put it in a wiki page: Summary Document
In my release 0.3, I tested three more functions on top of my release 0.2. A big improvement is that I got the unittest functions into my test code. In my 0.2, I had a difficult time to use assertion from unittest framework. The simple reason is this: in fedpkg, the function _run_command doesn't return anything. So I can't compare the result from the fedpkg function and from the local os. Finally, I figure out a solution. I create a bash script on the fly and run the bash script. I redirect the output of the bash script to a file. Now I can extract data from the output file and compare it to the result from the fedpkg function. I clean up the temp files in the end.
One function I tested is _find_branch. This function is to find the current branch. An easy assertion is the default branch master (assertEqual). Then I use a bash script change the branch name (git branch and git checkout commands). When I run _find_branch again, I asserted that the branch is not master anymore (assertNotEqual).
Next function I tested is _hash_file. This function is to generate a hash for a file, with a providing hash type. I use a bash script to calculate the hash of a file and output to a local file. I extracted hash to a variable. Then I run _hash_file on the same file using the same hash type. The two hash values should be equal (assertEqual, again).
The third function I tested is _verify_file. The function will verity the hash value against a predetermined value. Return true is they are the same. I generate a hash for a file using a bash script. Then use the hash and the file as arguments in _verify_file. Assertion should be true (assertTrue).
To run my test, you can go to chile.proximity.on.ca. I have my test script and required packages in /var/www/html/fedpkg directory. Run ./test_fedpkg.py will go through 7 tests.
Friday, December 10, 2010
Wednesday, December 8, 2010
A little Python script
In our OPS535 class, we need to extract the records from a wiki page and generate a zone file. Normally, bash script is the choice for such a job. You use wget to download the file. Then you grep, cut, and output to another file. (You can download the page manually because the contents changes from time to time.)
Since I'm new in Python, I decide to do a little practice in Python scripting. There is a nice feature in Python in dealing with web pages. You don't have to download the page. You can just open it, process it and then write your output to a local file. You need import a package called urllib for that purpose. Anyway, below is the script I wrote for the task.
#!/usr/bin/python
'''generate a root.zone fine from the wiki page of DNS registration'''
import urllib
url = "http://zenit.senecac.on.ca/wiki/index.php/Domainreg"
#open the webpage and read the registration info into a list and then close the webpage
webpage = urllib.urlopen(url)
lst = []
lst2 = []
for line in webpage:
line = line.strip()
if line.startswith("</td><td>"):
lst.append(line[9:].strip())
webpage.close()
#read the useful items from lst to lst2
for i in range(0,len(lst)):
if i % 5 == 0 or i % 5 == 2 or i % 5 == 3:
if lst[i] != '':
lst2.append(lst[i])
#now generate root.zone file
output = open("root.zone", "w")
output.write("$TTL 86400\n")
output.write("@ IN SOA host.mydomain.com. root.host.mydomain.com. (\n")
output.write(" 42 ; serial\n")
output.write(" 3H ; refresh\n")
output.write(" 15M ; retry\n")
output.write(" 1W ; expiry\n")
output.write(" 1D ) ; minimum\n")
output.write("@ IN NS host.mydomain.com.\n")
num = len(lst2)/3
for i in range(0, num):
output.write(lst2[3*i])
output.write(". IN NS ")
output.write(lst2[3*i + 2])
output.write(".\n")
output.write(lst2[3*i + 2])
output.write(". IN A ")
output.write(lst2[3*i + 1])
output.write("\n")
output.close()
Since I'm new in Python, I decide to do a little practice in Python scripting. There is a nice feature in Python in dealing with web pages. You don't have to download the page. You can just open it, process it and then write your output to a local file. You need import a package called urllib for that purpose. Anyway, below is the script I wrote for the task.
#!/usr/bin/python
'''generate a root.zone fine from the wiki page of DNS registration'''
import urllib
url = "http://zenit.senecac.on.ca/wiki/index.php/Domainreg"
#open the webpage and read the registration info into a list and then close the webpage
webpage = urllib.urlopen(url)
lst = []
lst2 = []
for line in webpage:
line = line.strip()
if line.startswith("</td><td>"):
lst.append(line[9:].strip())
webpage.close()
#read the useful items from lst to lst2
for i in range(0,len(lst)):
if i % 5 == 0 or i % 5 == 2 or i % 5 == 3:
if lst[i] != '':
lst2.append(lst[i])
#now generate root.zone file
output = open("root.zone", "w")
output.write("$TTL 86400\n")
output.write("@ IN SOA host.mydomain.com. root.host.mydomain.com. (\n")
output.write(" 42 ; serial\n")
output.write(" 3H ; refresh\n")
output.write(" 15M ; retry\n")
output.write(" 1W ; expiry\n")
output.write(" 1D ) ; minimum\n")
output.write("@ IN NS host.mydomain.com.\n")
num = len(lst2)/3
for i in range(0, num):
output.write(lst2[3*i])
output.write(". IN NS ")
output.write(lst2[3*i + 2])
output.write(".\n")
output.write(lst2[3*i + 2])
output.write(". IN A ")
output.write(lst2[3*i + 1])
output.write("\n")
output.close()
Friday, November 26, 2010
SBR600 Project "fedpkg Test Suite" -- release 0.2
For testing fedpkg library functions, I wrote 4 tests. Nothing is significant. It's really about "finding the way in the wood". I mean, even I got myself familiar with Python syntax, when come to a real program, I still can't understand half of it. So it took some time to find the clue in the program. Another thing is to learn "unittest". I scrach the surface of it but still need time to know how to use the functions.
I put my test file in my matrix account. I took the __init__.py file from fedpkg and renamed it to pyfedpkg.py. I have two tests on clone, a test on commit and a test on _run_command. They are quite trivel. I will continue to work on them toward my next release. The output of my test is pasted as following.
[zwang98@dell fedpkg]$ ./test_fedpkg.py
Cloning into nled...
remote: Counting objects: 82, done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 82 (delta 38), reused 82 (delta 38)
Receiving objects: 100% (82/82), 8.20 KiB, done.
Resolving deltas: 100% (38/38), done.
nled/nled.spec
.Cloning into bind...
ssh_exchange_identification: Connection closed by remote host
fatal: The remote end hung up unexpectedly
user is not valid
.fatal: Not a git repository (or any parent up to mount parent )
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
commit must have an argument
.nled/passwd
.
----------------------------------------------------------------------
Ran 4 tests in 7.279s
OK
I put my test file in my matrix account. I took the __init__.py file from fedpkg and renamed it to pyfedpkg.py. I have two tests on clone, a test on commit and a test on _run_command. They are quite trivel. I will continue to work on them toward my next release. The output of my test is pasted as following.
[zwang98@dell fedpkg]$ ./test_fedpkg.py
Cloning into nled...
remote: Counting objects: 82, done.
remote: Compressing objects: 100% (36/36), done.
remote: Total 82 (delta 38), reused 82 (delta 38)
Receiving objects: 100% (82/82), 8.20 KiB, done.
Resolving deltas: 100% (38/38), done.
nled/nled.spec
.Cloning into bind...
ssh_exchange_identification: Connection closed by remote host
fatal: The remote end hung up unexpectedly
user is not valid
.fatal: Not a git repository (or any parent up to mount parent )
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
commit must have an argument
.nled/passwd
.
----------------------------------------------------------------------
Ran 4 tests in 7.279s
OK
Thursday, November 4, 2010
SBR600 Project "fedpkg Test Suite" -- release 0.1
I decide to do the release 0.1 of my project although there are not as many functions being tested as I planned to. I believe that the stage one of fedpkg Test Suite is mainly to learn Python. I spent, cumulatively, about 3 - 4 weeks to get myself familiar with the syntax of Python.
The stage one of my project is to write some simple functions and use nose to run the test. As exercise of programming Python, I wrote three functions on list. List is relatively simpler than dictionary. Maybe I will add one or two functions of dictionary to run the nose test on them. But the nose principle is the same. Just coding on dictionary should be more complex (so, it's about the skill of Python).
I wrote three functions on list. They are all very simple. One is to remove negative items from a number list. I blogged some details in my last post. Another function is to find duplicate items in a list and return a set of these duplicates. The third function is bubble sort.
The test functions are in the same file as the functions being tested. Because they all very small, it's easier to debugging this way. The name of the test function has to start with test. I mainly use keyword assert to test the results.
From here, I need to dive in the fedpkg code. I also need to get familiar with the functionality of fedpkg, which probably need certain knowledge of Git. Another thing I am still not clear is the relationship between nose and unittest framework. I thought they are the same at beginning, but probably they are not.
I dump the test code in my matrix account:
http://matrix.senecac.on.ca/~zwang98/nosetest.html
The stage one of my project is to write some simple functions and use nose to run the test. As exercise of programming Python, I wrote three functions on list. List is relatively simpler than dictionary. Maybe I will add one or two functions of dictionary to run the nose test on them. But the nose principle is the same. Just coding on dictionary should be more complex (so, it's about the skill of Python).
I wrote three functions on list. They are all very simple. One is to remove negative items from a number list. I blogged some details in my last post. Another function is to find duplicate items in a list and return a set of these duplicates. The third function is bubble sort.
The test functions are in the same file as the functions being tested. Because they all very small, it's easier to debugging this way. The name of the test function has to start with test. I mainly use keyword assert to test the results.
From here, I need to dive in the fedpkg code. I also need to get familiar with the functionality of fedpkg, which probably need certain knowledge of Git. Another thing I am still not clear is the relationship between nose and unittest framework. I thought they are the same at beginning, but probably they are not.
I dump the test code in my matrix account:
http://matrix.senecac.on.ca/~zwang98/nosetest.html
Wednesday, November 3, 2010
Learning Python
My project for SBR600 is fedpkg Test Suite. For 0.1 release, I plan to write some simple functions in manipulating list, and then use nose to test the functions. The first function I try to write is a simple exercise: given a list of numbers, the function will remove the negative items.
The first try
def rem_neg(list):
for i in list:
if list[i] < 0:
list.remove(list[i])
before I use nose to test, I did some regular test. Give a list [1, 2, -1, 3, -2], the result will be [1,2,3]. Not bad. But when I use [5,-1,-2,3,4,-8], the result is [5-2,3,4]. So the function is not right. I went to check the book and realized that when a list remove an item, the rest will shift to the left. Therefore, the for loop missing the second negative number if two negative numbers next to each other.
The second try
I tried to find a way to work around the shifting problem. How about do the loop backward? Then the shift won't cause problem anymore.
def rem_neg(list)
for i in range(len(list)-1, 0, -1):
if list[i] < 0:
list.remove(list[i])
I tested the previous example. It worked. But I should test more examples. The text book I am read said to use special case or extreme case to test. When I test list [-1], it failed again. The result is [-1]. So the function didn't reach the first item (It's the last if we consider the loop goes backward). It's typical "off by 1" error in programming. My for loop range should be (len(list)-1, -1, -1). The range function always stop 1 before the last item.
Come to nose
Now I put the function to nose test.
import nose
def rem_neg(list)
for i in range(len(list)-1, -1, -1):
if list[i] < 0:
list.remove(list[i])
def test_rem_neg():
assert rem_neg([-1]) == []
if __name__ == '__main__' :
nose.runmodule()
The test failed. It took a while for me to figure out what is wrong. The function rem_neg(list) return none. I wrote the function this way just to follow the style of list built-in functions. This is: the original list get modified and nothing returned (except the pop function).
Come next: 0.1 release
This is just the first example I plan to use for the nose test. I will test more examples in two days and finish my 0.1 release.
The first try
def rem_neg(list):
for i in list:
if list[i] < 0:
list.remove(list[i])
before I use nose to test, I did some regular test. Give a list [1, 2, -1, 3, -2], the result will be [1,2,3]. Not bad. But when I use [5,-1,-2,3,4,-8], the result is [5-2,3,4]. So the function is not right. I went to check the book and realized that when a list remove an item, the rest will shift to the left. Therefore, the for loop missing the second negative number if two negative numbers next to each other.
The second try
I tried to find a way to work around the shifting problem. How about do the loop backward? Then the shift won't cause problem anymore.
def rem_neg(list)
for i in range(len(list)-1, 0, -1):
if list[i] < 0:
list.remove(list[i])
I tested the previous example. It worked. But I should test more examples. The text book I am read said to use special case or extreme case to test. When I test list [-1], it failed again. The result is [-1]. So the function didn't reach the first item (It's the last if we consider the loop goes backward). It's typical "off by 1" error in programming. My for loop range should be (len(list)-1, -1, -1). The range function always stop 1 before the last item.
Come to nose
Now I put the function to nose test.
import nose
def rem_neg(list)
for i in range(len(list)-1, -1, -1):
if list[i] < 0:
list.remove(list[i])
def test_rem_neg():
assert rem_neg([-1]) == []
if __name__ == '__main__' :
nose.runmodule()
The test failed. It took a while for me to figure out what is wrong. The function rem_neg(list) return none. I wrote the function this way just to follow the style of list built-in functions. This is: the original list get modified and nothing returned (except the pop function).
Come next: 0.1 release
This is just the first example I plan to use for the nose test. I will test more examples in two days and finish my 0.1 release.
Tuesday, October 12, 2010
rpath - an interesting story
Finally, I got my koji build done. Thanks to Chris Tyler for the help to correct my last error in the spec file. It's quite a long story to work through wireshark package. In the early stage, I encountered the error with rpath. It's basically the library path being hard corded in the file, which causes the build process confusion when it can't find the path (or something like that).
I got the rpath error in the lab and didn't have enough time to resolve it before I had to leave the lab. Later, I test the build at home, where I use a 32 bit processor. The rpath error didn't show (lucky!). I dealt with other problems and got the build done. I wasn't fully understand the rpath nature because I didn't try it again in the lab.
Then I got the access to the machines in CDOT. I tried to do the mock build in ireland. Now I remembered that I still have the rpath problem. First I try to follow the hint in the error massage to run:
$ QA_RPATHS=$[ 0x0001|0x0010 ] rpmbuild -ba wireshark.spec
The errors are gone, but when run rpmlint afterward, the rpath errors come back. So, the above command just ignore the rpath. It didn't solve the problem.
I found the page Packaging:Guidelines. I realized that I need to run chrpath on each file in the error line. Please go the that page to get the details if you are interested. Now the rpmlint is happy and errors are gone.
The mock build is kind of tedious. I repeated half a dozen times since I added the build requires each time I run into an error. Eventually, all the build requirements are satisfied and mock build succeeded.
Finally, I moved to koji. First a certificate was applied and pasted to ireland. Then I run koji build, and I failed. I went to check the build.log. Guess who is making trouble? rpath, again? In my spec file, the chrapth command run for each file has rpath in it. Two of these files are in /usr/lib64/.... The koji build failed in i686 architecture since lib64 doesn't exist in that case. Today I asked Chris for help. He told me to use %{_libdir} instead of /usr/lib64. That's it, koji went through.
Looking back, it's quite an interesting learning curve. Looking back further, It's breath taking. I started Seneca in summer 2009. I remember at the time half of the class were struggling with ULI101: sort, tail, pipe ... etc. Can you imaging to deal with a course like SBR600 in such a short time. Oh well...
I got the rpath error in the lab and didn't have enough time to resolve it before I had to leave the lab. Later, I test the build at home, where I use a 32 bit processor. The rpath error didn't show (lucky!). I dealt with other problems and got the build done. I wasn't fully understand the rpath nature because I didn't try it again in the lab.
Then I got the access to the machines in CDOT. I tried to do the mock build in ireland. Now I remembered that I still have the rpath problem. First I try to follow the hint in the error massage to run:
$ QA_RPATHS=$[ 0x0001|0x0010 ] rpmbuild -ba wireshark.spec
The errors are gone, but when run rpmlint afterward, the rpath errors come back. So, the above command just ignore the rpath. It didn't solve the problem.
I found the page Packaging:Guidelines. I realized that I need to run chrpath on each file in the error line. Please go the that page to get the details if you are interested. Now the rpmlint is happy and errors are gone.
The mock build is kind of tedious. I repeated half a dozen times since I added the build requires each time I run into an error. Eventually, all the build requirements are satisfied and mock build succeeded.
Finally, I moved to koji. First a certificate was applied and pasted to ireland. Then I run koji build, and I failed. I went to check the build.log. Guess who is making trouble? rpath, again? In my spec file, the chrapth command run for each file has rpath in it. Two of these files are in /usr/lib64/.... The koji build failed in i686 architecture since lib64 doesn't exist in that case. Today I asked Chris for help. He told me to use %{_libdir} instead of /usr/lib64. That's it, koji went through.
Looking back, it's quite an interesting learning curve. Looking back further, It's breath taking. I started Seneca in summer 2009. I remember at the time half of the class were struggling with ULI101: sort, tail, pipe ... etc. Can you imaging to deal with a course like SBR600 in such a short time. Oh well...
Thursday, September 30, 2010
Testing j-value with a big package
I guess Chris Tyler is not convinced about j1 is faster than j3 or j2. There are two such cases, include my own, were posted on the Planet. I decide to test a big package. I chose gcc-c++. The size of the rpm source package is about 50M. It seems not that big. So the other day I tried to run rpmbuild on this package. It run more than 2 hours and didn't finish. I had to kill it.
I decide to write a small script to run automatically. First I use a simple for loop (for i in 1 2 3 4 5). But how to catch the time. If I use time before rpmbuild, I need to redirect the output to catch the time. The redirection file will be huge. But I only need the last 3 lines. The funny thing is that 3 lines of time is not redirected (when I test using a small package). So I better use time stamp, recording time before build (date +%s) and recording time after. Then out put the difference.
To change the j value in .rpmmacro file, I couldn't think of a better solution. OPS435 was quite some time ago. I simply just echo the 3 lines to the file before run the rpmbuild command. I think it's not a good approach in scripting. But I want to start run the test sooner. After I tested the script, I started to run the build this evening and planed to run over night. I set to run 5 cases (from 1 to 5). I don't know if it will finish by tomorrow morning. I will add the result here as soon as the test is done.
......The test result finally comes out. The output is like this:
This is testing of j1.
Time spent is: 37835 seconds
This is testing of j2.
Time spent is: 38562 seconds
This is testing of j3.
Time spent is: 39019 seconds
This is testing of j4.
Time spent is: 39316 seconds
This is testing of j5.
Time spent is: 39301 seconds
It seems j1 is still faster. I wonder what could be an explaination for that.
I decide to write a small script to run automatically. First I use a simple for loop (for i in 1 2 3 4 5). But how to catch the time. If I use time before rpmbuild, I need to redirect the output to catch the time. The redirection file will be huge. But I only need the last 3 lines. The funny thing is that 3 lines of time is not redirected (when I test using a small package). So I better use time stamp, recording time before build (date +%s) and recording time after. Then out put the difference.
To change the j value in .rpmmacro file, I couldn't think of a better solution. OPS435 was quite some time ago. I simply just echo the 3 lines to the file before run the rpmbuild command. I think it's not a good approach in scripting. But I want to start run the test sooner. After I tested the script, I started to run the build this evening and planed to run over night. I set to run 5 cases (from 1 to 5). I don't know if it will finish by tomorrow morning. I will add the result here as soon as the test is done.
......The test result finally comes out. The output is like this:
This is testing of j1.
Time spent is: 37835 seconds
This is testing of j2.
Time spent is: 38562 seconds
This is testing of j3.
Time spent is: 39019 seconds
This is testing of j4.
Time spent is: 39316 seconds
This is testing of j5.
Time spent is: 39301 seconds
It seems j1 is still faster. I wonder what could be an explaination for that.
Subscribe to:
Posts (Atom)