Use either:
1. Instead of building a web server, you can consider to write a fastcgi
or scgi application.
http://www.fastcgi.com/devkit/doc/fastcgi-prog-guide/ap_guide.htm
2. You could look at the lua module. I hardly ever write C for nginx now.
http://openresty.org/ has some examples.
Also, I'm a big fan of using HTTP over things like fastcgi,scgi, etc.
Much easier to test, debug, etc. But that's just my opinion.
3. fcgiwrap http://wiki.nginx.org/Fcgiwrap
4. thttpd http://wiki.nginx.org/ThttpdCGI
5. mini-httpd
Nginx doesn't support CGI.
The best solution probably depends on what you're trying to achieve and your particular circumstances. As well as the solutions already suggested, one method I've used is to build the C program against libevent, effectively using libevent to build a mini webserver that is running all the time. nginx can then use that as an upstream proxy.
Start here:
http://libevent.org/
libevent has all of the hard stuff done for you. You basically just need to use it's API to get hold of all the HTTP request information and to let it handle the response to the user.
Reference:
http://www.ruby-forum.com/topic/3135016
Wednesday, April 17, 2013
Narrowing Down Performance Problems in Managed Code
My last entry was some generic advice about how to do a good performance investigation. I think actually it's too generic to be really useful -- in fact I think it fails my Peanut Butter Sandwich Test.
Digression to discuss the Peanut Butter Sandwich Test
I review a lot of documents and sometimes they say things that are so obvious as to be uninteresting. The little quip I have for this situation is, "Yes what you are saying is true of [the system] but it's also true of peanut butter sandwiches." Consider a snippet like this one, "Use a cache where it provides benefits," and compare with, "Use a peanut butter sandwich where it provides benefits." Both seem to work... that's a bad sign.
You certainly don't want to get an F on the Peanut Butter Sandwich Test but hopefully you won't settle for just a C-.
Back on topic
I thought it would be good to follow up the generic advice with some specific suggestions for things to look at. These are things I look at in step 2 or 3 of the investigation.
Under .NET CLR Memory, check "% Time in GC" if it's getting near 10% or higher you may have some memory issues, consider these secondary tests:
is the raw allocation rate "Allocated Bytes/sec" too high? -> reduce total allocations
is the promotion rate "Promoted Memory from Gen 1" too high? -> be careful about object lifetimes, avoid "mid-life crisis"
is the finalization rate "Finalization Survivors" too high? -> make sure you are disposing the key objects
is the heap growing when it shouldn't "# Bytes in all Heaps" -> check for reference leaks
Is the CPU not saturated when it should be? Look under .NET CLR LocksAndThreads
is the "Contention Rate / sec" counter high compared to your throughput rate? -> you should re-examine your locking strategy
is the "# of current physical Threads" too low for the problem? -> (ammended) more parallelism may be helpful, consider using the ThreadPool if not already in use, possibly adjust ThreadPool parameters to get more threads (not usually needed)
in the "Thread" category examine "Context Switches / sec", is this high compared to your throughput rate? -> perhaps the workitem you are giving threads in the thread pool is too small, consider something chunkier
Is the throughput rate low even though the CPU is saturated?
look under ".NET CLR Exceptions", is "# of Excepts Thrown / sec" high compared to your throughput? -> consider reducing use of exceptions in common paths
look under ".NET CLR Interop", is "# of marshalling" growing too fast? -> consider simplifying the arguments passed in interop cases so that marshalling is cheaper
look under ".NET CLR Security", is "% Time in RT checks" significant? -> consider simplying the demands being placed on the security system to lower the cost of security checks
look under ".NET CLR Jit", is "% Time in Jit" significant? This counter shouldn't stay high because jitting should settle out, if it remains high then perhaps there is dynamic code generation via reflection going on -> simply dynamic code cases
This just a taste of course, and each of these items would likely lead to further investigation with a profiling tool that is suitable to drilling into that particular kind of problem but these are examples of leading indicators that I use.
For more information on the GC Performance counters specifically see Maoni's blog entry on that subject. Her most recent article is on using the GC efficiently also very interesting, lots of good details there.
Reference:
http://blogs.msdn.com/b/ricom/archive/2005/05/25/421926.aspx
Digression to discuss the Peanut Butter Sandwich Test
I review a lot of documents and sometimes they say things that are so obvious as to be uninteresting. The little quip I have for this situation is, "Yes what you are saying is true of [the system] but it's also true of peanut butter sandwiches." Consider a snippet like this one, "Use a cache where it provides benefits," and compare with, "Use a peanut butter sandwich where it provides benefits." Both seem to work... that's a bad sign.
You certainly don't want to get an F on the Peanut Butter Sandwich Test but hopefully you won't settle for just a C-.
Back on topic
I thought it would be good to follow up the generic advice with some specific suggestions for things to look at. These are things I look at in step 2 or 3 of the investigation.
Under .NET CLR Memory, check "% Time in GC" if it's getting near 10% or higher you may have some memory issues, consider these secondary tests:
is the raw allocation rate "Allocated Bytes/sec" too high? -> reduce total allocations
is the promotion rate "Promoted Memory from Gen 1" too high? -> be careful about object lifetimes, avoid "mid-life crisis"
is the finalization rate "Finalization Survivors" too high? -> make sure you are disposing the key objects
is the heap growing when it shouldn't "# Bytes in all Heaps" -> check for reference leaks
Is the CPU not saturated when it should be? Look under .NET CLR LocksAndThreads
is the "Contention Rate / sec" counter high compared to your throughput rate? -> you should re-examine your locking strategy
is the "# of current physical Threads" too low for the problem? -> (ammended) more parallelism may be helpful, consider using the ThreadPool if not already in use, possibly adjust ThreadPool parameters to get more threads (not usually needed)
in the "Thread" category examine "Context Switches / sec", is this high compared to your throughput rate? -> perhaps the workitem you are giving threads in the thread pool is too small, consider something chunkier
Is the throughput rate low even though the CPU is saturated?
look under ".NET CLR Exceptions", is "# of Excepts Thrown / sec" high compared to your throughput? -> consider reducing use of exceptions in common paths
look under ".NET CLR Interop", is "# of marshalling" growing too fast? -> consider simplifying the arguments passed in interop cases so that marshalling is cheaper
look under ".NET CLR Security", is "% Time in RT checks" significant? -> consider simplying the demands being placed on the security system to lower the cost of security checks
look under ".NET CLR Jit", is "% Time in Jit" significant? This counter shouldn't stay high because jitting should settle out, if it remains high then perhaps there is dynamic code generation via reflection going on -> simply dynamic code cases
This just a taste of course, and each of these items would likely lead to further investigation with a profiling tool that is suitable to drilling into that particular kind of problem but these are examples of leading indicators that I use.
For more information on the GC Performance counters specifically see Maoni's blog entry on that subject. Her most recent article is on using the GC efficiently also very interesting, lots of good details there.
Reference:
http://blogs.msdn.com/b/ricom/archive/2005/05/25/421926.aspx
How To Do A Good Performance Investigation
I find that sometimes people have difficultly just getting started when doing a performance analysis – meaning they’re faced with a potentially big problem and don’t know where to begin. Over the years many people have come to me under those circumstances and asked me how I would approach the problem. So today I’m trying to distill those bits of advice – my Modus Operandi – into some simple steps in the hope that it might help others to get off to a good start.
So here it is, Rico’s advice on how to do a good performance investigation.
Preliminaries
The first thing to remember is not to try to do this in a rushed way. The more of a hurry you are in to get to the bottom of the problem the less you can afford to be hasty. Be deliberate and careful. Block out a good chunk of time to think about the problem. Make sure you have the resources you need to succeed – that means enough access to the right hardware and the right people. Prepare a log book – electronic if you like – so that you can keep notes and interim results at each step. This will be the basis of your final report and it will be an invaluable reference to anyone that follows in your footsteps – even if (especially if?) that someone is you.
Step 1 – Get the Lay of the Land
Before you look at any code, talk to the people involved. You’ll want to get a feel for what they are trying to accomplish. What are their key difficulties? What is inherently hard about this problem? What is the overall organization of the system? What are the inherent limitations of those choices? What is the chief complaint with the current system? What would a successful system look like? How do they measure success?
In addition to a basic understanding of how the system is intended to work the key question you want to answer is this: Which resource is the one that should be critical in this system? Is the problem fundamentally CPU bound, disk bound, network bound? If things were working perfectly what would be constraining the performance?
Step 2 – Identify the current bottlenecks
In this step we cast a broad net to see what resource is the current limiting factor. The tool I reach for first is PerfMon. Look at key counters like CPU Usage, Memory Usage, Disk and Network I/O. Examine these on all the various different machines involved if this is a server problem (i.e. check all the tiers). Check for high levels of lock contention.
At this point you should be able to identify which resource is currently the one that is limiting performance. Often it is not the resource identified in Step 1.
If it is the correct resource – the one that is supposed to be the limiting factor in this kind of computing – that’s a good preliminary sign that sensible algorithms have been selected. If it’s the wrong resource it means you are going to be looking for design problems where a supposedly non-critical resource has been overused to the point that it has become the critical resource. The design will have to be altered such that it does not have this (fundamentally unnecessary) dependence.
Step 3 – Drill down on the current bottleneck
A common mistake in a performance analysis is to try to do step 3 before step 2. This is going to lead to a good bit of waste because you could do a deep analysis of say CPU usage when CPU usage isn’t the problem. Instead, choose tools that are good at measuring the problem resource and don’t worry so much about the others for now. If it’s hard to measure the resource in question, add instrumentation for this resource if possible. The goal is to find out as much as you can about what is causing the (over) use of the bottleneck resource.
When doing this analysis, consider factors that control the resources at different system levels starting from the largest and going to the smallest. Is there something about the overall system architecture that is causing overuse of this resource? Is it something in the overall application design? Or is it a local problem with a module or subcomponent? Most significant problems are larger in scope, look at those first. They are the easiest to diagnose and sometimes the easiest to correct. E.g. if caching were disabled on a web server you could expect big problems in the back end servers. You’ll want to make sure caching is working properly before you decide to (e.g.) add more indexes to make some query faster.
Your approach will need to be tailored to the resource and the system. For CPU problems a good time profiler can be invaluable. For SQL problems there’s of course SQL Profiler (find the key queries) and Query Analyzer (view the plans). For memory issues there are abundant performance counters that can be helpful, including the raw memory counters and .NET memory management ones, and others. Tracking virtual memory use over time can be helpful – sampling with vadump is handy. Examination of key resources by breaking into the system with a debugger can also be useful.
Step 4 – Identify anomalies in the measurements
The most interesting performance problems are almost always highlighted by anomalous observed costs. Things are happening that shouldn’t be happening or need not happen. If the critical resource isn’t the “correct” one as identified in Step 1 it’s almost certainly the case that your root-cause analysis will find an undesired use of the resource. If the resource was the “correct” one then you’ll be looking for overuse to get an improvement.
In both cases it is almost always helpful to look at the resource costs per unit-of-work. What is the “transaction” in this system? Is it a mouse click event? Is it a business transaction of some kind? Is it an HTML page delivered to the user? A database query performed? Whatever it is look at your critical resource and consider the cost per unit of work. E.g. consider CPU cycles per transaction, network bytes per transaction, disk i/o’s per transaction, etc. These metrics will often show you the source of the mistake – consider in my recent analysis of Performance Quiz #6 where I looked at the number of string allocations per line of input and then bytes allocate per byte in the input file. Those calculations were both easy and revealing.
Expressing your costs in a per-unit-of work fashion will help you to see which costs are reasonable and which are problems.
Step 5 – Create a hypothesis and verify it
Based on the analysis in step 4 you should be able to find a root cause for the current bottleneck. Design an experiment to validate that this is the case. This can be a very simple test such as “if we change the settings [like so], it should make the problem much worse.” Take advantage of any kind of fairly quick validation that you can do… the worst thing to do is to plunge into some massive correction without being sure that you’ve really hit the problem. If there is nothing obvious, consider doing only a small fraction of the corrective work. Perhaps just enough for one test case to function – validate that before you proceed.
The trick to doing great performance work is to be able to try various things and yet spend comparatively little time on the losing strategies while quickly finding and acting on the winners.
Step 6 – Apply the finished corrections, verify, and repeat as needed
After Step 5 you should be highly confident that you have a winning change on your hands. Go ahead and finish it up to production quality and apply those changes. Make sure things went as you expected and only then move on. If your changes were not too sweeping you can probably resume at Step 2, or maybe even Step 3. If they were very big changes you might have to go back to Step 1.
Step 7 – Write a brief report
Summarize your method and findings for your teammates. It’s invaluable as a learning exercise for yourself and as a long-term resource for your team.
Post Script
I wrote a followup article which offers more prescriptive advice about managed code specifically -- see Narrowing Down Performance Problems in Managed Code
Reference:
http://blogs.msdn.com/b/ricom/archive/2005/05/23/421205.aspx
So here it is, Rico’s advice on how to do a good performance investigation.
Preliminaries
The first thing to remember is not to try to do this in a rushed way. The more of a hurry you are in to get to the bottom of the problem the less you can afford to be hasty. Be deliberate and careful. Block out a good chunk of time to think about the problem. Make sure you have the resources you need to succeed – that means enough access to the right hardware and the right people. Prepare a log book – electronic if you like – so that you can keep notes and interim results at each step. This will be the basis of your final report and it will be an invaluable reference to anyone that follows in your footsteps – even if (especially if?) that someone is you.
Step 1 – Get the Lay of the Land
Before you look at any code, talk to the people involved. You’ll want to get a feel for what they are trying to accomplish. What are their key difficulties? What is inherently hard about this problem? What is the overall organization of the system? What are the inherent limitations of those choices? What is the chief complaint with the current system? What would a successful system look like? How do they measure success?
In addition to a basic understanding of how the system is intended to work the key question you want to answer is this: Which resource is the one that should be critical in this system? Is the problem fundamentally CPU bound, disk bound, network bound? If things were working perfectly what would be constraining the performance?
Step 2 – Identify the current bottlenecks
In this step we cast a broad net to see what resource is the current limiting factor. The tool I reach for first is PerfMon. Look at key counters like CPU Usage, Memory Usage, Disk and Network I/O. Examine these on all the various different machines involved if this is a server problem (i.e. check all the tiers). Check for high levels of lock contention.
At this point you should be able to identify which resource is currently the one that is limiting performance. Often it is not the resource identified in Step 1.
If it is the correct resource – the one that is supposed to be the limiting factor in this kind of computing – that’s a good preliminary sign that sensible algorithms have been selected. If it’s the wrong resource it means you are going to be looking for design problems where a supposedly non-critical resource has been overused to the point that it has become the critical resource. The design will have to be altered such that it does not have this (fundamentally unnecessary) dependence.
Step 3 – Drill down on the current bottleneck
A common mistake in a performance analysis is to try to do step 3 before step 2. This is going to lead to a good bit of waste because you could do a deep analysis of say CPU usage when CPU usage isn’t the problem. Instead, choose tools that are good at measuring the problem resource and don’t worry so much about the others for now. If it’s hard to measure the resource in question, add instrumentation for this resource if possible. The goal is to find out as much as you can about what is causing the (over) use of the bottleneck resource.
When doing this analysis, consider factors that control the resources at different system levels starting from the largest and going to the smallest. Is there something about the overall system architecture that is causing overuse of this resource? Is it something in the overall application design? Or is it a local problem with a module or subcomponent? Most significant problems are larger in scope, look at those first. They are the easiest to diagnose and sometimes the easiest to correct. E.g. if caching were disabled on a web server you could expect big problems in the back end servers. You’ll want to make sure caching is working properly before you decide to (e.g.) add more indexes to make some query faster.
Your approach will need to be tailored to the resource and the system. For CPU problems a good time profiler can be invaluable. For SQL problems there’s of course SQL Profiler (find the key queries) and Query Analyzer (view the plans). For memory issues there are abundant performance counters that can be helpful, including the raw memory counters and .NET memory management ones, and others. Tracking virtual memory use over time can be helpful – sampling with vadump is handy. Examination of key resources by breaking into the system with a debugger can also be useful.
Step 4 – Identify anomalies in the measurements
The most interesting performance problems are almost always highlighted by anomalous observed costs. Things are happening that shouldn’t be happening or need not happen. If the critical resource isn’t the “correct” one as identified in Step 1 it’s almost certainly the case that your root-cause analysis will find an undesired use of the resource. If the resource was the “correct” one then you’ll be looking for overuse to get an improvement.
In both cases it is almost always helpful to look at the resource costs per unit-of-work. What is the “transaction” in this system? Is it a mouse click event? Is it a business transaction of some kind? Is it an HTML page delivered to the user? A database query performed? Whatever it is look at your critical resource and consider the cost per unit of work. E.g. consider CPU cycles per transaction, network bytes per transaction, disk i/o’s per transaction, etc. These metrics will often show you the source of the mistake – consider in my recent analysis of Performance Quiz #6 where I looked at the number of string allocations per line of input and then bytes allocate per byte in the input file. Those calculations were both easy and revealing.
Expressing your costs in a per-unit-of work fashion will help you to see which costs are reasonable and which are problems.
Step 5 – Create a hypothesis and verify it
Based on the analysis in step 4 you should be able to find a root cause for the current bottleneck. Design an experiment to validate that this is the case. This can be a very simple test such as “if we change the settings [like so], it should make the problem much worse.” Take advantage of any kind of fairly quick validation that you can do… the worst thing to do is to plunge into some massive correction without being sure that you’ve really hit the problem. If there is nothing obvious, consider doing only a small fraction of the corrective work. Perhaps just enough for one test case to function – validate that before you proceed.
The trick to doing great performance work is to be able to try various things and yet spend comparatively little time on the losing strategies while quickly finding and acting on the winners.
Step 6 – Apply the finished corrections, verify, and repeat as needed
After Step 5 you should be highly confident that you have a winning change on your hands. Go ahead and finish it up to production quality and apply those changes. Make sure things went as you expected and only then move on. If your changes were not too sweeping you can probably resume at Step 2, or maybe even Step 3. If they were very big changes you might have to go back to Step 1.
Step 7 – Write a brief report
Summarize your method and findings for your teammates. It’s invaluable as a learning exercise for yourself and as a long-term resource for your team.
Post Script
I wrote a followup article which offers more prescriptive advice about managed code specifically -- see Narrowing Down Performance Problems in Managed Code
Reference:
http://blogs.msdn.com/b/ricom/archive/2005/05/23/421205.aspx
Tuesday, April 16, 2013
wget alternative in windows command line
Run the PowerShell V1.0 script directly in command line:
cmd> PowerShell -Command "(new-object System.Net.WebClient).DownloadFile( 'http://download.microsoft.com/download/2/0/E/20E90413-712F-438C-988E-FDAA79A8AC3D/dotnetfx35.exe', 'D:\dotnetfx35.exe')"
Run the PowerShell V1.0 script from a file:
PowerShell -Command "& {c:\users\john\myscript.ps1}"
myscript.ps1:
$client = new-object System.Net.WebClient
$client.DownloadFile( $url, $path )
or myscript.ps1 in one line:
(new-object System.Net.WebClient).DownloadFile($url, $path)
or myscript.ps1 in PowerShell V3.0:
Invoke-WebRequest http://www.google.com/ -OutFile c:\google.html
or myscript.ps1 in PowerShell V3.0:
Invoke-WebRequest http://www.google.com/ > c:\google.html
Run the PowerShell V2.0 script from a file:
PowerShell.exe -File c:\users\john\myscript.ps1
Run Wget for Windows:
http://gnuwin32.sourceforge.net/packages/wget.htm
Thursday, April 11, 2013
那些台灣軟體產業所缺少的 – 自動化測試
你是否有計算過,你在寫專案的過程中,測試過了多少次的程式? 我想是沒有,我也沒有,但是你是否有曾想過,或是感覺過,隨著專案的膨漲,你要測試的項目也跟著變多了? 這是理所當然的事情,當專案小,測試還算很輕鬆,因為程式的功能不外乎就那幾樣,一轉眼就測完了,常見的寫程式流程會像這樣
這表示你每寫一行新程式的成本增加了,身為以減低成本為傲的島國 國民: 台灣人…,你說,簡單! 不要測舊功能不就好了? 是的,我想這可能就是最常見的情況,不要測試舊功能理所當然地,每寫一行的程式成本都保持一樣很低,但這代表著舊程式可能出錯的風險也跟著增加了,當你喜 滋滋地覺得你幫公司省了成本,結果在一個月後因為舊程式缺乏測試,因改動了核心的部份造成舊的功能將所有資料外洩,公司損失慘重,這就是不重視軟體品質的 後果
舉真實生活上發生過的例子,PTT曾經有過改程式未經好好地測試,造成每個人都能以管理員的權限登入的事情,知名的檔案同步平台Dropbox,也曾經發生過因為認證的程式改版有bug,造成任何人都可以登入別人帳號的事,我也有曾聽聞一些網站因為工程師為了測試方便,把認證的函數暫時改成
在未來,網路的應用越來越多,而軟體的品質重要程度只會越來越高,所以,要如何維持軟體的品質又同時能不讓測試的成本隨著專案的擴張而跟著無限制地成長呢? 答案就是 – 自動化測試
引入了自動化測試不代表程式就不會出錯,它不是萬能的,但是它至少保證了程式一定的品質,只要使用得當,就能降低測試的成本,也能讓大部份有經過自動測試的程式都不會出現太離譜的錯誤,至於要怎麼做,讓我們看下去
雖然單元測試在相當單純的模擬環境下測過了我們的程式,然而世界並不是那樣的美好,總有些事情沒有經過真槍實彈操演過可能會有差錯,因此有時我們會 引入部份受控制的真實環境來測試,例如你想測試網路連線,或許你可以寫一段script在Amazon EC2上建起幾個instance,並上傳程式到那些機器中,自動讓他們連線來確保這些功能是正常的,然而越真實的環境變因就越多,因此測試也就相對困難
記得,工具是為我們服務的,不是我們為工具服務,這樣重複的瑣事理所當然也是由工具來幫忙,謝天謝地現今有好用又免費的工具,可以幫你做到這點,那就是Jenkins, 它是一套基於網頁的自動化測試管理工具,它可以做到什麼呢? 它可以做到幫你定時去版本控制系統取程式回來,用預先設定好的流程進行測試,並且記錄測試的結果,如果有某個測試出錯了,當然也可以發Email通知你, 以Now.in的開發為例,因為專案為數眾多,其中又有依賴關係,有了Jenkins的幫忙,程式只要一改送到BitBucket,它就會自動進行測試
如此一來就省下了大量的時間,同時,也可以專心在於開發上
Jenkins除了功能強大以外,他還有一項特色令我驚訝,就是非常簡單易用,從安裝到設定完所有的測試,除了clone hg檔案庫和設置測試環境以外,我從沒因為Jenkins打過一行指令,全部都可以透過它友善的網頁介面完成,同時它也有內建資料庫,也沒因此設定 MySQL,在Windows下安裝更是容易,一個安裝檔執行完就是安裝完成,如果你希望有工具幫你自動定時測試或是建構,請不要懷疑,Jenkins是 你最佳選擇
除此之外,寫測試事實上也是成本,因此如果時間有限,請
雖然你的程式可能大多都已經有自動化測試在幫你測試,但即使如此,你還是會發現新的bug,如果說,你直接改了bug,就這樣了事,很有可能在下幾次改版bug又回來了,因此
撰寫新功能當然,也有修正bug的情況
測試新功能
修正bug如此一直循環,當你寫了新功能,理所當然地會去測試新功能,看是否如你預期地執行,那舊功能呢? 或許你記憶力不錯,在寫新功能的同時,想到先前某個舊功能是依賴現在改的東西,這麼一改可能會造成舊的功能出問題,於是你也順便測了一下舊的功能,當程式還小
測試bug
撰寫新功能嘿,不怎麼樣吧? 只佔了開發時間的三分之一,好吧那如果有更多的舊功能要測呢?
測試新功能
測試舊功能
撰寫新功能發現了沒有? 隨著你的專案越來越大,如果要確保整個系統所有的功能都是正常運作的,無可避免地,在你修改程式之後要測試的項目會越來越多
測試新功能
測試舊功能
測試舊功能
測試舊功能
….
這表示你每寫一行新程式的成本增加了,身為以減低成本為傲的島國 國民: 台灣人…,你說,簡單! 不要測舊功能不就好了? 是的,我想這可能就是最常見的情況,不要測試舊功能理所當然地,每寫一行的程式成本都保持一樣很低,但這代表著舊程式可能出錯的風險也跟著增加了,當你喜 滋滋地覺得你幫公司省了成本,結果在一個月後因為舊程式缺乏測試,因改動了核心的部份造成舊的功能將所有資料外洩,公司損失慘重,這就是不重視軟體品質的 後果
舉真實生活上發生過的例子,PTT曾經有過改程式未經好好地測試,造成每個人都能以管理員的權限登入的事情,知名的檔案同步平台Dropbox,也曾經發生過因為認證的程式改版有bug,造成任何人都可以登入別人帳號的事,我也有曾聽聞一些網站因為工程師為了測試方便,把認證的函數暫時改成
function authenticate(user_id, password) {
return true;
// do authentication here
// ....
}然後又不小心commit,因此讓任何人都通過認證的事情,但這些都不能只怪工程師本身,誰能無過? 人總會犯錯的,問題出問於工程本身的制度、專案的管理、和工具的使用上在未來,網路的應用越來越多,而軟體的品質重要程度只會越來越高,所以,要如何維持軟體的品質又同時能不讓測試的成本隨著專案的擴張而跟著無限制地成長呢? 答案就是 – 自動化測試
自動化測試
自動化測試聽起來好像很美妙,讓電腦自動幫你測試程式? 有這麼好的事情嗎? 事實上不是那樣,自動化測試,是透過寫好的規則,自動對於程式進行測試,所以終究還是得需要人力的介入,那你或許會問,結果倒頭來還不是得用人力? 那到底有什麼好處? 答案就跟我們先前提到的一樣,如果你的專案很小,用人力測試其實可能就已足夠,但當你的專案夠大,如果沒有自動化測試,那麼光是在測舊的程式就是相當龐大 的成本上引入了自動化測試不代表程式就不會出錯,它不是萬能的,但是它至少保證了程式一定的品質,只要使用得當,就能降低測試的成本,也能讓大部份有經過自動測試的程式都不會出現太離譜的錯誤,至於要怎麼做,讓我們看下去
單元測試
最常見的測試,就是單元測試(Unit test),通常是針對單一個或是少數類別,確保這些類別單獨運作是正確的,舉個例子,你寫了一個類別,是用來找輸入的地圖的最短路徑,那麼你就得替這類 別,寫一個單元測試,餵入你準備好的資料,然後取得輸出的結果,看是否和你準備的預期答案是一樣的,舉一個最簡單的例子,一個用Python來將輸入文字 拆解成一行一行的解析器class LineParser(object):
def __init__(self, newline='rn', remain=''):
self.newline = newline
self._buffer = [remain]
self._size = len(remain)
def feed(self, data):
self._buffer.append(data)
self._size += len(data)
def getLine(self):
data = ''.join(self._buffer)
index = data.find(self.newline)
if index != -1:
line = data[:index]
self._buffer = [data[index + len(self.newline):]]
self.length = len(self._buffer[0])
return line
def iterLines(self):
line = self.getLine()
while line is not None:
yield line
line = self.getLine()它的單元測試就長這樣import unittest
class TestLineParser(unittest.TestCase):
def makeOne(self):
return LineParser()
def testParser(self):
p = self.makeOne()
p.feed('abc')
line = p.getLine()
self.assertEqual(line, None)
p.feed('rn')
line = p.getLine()
self.assertEqual(line, 'abc')
line = p.getLine()
self.assertEqual(line, None)
# write lots line
p.feed('111rn222rn3333')
lines = list(p.iterLines())
self.assertEqual(['111', '222'], lines)
line = p.getLine()
self.assertEqual(line, None)
p.feed('rntext')
line = p.getLine()
self.assertEqual(line, '3333')
line = p.getLine()
self.assertEqual(line, None)
# write nothing
p.feed('')
line = p.getLine()
self.assertEqual(line, None)
p.feed('rn')
lines = list(p.iterLines())
self.assertEqual(lines, ['text'])
self.assertEqual(list(p.iterLines()), [])
def suite():
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(TestLineParser))
return suite
if __name__ == '__main__':
unittest.main(defaultTest='suite')很簡單的想法就是列出幾種常見的case,還有你能想到的特例代進去,好的測試資料要能夠測到每一行程式,但是要做到那樣需要花不少心力,其實能夠做到大部份常見的情況和常見的特例,就已經相當足夠整合測試
有些程式,無可避免地會依賴其它程式,如果我們針對這兩個程式同時測試,會無法分出出錯到底是誰的錯,再者,很多依賴的部份可能會牽扯到IO或是其 它系統資源,讓測試變得更複雜,例如有個類別是負責輸出文件到印表機的,那你要如何確認印表機印出來的東西是正確的? 答案就是做一個假的 (Mocking)印表機丟給那個類別去做列印的動作,再去讀取裡面的資料,確認跟你預期的一樣雖然單元測試在相當單純的模擬環境下測過了我們的程式,然而世界並不是那樣的美好,總有些事情沒有經過真槍實彈操演過可能會有差錯,因此有時我們會 引入部份受控制的真實環境來測試,例如你想測試網路連線,或許你可以寫一段script在Amazon EC2上建起幾個instance,並上傳程式到那些機器中,自動讓他們連線來確保這些功能是正常的,然而越真實的環境變因就越多,因此測試也就相對困難
每日建構的好幫手 – Jenkins
Joel有說過 每日建構是你的朋友 (Daily build is your friend),也有提過 軟體開發成功的12個法則 (The Joel Test: 12 Steps to Better Code), 裡面的daily build是指利用工具每天自動建構整個專案,通常對於編譯式的語言,如C語言寫的大型專案會較需要這類的工具,但是這樣的工具還有一個目的,在於確保程 式是可以正常編譯的,並且讓測試員容易拿到最新的程式進行測試,然而自動化測試,同樣的也需要類似的工具,因為通常你在改完程式就會進行測試,那每次一改 完程式就得跑一次測試指令,這不是一件很煩的事情嗎?記得,工具是為我們服務的,不是我們為工具服務,這樣重複的瑣事理所當然也是由工具來幫忙,謝天謝地現今有好用又免費的工具,可以幫你做到這點,那就是Jenkins, 它是一套基於網頁的自動化測試管理工具,它可以做到什麼呢? 它可以做到幫你定時去版本控制系統取程式回來,用預先設定好的流程進行測試,並且記錄測試的結果,如果有某個測試出錯了,當然也可以發Email通知你, 以Now.in的開發為例,因為專案為數眾多,其中又有依賴關係,有了Jenkins的幫忙,程式只要一改送到BitBucket,它就會自動進行測試
Jenkins除了功能強大以外,他還有一項特色令我驚訝,就是非常簡單易用,從安裝到設定完所有的測試,除了clone hg檔案庫和設置測試環境以外,我從沒因為Jenkins打過一行指令,全部都可以透過它友善的網頁介面完成,同時它也有內建資料庫,也沒因此設定 MySQL,在Windows下安裝更是容易,一個安裝檔執行完就是安裝完成,如果你希望有工具幫你自動定時測試或是建構,請不要懷疑,Jenkins是 你最佳選擇
部署前的自動化測試
執行自動化測試的時機,除了剛改完程式,還有一個重要的時機,那就是在你把程式部署到伺服器以前,讓你的自動化部署的script先跑過一次自動測 試,確認測試通過了再進行部署,為什麼要這樣做呢? 還記得先前提到的PTT和Dropbox以及一些網站對於authentication的return true慘劇嗎? 為了不讓那種事情發生,或著至少讓機會降低,在deploy前讓自動化測試跑過一次,確保測試的範圍內都是正確的,可以大大降低那種情況發生的機會,除此 之外,也比較不會因為改出bug,自己沒發現,等到使用者來抱怨了才知道問題在哪測試的幾項重點
自動化測試雖然是一項利器,但是得經過正確的使用才會有好的效果,自動化測試有所謂的覆蓋率,也就是你的程式裡以行為單位,有多少行是在跑測試時有執行過的? 這些工具都可以幫你統計出來,但是切記不要為了追求高測試覆蓋率,替foo bar寫測試這只是在浪費時間,如果某段程式已經簡單到沒測試的必要,你寫了也是多餘
除此之外,寫測試事實上也是成本,因此如果時間有限,請
優先針對重要的核心、資料模型、商業邏輯測試因為就算你測再多無關緊要的程式,最重要的核心出錯了,可能整個系統就完蛋了,所以盡量以重要的程式做為測試的優先考量
優先針對安全性相關、存取權限、身份認證、常見攻擊手法測試雖然身份認證這種事情算不上是核心,但這關係到你的系統會不會被輕易地攻擊,除此之外,如果你的程式是網站應用程式,SQL Injection、XSS、buffer overflow這類攻擊也會很常發生,因此,你也需要優先自行設計一些攻擊,針對這些常見的問題餵一些資料,雖然這無法保證一定不會犯錯,至少確保不會 發生太低等級的錯誤,因為常見的case都已經有自動測試過了,搭配先前所提到的,deploy前跑過一次測試,如此一來就能將犯錯的機會降低許多
雖然你的程式可能大多都已經有自動化測試在幫你測試,但即使如此,你還是會發現新的bug,如果說,你直接改了bug,就這樣了事,很有可能在下幾次改版bug又回來了,因此
每當你發現你先前沒想到的bug,請加到你的測試中如此一來,隨著你針對的bug測試case越多,你的程式品質就越高,未發現的bug也會越少,在未來確保這些bug不會再出現
最後
再一次,自動化測試不是萬能的,除此之外也需要正確的運用,如果台灣軟體業界能夠好好運用自動化測試,軟體的品質可以有所提升,開發者也不會因為除錯除到死加班到天亮,雖然寫測試是額外的負擔,但是對於大形專案長期看來是非常值得的投資Reference:
http://blog.ez2learn.com/2011/10/20/taiwan-software-lacking-of-auto-testing/
Tuesday, April 9, 2013
寫程式是一種超能力
昨天在Facebook上很多人分享這個「大部分學校都沒教的」影片,是由一個網站code.org拍 的,找來柯林頓、比爾蓋茲、Mark Zuckburg等等一堆名人強調編寫程式的重要性。上這個網站登錄了一下,它是推廣寫程式教育的non- profit網站。我覺得這件事蠻有意義的,現在很多人還不懂寫程式在這個時代的的真正意義,寫程式這項技術其實比一般人想像中重要得多
電腦結合連結全球的網路,電腦不斷增強的計算能力和不斷下跌的價格,加上網路連結速度愈來愈快,一個人寫的程式可以瞬間成為全世界人使用的工具。網路上的 內容,經過近二十年的累積,加上過去的文件也幾乎全部數位化,讓全世界的知識都存在網路上,經過適當的處理、建立索引,讓網路成為一個全知的智者,所有人 類知識能回答的問題,幾乎全能在此找出答案。這樣的環境使得寫程式成為現代非常重要的能力
寫程式就是控制電腦照你的心意去做事的能力。我想會寫程式和不會寫程式的差別,就如同石器時代會用石器的人和不會用石器的人的差別。還有文字和紙發明後,會讀寫文字和不會讀寫文字的人的差別。學會控制電腦的能力,在這個時代的優勢非比尋常
影片中讓我印象深刻的是有一個人說coding是最接近超能力的一種能力。我想寫軟體之所以很像超能力,是在於它的可擴展性,石器時代的人就算再會用石 器,他能做到的可能只是提升到三、五倍的生產力,但是現今軟體可以跑在網路上千百萬台計算設備上,你寫一個應用在AWS上、在Google App Engine上、在IOS上、在Android上,就能跑在千百萬台資料中心的伺服器,或是跑在幾百萬人的手機上面,這種近乎無限的擴展性,讓個人能發揮 出超級英雄般的影響力。一個人寫了一個APP,在APP store上如果百萬人去下載,使用它,就能改善百萬人的生活,生產力可以到千倍萬倍,這就是超能力
軟體還有一個特性就是它是能夠一個人從頭到尾(end to end)全部做完的,因為現在有AWS、Heroku、Google App Engine這些平台,你只要有idea,有執行力,就能獨力完成一個有用的東西。如果是硬體,你得做晶片設計、得有半導體廠、有人幫你做機械設計、還有 富士康大軍幫你製造。做軟體,只要一個人就可以做出來
以前在台灣高中以前不教程式(我已經很多年不在台灣生活了,不知道現在情況如何),一直到大學才有計算機概論這種課,但是也不很強調寫程式,只寫幾個很簡 單的作業,頂多幾十行的code。當時感覺台灣的教授並不熱衷這些東西,台灣的教授大多一輩子待在學校,就發表論文、做研究而言,程式的技能並不重要,所 以理所當然他們並不重視。出國之後,就我個人的觀察,我覺得台灣學生普遍程式能力低落,尤其是跟大陸學生和東歐學生比,差距非常大。我想另外的原因是台灣 的強項是電子製造業,好的人才大部分去做晶片設計、半導體製造,較少人做軟體
如果你想自我充實程式能力,我有幾個建議
1. 線上課程
網路上很多教程式、計算機的課程,都教得非常棒。較理論學術的,像是coursera、mit opencourse,比較實際應用的像是w3schools、codecademy、code school。如果你知道還有什麼很好的課程,也跟我講一下,我也想知道
2. 線上競賽
這類網站上有很多題目,你寫程式去滿足題目的輸入輸出,上傳程式後網站會跟你說有沒有做出來,我覺得這是很棒的練習,玩起來像電動般好玩。最有名的是 topcoder,很多大公司HR專門鎖定topcoder裡的高手,我組裡的一個捷克小弟就是這樣錄取的,直接從捷克錄取來
3. open source專案
參加open source的開發,這一點比較難做到。但是你至少可以把open source project下載,自己build、跑跑unit test看看,unit test是學習程式的好方法,你先學會跑test之後,就可以改東改西,跑跑test看看有什麼不同。再下一步是去找到專案的bug tracking page,這些是使用者回報的bug,open source上的人會想辦法解決這些bug,通常他們會很歡迎有人來幫忙,你可以找個簡單的trivial的bug看能不能解決它。參加開源專案是你學習 大型程式的機會。你在線上競賽、或是上課裡學到的東西都是像玩具一樣的小程式,真正的大型專案是非常不一樣的。想找open source project,可以去Apache Software Foundation找找,裡面有很多有名的project像是httpd和Hadoop
4. 學習script、editor、version control等等周邊工具
對bash、awk、grep、sed等等工具要有些基本的了解,才能快速處理檔案。至少要精通一個好的editor,例如vim或是emacs,這類 editor用到出神入化時就如同用意志直接控制游標,看code、改code、找code十分快速。最後要會用版本控制,像是git、svn這類東西, 這是軟體開發不可缺的工具
Reference:
http://pinky-monkey.blogspot.ca/2013/02/blog-post_27.html
How to Create a Bootable Windows 7 Installation USB Flash Drive
Step 1. Format the USB Flash Disk
Step 2. Copy Windows 7 or Vista's DVD content to the Flash Disk
Still on CMD, assuming your dvd is drive d: and your usb is drive e:
xcopy d:\*.* /s/e/f e:\
You're done. Works perfectly.You'll make the fastest install ever.
Reference:
http://www.sevenforums.com/tutorials/2432-usb-windows-7-installation-key-drive-create.html
Run CMD (elevated) and type:
diskpart
list disk (*now find your USB disk number, you'll find it by its size)
select disk 1 (*if your USB is disk 1)
clean
create partition primary
select partition 1 (*this is 1, no matter what number is your USB disk)
active
format fs=fat32 (or ntfs, works with both)
assign
exit
Note: Don't type the things in parentheses above!!Step 2. Copy Windows 7 or Vista's DVD content to the Flash Disk
Still on CMD, assuming your dvd is drive d: and your usb is drive e:
xcopy d:\*.* /s/e/f e:\
You're done. Works perfectly.You'll make the fastest install ever.
Reference:
http://www.sevenforums.com/tutorials/2432-usb-windows-7-installation-key-drive-create.html
Subscribe to:
Posts (Atom)