Thursday, December 9, 2010

Apache DTrace module

The Apache DTrace module (mod_dtrace) utilizes the hook framework to add DTrace
probes to the Apache web server. These probes can be used to observe and correlate
web server and system behavior, and allow easy access to numerous pieces of realtime
Apache data. The module comes as as two files. The first file contains the C code
required to hook the module into Apache. The second file is the probe definition file.
mod_dtrace has been tested with the Apache source distribution (2.0.55) and the
version of Apache that resides in /usr/apache2 on Solaris 10 servers. If you have
any comments, suggestions, scripts or feedback, shoot me an
E-mail.

Code and build instructions



The Apache DTrace module and a set of build instructions can be downloaded from the following site:



Scripts



The scripts on this page were created to monitor a running Apache process, and to
provide some insight into the Apache request processing flow. The results in some
cases may be slightly skewed, but the overall results should help web administrators
understand what is happening with an Apache web server.

*** NOTE***

Due to some issues with the DTrace "#include" support, these scripts currently
only work under SPARC Solaris (you can adjust the offsets in the scripts if
you are interested in using them under x86 or x64 Solaris)

View HTTP Requests



$ viewreqs.pl -d 5

Timestamp GETs POSTs HEADs TRACEs
02:05:59  0    0     0     0
02:06:04  12   0     0     0
02:06:09  15   0     0     0
02:06:14  59   0     0     0
02:06:19  68   0     0     0
02:06:24  70   0     0     0

View HTTP Requests By HTTPD Process




$ viewreqsbyproc.d

Execname  Process  Requests Processed
httpd     19103       16
httpd     19152       17
httpd     19101       17
httpd     19102       19
httpd     19104       20
httpd     19105       20

View HTTP Requests By Browser Type




$ viewbrowsers.pl 

Sampling browser data for 60 seconds...
Browser     GETs  POSTs  HEADs TRACEs
Firefox     34    0      0      0 
Safari      20    0      0      0 
Wget        7     0      0      0 


View HTTP Requests By Virtual Server




$ viewserver.pl -d 60

Sampling virtual server data for 60 seconds...
Server                     GETs  POSTs  HEADs  TRACEs
www.foo.com                101   0      0      0     
www.bar.com                23    0      0      0


View HTTP Response Codes




$ responsecodes.d
Sampling response code data (Ctrl-C to stop):
Response Code  Count
404            4                  
200            22


View HTTP Response Codes by file and URI




$ responsecodes.pl        
Response Code  URI                        File Name                
200            /                          /tmp/apache/htdocs/index.html.var
304            /apache_pb.gif             /tmp/apache/htdocs/apache_pb.gif
200            /                          /tmp/apache/htdocs/index.html.var
200            /apache_pb.gif             /tmp/apache/htdocs/apache_pb.gif
404            /favicon.ico               /tmp/apache/htdocs/favicon.ico


$ responsecodes.pl  -c 404
Response Code  URI                        File Name                
404            /favicon.ico               /tmp/apache/htdocs/favicon.ico
404            /favicon.ico               /tmp/apache/htdocs/favicon.ico
404            /favicon.ico               /tmp/apache/htdocs/favicon.ico
404            /favicon.ico               /tmp/apache/htdocs/favicon.ico


View HTTP 301/302 redirects




$ viewredirects.pl
Response Code  URI Requested              Redirect Sent To Client
301            /yikes                     http://192.168.1.3:8080/yikes2 
301            /foo                       http://192.168.1.3:8080/bar 
301            /bling                     http://192.168.1.3:8080/crunk


View request sizes in a quantized distribution




$ requestsize.d
  dtrace: script 'requestsize.d' matched 1 probe

  httpd                                                  5549
           value  ------------- Distribution ------------- count
              -1 |                                         0
               0 |@@@@@@@@@@@@@@@@@@@@@@@@                 43
               1 |                                         0
               2 |                                         0
               4 |                                         0
               8 |                                         0
              16 |                                         0
              32 |                                         0
              64 |                                         0
             128 |                                         0
             256 |@@@@                                     8
             512 |                                         0
            1024 |                                         0
            2048 |@@@@@@@@@@@@                             22
            4096 |                                         0

View requests in a top-like display (similar to the apachetop C program)




$ apachetop.pl

 21:12:39    Requests:    347 (  69/sec)    Bytes:  35349b (7069b/sec,  101b/request)
 Requests:   GETs:  347    POSTs:    0    HEADs:    0    TRACEs:    0
 Responses:   1XX:    0      2XX:  301      3XX:   46       4XX:    0      5XX:    0

 Requests   Requests/Sec   Bytes Sent    URI
       13            2.6        13312    /blah
        6            1.2            0    /manual/ja/mod/directives.html
        6            1.2            0    /manual/ru/mod/directives.html
        6            1.2            0    /manual/de/mod/directives.html
        6            1.2            0    /manual/en/mod/directives.html
        6            1.2            0    /manual/es/mod/directives.html
        6            1.2            0    /manual/ko/mod/directives.html
        2            0.4            0    /manual/ja/mod/
        2            0.4            0    /manual/es/mod/
        2            0.4            0    /manual/de/mod/
        2            0.4            0    /manual/ko/mod/
        2            0.4            0    /manual/en/mod/
        1            0.2            0    /manual/ja/mod/mod_negotiation.html
        1            0.2            0    /manual/ja/mod/mod_isapi.html
        1            0.2            0    /manual/ko/mod/mod_file_cache.html
        1            0.2          344    /manual/ko/ru/mod/directives.html
        1            0.2            0    /manual/ja/mod/mod_asis.html
        1            0.2            0    /manual/es/mod/mod_autoindex.html
        1            0.2            0    /manual/ru/mod/beos.html
        1            0.2            0    /manual/ko/mod/mod_setenvif.html


View Apache module function processing time




$ apachemodulefunctime `pgrep httpd` mod_alias.so
Tracing Apache Module mod_alias.so (Control-C to stop)
Module                Function                        Time      
mod_alias.so          alias_matches                   558679
mod_alias.so          fixup_redir                     623683
mod_alias.so          translate_alias_redir           938919
mod_alias.so          try_alias_list                  1169778

Total Time:           3291059


View Apache module processing time




$ apachemoduleprof `pgrep httpd`
Started Apache Request Profiler (Control-C to stop...)


            *** New connection ***

Connection start: 2005 Dec 10 19:56:39
Connection stop : 2005 Dec 10 19:56:54

Method      URI                                       Count
GET         /apache_pb.gif                            1
GET         /                                         1

Module                     Processing Time
mod_actions.so                  43093 ns
mod_expires.so                  66941 ns
mod_dir.so                      92026 ns
mod_speling.so                 100357 ns
mod_log_forensic.so            102187 ns
mod_userdir.so                 102598 ns
mod_vhost_alias.so             104377 ns
mod_info.so                    108378 ns
mod_autoindex.so               110851 ns
mod_usertrack.so               117790 ns
mod_asis.so                    134288 ns
mod_cern_meta.so               138016 ns
mod_cgi.so                     138344 ns
mod_imap.so                    142541 ns
mod_headers.so                 189257 ns
mod_status.so                  205965 ns
mod_env.so                     206865 ns
mod_dav.so                     240208 ns
mod_access.so                  245391 ns
mod_unique_id.so               321004 ns
mod_negotiation.so             331457 ns
mod_auth_digest.so             333495 ns
mod_dtrace.so                  467512 ns
mod_rewrite.so                 527799 ns
mod_include.so                 874804 ns
mod_alias.so                  1886377 ns
mod_log_config.so             3044315 ns
mod_mime.so                   3684061 ns
mod_setenvif.so               4318186 ns
httpd                        22451784 ns
Total                              40 ms

Hook Name                     Module            Hook Function                   Time 
ap_run_pre_connection         mod_dtrace.so     apache_pre_connection           24398 ns
ap_run_create_connection      mod_dtrace.so     apache_create_connection        25998 ns
ap_run_handler                mod_actions.so    action_handler                  43093 ns
ap_run_log_transaction        mod_log_forensic.so  log_after                       43118 ns
ap_run_access_checker         httpd             do_nothing                      46784 ns
ap_run_header_parser          mod_setenvif.so   match_headers                   57872 ns
ap_run_post_read_request      mod_log_forensic.so  log_before                      59069 ns
ap_run_insert_filter          httpd             core_insert_filter              61771 ns
ap_run_post_read_request      mod_dtrace.so     apache_receive_request          66115 ns
ap_run_access_checker         mod_dtrace.so     apache_check_access             66201 ns
ap_run_insert_filter          mod_expires.so    expires_insert_filter           66941 ns
ap_run_insert_filter          mod_headers.so    ap_headers_insert_output_filter  78055 ns
ap_run_handler                mod_negotiation.so  handle_map_file                 90869 ns
ap_run_fixups                 mod_dir.so        fixup_dir                       92026 ns
ap_run_handler                mod_dav.so        dav_handler                     96851 ns
ap_run_handler                mod_rewrite.so    handler_redirect                98021 ns
ap_run_fixups                 mod_speling.so    check_speling                   100357 ns
ap_run_translate_name         mod_userdir.so    translate_userdir               102598 ns
ap_run_translate_name         mod_vhost_alias.so  mva_translate                   104377 ns
ap_run_translate_name         mod_rewrite.so    hook_uri2file                   106062 ns
ap_run_handler                mod_info.so       display_info                    108378 ns
ap_run_fixups                 mod_negotiation.so  fix_encoding                    109785 ns
ap_run_handler                mod_autoindex.so  handle_autoindex                110851 ns
ap_run_fixups                 mod_headers.so    ap_headers_fixup                111202 ns
ap_run_fixups                 mod_rewrite.so    hook_fixup                      111632 ns
ap_run_fixups                 mod_usertrack.so  spot_cookie                     117790 ns
ap_run_fixups                 mod_auth_digest.so  add_auth_info                   118533 ns
ap_run_fixups                 httpd             core_override_type              124018 ns
ap_run_map_to_storage         httpd             ap_send_http_trace              129160 ns
ap_run_type_checker           mod_negotiation.so  handle_multi                    130803 ns
ap_run_handler                mod_asis.so       asis_handler                    134288 ns
ap_run_log_transaction        mod_dtrace.so     apache_log_request              134483 ns
ap_run_fixups                 mod_cern_meta.so  add_cern_meta_data              138016 ns
ap_run_handler                mod_cgi.so        cgi_handler                     138344 ns
ap_run_handler                mod_imap.so       imap_handler                    142541 ns
ap_run_fixups                 mod_dav.so        dav_fixups                      143357 ns
ap_run_create_request         mod_dtrace.so     apache_create_request           150317 ns
ap_run_pre_connection         httpd             core_pre_connection             153533 ns
ap_run_fixups                 mod_alias.so      fixup_redir                     199353 ns
ap_run_handler                mod_status.so     status_handler                  205965 ns
ap_run_fixups                 mod_env.so        fixup_env_module                206865 ns
ap_run_type_checker           mod_rewrite.so    hook_mimetype                   212084 ns
ap_run_post_read_request      mod_auth_digest.so  parse_hdr_and_update_nc         214962 ns
ap_run_access_checker         mod_access.so     check_dir_access                245391 ns
ap_run_post_read_request      mod_unique_id.so  gen_unique_id                   321004 ns
ap_run_create_request         httpd             http_create_request             505504 ns
ap_run_translate_name         httpd             ap_core_translate               542836 ns
ap_run_create_request         httpd             core_create_req                 728807 ns
ap_run_create_connection      httpd             core_create_conn                874776 ns
ap_run_fixups                 mod_include.so    include_fixup                   874804 ns
ap_run_translate_name         mod_alias.so      translate_alias_redir           1687024 ns
ap_run_log_transaction        mod_log_config.so  multi_log_transaction           3044315 ns
ap_run_type_checker           mod_mime.so       find_ct                         3684061 ns
ap_run_post_read_request      mod_setenvif.so   match_headers                   4260314 ns
ap_run_map_to_storage         httpd             core_map_to_storage             8709887 ns
ap_run_handler                httpd             default_handler                 10574708 ns

http://prefetch.net/projects/apache_modtrace/index.html

No comments: