Friday, May 28, 2010
SplFastArray to speed up your PHP arrays
The Standard PHP Library was recently completed by a couple of data structures, namely heaps and doubly linked lists. You can find more information about those on
php.net/spl.datastructures
Reference:
http://www.colder.ch/news/06-07-2008/33/splfastarray-to-speed-up-.html
http://php.net/spl.datastructures
Preferred method to store PHP arrays (json_encode vs serialize)
Would it be more efficient to store the array as JSON or as a PHP serialized array in this text file? I've looked around and it seems that in the newest versions of PHP (5.3), json_decode is actually faster than unserialize.
I'm currently leaning towards storing the array as JSON as I feel its easier to read by a human if necessary, it can be used in both PHP and JavaScript with very little effort, and from what I've read, it might even be faster to decode (not sure about encoding, though).
Does anyone know of any pitfalls? Anyone have good benchmarks to show the performance benefits of either method?
Thanks in advance for any assistance.
php performance arrays json serialization
flag
asked Apr 29 '09 at 20:09
KyleFarris
2,184412
8 Answers
oldest newest votes
17
Depends on your priorities.
If performance is you absolute driving characteristic, then by all means use the fastest one. Just make sure you have a full understanding of the differences before you make a choice
* JSON converts UTF-8 characters to unicode escape sequences. serialize() does not.
* JSON will have no memory of what the object's original class was (they are always restored as instances of stdClass).
* You can't leverage __sleep() and __wakeup() with JSON
* Only public properties are serialized with JSON
* JSON is more portable
And there's probably a few other differences I can't think of at the moment.
EDIT
A simple speed test to compare the two
<?php ini_set( 'display_errors', 1 ); error_reporting( E_ALL ); // Make a bit, honkin test array // You may need to adjust this depth to avoid memory limit errors $testArray = fillArray( 0, 5 ); // Time json encoding $start = microtime( true ); json_encode( $testArray ); $jsonTime = microtime( true ) - $start; echo "JSON encoded in $jsonTime seconds "; // Time serialization $start = microtime( true ); serialize( $testArray ); $serializeTime = microtime( true ) - $start; echo "PHP serialized in $serializeTime seconds "; // Compare them if ( $jsonTime < $serializeTime ) { echo "json_encode() was roughly " . number_format( ($serializeTime / $jsonTime - 1 ) * 100, 2 ) . "% faster than serialize()"; } else if ( $serializeTime < $jsonTime ) { echo "serialize() was roughly " . number_format( ($jsonTime / $serializeTime - 1 ) * 100, 2 ) . "% faster than json_encode()"; } else { echo 'Unpossible!'; } function fillArray( $depth, $max ) { static $seed; if ( is_null( $seed ) ) { $seed = array( 'a', 2, 'c', 4, 'e', 6, 'g', 8, 'i', 10 ); } if ( $depth < $max ) { $node = array(); foreach ( $seed as $key ) { $node[$key] = fillArray( $depth + 1, $max ); } return $node; } return 'empty'; } ?>
link|flag
edited Apr 29 '09 at 21:30
answered Apr 29 '09 at 20:21
Peter Bailey
20.1k11643
You make some great points. Fortunately, for my case, I'm storing simple arrays (of other arrays, ints, bools, and strings) no objects. If I were storing objects, IMO, serialize would be the way to go. – KyleFarris Apr 29 '09 at 20:32
1
Excellent work dude. This will benefit everyone. I ran the test about 30 times and json_encode won every single time with around 100% (average) performance increase over serialize. I added json_decode and unserialize tests and json_decode won everytime in about 10 tests with an average performance benefit of ~20% over unserialize. Thanks for this. – KyleFarris Apr 30 '09 at 13:25
Too bad json_encode/json_decode is php 5.2 and above only. disgusts_uncover_akin_umbriel – dreftymac Feb 23 at 1:28
12
JSON is simpler and faster than PHP's serialization format and should be used unless:
* You're storing deeply nested arrays: json_decode(): "This function will return false if the JSON encoded data is deeper than 127 elements."
* You're storing objects that need to be unserialized as the correct class
* You're interacting with old PHP versions that don't support json_decode
link|flag
answered Apr 29 '09 at 20:13
Greg
48.1k45284
Great answer. Haha, 127 levels deep seems a bit insane; thankfully I'm only going like 2-3 at the most. Do you have any data to back up the fact that json_decode/json_encode is faster than unserialize/serialize? – KyleFarris Apr 29 '09 at 20:34
I did test it a while ago and json came out faster - I don't have the data any more though. – Greg Apr 29 '09 at 21:02
@Kyle - I added a speed test to my answer. On my server, json_encode() is averaging about 100% faster that serialize() – Peter Bailey Apr 29 '09 at 21:32
5
I've written a blogpost about this subject: "Cache a large array: JSON, serialize or var_export?". In this post it is shown that serialize is the best choice for small to large sized arrays. For very large arrays (> 70MB) JSON is the better choice.
link|flag
answered Jul 7 '09 at 14:20
Takkie
5111
Very cool man. Thanks. ;-) – KyleFarris Jul 8 '09 at 21:07
2
If you are caching information that you will ultimately want to "include" at a later point in time, you may want to try using var_export. That way you only take the hit in the "serialize" and not in the "unserialize".
link|flag
answered Apr 29 '09 at 23:04
Jordan S. Jones
3,192513
2
I augmented the test to include unserialization performance. Here are the numbers I got.
Serialize
JSON encoded in 2.5738489627838 seconds
PHP serialized in 5.2861361503601 seconds
Serialize: json_encode() was roughly 105.38% faster than serialize()
Unserialize
JSON decode in 10.915472984314 seconds
PHP unserialized in 7.6223039627075 seconds
Unserialize: unserialize() was roughly 43.20% faster than json_decode()
So json seems to be faster for encoding but slow in decoding. So it could depend upon your application and what you expect to do the most.
link|flag
answered Nov 23 '09 at 19:14
Jeff Whiting
211
0
This is really awsome. and I think json_encode /json_decode is absouletly superb for me!
link|flag
answered Oct 21 '09 at 3:08
arunoda
3
0
Before you make your final decision, be aware that the JSON format is not safe for associative arrays - json_decode() will return them as objects instead:
<?php $config = array( 'Frodo' => 'hobbit', 'Gimli' => 'dwarf', 'Gandalf' => 'wizard', ); print_r($config); print_r(json_decode(json_encode($config))); ?>Output is:
<?php Array ( [Frodo] => hobbit [Gimli] => dwarf [Gandalf] => wizard ) stdClass Object ( [Frodo] => hobbit [Gimli] => dwarf [Gandalf] => wizard ) ?>link|flag
answered Oct 21 '09 at 3:34
too much php
8,63431938
Indeed, you are right. I mean, it is Javascript object notation afterall! Thankfully, if you know that what you encoded using json_encode was an associative array, you can easily force it back into an array like so: $json = json_encode($some_assoc_array); $back_to_array = (array)json_decode($json); Also it's good to note that you can access objects the same way as arrays in PHP so in a typical scenario, one wouldn't even know the difference. Good point though! – KyleFarris Dec 7 '09 at 21:10
1
@toomuchphp, sorry but you are wrong. There is a second parameter for json_decode 'bool $assoc = false' that makes json_decode produce an array. @KyleFarris, this should also be faster than using the typecast to array. – Jan P. Jan 20 at 14:30
@Jan thanks for the correction – too much php Jan 21 at 1:31
0
Thanks all for comparing two method :)
usually, i use json just for ajax, and serialize for storing in database .... regards
Quicker Than window.onload
Quicker Than window.onload
21
May
Posted by Jack as Tutorials
Window.onload() is a workhorse of traditional javascript code. It’s been used by coders for years to kick start the client side magic as soon as everything on the page loads up.
But sometimes waiting for a page to load just isn’t quick enough.
A few large image files will quickly reveal that window.onload() can be painfully slow. So when I was creating a web app for internet marketers recently, I had to have something faster.
Some quick research into possible workarounds for the window.onload() issue brought me to some code by Brother Cake. If all you need is a fast way to kick start your javascript then their code might be something to try.
But if you’re going to be doing some DOM (Document Object Model) javascript coding then why not use jQuery and have your cake and eat it too (horrible pun - sorry).
jQuery has a handy little function that launches your javascript as soon as the Document Object Model is ready… which happens before the page has finished loading.
$(document).ready(function(){ // Your code here... });
You can use it to launch any kind of javascript you like. It doesn’t have to be reserved for jQuery style coding. And there’s nothing wrong with telling jQuery to launch several different functions at once.
Similar to many init() functions you may have seen before… just a lot faster.
You’ll see this code used again and again in the examples I give you on 15 Days of jQuery.
[tags]jQuery, DOM, javascript, window.onload[/tags]
67 Responses
wesley
1
Too bad that there’s all sorts of issues with it:
http://jquery.com/discuss/2006-May/001280/
Jack
2
Wesley,
From what I’ve read, it seems to be limited to situations where there is a document.write in the code.
Joel Birch
3
Yes, I’ve mentioned the $(document).ready() shortcomings before myself, but jQuery is really young - these sorts of teething problems are being fixed at an amazing rate. $(document).ready() is still my first choice when initialising scripts that need to fire as early as possible. When it works (which usually means when other scripts are not interfering) its an absolute gem.
4
The YUI equivalent of this would be YAHOO.util.Event.onAvailable which acts just like an eventListener and will fire upon ‘load’ of whatever HTMLElement that you pass in.
Cheers
Jack
5
Dustin,
As you know from my email to you, I’m a fan of yours. Great to have you stop by!
6
> From what I’ve read, it seems to be limited to situations
> where there is a document.write in the code.There are more problems than that. The back button on IE causes the page to render in a slightly different way. This often breaks timer based detects.
Jack
7
Dean,
First of all, great to have you stop by.I wasn’t aware of this issue. Although I haven’t noticed it myself, I’ll take you at your word.
Do you know if the other methods of triggering quick javascript execution before window load (Brother Cake, YUI, etc.) have the same issue(s)?
Jack
8
Dean has supplied me with two links to his site where he has code that may work for those of you that want something other than jQuery’s solution:
http://dean.edwards.name/weblog/2005/09/busted/
http://dean.edwards.name/weblog/2005/09/busted2/According to Dean, the Brother Cake solution as well as the YUI one (and I suppose all other solutions) are not as relaible as the ones he has come up with.
9
The YUI one is OK because it uses
document.getElementById
to check for the existence of elements, so it is not really dependant on a complete DOM. The downside is that you have to assign an ID on every element you want to address.
10
While jQuery’s $(document).ready(); seems to be working excellently in Firefox and IE (it’s way better than the onload), it miserably fails in Opera9 resulting in a crash of my layout. You can check the code in my website actually. I’ll be using the onload one till that gets sorted out because I main test on Opera :/
Jack
11
Hercules,
Have you tried the latest release?
http://jquery.com/src/jquery-1.0a.jsJohn announced it on his blog just a few days ago.
http://jquery.com/blog/2006/06/30/jquery-10-alpha-release/And according to John, one of the improvements is in the (document).ready function:
$(document).ready() has been drastically improved. It now works in all modern browsers completely as you’d expect, even with Adsense in the page.
I have never been able to replicate the problems reported with document.ready so I can’t really attest to the improvement… but looking at the code leads me to believe that it uses the typical window.onload as a backup “Plan B”.
12
I tried what you suggested but it now behaves as the plain old onload, even in FF, which was the only one of the big 3 handling the .ready as it should… I guess I;ll just wait for the finalized release.
13
The YUI one is OK because it uses document.getElementById to check for the existence of elements, so it is not really dependant on a complete DOM. The downside is that you have to assign an ID on every element you want to address.
14
Is this different to just using
$(function() { // onload scripts });
15
Wesley,
From what I’ve read, it seems to be limited to situations where there is a document.write in the code.
16
good article
17
Great post. You explain the issues and solution clearly. YUI does work reasonably well, but in my experience they can be a little pushy in trying to get programmers to use it. Their basic approach seems to be that anything that makes your page faster trumps all other considerations. In contrast, I would suggest that content counts for a lot and all the speed in the world can’t overcome a lousy looking or functioning page. Additionally, this seems like a simpler, more elegant solution, even if it is still in the process of working the kinks out. And after all, what are a few kinks among friends, right? After all, it isn’t like there aren’t bugs in virtually every piece of software out there.
18
Great article. However, I still seem to be having problems in IE6 + IE7 as it somethings does not .hide() elements at all. But when I refresh the page it all works fine. Strange!?
19
Our problem with Window.onload() is with images. Since our site includes a lot of images, I noticed a lag before the page becomes active. What we want is a way to determine when the DOM has fully loaded without waiting for all those pesky images to load also. System is OK with IE but we’re having problems with FF.
September 11th, 2007 at 11:10 pm
20
Maybe its only matter of hardware not software .Maybe when we gonna use p9 47Ghz 1600 gxz 128000ram then its gonna be flawless like a ….or hmmm
Lorenzo
September 12th, 2007 at 4:52 pm
21
Can you please put all the html code in this page so I can copy-paste it and try jquery? This is for extremelly dummies like me? Thanks
September 25th, 2007 at 8:58 am
22
On Ie it works great! Thanks!
Jules
September 30th, 2007 at 12:04 am
23
WHy is this a whole day? You can market your site better if you say 15 minutes of jquery. Nobody wants to spend 15 days learning a javascript library.
Jack
24
It’s a takeoff on another site, and the concept is/was to have one tutorial a day. Not that it should take 15 days to wrap your head around jquery, or any other js library for that matter.
December 19th, 2007 at 10:00 pm
25
Great article. However, I still seem to be having problems in IE6 + IE7 as it somethings does not .hide() elements at all.
January 31st, 2008 at 11:28 am
26
Very nice, This came in very handy with speeding up a site I’m working on.
27
I use jQuery in my project and Im very impressed of its abilities. I recommend jQuery for all webdevelopers.
greetings,
agencja modelek
28
How does the jQuery $(document).ready() compare to MooTools’ ‘domready’ event, or Prototype ‘dom:loaded’ ?
Brian
29
can you use $(document).ready() on a page that has a body.onload function call?
I’m in a situation where I cant remove the old JS that has an onload on the body element… from my reading the JQuery ready event gets fired before the old JS would..
30
I’m hoping that the new browsers out there (FireFox 3, IE8, Opera 10 when it happens etc.) will put an end to some of the loading times and somehow implement this sort of coding on websites themselves. I have a feeling that regardless of what we do with JQUERY, we’ll still find that IE7 or IE6 (which, apparently, most people still run on which is ridiculous) might still just put it’s habits on the page – meaning that it’s pretty useless at the end of the day until the developers for the browsers do something instead.
NETTUTS - 15 Resources To Get You Started With jQuery From Scratch
31
[...] Visit Article [...]
15 Resources To Get You Started With jQuery From Scratch | Mexzhouse Design Studio
32
[...] Visit Article [...]
Jquery For Dunmmies | mashup design | pisake | web design | css | playground
33
[...] Visit Article [...]
“The Complete Guide” for jQuery Developer- Reblog « Dynamic Disruption
34
[...] Quicker Than Window.Onload()Load your Javascript before every element of the page is loaded. [...]
35
u can make ur jqurry n other stuf in html readymade here http://66lives.ueuo.com
November 18th, 2008 at 10:17 pm
36
good article!
Michael Everitt
December 15th, 2008 at 1:10 am
37
Thanks for the article. I’m just learning jQuery and to tell you the truth 15 days of jQuery is plenty fast for me. Posts like yours sure help.
38
good!
39
On Ie it works great! Thanks!
jQuery Ö?retecek 15 Kaynak | MuGi Graphic System
40
[...] 15DaysOfJquery.com: 1. Gün: jQuery’nin tüm özelliklerini iki haftada kavramak gibi cesur bir giri?imde bulunan “15 Days of jQuery” (jQuery ile 15 Gün) web sitesi her gün bir konuyu i?liyor. Bu makalede “document.ready” yöntemini nas?l kullanman?z gerekti?ini ö?reneceksiniz. [...]
pozycjonowanie i optymalizacja
January 17th, 2009 at 12:06 pm
41
Wielu twórców stron zastanawia si? nad tym, w jaki sposób tworzy? witryny przyjazne wyszukiwarkom takim jak Google. Które rzeczy s? wa?ne z punktu widzenia indeksowania strony internetowej, a które s? nieistotne
Serge
42
When DOM loads that doesn’t mean that your images are loaded. I wouldn’t use ready() even on a site with small graphics. If your end-user is on a slow connection they can get an JS error while interacting with the UI on the site before anything even loads. Be careful with it. I think ready() would be more applicable for intranet stuff.
43
Its really good for learner for basic of the Jquery. Thanks!!
February 5th, 2009 at 10:07 am
44
[...] 15DaysOfJquery.com: 1. Gün: jQuery’nin tüm özelliklerini iki haftada kavramak gibi cesur bir giri?imde bulunan “15 Days of jQuery” (jQuery ile 15 Gün) web sitesi her gün bir konuyu i?liyor. Bu makalede “document.ready” yöntemini nas?l kullanman?z gerekti?ini ö?reneceksiniz. [...]
LOL
45
> WHy is this a whole day? You can market your site better if you say 15 minutes of jquery. Nobody wants to spend 15 days learning a javascript library.
Oh man, I have this idea for 6 minute abs. It will blow 7 minute abs out of the water!
Maarten
46
Better is not to use this function at all! If you place the jquery script at the end of your html (before you close your body) it will fire even faster as document ready (since that one waits for example loading of external banners or content iframe. It then also follows YIU guidelines!! I declare jquery in the head and then use all dom events in one file which i place below all html. Works faster in all occasions!!
sundowach
47
That JQuery is very good library.
meneye
48
Great article.thanks.
0 ( s?f?rdan ) Jquery Ö?reten 15 Kaynak | ONLINE SCRIPT YUVANIZ - ASP,PHP,C,JS,FLASH,AJAX..
49
[...] 15DaysOfJquery.com: 1. Gün: jQuery’nin tüm özelliklerini iki haftada kavramak gibi cesur bir giri?imde bulunan “15 Days of jQuery” (jQuery ile 15 Gün) web sitesi her gün bir konuyu i?liyor. Bu makalede “document.ready” yöntemini nas?l kullanman?z gerekti?ini ö?reneceksiniz. [...]
50
Starting with Jquery and now i know a little bit more about the difference between document ready and window.onload
51
This is the first JQuery article I’ve read. Normally I use Prototype or Dojo but I’ve been seeing way to much about JQuery everywhere to not learn it. Thanks and I’m looking forward to the next 14 articles.
52
Jquery with Opera was always a bit problematic.
53
Quicker Than window.onload…
Window.onload() is a workhorse of traditional javascript code. It’s been used by coders for years to kick start the client side magic as soon as everything on the page loads up.
But sometimes waiting for a page to load just isn’t quick enough.
A …
54
@Cody Taylor: I’ve tried both of those and JQuery is by far the more intuitive framework to use. I really like how you can chain function together too, really useful.
September 23rd, 2009 at 12:09 pm
55
This works for me:
$(document).ready( function () {
alert(”test”):
});
56
I like Jqueries.
Bob
57
Arjun, why would you do that if you can do: $(function(){alert(’h');});
jack
58
when adding comments USE ENGLISH!!!! don’t just put unrelated crap so you can get a link to you pathetic website you dumb spammers!
jack
59
sorry, just to make it clear…, last comment above is not from Jack the author of the article we are two different people
60
I have used CrossSlide Jquery plugin in my website and my website is taking too long to load.Please tell me the solution.
61
nice article! short , but good
62
I just learn Jquery and I am still confuse when I have many jquery effects on multiple pages.
So far, I always put
$(document).ready(function(){
// Your code here…
});on every pages… although I have a single template for my web, but you know, every pages has different effect…
Do you know the best solution to manage multiple effect of Jquery on multiple pages?
thanks in advance….
Kody
63
Hi! I have some problem with Jquery and body onLoad… how can i do something like call a function in the body onLoad of the htm page, wich pass a parameter too, and then in the js file assign that param to a div.
Something like:page.php
———–
<body onLoad=”updateArea(”);”funcs.js
———–
$(document).ready(function(){updateArea(area) { $(’#mydiv’).html(area); }
});
Obviusly that example doesn’t work cause it’s wrong.
Thanks in advance.
64
[...] can execute a bit faster when compared to using the traditional window.onload method. More on that here and [...]
Pattern Inc » Learning jQuery – Tutorial Roundup for Beginners
65
[...] 15 Days Of jQuery [...]
Learning jQuery – Tutorial Roundup for Beginners | Duong Luat
66
[...] 15 Days Of jQuery [...]
Homeschool
67
@Asmita make sure you use
$(window).load(function() { #slider.crossSlide(); });
to make sure all of the images are loaded. $(document).ready() won’t wait for the images to load.
I suggest Nivo Slider (google it)
Thursday, May 27, 2010
How do I place a Drupal block's contents into a node?
If we wanted to place this custom block into a page, we would simply substitute these values into our PHP code snippet (which we place in a node (page or other content type)) as follows:
<?php $block = module_invoke('block', 'block', 'view', 26); print $block['content']; ?>
Notice that the middle two terms ('block', 'view') always stay the same. You're simply updating the first and last term.
Voila
Monday, May 24, 2010
jQuery Note
<?php // Scroll to the bottom of div $("#div-id").scrollTop($("#div-id").get(0).scrollHeight); // set a value to a form element $("#element-id").val("TEST"); // set a value to a item $("#item-id").html("TEST"); // ########################################## // This should do the trick if the ID of your form is myform: $(':input','#myform') .not(':button, :submit, :reset, :hidden') .val('') .removeAttr('checked') .removeAttr('selected'); // It is using the :input selector which will match all input, textarea, select and button elements. Since we are passing #myform as the second argument, it will only find inputs inside this form element. Then it filters out all buttons, submits, resets and hidden inputs using not(). Then it is using val() to set the value of the remaining fields to an empty string, and then it uses removeAttr to remove the checked and selected attribute of the fields in case you have any radio/checkbox/select inputs. function clear_form_elements(ele) { $(ele).find(':input').each(function() { switch(this.type) { case 'password': case 'select-multiple': case 'select-one': case 'text': case 'textarea': $(this).val(''); break; case 'checkbox': case 'radio': this.checked = false; } }); } // ########################################## // From http://groups.google.com/group/jquery-dev/msg/2e0b7435a864beea: $('#myform')[0].reset(); // setting .val('') might not emulate "reset" 100% if you have an input like this: // <input name="percent" value="50"/> // Eg calling .val() on an input with a default value of 50 would set it to an empty string, whereas calling reset() would reset it to its initial value of 50. // ########################################## // Here is something to get you started $('form') // match your correct form .find('input[type!=submit], input[type!=reset]') // don't reset submit or reset .val(''); // set their value to blank // Of course, if you have checkboxes/radio buttons, you'll need to modify this to include them as well and set .attr({'checked': false}); // edit Paolo's answer is more concise. My answer is more wordy because I did not know about the :input selector, nor did I think about simply removing the checked attribute. // ########################################## // Clearing forms is a bit tricky and not as simple as it looks. // Suggest you use the jQuery form plugin and use its clearForm or resetForm functionality. It takes care of most of the corner cases. // ########################################## // I usually do: $('#formDiv form').get(0).reset() // or $('#formId').get(0).reset() ?>
Reference: http://stackoverflow.com/questions/680241/reset-form-with-jquery
Friday, May 21, 2010
INSERT ON DUPLICATE KEY UPDATE and summary counters
INSERT ... ON DUPLICATE KEY UPDATE is very powerful but often forgotten MySQL feature. It was introduced in MySQL 4.1 but I still constantly see people unaware of it.
Myself I like this feature big deal because it is designed in truly MySQL style - very efficient solution for freqent task while keeping it beautiful and easy to use.
So what is this feature great for ? Well any kind of maintaining counters. If you're writing traffic accounting it could be traffic and number of packet passed for given port or IP address. For Web Applications it could be counting number of visits per page or IP address, number of times particular keyword was searched etc.
This functionality also makes it very easy to do incremental single pass log file processing and building summary tables.
Here is example:
CREATE TABLE ipstat(ip int UNSIGNED NOT NULL PRIMARY KEY,
hits int UNSIGNED NOT NULL,
last_hit timestamp);
INSERT INTO ipstat VALUES(inet_aton('192.168.0.1'),1,now())
ON duplicate KEY UPDATE hits=hits+1;
The third feature this example takes advantage of is TIMESTAMP field. By default first TIMESTAMP column will have its value automatically updated to current timestamp on insert and update. We actually could have omitted now() in insert clause but this would require to specify list of columns which we skipped for sake of example.
So how would this example work ? Well just you would expect it. If there is no such IP address in the table it will be added with hits=1 if it is already where (note ip is PRIMARY KEY) it would be just incremented and last visit timestamp updated.
The benefit of using this feature insted of INSERT + UPDATE could be different, depending on number of new rows and data set size. 30% speedup should be typical. Performance increase is not the only benefit - what is even more important the application code becomes simplier - less error prone and easy to read.
Thursday, May 20, 2010
Top 10 Concepts That Every Software Engineer Should Know
is about good craftsmen. With infrastructure like Amazon Web Services and an abundance of basic libraries, it no longer takes a village to build a good piece of software. These days, a couple of engineers can deliver complete systems.
These days, a couple of engineers that know what they are doing can deliver complete systems. In this post, we discuss the top 10 concepts software engineers should know to achieve that.
A successful software engineer knows and uses design patterns, actively refactors code, writes unit tests and religiously seeks simplicity. Beyond the basic methods, there are concepts that good software engineers know about. These transcend programming languages and projects – they are not design patterns, but rather broad areas that you need to be familiar with. The top 10 concepts are:
- Interfaces
- Conventions and Templates
- Layering
- Algorithmic Complexity
- Hashing
- Caching
- Concurrency
- Cloud Computing
- Security
- Relational Databases
10. Relational Databases
Relational Databases have
recently been getting a bad name because they cannot scale well to support massive web services. Yet this was one of the most fundamental achievements in computing that has carried us for two decades and will remain for a long time. Relational databases are excellent for order management systems, corporate databases and P&L data.
At the core of the relational database is the concept of representing information in records. Each record is added to a table, which defines the type of information. The database offers a way to search the records using a query language, nowadays SQL. The database offers a way to correlate information from multiple tables.
The technique of data normalization is about correct ways of partitioning
the data among tables to minimize data redundancy and maximize the speed of retrieval.
9. Security
With the rise of hacking and data sensitivity, the security is paramount. Security is a broad topic that includes authentication,
authorization, and information transmission.
Authentication is about verifying user identity. A typical website prompts for a password. The authentication typically happens over SSL (secure socket layer), a way to transmit encrypted information over HTTP. Authorization is about permissions and is important in corporate systems, particularly
those that define workflows. The recently developed OAuth protocol helps web services to enable users to open access to their private information. This is how Flickr permits access to individual photos or data sets.
Another security area is network protection. This concerns operating systems, configuration and monitoring to thwart hackers. Not only network is vulnerable, any piece of software is. Firefox browser, marketed as the most secure, has to patch the code continuously. To write secure code for your system requires understanding specifics and potential problems.
8. Cloud Computing
In our recent post Reaching For The Sky Through Compute Clouds
we talked about how commodity cloud computing is changing the way we deliver large-scale web applications. Massively parallel, cheap cloud computing reduces both costs and time to market.
Cloud computing grew out of parallel computing, a concept that many problems can be solved faster by running the computations in parallel.
After parallel algorithms came grid computing, which ran parallel computations on idle desktops. One of the first examples was SETI@home project out of Berkley, which used spare CPU cycles to crunch data coming from space. Grid computing is widely adopted by financial companies, which run massive risk calculations. The concept of under-utilized resources, together with the rise of J2EE platform, gave rise to the precursor of cloud computing: application server virtualization. The idea was to run applications
on demand and change what is available depending on the time of day and user activity.
Today’s most vivid example of cloud computing is Amazon Web Services, a package
available via API. Amazon’s offering includes a cloud service (EC2), a database for storing and serving large media files (S3), an indexing service (SimpleDB), and the Queue service (SQS). These first blocks already empower an unprecedented way of doing large-scale computing, and surely the best is yet to come.
7. Concurrency
Concurrency is one topic engineers notoriously get wrong, and understandably so, because the brain does juggle many things at a time and in schools linear thinking is emphasized. Yet concurrency is important in any modern system.
Concurrency is about parallelism, but inside the application. Most modern languages have an in-built concept of concurrency; in Java, it’s implemented using Threads.
A classic concurrency example is the producer/consumer, where the producer generates data or tasks, and places it for worker threads to consume and execute. The complexity in concurrency programming stems from the fact Threads often needs to operate on the common data. Each Thread has its own sequence of execution, but accesses common data.
One of the most sophisticated concurrency libraries has been developed
by Doug Lea and is now part of core Java.
6. Caching
No modern web system
runs without a cache, which is an in-memory store that holds a subset of information typically stored in the database. The need for cache
comes from the fact that generating results based on the database is costly. For example, if you have a website that lists books that were popular last week, you’d want to compute this information once and place it into cache. User requests fetch data from the cache instead of hitting the database and
regenerating the same information.
Caching comes with a cost. Only some subsets of information can be stored in memory. The most common data pruning strategy is to evict items that are least recently used (LRU). The pruning needs to be efficient, not to slow down the application.
A lot of modern web applications, including Facebook, rely on a distributed caching system called Memecached, developed by Brad Firzpatrick
when working on LiveJournal. The idea was to create a caching system that utilises spare memory capacity on the network. Today, there are Memecached libraries for many popular languages, including Java and PHP.
5. Hashing
The idea behind hashing is fast access to data. If the data is stored sequentially, the time to find the item is proportional to the size of the list. For each element, a hash function calculates a number, which is used as an index into the table.
Given a good hash function that uniformly spreads data along the table, the
look-up time is constant. Perfecting hashing is difficult and to deal with that hashtable implementations support collision resolution.
Beyond the basic storage of data, hashes are also important in distributed systems.
The so-called uniform hash is used to evenly allocate data among computers in a cloud database. A flavor of this technique is part of Google’s indexing service; each URL is hashed to particular computer. Memecached similarly uses a hash function.
Hash functions can be complex and sophisticated, but modern libraries have good defaults. The important thing is how hashes work and how to tune them for maximum performance benefit.
4. Algorithmic Complexity
There are just a handful of things
engineers must know about algorithmic complexity. First is big O notation. If something takes O(n) it’s linear in the size of data. O(n^2) is quadratic. Using this notation, you should know that search through a list is O(n) and binary search (through a sorted list) is log(n). And sorting of n items would take n*log(n) time.
Your code should (almost) never have multiple nested loops (a loop inside a loop
inside a loop). Most of the code written today should use Hashtables, simple lists and singly nested loops.
Due to abundance of excellent libraries, we are not as focused on efficiency these days. That’s fine, as tuning can happen later on, after you get the design right.
Elegant algorithms and performance is something you shouldn’t ignore. Writing
compact and readable code helps ensure your algorithms are clean and simple.
3. Layering
Layering is probably the simplest way to discuss software architecture. It first got serious attention when John Lakos published his book about Large-scale C++ systems. Lakos argued that software consists of layers. The book introduced the concept of layering. The method is this. For each software component, count the number of other components it relies on. That is the metric of how complex the component is.
Lakos contended a good software follows the shape of a pyramid; i.e., there’s a progressive increase in the cumulative complexity of each component, but not in the immediate complexity. Put differently, a good software system consists of small, reusable building blocks, each carrying its own responsibility. In a good system, no cyclic dependencies between components are present and the whole system is a stack of layers of functionality, forming a pyramid.
Lakos’s work was a precursor to many developments in software engineering, most notably Refactoring.
The idea behind refactoring is continuously sculpting the software to ensure it’s structurally sound and flexible. Another major contribution was by Dr Robert Martin from Object Mentor, who
wrote about dependencies and acyclic architectures
Among tools that help engineers deal with system architecture are Structure 101 developed by Headway software, and SA4J developed by my former company, Information Laboratory, and now available from IBM.
2. Conventions and Templates
Naming conventions and basic templates are the most overlooked software
patterns, yet probably the most powerful.
Naming conventions enable software automation. For example, Java Beans framework is based on a simple naming convention for getters and setters.
And canonical URLs in del.icio.us: http://del.icio.us/tag/software take the user to the page that has all items tagged software.
Many social software utilise naming conventions in a similar way. For example, if your user name is johnsmith then likely your avatar is johnsmith.jpg and your rss feed is johnsmith.xml.
Naming conventions are also used in testing, for example JUnit automatically recognizes all the methods in the class that start with prefix test.
The templates are not C++ or Java language constructs. We’re talking about template files
that contain variables and then allow binding of objects, resolution, and rendering the result for the client.
Cold Fusion was one of the first to popularize templates for web applications. Java followed with JSPs, and recently Apache developed handy general purpose templating for Java called Velocity. PHP can be used as its own templating engine because it supports eval function (be careful with security). For XML programming it is standard to use XSL language to do templates.
From generation of HTML pages to sending standardized support emails, templates are an essential helper in any modern software system.
1. Interfaces
The most important concept in software is interface. Any good software is a model of a real (or imaginary) system. Understanding how to model the problem
in terms of correct and simple interfaces is crucial. Lots of systems suffer from the extremes: clumped, lengthy code with little abstractions, or an overly designed system with unnecessary complexity and unused code.
Among the many books, Agile Programming by Dr Robert Martin stands out because of focus on modeling correct interfaces.
In modeling, there are ways you can iterate towards the right solution. Firstly, never add methods that might be useful in the future. Be minimalist, get away with as little as possible. Secondly, don’t be afraid to recognize today that what
you did yesterday wasn’t right. Be willing to change things. Thirdly Ultimately, be patient and enjoy the process. you will arrive at a system that feels right. Until then, keep iterating and don’t settle.
Conclusion
Modern software engineering is sophisticated and powerful, with decades of experience, millions of lines of
supporting code and unprecidented access to cloud computing. Today, just a couple of smart people can create software that previously required the efforts of dozens of people. But a good craftsman still needs to know what tools to use, when and why.
In this post we discussed concepts that are indispensable for software engineers. And now tell us please what you would add to this list. Share with us what concepts you find indispensible in your daily software engineering journeys.
What skills are worth learning for a programming career and/or resume?
What skills, languages, tools, etc. would you suggest someone interested in a career in programming learn?
=========================================
The things I look for when reviewing a resume are:
SCM (Subversion, CVS, Git, Perforce, etc)
Unit Testing (JUnit, JMock, etc)
Issue Tracking (Bugzilla, Mantis, etc)
A diverse set of languages (Java, Lisp, Ruby, PHP, etc)
I think having these basic skill sets lets you know that the potential employee has been around the block a few times. As with all things in an interview, make sure you ask follow up questions because resumes are not always the whole story.
=========================================
Failure: and this is a big one. If you do not like to fail, then do not start your future in programming. I fail over and over every day. You write some code, compile, fail. Repeat that process until you get it to compile. Then you run your program, fail. Repeat the cycle until it compiles again and then re-test the software.
Problem Solving: programming is all about solving problems. All of those failures that I mentioned in item 1 above... you have to find solutions to each failure. Every programming task you work on will be a set of problems which need to be overcome.
Coding Style and Documentation: Work on this from the very beginning. Other programmers will love working with you if they can a) read your code and b) figure out what it is supposed to do. Not only that, but adhering to good coding style and best practices will often prevent a bunch of mistakes and hard to find bugs in your programs.
Finally, don't be caught up on what language is the hottest or not. Learn a language and learn it well. Learn to solve problems. Learn the concepts of programming. After that it's all semantics.
Here is some further reading material from Alex Iskold which is very good:
Top 10 Concepts That Every Software Engineer Should Know
====
Learn to write well in the native language of your company (for me that's English). This will help with all the other skills you want (including resume writing!). That includes bug reporting, writing and reading specs and requirements, asking questions, commenting code and any report writing or paperwork you may need to do. In an indirect way, it may also help your coding.
This is important: learn to be concise. Remove all unnecessary words in your writing.
Also, learn to ask questions and seek help, especially in areas that are new to you. If you dither for an hour, that's an hour lost. And that's a shame when it can be prevented.
====
Here are my top few suggestions:
1) Make sure you know how you learn. Do you prefer examples, abstract concepts, learning visually or orally, etc. Chances are you will likely have to learn some technology over your career and this may make it much easier.
2) Understand some basics of software development: Testing, basic data structures like stacks and queues, as well as a methodology or two like Waterfall or Agile, would be another key point.
3) Find your market. Do you like web development, games, databases, neural networks, numerical analysis, embedded systems, mobile devices, scripting or something else? There are lots of different areas one can focus which I'd suggest finding which you'd like to specialize in on as few programmers are good at all of these.
4) Know how to investigate a problem and solve it. This skill may require some work but it can be useful when given a problem that can be solved many ways where some may work better than others. This is also how to get through the part of a job interview where you have a whiteboard and you're asked something that may seem fairly simple that turns out was given vaguely enough to test how well you communicate or do you just assume things that may or may not be true in a hypothetical case.
===
Here are some basic starting areas:
Common Programming Patterns (MVC, Bridge, Observer)
Languages: ASP.NET, PHP
IDE: Eclipse, Visual Studio
Regular Expressions
Unit Testing: nANT, NUnit, JUnit
===
Get involved with open source projects. This can give you experience with teamwork, source control, and coding standards.
Look for local user groups in your area you can attend. Showing an interest in participating in the community will show your enthusiasm.
Be prepared to answer questions about everything you mention on your resume. Try to have some real world examples you can speak off about the technologies you mention. Just learning the facts about a technology will not show that you can apply the knowledge.
===
The book "The Pragmatic Programmer" has a great section on that! If you haven't read it, I would highly suggest that you do. Basically though, stay current with technology. Try and learn a new language every year. Beware of adapting bleeding edge technology. And try many different tools. Don't become too reliant on one technology, cause it might not be the right technology for the job or project.
===
Reference: http://stackoverflow.com/questions/67794/what-skills-are-worth-learning-for-a-programming-career-and-or-resume
Doxygen - Source code documentation generator tool
http://www.stack.nl/~dimitri/doxygen/
JQuery HowTo: How to bind events to AJAX loaded elements
We all use jQuery's AJAX helpers like $.get()
, $.post()
and $.load()
. We also all use jQuery's event bindings. When you use them both, problems such as your event bindings like click()
are not bided to your new AJAX loaded elements. To solve this problem you need to bind events to your newly loaded elements.
NOTE:
Make sure you don't bind the same event to the elements with this event already bound.
TIP:
You should somehow separate newly created elements and then bind event in callback of an AJAX function.
Regarding the tip above. Here are two ways of separating AJAX loaded content:
- You can load into some specific container
<div id="ajax-loaded"></div>
and then apply your events to the elements inside this container (Ex:$('#ajax-loaded .my-button').click(...)
). - Add some class to your elements that are loaded on the server side.
You might also be interested in:
- Cross domain AJAX querying
- AJAX update content every X seconds
- Display loading GIF image while loading through AJAX
Update: Added a demo page that shows how to bind events to AJAX loaded elements here.
Wednesday, May 19, 2010
Some useful Drupal modules you may not have heard about
We’re using many of these modules on our newly upgraded to drupal 6 department website at http://itls.usu.edu/.
Even if you’re experienced with Drupal, you may not have heard of some of these modules. I encourage you to check them out to see if they are of use to your site(s). I’ve marked the ones I consider essential.
If you’d like to know how we did some of the features on our site just contact me, or wait a few weeks and I’ll be releasing a generic version of our site (called “Department 2.0″). For example there is a job board, upcoming calendar events block, slideshow, customized listing of people, and our groups have numerous custom views and blocks and features (see my Foundations of Educational Technology class group page for example).
- FriendFeed – Show other social networking activity via the FriendFeed.com site. I had to make a few patches to this one
- activity – Show all activity on the site on one page, including comments and profiles edits (stuff that views still can’t show together in one view unless you use nodecomment and content_profile which convert comments and profiles to nodes)
- admin_menu – we’re using simplemenu instead actually
- admin_theme – so you have a clean built-in theme when doing administration work
- advanced_help
- ajax – works with logintoboggan and numerous other modules to make forms submit without a page reload
- alt_login – not using now, but we needed this for ldap support before
- auto_nodetitle – we use this to auto-title our course nodes and other things
- autologout – in case many of your users are using shared computers and forget to log out
- backup_migrate – ESSENTIAL
- better_formats – ESSENTIAL if you have multiple input formats/filters – so that you don’t default to plain text for example
- block_edit
- calendar – works with the date module. See alternatively the event module.
- cck – ESSENTIAL
- comment_notify
- commentrss – ESSENTIAL if you want to track all site activity including comments without having to visit administration every time. Alternatively you can create a view of recent comments yourself with an rss feed.
- contemplate – important if you want site administrators to be able to make changes to the site without shell access
- context – not using yet
- creativecommons – not using, but we may switch from the next one
- creativecommons_lite – a bit buggy but we were using it before in drupal 5
- custom_pagers – alternative to book
- date – ESSENTIAL for cck types with date/time info like our job board (expiration date), calendar and so forth
- devel – useful for debugging issues
- diff – useful for our wiki and for tracking edits
- draft – lets users have a ‘save as draft’ button, plus features autosave
- drush – ESSENTIAL for easier module/theme installation and upgrading. Also works with features module. This is the only module you’ll have to install by hand.
- ed_readmore – Improve the ‘read more’ teaser link that nobody clicks.
- email – for email cck fields
- extlink – distinguish internal from external links
- fckeditor – ESSENTIAL for wysiwyg editing. See also the TinyMCE module.
- filefield – I think this is ESSENTIAL but we aren’t using it yet. It puts uploaded files in separate folders rather than having a huge mess of files like we do now.
- freelinking – for [[wiki links]].
- globalredirect – ESSENTIAL for making sure you don’t have 2 or more paths with identical content. ‘node/1234′ will redirect to the nicer path for example if you use pathauto.
- google_analytics – ESSENTIAL for tracking statistics on your site, but piwik is a recommended open source alternative.
- image – ESSENTIAL unless you are using imagecache instead
- imce – ESSENTIAL for making it easier for users to insert images in fckeditor. Also gives each user their own personal file space.
- interwiki – if using a wiki
- invisimail – obscure any email addresses users might type in a comment or node
- javascript_aggregator – this is now mostly built-in to drupal 6, but has a few extra features
- job_queue
- jq
- jquery_ui
- ldap_integration
- ldap_provisioning
- link – cck field type
- linkchecker – haven’t tried, but checks for broken links on your site
- live – live preview of posts/comments – used on the drupal.org site
- location – cck field type
- logintoboggan – ESSENTIAL to me anyway – lets people login with email address instead of username among other nice features.
- menu_block – see also nodehierarchy, submenutree and other related modules
- messaging – ESSENTIAL along with notifications if using organic groups (og) especially
- mimemail – we were using this, but now we are using HTMLMail with messaging/notifications instead
- moduleinfo – ESSENTIAL – gives more info about modules on module list page
- mollom – ESSENTIAL or else find another spam prevention solution if your site is open to any public contributions such as comments (or for example our job board)
- nice_menus – not using, but provides suckerfish menus if not provided by your theme directly
- node_import – importing content
- nodecomment – turns comments into nodes – I tried this to I could show comments and node updates together in a view block, but I had to make comments og-enabled, too, which wouldn’t work for anonymous commenting, so I’m not using it now.
- nodehierarchy – Integrates the organization of your content with the menu system
- notifications – works with messaging – overly complicated interface though for end-users to customize their preferences
- og – organic groups
- og_forum – makes it easier to set up per group forums
- og_mandatory_group – nice to force end-users to be in a group before they can contribute stuff
- path_redirect
- pathauto – ESSENTIAL for nicer looking, automatically generated URLs
- permissions_api – easier administration of permissions
- poormanscron – if you don’t have cron or it is unreliable (like on Mac OS X)
- prepopulate – if you want to use bookmarklets for things like shared links and so forth
- print – ESSENTIAL for printable versions of your pages
- quicktabs – we ended up using views_slideshow instead
- quiz – not using, but nice module
- realname – we just have people use their real name (First Last) as their regular username, so we didn’t require this module.
- revision_deletion – ESSENTIAL if you are using a wiki or a site with many people editing, so you can occasionally clean up stuff.
- rules – Much nicer, easier to use alternative to workflow. We use it to do numerous things on our site, for example notifying someone when a job is posted, promoting items in certain groups to the front page, etc. Formerly called workflow_ng in drupal 5 (not upgradeable though)
- scanner – ESSENTIAL find/replace text on your site. Regex support.
- scheduler – set your node to be published at a future date
- securepages – ESSENTIAL if using an SSL server (which you should)
- similarterms
- simplemenu – this is our end-users’ toolbar for creating content, visiting groups, etc.
- site_map
- site_tour
- smtp – ESSENTIAL if sendmail doesn’t work or is restricted from your server. I label it essential because stuff like this is already included in other tools like moodle even though most won’t need it (moodle includes built-in ldap support and other things too).
- submenutree – alternative to book module – displaying child items below a node
- tagadelic
- talk – if wanted for wiki pages
- theme_editor – ESSENTIAL if you want administrators to be able to edit theme files without having shell access
- token – ESSENTIAL – required by many other modules including pathauto
- trash – essential but broken? We need a way for end-users to delete stuff like wiki pages in a way that administrators can undo it if they deleted something important. At least drupal 6 finally separated the edit and delete permissions.
- twitter – if you want to have a site-wide twitter account that announces stuff on twitter
- upload_replace – ESSENTIAL – should be built-in to drupal. Right now if you upload a new version of a file, it gets a new name, rather than moving the old file.
- user_import – import users from a spreadsheet – warning, backup everything first – this module created duplicate entries in our profile tables a year ago
- userplus
- userprotect – prevent users from changing certain things in their accounts
- video
- video_filter – easy linking and displaying of videos from external sites
- views – ESSENTIAL – works with cck
- views_bulk_operations – lets you do batch operations on nodes and other things
- views_calc
- views_slideshow – see other alternatives for doing slideshows like slider
- webform – ESSENTIAL if you want to do things like surveys and so forth on your site – 1000x better than the built-in poll module
- wikitools – for wikis – move protection, etc.
- I also posted a comparison of drupal modules for uploading large files recently, including swfupload, image_fupload, and others
Monday, May 17, 2010
How to remove the first Exchange Server 2003 computer from the administrative group
This article describes the steps to remove the first Microsoft Exchange Server 2003 computer from an administrative group. The first Exchange Server 2003 computer that is installed in an administrative group holds certain important roles. For example, the first server hosts the Offline Address Book folder, the Schedule+ Free Busy folder, the Events Root folder, and other folders. Therefore, you must use caution when you remove this server from the administrative group that it belongs to.
Back to the top
Replicate all public folders to another server
All public folders and system folders that are housed on the first Exchange 2003 computer must be replicated to another Exchange 2003 computer that is in the site. The remainder of this article refers to this other computer as the destination server. The server that is being removed is referred to as the source server.
Click Start, point to Programs, point to Microsoft Exchange, and then click System Manager.
If the Display administrative groups option is turned on, expand Administrative Groups, and then expand First Administrative Group.
Note To display administrative groups, right-click Your_Organization, click Properties, click to select the Display administrative groups check box, click OK two times, and then restart Exchange System Manager.
Expand Folders, expand Public Folders, right-click a top-level public folder, and then click Properties.
Click the Replication tab, and then click Add.
In the Select a Public Store box, select the name of the server where you want a replica of this public folder (the destination server), and then click OK.
Click Apply, and then click OK. If this public folder has any subfolders and if you want these subfolders to have a replica on the destination server, right-click the top-level public folder, click All Tasks, and then click Manage Settings.
Note When you are running Microsoft Exchange Server 2003 Service Pack 2 (SP2), the Manage Public Folders Settings Wizard page may appear. You can use this wizard to modify client permissions, to modify lists of replica servers, and to overwrite public folder settings.
In the Propagate Folder Settings box, click to select the Replicas check box, and then click OK. When you complete this step, all subfolders of that top-level folder have a replica on the destination server.
Note For performance reasons, you may not always want to have replicas of all subfolders of a top-level folder on another server. You may want to consider this before you complete this procedure.
For more information, click the following article number to view the article in the Microsoft Knowledge Base:
273479 (http://support.microsoft.com/kb/273479/ ) Description of the public folder referral functionality in Exchange 2000 Server and in Exchange Server 2003
Repeat steps 1 through 7 for all top-level folders and all subfolders that you want to replicate.
After replicas have been made on the destination server, wait for replication to complete, and then make sure that the replica folders are synchronized with the source folders. To do so, follow these steps:
Right-click the public folder that you want to verify, and then click Properties.
Click the Replication tab, and then click Details.
When replication has completed, the Replication Status column indicates In Sync.
After you have confirmed that replication is complete for each folder, you must turn off replication for each public folder. To do this, follow these steps:
Expand Public Folders, right-click a top-level public folder, and then click Properties.
Click the Replication tab.
In the Replicate content to these public stores section, click the name of the public folder store that is on the source server, click Remove, click Apply, and then click OK.
Back to the top
Rehome the Offline Address Book folder
Click Start, point to Programs, point to Microsoft Exchange, and then click System Manager.
If the Display administrative groups option is turned on, expand Administrative Groups, and then expand First Administrative Group (where First Administrative Group is the name of your administrative group).
Note To display administrative groups, right-click Your_Organization, click Properties, click to select the Display administrative groups check box, click OK two times, and then restart Exchange System Manager.
Expand Folders, right-click Public Folders, and then click View System Folders.
Note If you do not see a View System Folders option, you are already viewing the system folders.
Expand Public Folders, and then expand Offline Address Book.
Right-click /o=Organization Name/cn=addrlists/cn=oabs/cn=Default Offline Address List, and then click Properties.
Note If you receive a message that the mail proxy for this folder was not found, click OK.
Click the Replication tab, and then click Add.
In the Select a Public Store box, select the name of the server where you want a replica of this public folder (the destination server), and then click OK.
Click Apply, and then click OK. If this public folder has any subfolders and if you want these subfolders to have a replica on the destination server, right-click the top-level public folder, click All Tasks, and then click Manage Settings.
Note When you are running Exchange Server 2003 SP2, the Manage Public Folders Settings Wizard page may appear. You can use this wizard to modify client permissions, to modify lists of replica servers, and to overwrite public folder settings.
In the Propagate Folder Settings box, click to select the Replicas check box, and then click OK. When you complete this step, all subfolders of that top-level folder have a replica on the destination server.
After replicas have been made on the destination server, wait for replication to complete, and then make sure that the replica folders are synchronized with the source folders. To do so, follow these steps:
Right-click the public folder that you want to verify, and then click Properties.
Click the Replication tab, and then click Details.
When replication has completed, the Replication Status column indicates In Sync.
After you have confirmed that replication is complete for each folder, you must remove the replica from the source server. To do this, follow these steps:
Right-click /o=Organization Name/cn=addrlists/cn=oabs/cn=Default Offline Address List, and then click Properties.
Click the Replication tab.
In the Replicate content to these public stores section, click the name of the public folder store on the source server, click Remove, click Apply, and then click OK.
Repeat step 11 for each subfolder of the offline Address Book and for each additional offline address list.
Back to the top
Change the server that is responsible for generating the Offline Address List
Start Exchange System Manager, expand Recipients, and then click the Offline Address Lists container.
In the right pane, right-click Default Offline Address List, and then click Properties.
In the Default Offline Address List Properties dialog box, the server that is going to be removed from the administrative group will be in the Offline address list server list.
Click Browse, and then type the name of the server that the replica of the Offline Address Book was added to in the "Rehome the Offline Address Book folder" section.
Click OK.
The new server is now listed as the Offline address list server.
In the Default Offline Address List Properties dialog box, click OK. You may have to quit and restart the Exchange System Manager before the change is displayed.
Back to the top
Rehome the Schedule+ Free Busy folder
Click Start, point to Programs, point to Microsoft Exchange, and then click System Manager.
If the Display administrative groups option is turned on, expand Administrative Groups, and then expand First Administrative Group (where First Administrative Group is the name of your administrative group).
Note To display administrative groups, right-click Your_Organization, click Properties, click to select the Display administrative groups check box, click OK two times, and then restart Exchange System Manager.
Expand Folders, and then click Public Folders.
Right-click Public Folders, and then click View System Folders.
Note If you do not see a View System Folders option, you are already viewing system folders.
Expand Schedule+ Free Busy, right-click EX:/o=Organization_Name/ou=Administrative_Group_Name folder, and then click Properties.
Note If you receive a message that the mail proxy for this folder was not found, click OK.
Click the Replication tab, and then click Add.
In the Select a Public Store box, select the name of the server where you want a replica of this public folder (the destination server), and then click OK.
Click Apply, and then click OK. If this public folder has any subfolders and if you want these subfolders to have a replica on the destination server, right-click the top-level public folder, click All Tasks, and then click Manage Settings.
Note When you are running Exchange Server 2003 SP2, the Manage Public Folders Settings Wizard page may appear. You can use this wizard to modify client permissions, to modify lists of replica servers, and to overwrite public folder settings.
In the Propagate Folder Settings box, click to select the Replicas check box, and then click OK. When you complete this step, all subfolders of that top-level folder have a replica on the destination server.
After replicas have been made on the destination server, wait for replication to complete, and then make sure that the replica folders are synchronized with the source folders. To do this, follow these steps:
Right-click the public folder that you want to check, and then click Properties.
Click the Replication tab, and then click Details.
When replication has completed, the Replication Status column indicates In Sync.
After you have confirmed that replication is complete, you must turn off replication for this folder. To do this, follow these steps:
Expand Public Folders, right-click EX:/o=Organization_Name/ou=Administrative_Group_Name, and then click Properties.
Click the Replication tab.
In the Replicate content to these public stores section, click the name of the public folder store on the source server, click Remove, click Apply, and then click OK.
Repeat step 11 for each subfolder of the Schedule+ Free Busy folder and for each additional folder that contains free and busy information that has been replicated to the destination server.
Back to the top
Rehome the Organization Forms folder
If the server that is being decommissioned has the Organization Forms folder homed and if this server has the only replica, replicate the Organization Forms folder to another server in the site. If you do not replicate this folder, clients may stop responding (hang) when non-delivery report (NDR) messages and read receipt messages are sent.
Start Exchange System Manager, expand Administrative Groups, and then expand the name of your administrative group.
Expand Folders, right- click Public Folders, and then click View System Folders.
Note If you do not see the View System Folders option, you are already viewing system folders.
Expand the EFORMS REGISTRY folder.
If an Organization Forms library folder does not exist, ignore the rest of the steps in this section.
Right-click the Organizational Forms folder, and then click Properties.
Note If you receive a message that states that the mail proxy for this folder cannot be found, click OK.
Add a replica of this folder to the destination server.
For more information about how to add a replica of this folder to the destination server, see the "Replicate all public folders to another server" topic earlier in this article.
After replicas have been made to the destination server, wait for replication to complete, and then make sure that the replica folders are synchronized with the source folders. To do this, view the properties of the specific public folder, click the Replication tab, and then click Details. If the Replication Status column indicates In Sync, the replica folders have been synchronized with the source folders.
After you confirm that the replicas are synchronized, remove the replica of this folder from the source server.
For more information about how to remove the replica folder from the source server, see the "Replicate all public folders to another server" topic earlier in this article.
Back to the top
Rehome the Recipient Update Service
In Exchange System Manager, expand Recipients, and then click Recipient Update Services.
In the column headings that are in the right pane, click Exchange Server to sort the available Recipient Update Services by the Exchange Server computer that hosts them.
Right-click the Recipient Update Service that is hosted on the Exchange Server 2003 computer that you plan to remove, and then click Properties.
On the General tab, next to the Exchange server box, click Browse.
In the Select Exchange Server dialog box, click the name of another Exchange Server computer as the new server to host the Recipient Update Service, and then click OK two times.
Back to the top
Designate another server to be the routing group master
If this server is the routing group master, you must designate another server to be the routing group master. To do this, follow these steps:
On the Exchange 2003 computer that you want to designate as the routing group master, start Exchange System Manager.
If the Display administrative groups check box and the Display routing groups check box are selected, expand Administrative Groups, expand First Administrative Group, expand Routing Groups, and then expand First Routing Group. (First Routing Group is a placeholder for the name of the routing group where you want to change the routing group master.)
Notes
To display administrative groups, right-click First Organization (First Organization is a placeholder for the name of your organization), click Properties, click to select the Display administrative groups check box, click OK two times, and then restart Exchange System Manager.
To display rooting groups, right-click First Organization, click Properties, click to select the Display rooting groups check box, and then click OK two times.
Click Members, right-click the server in the right pane that you want to make the routing group master, and then click Set as Master.
Back to the top
Create another Site Replication Service (SRS) instance
If this Exchange 2003 computer has the Site Replication Service (SRS) installed and running on it, you must create a new SRS in Exchange System Manager. To do this, follow these steps:
Start Exchange System Manager on the Exchange Server 2003 computer where you want to create the SRS, and then expand Tools.
Right-click Site Replication Service, click New, and then click Site Replication Service.
When you receive the Are you sure you want to install a new Site Replication Service on this server (ServerName) prompt, click Yes.
When you receive the In order to create a Site Replication Service on the local machine, enter the password for the Exchange service account (Domain\service account) prompt, type the password for the Exchange service account.
This creates the SRS, and it also creates an associated configuration Connection Agreement.
For more information, click the following article number to view the article in the Microsoft Knowledge Base:
255285 (http://support.microsoft.com/kb/255285/ ) How to create an additional Site Replication Service for a mixed site
Back to the top
Rehome connectors to another server
If this Exchange Server 2003 computer has any connectors that are hosted on it, these connectors must be rehomed to another server before you remove this Exchange 2003 computer from the routing group. For example, if the source server hosts an X.400 connector, you must set up this connector on the destination server.
Back to the top
Move mailboxes to another server
If this Exchange Server 2003 computer hosts any mailboxes, these mailboxes must be moved to another server before you remove this Exchange 2003 computer. To do this, follow these steps:
Start Exchange System Manager on the Exchange 2003 source computer.
Double-click Servers, and then locate the server container where the mailboxes of the users are located.
For example, if you want to move mailboxes from the default storage group and from the mailbox store, double-click First Storage Group, double-click Mailbox Store, and then click Mailboxes.
In the right pane, click the mailboxes that you want to move.
Right-click the selected users, and then click Exchange Tasks.
In the Exchange Task Wizard, click Next.
On the Available Tasks page, under Select a task to perform, click Move Mailbox, and then click Next.
On the Move Mailbox page, click Cross Administrative Group Move under Select the Type of Move, and then click Next.
On the Move Mailbox page, click the destination server that is in the Server list, click a mailbox store in the Mailbox Store list, and then click Next.
Configure how you want any corrupted messages that are found during the move to be handled, and then click Next two times.
For more information about how to move mailboxes in Exchange Server 2003, click the following article number to view the article in the Microsoft Knowledge Base:
821829 (http://support.microsoft.com/kb/821829/ ) Moving mailboxes in Exchange Server 2003
Back to the top
Remove the first Exchange Server 2003 computer
Insert the Exchange Server 2003 CD in the CR-ROM drive in the Exchange 2003 computer, and then click Remove for each component that has been installed.
For more information, click the following article numbers to view the articles in the Microsoft Knowledge Base:
235396 (http://support.microsoft.com/kb/235396/ ) How to determine the first Exchange Server computer in the site
152959 (http://support.microsoft.com/kb/152959/ ) How to remove the first Exchange Server in a site
284148 (http://support.microsoft.com/kb/284148/ ) How to remove the last Exchange Server 5.5 computer from an Exchange 2000 administrative group
282061 (http://support.microsoft.com/kb/282061/ ) How to rebuild a Site Replication Service without a backup
For more information about how to rehome public folders in Exchange 2000, click the following article number to view the article in the Microsoft Knowledge Base:
288150 (http://support.microsoft.com/kb/288150/ ) How to rehome public folders in Exchange 2000
For more information about this topic for Microsoft Exchange Server 5.5 and for Microsoft Exchange 2000 Server, click the following article numbers to view the articles in the Microsoft Knowledge Base:
152959 (http://support.microsoft.com/kb/152959/ ) How to remove the first Exchange server in a site
275171 (http://support.microsoft.com/kb/275171/ ) How to reset system folders on an Exchange 2000 server