Try/Catch for SQL!?
Thanks to Nick Sertis for this trick – who knew TSQL could do try/catch statements!
Very useful when you need to write data manipulation scripts for production databases.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | BEGIN TRY
BEGIN TRAN
--Some SQL
COMMIT TRAN
END TRY
-- Catch the errors on the inserts
BEGIN CATCH
ROLLBACK TRAN
SELECT ERROR_MESSAGE()
END CATCH |
For a software craftsman, reducing technical debt should be as much of a habit as typing
Filed under: Refactoring, Software Craftmanship, Technical Debt
I was involved in an interesting group discussion with fellow craftsmen yesterday on Technical Debt at the 2009 Software Craftsmanship conference.
The question put to the group was: “How should a team make time to reduce technical debt?”
I was interested that there was a totally unanamious response – “You shouldn’t”. “You should be doing tiny pieces of technical debt reduction all the time”.
Previously I have advocated creating technical debt reduction stories, and trying to schedule those into the iteration plan. People thought this was in principal the wrong strategy; and indeed in my experience this approach doesn’t work.
The group felt that in general tackling technical debt reduction though large scale refactorings was the wrong approach – rather a craftsman should be making incremental improvements every time they touch the code.
Bob Martin’s Boy Scout Rule: – check in your code a little cleaner than what you checked out – encapsulates this principal. Its the little refactorings that you make – removing a tiny piece of duplication, changing a variable name to better reveal intent; extracting an expression into a intention revealing method – that, over time, result in a clean, maintainable code base.
In a way, this is similar to implementing the “Fixing Broken Windows Theory” in software development. The theory is that having a zero tolerance attitude towards the little things makes a huge impact on the so called “bigger” things.
Its perhaps easier understood if you consider what happens if you don’t care about the little things. Its about the attitude – if I couldn’t care enough to clean up a messy bit of code; will my team mates care about a few broken tests? If its okay to have a few broken tests; then it’s probably okay to ignore some bugs. If its okay to ignore bugs; then who really needs to care about well defined acceptance tests? And if the team doesn’t care about precise acceptance tests; why should the business care about unambiguous requirements. You get the picture.
Its the little things, added up, that result in technical debt reduction.
Functions with side effects are just rude!
Today I fell into a trap when using a function that had a side effect – it unexpectedly changed an input parameter; causing a later statement to fail. Debugging took an age!
For example, consider the following function:
1 | string StringReplace(string haystack, string needle) |
If this function is side-effect free, we can use it without fear like this:
1 2 3 4 5 6 | string menagerie = "cat,dog,bee,llama"; string catFreeMenagerie = StringReplace(menagerie, "cat"); string beeFreeMengerie = StringReplace(menagerie, "eric"); Assert.AreEqual(",dog,fish,llama", catFreeMenagerie); Assert.AreEqual("cat,dog,,llama", beeFreeMengerie); |
However, if StringReplace() had the side effect of also changing the passed in haystack, then the second Assert would fail, because the first StringReplace has the unexpected side effect of changing one of its arguments.
Evans in the DDD book has quite a bit to say about this; arguing that having side effect free functions goes a long way towards making a supple design
Side effect free functions also make testing & refactoring easier (less state to worry about etc)
Remember, a function that changes its parameters is rude, and should not be trusted!
PS: Eric the half a bee lyrics
Selenium gotcha – selenium.GetHtmlSource() returns processed HTML
Whilst writing some Selenium based acceptance tests today; I bumped into a hair pulling gotcha. Hopefully this post will prevent you from the same pain.
The test was to check whether some tracking tag javascript was being inserted into the page correctly or not.
I assumed that I could get the page source as it was being delivered to the browser by calling selenium.GetHtmlSource(); and then check that for the javascript string I was expected.
Unfortunately, GetHtmlSource is just a proxy for the browsers DOM.InnerHTML method; and that returns the Html after it has been preprocessed by the browser.
Turns out that preprocessing does a couple of funky things, including
- Changing line-endings (Firefox)
- Changing capitalization (IE6)
- Seemingly random removal / insertion of ” & ‘ (IE6)
So, when I was expecting a string like this:
1 2 3 4 5 6 | <script language="javascript" type="text/javascript"> <!-- var amPid = '206''; var amPPid = '4803'; if (document.location.protocol=='https:') ...[snip]... |
IE6 was presenting me with:
1 2 3 4 5 6 | <SCRIPT language=javascript type=text/javascript> <!-- var amPid = '206''; var amPPid = '4803'; if (document.location.protocol=='https:') ...[snip]... |
A possible solution is to ignore case, whitespace and quotes when doing the comparison, with a helper method like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 | /// <summary> /// Use this to compare strings to those returned from selenium.GetHtmlSource for an Internet Explore instance /// (IE6 seems to change case and inclusion of quotes, especially for Javascript.?) /// </summary> /// <param name="expected"></param> /// <param name="actual"></param> private static void AssertStringContainsIgnoreCaseWhiteSpaceAndQuotes(string expected, string actual) { string expectedClean = Regex.Replace(expected, @"\s", "").ToLower().Replace("\"","").Replace("'",""); string actualClean = Regex.Replace(actual, @"\s", "").ToLower().Replace("\"", "").Replace("'", ""); StringAssert.Contains(expectedClean,actualClean, string.Format("Expected string \n\n{0} \n\nis not contained within \n\n{1}", expected, actual)); } |
It was the line endings that really floored me; because they were automatically normalized/corrected by my test runner when displaying the error. Aaargh!
Apache2 on Ubuntu 8.04LTS; restrict access to PAM authenticated users
I have a couple of static pages that I want to restrict access to.
I don’t want to manage another set of usernames & passwds, so I’d like apache2 to authenticate off the standard users on my system, via PAM.
To get this to work, you need to install and configure mod_auth_pam and mod_auth_shadow
aptitude install libapache2-mod-auth-pam libapache2-mod-auth-shadow
Ensure the www-data user is part of the shadow group, so apache2 can read the passwords
usermod -G shadow www-data
And set up the relevent virtual host:
AuthPAM_Enabled On AuthShadow on AuthPAM_FallThrough Off AuthBasicAuthoritative Off AuthType Basic AuthName "Restricted to group: sysadmins" AuthUserFile /dev/null Require group sysadmins
Restart apache, and you’re done!
Self Cert SSL certificate for Apache2 on Ubuntu 8.04LTS
Generate a self cert certificate:
https://help.ubuntu.com/8.04/serverguide/C/certificates-and-security.html
Create a new virtual host (you can only have one SSL virtual host / IP)
sudo cp /etc/apache2/sites-available/default /etc/apache2/sites-available/ssl
Edit ssl sothat it looks like this:
NameVirtualHost *:443
ServerName webangle-www1.everyangle.co.uk
ServerAdmin webmaster@localhost
DocumentRoot /var/www/
SSLEngine on
SSLOptions +StrictRequire
SSLCertificateFile /etc/ssl/certs/server.crt
SSLCertificateKeyFile /etc/ssl/private/server.key
Finally, if you want to force redirect of all traffic to a certain folder via SSL (e.g, /phpmyadmin), add the following to /etc/apache2/sites-available/default
#Redirect traffic to /phpmyadmin through https
RewriteEngine on
RewriteCond %{SERVER_PORT} ^80$
RewriteRule ^/phpmyadmin(.*)$ https://%{SERVER_NAME}/phpmyadmin$1 [L,R]
Enable it:
sudo a2ensite ssl sudo /etc/init.d/apache2 reload
Automount remote filesystem over SSH
Previously I posted on how I backup my server’s data to rsync.net’s remote storage.
A convienient way to access that remote storage is to configure rsync over sshfs:
sudo aptitude install sshfs mkdir /mnt/sshfs mkdir /mnt/sshfs/rsync.net sshfs **username**@ch-s011.rsync.net: /mnt/rsync.net
First, unmount
fusermount -u /mnt/rsync.net
Then, install autofs, and edit the config file
sudo aptitude install autofs sudo vi /etc/auto.master
Add the following line
/mnt/sshfs /etc/auto.sshfs --timeout=30,--ghost
Then,
sudo vi /etc/auto.sshfs
Add
rsync.net -fstype=fuse,rw,nodev,nonempty,noatime,allow_other,max_read=65536 :sshfs\#**username**@ch-s011.rsync.net\:
And finally restart autofs
sudo /etc/init.d/autofs restart
Now, when you cd /mnt/sshfs/rsync.net, after a short delay you will automatically be connected to the remote filesystem over SSH. After 30 seconds of inactivity, the connection will be closed.
Backup Ubuntu 8.04LTS to rsync.net using backup-manager (at linode.com)
I’m setting up a new linode360 VPS, based of the Ubuntu 8.04LTS image.
For backups, I want to do weekly backups and daily incrementals of the data files, and sync these off to an external backup location.
Broadly, there are two parts to the backup, creating the backed up files, and then copying them offsite.
Creating the backups
I’m using backup-manager 0.7.6-debian1, which handles backing up sets of files and MySQL databases to tar.gz files.
sudo aptitude install backup-manager sudo /usr/sbin/backup-manager --version
The comments in the config file make editing it quite straight forward.
sudo vi /etc/backup-manager.conf
One minor points:
- Separate multiple backup methods with a space, eg:
export BM_ARCHIVE_METHOD="tarball-incremental mysql"
To test:
sudo /usr/sbin/backup-manager --verbose
The output folder you specified (/var/archives) should now contain some .tar.gz versions of your data. Hurrah!
Getting the files offsite
Originally I intended to use Amazon’s S3 as a backup store, following Michael Zehrer’s instructions on how to rsync with S3. However, I couldn’t get this to work reliably; so I opted instead for rsync.net which offers standard scp, ftp, WebDav and sshfs access to their geographic backup locations.
Backup-manager can rsync over ssh, which is a quick and efficient way to sync changes over to the remote host..
The first step is get your rsync.net account setup; and set up your ssh so you can access without typing in a password
Then, set the BM_UPLOAD_METHOD to rsync, and configure both the scp and the rsync settings in /etc/backup-manager.conf (pay attention not to prefix remote folders with / ).
Test with:
sudo /usr/sbin/backup-manager --verbose
Once its all working, set up a cron job to call backup-manager daily.
crontab -e
I run backup-manager once per day in the wee hours, and log output to /root/crontab/daily_backup-manager.logs
0 3 * * * /usr/sbin/backup-manager -v > /root/cronlogs/daily_backup-manager.log
Viola!
Farside re-enactment group
The evolution of the giraffe.
In case you were wondering. More sidesplitting stuff at http://www.flickr.com/groups/farside/pool/
The Correlation between Schedule Pressure & Low Quality
Research suggests that
- 40% of all software errors are caused by pressure on developers to complete quicker (Glass 2004)
- Under extreme schedule pressure, code defects increase by 400% (Jones 2004)
- Projects which aim to have the lowest number of defects also have the shortest schedules (Jones 2000)
This makes sense is you consider that good engineering practises are the first to leave the building under pressure to finish, and most teams will revert to quick & dirty hacks to get things implemented, without complete testing etc.
My personal opinion is that the only way to shorted development cycles is to reduce the feature set. Its pleasing for me to see that the research seems to back this up.
When deciding which features will be dropped; I think its worth revisiting the business requirements that are driving a particular set of features. In many cases a simpler “design” could suffice; for example a fancy calendar widget could be replaced with a simple textbox; a little used settings screen could be retired in favour of manually changing config files; or overly complex but little used workflows could be put on the back burner.
I maintain that a lot of “features” can be dropped, without actually impairing the business functionality of the system.
Just remember, what every you do DON’T consider dropping testing or QA in an effort to meet your deadline; unless you want to guarantee that you will continue to miss all future deadlines until the project gets cancelled!
