post

Apache2 on Ubuntu 8.04LTS; restrict access to PAM authenticated users

I have a couple of static pages that I want to restrict access to.

I don’t want to manage another set of usernames & passwds, so I’d like apache2 to authenticate off the standard users on my system, via PAM.

To get this to work, you need to install and configure mod_auth_pam and mod_auth_shadow

aptitude install libapache2-mod-auth-pam libapache2-mod-auth-shadow

Ensure the www-data user is part of the shadow group, so apache2 can read the passwords

usermod -G shadow www-data

And set up the relevent virtual host:


                AuthPAM_Enabled On
                AuthShadow on
                AuthPAM_FallThrough Off
                AuthBasicAuthoritative Off
                AuthType Basic
                AuthName "Restricted to group: sysadmins"
                AuthUserFile /dev/null
                Require group sysadmins

Restart apache, and you’re done!

post

Self Cert SSL certificate for Apache2 on Ubuntu 8.04LTS

Generate a self cert certificate:

https://help.ubuntu.com/8.04/serverguide/C/certificates-and-security.html

Create a new virtual host (you can only have one SSL virtual host / IP)

sudo cp /etc/apache2/sites-available/default /etc/apache2/sites-available/ssl

Edit ssl sothat it looks like this:
NameVirtualHost *:443

ServerName webangle-www1.everyangle.co.uk
ServerAdmin webmaster@localhost

DocumentRoot /var/www/

SSLEngine on

SSLOptions +StrictRequire

SSLCertificateFile /etc/ssl/certs/server.crt
SSLCertificateKeyFile /etc/ssl/private/server.key

Finally, if you want to force redirect of all traffic to a certain folder via SSL (e.g, /phpmyadmin), add the following to /etc/apache2/sites-available/default

#Redirect traffic to /phpmyadmin through https
        RewriteEngine   on
        RewriteCond     %{SERVER_PORT} ^80$
        RewriteRule     ^/phpmyadmin(.*)$ https://%{SERVER_NAME}/phpmyadmin$1 [L,R]

Enable it:

sudo a2ensite ssl
sudo /etc/init.d/apache2 reload
post

Automount remote filesystem over SSH

Previously I posted on how I backup my server’s data to rsync.net’s remote storage.

A convienient way to access that remote storage is to configure rsync over sshfs:

sudo aptitude install sshfs
mkdir /mnt/sshfs
mkdir /mnt/sshfs/rsync.net
sshfs **username**@ch-s011.rsync.net: /mnt/rsync.net
Now, test that you can access /mnt/rsync.net, and copy a few files to your remote storage.  if all works well, the next step is to have sshfs automatically connect whenever we try to access the directory

First, unmount

fusermount -u /mnt/rsync.net

Then, install autofs, and edit the config file

sudo aptitude install autofs
sudo vi /etc/auto.master

Add the following line 

/mnt/sshfs /etc/auto.sshfs --timeout=30,--ghost

Then,  

sudo vi /etc/auto.sshfs

Add

rsync.net -fstype=fuse,rw,nodev,nonempty,noatime,allow_other,max_read=65536 :sshfs#**username**@ch-s011.rsync.net:

 

And finally restart autofs 

sudo /etc/init.d/autofs restart

 

Now, when you cd /mnt/sshfs/rsync.net, after a short delay you will automatically be connected to the remote filesystem over SSH.  After 30 seconds of inactivity, the connection will be closed.

post

Backup Ubuntu 8.04LTS to rsync.net using backup-manager (at linode.com)

I’m setting up a new linode360 VPS, based of the Ubuntu 8.04LTS image.

For backups, I want to do weekly backups and daily incrementals of the data files, and sync these off to an external backup location.

Broadly, there are two parts to the backup, creating the backed up files, and then copying them offsite.

Creating the backups

I’m using backup-manager 0.7.6-debian1, which handles backing up sets of files and MySQL databases to tar.gz files.

sudo aptitude install backup-manager
sudo /usr/sbin/backup-manager --version

The comments in the config file make editing it quite straight forward.

sudo vi /etc/backup-manager.conf

One minor points:

  • Separate multiple backup methods with a space, eg:
    export BM_ARCHIVE_METHOD="tarball-incremental mysql"

To test:

sudo /usr/sbin/backup-manager --verbose

The output folder you specified (/var/archives) should now contain some .tar.gz versions of your data. Hurrah!

Getting the files offsite

Originally I intended to use Amazon’s S3 as a backup store, following Michael Zehrer’s instructions on how to rsync with S3. However, I couldn’t get this to work reliably; so I opted instead for rsync.net which offers standard scp, ftp, WebDav and sshfs access to their geographic backup locations.

Backup-manager can rsync over ssh, which is a quick and efficient way to sync changes over to the remote host..

The first step is get your rsync.net account setup; and set up your ssh so you can access without typing in a password

Then, set the BM_UPLOAD_METHOD to rsync, and configure both the scp and the rsync settings in /etc/backup-manager.conf (pay attention not to prefix remote folders with / ).

Test with:

sudo /usr/sbin/backup-manager --verbose

Once its all working, set up a cron job to call backup-manager daily.

crontab -e

I run backup-manager once per day in the wee hours, and log output to /root/crontab/daily_backup-manager.logs

  0 3   *   *   *    /usr/sbin/backup-manager -v > /root/cronlogs/daily_backup-manager.log

Viola!

post

ASP.NET MVC Beta – Setting properties on ViewControls

In ASP.NET MVC Beta, it isn’t possible to set properties on partials when calling them with Html.RenderPartial.

Rusty Zarse blogged about a useful ViewData helper class, which allows you to set properties by passing values to the partial through the ViewData.

I’ve extended this slightly to enable the following syntax:


Which sets properties on a ViewUserControl like this:

     public partial class YUIDataTable : ViewUserControl
    {
        public string ConfigNamespace { get; set; }
        public string DataTableId { get; set; }
        public bool HideFilter { get; set; }

        protected void Page_Load(object sender, EventArgs e)
        {
            ViewDataDictionaryBuilder.SetPropertiesToViewDataValues(this);
        }
    }

Here is the full helper code.

using System;

namespace MvcHelpers
{
    ///
    /// With thanks to http://www.vitaminzproductions.com/technology-blog/index.php/2008/11/12/setting-properties-using-aspnet-mvc/
    ///
    public static class ViewDataDictionaryBuilder
    {
        public static System.Web.Mvc.ViewDataDictionary Create(object data, ModelType model) where ModelType : class
        {
            return (System.Web.Mvc.ViewDataDictionary)CreateInternal(new System.Web.Mvc.ViewDataDictionary(model), data);
        }

        public static System.Web.Mvc.ViewDataDictionary Create(object data, object model)
        {
            return CreateInternal(new System.Web.Mvc.ViewDataDictionary(model), data);
        }

        public static System.Web.Mvc.ViewDataDictionary Create(object data)
        {
            return CreateInternal(new System.Web.Mvc.ViewDataDictionary(), data);
        }

        private static System.Web.Mvc.ViewDataDictionary CreateInternal(System.Web.Mvc.ViewDataDictionary dictionary, object data)
        {
            AddPropertiesToViewData(dictionary, data);
            return dictionary;
        }

        private static void AddPropertiesToViewData(System.Web.Mvc.ViewDataDictionary dictionary, object data)
        {
            if (data == null) return;

            System.Reflection.PropertyInfo[] properties = data.GetType().GetProperties();

            foreach (var property in properties)
            {
                dictionary.Add(property.Name, property.GetValue(data, null));
            }
        }

        public static void SetPropertiesToViewDataValues(System.Web.Mvc.ViewUserControl viewUserControl)
        {
            foreach (var property in viewUserControl.GetType().GetProperties())
            {
                if (viewUserControl.ViewData[property.Name] != null)
                    property.SetValue(viewUserControl, Convert.ChangeType(viewUserControl.ViewData[property.Name], property.PropertyType), null);
            }
        }

    }
}

Hope that’s useful to you!

post

Announcing the TDD TestHelpers opensource project

Whenever I start working on a project; I invariably find myself writing a collection of TDD test helper methods.  I quick survey of other TDDers reveals the same; and thus the birth of my latest opensource project, TestHelpers (http://code.google.com/p/testhelpers/).

The aim of the project is to centralise all those little test helper methods you end up creating into a useful assembly you can use to jumpstart your next project.  Things like:

  • Comparers
    • Generic object comparers
    • DataSet comparers
  • Test Data generators
    • Builder pattern
  • Automocking containers

For example, I’ve just added an “AssertValues” functor; which helps you check whether the values of who object instances are the same. 

One area I keep using asserts like this is in integration tests; where I want to check that the objects I’m persisting to the database via my ORM actually end up in the database in a non-mangled form.  In this case, I new up entityA, persist it, reload it into entityB and then need to check that all the values in entityB are the same as those in entityA.

A standard Assert.AreEqual will fail, because entityA and entityB are different instances.  But, my helper method AssertValues.AreEqual will pass, because it checks the (serialized) string values of entityA and entityB.

Here is another, simpler example to illustrate the concept.

    [TestFixture]
    public class StandardObjectsTests
    {
        public class StringContainer
        {
            public string String1 { get; set; }
            public string String2 { get; set; }
        }

        [Test]
        public void ObjectsWithSameValue_ShouldBeEqual()
        {
            var stringContainer1 = new StringContainer {String1 = "Test String1", String2 = "Test String 2"};
            var stringContainer2 = new StringContainer {String1 = "Test String1", String2 = "Test String 2"};

            Assert.AreNotEqual(stringContainer1, stringContainer2);

            AssertValues.AreEqual(stringContainer1, stringContainer2);
        }
   }

I’m sure you have a bunch of similar helper methods lying about your projects.

How about contributing them to the TestHelper project?

post

ALT.NET; London; 13 Sept 2008

Intro

Debate over what ALT.NET is; should it have a set of guiding principles like the Agile manifesto?

Continuous integration & deployment

There seemed to be 3 major areas where people encountered difficulties doing continuous integration & deployment.

 

  1. Configuration files
  2. DB schema migrations
  3. Data migrations.
Best practise approaches discussed were:
Config files
  1. Make sure that  your config files are small. and contain only that config data that changes often (DB connection strings, file paths etc).  Put all your “static” config data into separate files (DI injection config etc).
  2. Consider templated config files; where specific values are injected during deploy process.
  3. Keep all config in simple text files in source control.
DB schema migrations
  1. Migration techniques borrowed from Ruby on Rails – generate change scripts by hand or using tools like SQL Compare; and then apply them using a versioning tool like dbdeploy.
DB data migrations
  1. Take backup before data migration.
  2. Ensure app fails quickly if is a problem; cause if data has changed since deployment then cannot rollback.
  3. Consider apps upgrading themselves and running smoke tests upon startup – and refusing to run if there is a problem – this technique is used by established opensource projects – WordPress, Drupal, Joomla.
Mentioned tools: TFS, Subversion, CC.NET, Jetbrains TeamCity, dbdeploy, SQL compare.
Acceptance testing
It seemed to me that the majority of pain experienced in this area results from a lack of a ubiquitous domain specific language:
  • Build a DSL incrementally during short iterations.  Gives you opportunity to refine, fill in gaps, and train whole team to use same language.
  • Without a DSL, acceptance testing via the UI testing becomes brittle, as you end up specifying your tests at too low a level, (click button A, then check for result in cell B); rather than having a translation from acceptance tests in a higher DSL language to specific UI components.
  • Consider prioritised tests – have a set of facesaving tests / smoke tests that always work, and ensure major things are still working (company phone number correct?  Submit order button still work?).  Acceptance tests can be thrown away if they have served their function of evolving the design / team understanding.
  • The acceptance testing trio – Developers test for success – thus automated testing only tests happy flow – still need exploritory testing by someone with testing mindset; what happens if you do weird stuff?  Tester must have domain knowledge.  Business – what are should happen?  Don’t let developers be forced to make up business rules?
  • Ensure all layers of stack (tests, manuals, code, unit tests) use the same DSL language.
  • How do you get workable acceptance tests – see Requirements Workshops book
  • Short iterations – more focus, incremental specs, opportunity to discuss missing test examples.
  • Key is having a ubiquitous language encoded as a DSL (domain specific language) – develops over time, enables automated accpetance tests, 
  • Sign off against acceptance tests (Green Pepper tool – capture & approve acceptance tests)
  • Talk: The Yawning Gap of ?? doom – infoQ, Martin Fowler
  • Avoid describing these activities as “testing” – people avoid because testing has low social status.
Mentioned tools:  White for Windows GUI testing
Domain driven design
  • Discussion around the difference between DDD; where we treat the concepts & actions as central; vs DB centrered design, where we’re thinking about the data as central, and UI centered design, where the screens are considered central.
  • Concensus was that domain shouldn’t be tightly bound to the DB, or the UI.
  • Ideas around passing DTO objects up to view (UI, webservices etc), andchange  messages bad from view, indicating how the domain should be changed (rather than passing the whole DTO, where you don’t know what has changed).
BDD
  • Defined as Dan North’s Given, When, Then
  • Is it any difference from Acceptance testing? Only that it is better branding, because BDD doesn’t have the word “testing” in it; which prevents people being switched off hearing the word test when discussing specifications.
  • BDD is writing failing acceptance testing first; before writing code.  
  • Unit testing is ensuring that the code is built right, but acceptance testing / BDD ensures that the right code is built.
  • Toolset is still immature.  Fitnesse .NET & Java tooling is most mature toolset.  Many BDD tools (other than Ruby’s rSpec) have been started and abandoned (nBehave, nSpec etc)
  • BDD is not about testing, its about communicating and automating the DSL.  Be wary of implementing BDD in developer tools (e.g, nunit), which prevent other team members (business, customer, testers) from accessing them.
  • Refactoring can break fitnesse tests, because it isn’t part of the code base.
  • Executable specs (via acceptance tests) are the only way to ensure documentation / test suites are up to date & trustable
  • Agile is about surfacing problems early (rather than hiding them until its too late to address them).  So when writing acceptance tests up front is difficult; this is good, because you are raising the communication problems early.
  • The real value is in building a shared understanding via acceptance criteria; rather than building automated regression test suite.
  • Requirements workshops can degenerate into long boring meetings.  To mitigate this problem
Tools:  Ruby Rspec, JBehave, Twist, Green Pepper
Feedback
In the post conference feedback; everyone was overwhelmingly positive; and found the open spaces format very energising.  Fantastic sharing of real world experiences; introductions to new approaches, nuggets of information; great corridor conversations.  Format that allows human interaction.
Next ALT.NET beers on 14th Oct.
Next ALT.NET group therapy in Jan 2009, with larger ventue.
post

Domain mapping with WordPress MU, Plesk, Apache2 & Ubuntu

Given a WordPress MU install on Plesk running on Ubuntu with Apache2, we want to configure domain mapping so that

user1 can have myblog1.com mapping to their wordpress blog (myblog1.masterwpmu.com) and
user2 can have myblog2.com mapping to their wordpress blog (myblog2.masterwpmu.com)

We need to configure quite a few moving parts:

  1. DNS for masterwpmu.com – this should be an A record, pointing to the IP of your server
  2. DNS for myblog1.com & myblog2.com – these should be CNAME records, pointing to the A record in (1) – eg. masterwpmu.com
  3. Apache2 – we need to alter the apache vhost conf created by Plesk to setup a wildcard alias
  4. WordPressMU – we need to configure it to serve the right content when receiving a request for myblog2.com or myblog2.com

When someone makes a browser request for myblog2.com, the following sequence happens:

  1. myblog2.com is resolved to masterwpmu.com, which is resolved to the IP of your server.
  2. the browser makes a request to the IP, port 80, passing the host header of myblog2.com
  3. Apache intercepts the request to point 80, checks through all its known vhost server aliases, and not finding a match redirects to the wildcard alias pointing to our WPMU install
  4. WPMU gets the request, matches the host header to the correct blog content, and returns the relevant page.

So, how do we configure this?

  1. Create a new Plesk site, with its own domain name (eg. masterwpmu.com) & install WPMU.  Ensure this works.
  2. Create a new CNAME record myblog2.com which resolves to masterwpmu.com (Its also possible to setup an A record pointing to the same IP as masterwpmu.com; although this will break if the IP of masterwpmu.com ever changes).  Google has a nice set of instructions for doing this on most major DNS providers (obviously you’ll want to point to masterwpmu.com rather than ghs.google.com ;) )
  3. Edit the Apache2 vhost conf created by Plesk at: /var/www/vhosts/masterwpmu.com/conf/httpd.include, changing:
    ServerAlias *
    <Directory>
    AllowOverride FileInfo Options
  4. restart Apache2 ( /etc/init.d/apache2 restart)
  5. Log in to the WPMU install as admin, and create a new blog.  Edit the new blog, and change the Domain & FileUpload Url to myblog2.com and http://myblog2.com/files (all the other Urls are automatically updated when you save)
  6. Browse to http://myblog2.com !

Gotchas:

  • You can only have 1 wildcard Apache ServerAlias per IP

Hope that helps!

post

When to make technical stories?

During initial sprint planning, stories correspond to user features, and typically follow a

As a [user type]
I can [some action]
So that [some benefit]

structure.

Its important to keep the stories focused on features, rather than on tasks; because we need the users / product owner to be able to decide which stories to add or remove.  (A user cannot decide which tasks to add or remove, because the dependancies aren’t obvious).

However, during development of a particular story, you will often come across an area of the code that needs to be refactored.  A classic example is the case of removal of duplication; where as the design has evolved we discover additional areas of common functionality.

It can be tempting to attempt to work this refactoring into the current story, and if the refactoring is relatively small, this is a good idea.

However, in many cases the refactoring is too large to do without increasing the complexity of the story so much that it might not get finished in the current sprint.

This is the time to create a new “technical story”, which encompasses the refactoring (and perhaps any related work).

Its important that this block of work becomes a story to increase its visibility to the team, and to the product owner.  I’ve found that other team members always have useful input (hey, area Y of the team needs that too), and the product owner gets to prioritise the refactoring along with other stories.

This also makes plain to all the stakeholders why technical debt is increasing – if too many of these technical stories have been neglected in favour of new features.

post

Adding a prompt option to Html.Select in ASP.NET MVC

Digging around the source code to the ASP.NET MVC Source Refresh, I discovered a useful extension to the Html.Select helper method.

Say you want a select dropdown with a NULL case, which instructs the user to make a selection.

Eg: You want your first option to be:  ==Select==

To do so, call Html.Select with a htmlAttribute of prompt=”==Select==”

Like:

<%=Html.Select((“Quote.ClientId”), ViewData.Clients, “Name”,”Id”, ViewData.Quote.ClientId,0,false,new {prompt=”==Select==”})%>