Green Programming

LAPR - RoR on Hardy VM w/Postgres

2009-10-17T18:11:00.010-05:00

Well, if there is LAMP, LAPP and LAMR. What about LAPR - Linux, Apache, PostgreSQL and Ruby on Rails? My friend Ben said he tried out Ruby (he's a Python guy), but it didn't support Postgres very well. I wanted to see for myself, (having just arrived from Missouri- the ShowMe state). First I tried installing Postgres on an existing RoR VMWare appliance, but for some reason, could not get Ruby talking to the gem.

Next I tried the opposite approach. Converting an existing LAPP iso into LAPR. I am more familiar with installing Rails on Ubuntu than installing PG. That went much smoother, after, of course gnashing my teeth on the previous install.

Turnkey[1] makes some nice appliances for testing things out and for deployments. They have a Rails stack, a LAMP and the aforementioned LAPP stack. (They also have a Django Python one I'm dying to try out). They are based on Ubuntu and seem somewhat affiliated with them. They seem to all use Hardy because that is the latest LTS release. There is also a way to deploy your app in the 'cloud". For some appliance vendors this means an Amazon EC2 instance. For Turnkey, they are using VPS.NET (see the site for details).

What I like about Turnkey appliances is they are distributed as (Live/Installable) .isos and not pre-built vdks or other VM images. They will work anywhere even burned to a CD and booted from there. They also offer a Core appliance where you can build your own and utilize the web based interface they've built.

Once installed or booted from the Live CD, you point your web browser at the provided IP address or SSH into it. (using the root password you set up during the install). From there you apt-get install away! Now, Turnkey keeps their appliances up-to date daily with security updates, but not with application updates. I had to apt-get update and then apt-get upgrade. On my LAPP instance, this seemed to update PostrgeSQL to 8.3.8.

As I am wont to do, I installed Ruby and Rails using the tried and true method outlined on the Download page for Rails [2]. This involves compiling from source. To do this on Ubunt, see my earlier Post: Ruby with readline on Ubuntu [3].

Once you have Rails installed, you need to install the sqlite3-ruby gem before testing it out, as Rails is configured to use Sqlite3 out of the box. But to install it, you have to first install Sqlite3 and its development library.

sudo apt-get install sqlite3 libsqlite3-dev
sudo gem install sqlite3-rub

At this point, create a Rails app and start the Webrick server. Point your browser at the ip address:3000 and you should see the familiar Rails Welcome page. Clicking on "About your application environment" should reveal all the various gem versions including the DB adapter:Sqlite3.

Next, set up a model using the scaffold generator:

./script/generate scaffold Post name:sting comment:text
rake db:migrate

Going to ip address:3000/posts should show the empty posts model you just created. Now, on to installing Postgres support:

sudo apt-get install libpq-dev
sudo gem install pg

The correct gem to install is 'pg', no matter what you may have heard. Next setup config/database.yml:

production:
adapter: postgresql
database: app
username: postgres
encoding: utf8
pool: 5
timeout: 5000

Turnkey LAPP ships with the postgres user having no password. Also, Postgres defaults to US-ASCII encoding, but Ruby expects UTF-8. (Ruby 1.9 lets you specify different encodings).

Finally, do another:

rake RAILS_ENV=production db:create db:migrate
# close and restart the server in production mode:
./script/server -e production

Now, enter any posts in your web browser, and they should show up in the phpPgAdmin screen.

You have successfully created a LAPR appliance. Go collect your fame and fortune!

[1] http://www.turnkeylinux.org/

[2] http://rubyonrails.org/download

[3] http://greenprogrammer.blogspot.com/2006/05/ruby-wreadline-on-ubuntu.html

Saint Louis: The Pursuit of Rubyness ('06)

2006-12-30T11:49:00.000-06:00

Well, this has certainly been a breakout year for Ruby in the St. Louis, Missouri area. Our little Ruby brigade (Stlruby.org[1]) saw its attendance mushroom from an average 7-8/meeting to around 20. So much so that we are seeking a larger space to hold meetings. We have been thankful to Mark Volkman and the staff at OCI [3] for sponsoring our group and providing a place to meet since our second meeting. Thanks Mark and everyone at OCI.

Curt Hibbs kicked off our first meeting at a dinner with Dave Thomas at the Gateway Software Symposium (the "No Fluff, Just Stuff tour,") in March of 2005. I wish I could have been there, but we reprised the dinner this year in March. We got to meet Dave, Bruce Tate, Justin Gehtland, Neil Ford and some other NFJS'ers. These guys are all very young, super brainy and quite funny. I had a blast at the dinner. Pics here[4]

Curt also presented this year at RailsConf in Chicago, a talk titled "Inside Instant Rails," about his ubiquitous InstantRails project for Windows. Curt hasn't sat on his laurels for both the InstantRails and One-Click Ruby projects for Windows. 2006 saw a new book co-authored by Bruce Tate and Curt: Up and Running Ruby on Rails. [5] He also contributed to Ajax Hacks by Bruce Perry [6]. His One-Click Ruby project hit 1 million downloads in November. It is always near the top of the most active projects on Rubyforge. Sliiiiiick!

Our numbers may have grown, but the reason might not be solely due to the explosion of interest in Rails. Occasionally, we've asked our membership what they are doing with Ruby and the answer seems more or less split 50-50 between Ruby and Rails work/interests. St. Louis has a fairly large number of web development organizations and a lot of enterprise shops in the biotech, aerospace and financial industries. Ruby is beginning to make inroads in Java shops and even a few Microsoft shops (of which there are a lot in this area.) Mark has even used Ruby at Scott Air Force Base.

Speaking of Mark and NFJS tour, Mark gave a presentation titled "Ruby Tools" this year. Mark's talks are always well done and very interesting even if you've never heard of the subject. I remember his Distributed Ruby presentation (when we could still all fit in OCI's conference room,) with great appreciation. I went home and built a Rails Wiki using no DB, just the DRb stuff. Mark will return with "Ruby Plays Well With Others" at the 2007 Gateway Software Symposium. The talk will cover JRuby and writing C extensions. A few people from our group attended the talk by Charles Nutter on JRuby at the Gateway Java group. Ruby and Java are certainly two great tastes that taste great together!

Mark and Rob Smith have taught various Ruby and Rails classes at OCI in 2006 with more to come in 2007. Ruby training has seen its biggest increase in 2006. The interest has led two of our members, Mike Sullivan and Jeff Barczewski to form a new training consultancy (Inspired Horizons [7] specializing in Ruby and Rails training. It kicks off in 2007, registration is now open for the first workshops which will cover enterprise stuff with Rails and JRuby. (There is that Java thing again.)

Our membership sure gets around. In addition to Curt, Jeff, Mike and Kyle Cordes attended the RailsConf in Chicago. (I am counting Kyle as a member and sponsor at-large. He currently hosts our web site, and promotes Ruby in his Agile talks. Can't wait for your Lua Users Group to start up, Kyle!) Around town, Curt, Mark, Rob, Jeff and I have given talks to various other users groups. Besides Stlruby, we have presented at St. L Linux (stllug), StLUnix (sluug) St. Charles LUG (stclug) St. Louis Web Developer's (stlweb) and the St. Louis XP users group (xpstl) Several of them and other members also taught at the all day St. Louis Code Camp.[8] David Holsclaw gave a presentation on RoR and Jeff taught a class on Extending Rails featuring MasterView (more on that below.) Cory Foy (a member at-large in outstate Missouri,) taught a program on Ruby for C#/.Net developers comparing the languages and Rails and ASP.Net. I am sure there will be more of this in 2007. Stay tuned.

Speaking of Jeff, you may not be aware of his excellent Rails project MasterView MV is an attribute language for Rails. You can use WYSIWYG editors like DreamWeaver to edit your views as regular .html files and put attributes on tags like a, input and select. MasterView will render them as ERb in .rhtml files on the fly. 2006 saw the release of version 0.3 which featured a new DSL for writing your own customized attributes. Sweet!

Sean Carley has also been very active this year in Ruby projects. He contributed the cat2rb quiz to the weekly Ruby Quiz [9] He also founded the weekly hacking night in St. Louis. Attendance was low, so it is currently on hiatus. There are a lot of monthly IT related events in St. Louis, so much so that nearly every weeknight (and quite a few weekends as well) can entertain us. Balancing that with work and family can be quite tough. Sean has contributed to ruby projects like ParseTree, checkr, and Zentest. He has also been involved in many book and other writing projects. Be sure to check out his blog [10]

Quite a few of us are active in the blogosphere (oh how I loath that particular term. Forgive me.) Besides Sean's check out Curt's writing on O'Reilly's Ruby web log. [11] Curt also has his own Blog featuring a lot of Ruby wisdom [12] Also check out Pat Eyler's (a member on our mailing list, but lives in Utah,) blog [13] Pat also writes for the O'Reilly Ruby blog. Kyle and Cory also have blogs. They are at [14,15] respectively Yippee for self-named blogs!

Some of our members have taken what they have learned and applied it to actual stuff. Steve Molitor and John Hohlen have developed a fantasy golf site using Ruby on Rails. It is just getting off the ground, but they hope to grow the league this year. Check it out at [17]

What about yours truly? Well, I have given 4 presentations on Ruby and Rails at various users groups as I mentioned before. I am presently involved as a partner in a full time Rails consultancy (WDT Solutions [16]). We just finished a rather large Rails+Masterview commercial project for a company in the health care field. I contributed one bug to RSpec and a number of suggestions and feedback to Jeff and Deb on the Masterview team.

Our group has grown so much this past year, that we have decided to actually use a bit of organization. To that end we have formed a steering committee. We meet for about an hour at Friday's before the main meeting. Our first accomplishment is to lay out the presentations for the first few months of 2007. We also migrated the mailing list from Yahoo Groups to Google Groups, which has a much nicer management and user interface.

Well, I am sure I've missed something (or many somethings.) If I didn't cover your book, project, presentation in 2006, please email me and I'll incorporate it. That's it for ruby in St. Louis in 2006. 2007 is shaping up to be another fantastic year in the wide world of Ruby. Prognostication is not usually my forte, but if I had to guess at happenings in '07, I'd say that Curt will become even more famous. Dave and Bruce, et. al. will show up at the NFJS tour and have food with us. More projects will be contributed to, more blog posts written, more presentations will be given, more conferences will be attended. Outside of Japan, Seattle, Portland and Utah, St. Louis will be the place to be for all things Ruby.

Have a Happy New Year!

Ed
[1] StlRuby http://stlruby.org/ruwiki/ruwiki.cgi/
[2] New Google Groups for us: http://groups-beta.google.com/group/stlruby
[3] OCI http://www.ociweb.com/
[4] Pictures from our dinner with dave, Bruce and folks at the Gateway Software conference
http://www.cornetdesign.com/images/DSC00476.jpg (Dave is the one imbibing You can barely see the top of my head)
http://www.cornetdesign.com/images/DSC00477.jpg (Jeff is the one at the left closest to the camera.)
[5] Up and running Ruby on Rails http://www.amazon.com/Ruby-Rails-Running-Bruce-Tate/dp/0596101325/sr=1-1/qid=1167512231/ref=sr_1_1/103-6671475-6469421?ie=UTF8&s=books
[6] Ajax Hacks
http://www.amazon.com/Ajax-Hacks-Bruce-Perry/dp/0596101694/sr=8-1/qid=1167512145/ref=pd_bbs_sr_1/103-6671475-6469421?ie=UTF8&s=books
[7] Inspired Horizons, Ruby Training http://inspiredhorizons.com/
[8] St. Louis Code Camp http://www.stlcodecamp.org
[9] cat2rfb http://rubyquiz.com/quiz77.html
[10] O'Reilly Ruby Weblog http://www.oreillynet.com/ruby/
[11] Curt's Comments http://curthibbs.wordpress.com/
[13] Pat Eyler : On Ruby http://on-ruby.blogspot.com/
[14] Kyls's blog http://kylecordes.com/
[15] Cory's blog http://www.cornetdesign.com/
[16] WDT Solutions, LLC. http://www.wdtsolutions.com
[17] Fantasy Golf Site http://www.rgfgc.com

Testing XML-ish stuff in Rails

2006-08-23T17:22:00.000-05:00

Hi fellow Railsians,

Today post is in answer to a question from Chris T on the Rails list. Chris asks:
"Probably dead obvious, but are there any assertions for easing testing
of xml output, both for builder templates (for RSS feed -- something
like a version of assert_tag) and for the new restful stuff."

I've seen this question come up several times in the past. The trouble with assert_tag and assert_no_tag is that they are hooked directly to the response object of your controllers. Which means you can only use them in functional or integration tests. You could fake out a response object, but there is a simpler solution, just recreate them. They are essentially a call to the find method of HTML::Document.

Here is what I did: Add the following to the end your test/test_helper.rb:


  # Add more helper methods to be used by all tests here...
  def assert_xml_tag(xml, conditions)
    doc = HTML::Document.new(xml)
    assert doc.find(conditions), 
      "expected tag, but no tag found matching #{conditions.inspect} in:\n#{xml.inspect}"
  end

  def assert_no_xml_tag(xml, conditions)
    doc = HTML::Document.new(xml)
    assert !doc.find(conditions), 
      "expected no tag, but found tag matching #{conditions.inspect} in:\n#{xml.inspect}"
  end

I chose to use the actual value as the first parameter rather than the expected value because it allows for nicer looking paramter hash values. To use it, use just like assert_tag and assert_no_tag:


def test_xml_correct
   assert_xml_tag "", :tag => 'z', 
     :parent => {:tag => 'y',
                :parent => {:tag => 'x'}}

I usually have a default object under test with a single method to output. This allows for a micro-DSL-ish thing to reduce typing overhead:



...
protected
    def assert_render(conditions)
       assert_xml_tag @thing.render, conditions
    end

Simplified user roles in Rails

2006-08-01T16:53:00.000-05:00

Usually, the discussion around user authentication and authorization in Rails revolves around an authentication plugin, engine or a login generator. Next, you might get into the types of users and this is where you might see a discussion about Roles, UserRoles and Users. There is plenty of tutorials about this usually describing an example of HABTM or has_many :through. But for simple cases, you can get the effect of user roles without much effort at all, just through normal Rails associations.

In addition to your 'users' table, create tables for each type of role. These can contain attributes specific to that type of user. For instance, you can have a admin type of user with specific flags allowing different types of CRUD access. A business contact user might have additional ways to be contacted (phone extension, cell phone, mail stop, etc.) For each table, add a user_id foreign key. In the models, make sure these all 'belongs_to :user'.

Then in the User model, add a 'has_one :admin' and 'has_one :business_contact' etc. for each associated table. The has_one association adds methods to the model to make it easy to query the relation:


user = User.find 1
if user.admin
  # do admin stuff
end
...
details.save unless user.business_contact.nil?

Each user can participate in more than one role. This keeps the user login centralized and the authorization stuff very readable. Adding a new type only requires adding a new table and updating app/models/user.rb to add the has_one associtation.

This simple strategy might not be enough for your needs. It doesn't work if you anticipate adding user roles on the fly to a running system. In this case, you'd probably want to look into a permissions system tied to a HABTM based Role and UserRole model. I've only described a solution for a fixed amount of user types, more or less hard coded into the models.

Testing Helpers and Helping Testers

2006-07-07T17:42:00.000-05:00

The Rails helper_test plugin [1] is a useful thing that provides you with more tools to better test-cover your code. View helpers that live in app/helpers should be tested in isolation from unit and functional/integration tests. This is true just on the face of it. But additionally, you get to access Rails helper methods which you are usually combining with your own stuff. Hard to do that in normal unit tests, and the setup in integration tests is a little too bloated for my taste.

There are a few tricks that can be used to make this run even smoother. The first trick is to utilize assert_tag to check your HTML generation. assert_tag wants to check the body in the @response object which is expected to be a valid XHTML string. But you are not in the request/response environment of functional/integration tests, so @response is not available. Not to worry, we can just fake it out for now. Create a class with an accessible instance variable that is an instantiated string named body:


class Response
attr_writer :body
def initialize
  @body = ""
end

def body
  "<x>" + @body + "</x>"
end
end

In your test method, assign @response = Response.new. Assign the output of your helper method to @response.body. Now assert_tag will work in all it's glory.

The next trick involves testing the image_tag helper. If my helper generates one or more img tags, they can be hard to test with assert_tag. The reason for this is due to the change in Rails to help the various asset tags work better with browser side caches. The helper
image_tag "graphic.gif"
results in something similar:
<img src="/images/graphic.gif?12345678">

where the parameter after the ? is a file based timestamp. This helps the browser to decide to get a new copy of the asset or not. But calling assert_tag :tag => 'img', :attributes => {:src => "/images/graphic.gif?12345678"} is brittle, since if you update the timestamp, (say the next time you get an update from Subversion,) your tests will break. The easy solution to this is to force the ASSET_ID to always be a fixed string. Add this line to config/environments/test.rb:

ENV['RAILS_ASSET_ID']=12345

Now this will be constant in your tests:


def test_image_asset
   @response = Response.new
   @response.body = my_helper_that_returns_an_img_tag
   assert_tag :tag => "img", :attributes => {:src => "/images/graphic.gif?12345"}
end

[1] http://nubyonrails.topfunky.com/articles/2006/04/07/test-your-helpers

StL.rb Hacking Night - Hacking the Ruby Quiz

2006-05-03T12:31:00.000-05:00

Well, once again Craig, Sean and I met this past Tuesday for the weekly hacking night of the St. Louis Ruby Brigrade. We decided to work on the current Ruby Quiz which Sean had submitted. It was a lot of fun and a good idea for what to work on if there is no other ideas.

This week's quiz [1] concerned cat2rfb.rb which from the RubyQuiz site he says : "The goal of cat2rafb is to implement a command line utility that will allow you to submit arbitrary text to http://rafb.net/paste/ and get the URL for the text back." You have to understand what rafb.net/paste is all about. This is a site where you can post snippets of code or text then it redirects you to a page where your code is displayed and syntax colorized. But the cool thing is the URL can be shared with others to collaborate.

From their FAQ [2]: "What is a 'nopaste' site?

A 'nopaste' site allows people to paste chunks of code for others to view. This is useful for situations such as asking for programming help on IRC, where it is frowned upon to paste chunks of code to the channel or to individual. With a nopaste site, the user pastes his or her code to the site and is given a url to provide to others so they can find the code."

What Sean wanted was an easy command line way to paste the code from files, or stdin w/o having to go through the web site. Bonus points if you convert the resulting URL from rafb.net through rubyurl.com which is a Rails equivalent to tinyurl.com that converts long URLs to short ones. (Not that the resulting rafb.net code display URLs are that long anyway.)

Sean hadn't actually come up with an answer for his own quiz yet, so that made me suspect he wanted to prime the WHN attendees with some actual cool work to do. In any case, it was a fairly good learning session for me. We actually set up an IRC session on freenode/#stl.rb to share links. Eating our own dogfood, I guess.

The basic idea is becoming familiar with Net::HTTP posts and gets. Sean went for a Ruby Golf approach, while I took the opposite approach. I wanted to provide options and error checking and read from an optional config file. The fields on the nopaste form allow you to choose from different languages, choose a nickname and give it a description. This makes it easier to find in the Recent pastes screen.

The other tricky part is understanding how to follow a redirect from Net::HTTP#post. Turns out Net::HTTPResponse has a bunch of subclasses. So you can do a 'case when' around the response object and then query its 'location' key (it behaves like a hash.)

I also figured out the issues wrt using GetoptLong and ARGF together. You need to process the arguments first, then set a post argument to ARGF.readlines. Also, RDoc::usage is a seriously cool thing. It looks at the comments at the top of your file and processes them into an usage statement and then exits. How can it see the comments in a file that calls it? Very deep Ruby-fu there. But this fits nicely with the DRY rule (comments and usage decared just once.)

So, we finished up right when the meeting was supposed to end. This kind of exploration to solve the quiz was well suited to the amount of time allocated for a hacking night. Look on ruby-talk for mine, Sean's and other answers to this week's quiz.

[1] http://www.rubyquiz.com/quiz77.html
[2] http://rafb.net/paste/faq.html

Ruby w/readline on Ubuntu

2006-05-01T12:41:00.000-05:00

Getting Ruby running on Ubuntu 5.10 (Breezy) can be tricky. The issue is Breezy wants Ruby 1.8.3 from apt-get, and that has known problems with Rails 1.x. I usually recommend people install ruby 1.8.4 from source to work with Rails 1.1. But the base install of Ubuntu doesn't have many source build tools and libraries. This shows up in various places. For instance, running 'script/console' barfs complaining about missing the readline require. That is because irb wasn't built with it in the 1.8.4 build tree. Another issue you may have run into is support for rubygems is lacking due to a missing zlib development library. This may have messed up your install of Rails or various plugins.

Before we start read this guide: [1] Be sure to add the extra repositories, if you haven't yet.

Here are a bunch of commands that I executed to get Ruby built.


apt-get install build-essential
apt-get install bison byacc gperf
apt-get install zlib1g-dev
apt-get install libreadline5 libreadline5-dev
apt-get install libncurses5 libncurses5-dev temcap-compat
apt-get install libssl-dev

If you have previously tried to build it, then rm -rf ruby-1.8.4 and tar zxvf ruby-1.8.4.tar.gz; cd ruby-1.8.4; ./configure; make; sudo make install

Retest in your rails app: script/console. Everything should work.

The last install is for openssl development headers and libs. This is a Rake dependency in certain cases. (Not Rails specific.)

[1] http://ubuntuguide.org/

Further adventures along our Trail

2006-04-20T12:47:00.000-05:00

More about Migrations.

Here are a few more things I've learned about Rails 1.1 DB Migrations, or I forgot to note in my last post.

Make small migrations. It is ok to have very many migrations. Goes with test a little, code a little, repeat, refactor, etc.
Don't put in the id column for the table. The create_table method does that by default.
When naming your migrations, only use "pure" CamelCase. E.g. This "AddSSNToClient" doesn't work because ./script/generate migration will creare 003_add_ssn_to_client but the name of the class will be AddSSNToClient, and 'rake migrate' will look for AddSsnToClient.
Use db/schema.rb to populate your production databases and other development servers. This will still preserve your version # and should allow you to back out to previous versions even on these DB instances.
Commit any code changes prior to your first migration. Tag this so you know where you came from.
Use rake db:schema:dump one time when you want to start using migations from a pre-existing SQL DDL generated schema. At this point you will be at version 0. Generate your next model or generate a new migration which will be version 1. If you back out via rake migrate VERSION=0 you will be at the last point before you started using migrations.
You can port your development environment to a new computer and a new instance of your DB server. Check out your head SVN trunk. I have a Rake task that generates my DBs and sets up config/database.yml*. run that or create at least your development and test dbs. Then run 'rake migrate'. You can backup and restore your old data into your new MySQL instance.

In my last post, I said you should start Rails Migrations from Day 1. That is true, but what if you have an existing schema? That's OK, too. It will just start at revision 0. From that point forward, you want to do migrations only.

If you are using MySQL, then I recommend doing a

mysqldump -t db_name >db/mydbstruct.sql

and commiting that before continuing. You should even make a full backup of your database, which is always a good idea when playing around with DBs.

The sequence (for legacy DB conversion to Rails Migrations) is:

mysqldump -t mydb >db/mydb_struct.sql
svn add db/mydb_struct.sql
svn commit -m "Commit DB Version 000 prior to first migration"
"svn copy" to tags/DBVersion_000
./script/generate migration AddColumnToMytabs
Edit db/migrate/001_add_column_to_mytabs.rb and add your column(s)
rake migrate
rake # to run tests
svn commit -m "Switched to Rails Migrations. DB Version 001. Added column to Mytabs."
svn copy to tags/DBVersion_001

Or, if you are creating a new model,

./script/generate model NewModel
Edit db/migrate/001_add_columns_to_mytabs.rb
... etc.

Some have suggested that you should do rake db:schema:dump then copy and paste the output inside db/schema.rb into a new migation. I am not exactly sure when this is useful. Perhaps if you want to start using migrations from scratch so that your first migration (001) creates all your legacy tables.

* This is my rake task. I copy it into lib/tasks (actually I have a generator to do that for me.) The fix_yaml task is needed in my case because I am running MySQL 4.1.x on Ubuntu. The socket is in a different place than Rails thinks and I don't want to be root for my personal DBs I create.

lib/tasks/create_db.rake


require 'yaml'
fn_yaml='config/database.yml'
myuserid='me'
yml=YAML::load(File.open(fn_yaml))

desc 'Creates databases in mysql'
task :create_db => [:fix_yaml] do
      databases do |db|
              system "echo create database #{yml[db]['database']} | mysql"
      end
end

desc 'Drops databases in mysql'
task :drop_db do
      databases do |db|
              system "echo drop database #{yml[db]['database']} | mysql"
      end

end

desc 'Fix database.yml'
task :fix_yaml do
      databases do |db|
              yml[db]['username'] = me
              yml[db]['socket'] = '/var/run/mysqld/mysqld.sock'
      end
      File.new(fn_yaml, 'w').write yml.to_yaml
end


def databases
      %w{development test production}.each do |db|
              yield db
      end
end

Happy Trails

2006-04-19T18:36:00.000-05:00

Happy Migrations.

Rails 1.1 has a really cool mechanism for dealing with updates to a database during development: Migrations. Actually, I think migrations predates Rails 1.1, but that is where I first discovered it and it is supposed to be improved. Anyway, migrations are the way you should do all your database development from day 1. It makes the handling database changes much more agile. I tried to wrap my head around migrations from reading online docs, but I was quite unsuited to that task. Thankfully, I attended the 2nd Weekly Hacking Night [1] of StL.rb with Sean Carely and Craig Buchek and I learned what a dunce I had been.

Here are some notes about migrations that I have collected. Others have written eloquently on the mechanics of migrations [2], [3]. I am primarily concerned about the process of migrations. What to do and when to do it.

Lets start with some advice from my little green friend:
Yoda says: If once you start down the migration path, forever will it dominate your destiny, consume you it will.

Lets also assume some starting parameters:

You are starting a new project.
You are using Rails 1.1
Your DB engine is supported by schema-type Ruby. IOW, you can use db/schema.rb instead of the old schema.sql style. (These are at least MySQL, PostgresSQL and SQLLite, mssql. And some rumored others.) I will be using MySQL here.
You are using Subversion for version control.

and finally:

You are committed to never use SQL to modify your DB structure again.

Whew, glad thats out of the way. Migrations and models and tests are well integrated in R1.1. You should use the model generator to create them. Lets begin at the point where we have created our fresh Rails app, we have created a repository with trunk, tags and branches folders in SVN for it, imported our created rails app into trunk and checked out a new working copy. We should have adjusted config/database.yml to reflect our connection and login parameters for MySQL. We may have done other stuff like create controllers and some views, but we haven't created our first model yet. We haven't even created our databases yet. This is where we start with our first migrations.

Our Rails app is called trails and it will feature campers, equipment, counselors, hikes, etc. Lets create our databases:

for d in development test production; do echo create database trails_${d}\;; done | mysql

Now lets add our first model: Camper.

./script/generate model Camper
exists  app/models/
exists  test/unit/
exists  test/fixtures/
create  app/models/camper.rb
create  test/unit/camper_test.rb
create  test/fixtures/campers.yml
create  db/migrate
create  db/migrate/001_create_campers.rb

Note that it created db/migrate and 001_create_campers.rb for us. This is how we are going to create our first table. Note that the version # is 001. As you create new migrations, these will increment, e.g. 002_xxx.rb, 003_yyy.rb, etc. When you run your migrations, rake will load them in that order and the highest versioned file in db/migrate will become the schema version #. More on that later.

Next you need to edit db/migrate/001_create_campers.rb and add columns to the table. Next run

rake migrate

This will create the table in the development db and create a new table called schema_info with a version column. That version will be set to 1. It will also create a file called db/schema.rb with all our current table settings. Schema.rb will contain the current version of the DB since our last rake migrate, so that is a quick way to discover the current version #. The test framework will use db/schema.rb to create the test database and set its version to 1 as well. Just running

rake

will do that for you, as well as running all your tests.

We next create a new model Equipment using the same procress of generate model, edit db/002_create_euipment.rb and rake migrate. Now our development schema_info.version is 2 and we have a new table called equipment.

Next we might decide that each camper is responsible for some pieces of equipment. We need to link our Campers to our Equipment on a foreign key which we forgot to add when we created the model. We do that by generating a new migration:

./script/generate migration AddCamperIdToEquipment

Add our column to db/migrate/003_add_camper_id_to_equipment,rb and then run

rake migrate

We are now at version 3. Adding some tests, then runing rake with no options will update our test DB to version 3 as well.

At this point, we have done a few migations and have a code base that is passing all our tests. Time to check in. Before we do, Sean pointed out that we can use Subversion metadata to keep commits in sync with our DB.

svn propset migrate-version 001 .

will do that for us. Later, we can svn propget migrate-version . to query it.

svn status

Add any missing files, then

svn commit -m "Database now at version 003"

Whenever I commit following one or more migrations, I like to tag it so it is recorded that the code base is working to a specific DB release.

svn copy -m "DB Version 003" svn+ssh://ip.of.host.box/path/to/repos/trunk svn+ssh://ip.of.host.box/path/to/repos/tags/DBVersion_003

After a few more models have been generated and migrations executed, we might want to revert to a previous database version. Assuming we are at version 7, we just migrate down with :

rake migate VERSION=6

We can run svn diff to discover what code changes we made since our last tag DBVersion_006 and back those out as well. I generally make it a policy to commit and tag in SVN when I've created a few models (or just one) or I've generated a special migration to add, rename or remove a column.

To summarize:

./script/generate model ModelName
- edit db/migrate/001_create_model_names.rb
rake migrate
rake        # runs tests, updates test DB to current schema.
./script/generate model NewModel
- edit db/migrate/002_create_new_models.rb
rake migrate
rake
# add a missing column
./script/generate migration AddColumnToNewModels
- edit db/migrate/003_add_column_to_new_models.rb
rake migrate
rake
# check in
svn propset migrate-version 003 .
svn status      # discover unversioned files
svn add db/migrate/001_ , 002 ...   etc.
svn commit -m "DB Version 003"
svn copy -m "DB Version 003" svn+ssh://ip.of.host.box/path/to/repos/trunk svn+ssh://ip.of.host.box/path/to/repos/tags/DBVersion_003
...
# revert to a previous version of the DB Current version is 7
rake migate VERSION=6
svn diff svn+ssh://ip.of.svn.box/path/to/repos/tags/DBVersion_006 svn+ssh://ip.of.svn.box/path/to/repos/trunk
# make changes or use svn revert
rake     # revert our test DB to version 6 and rerun our tests.

One last note. If you generate migrations, be sure to name them uniquely. That is because rake will load all the classes in db/migrate in order of version # and Ruby will overrite the first same named class with the last same named class. Use explicit names that reflect what the migration is going to do, e.g. AddClassNameToStudents. This is the ProgramingByIntention [4] principle of XP.

Overall, I have found migrations to fit nicely with the XP test-code-refactor cycle. It opens the database up to being able to be refactored. Subversion is a tool that helps me when I inevitably shoot myself in the foot.

[1] http://sean-carley.blogspot.com/2006/04/stlrb-hacking-nights.html#links
[2] http://wiki.rubyonrails.com/rails/pages/UsingMigrations
[3] http://rails.rubyonrails.org/classes/ActiveRecord/Migration.html
[4] http://c2.com/cgi/wiki?IntentionalProgramming

What is Green Programming?

2006-01-11T12:39:00.000-06:00

This is just an initial post.

Green programming is an attempt to synthesize the principals of environmentally sound policy and the Agile Software set. IOW, just like we should aspire to use renewable energy sources to help the health of the planet, we should also use reusable software elements to create robust, healthy code for our customers. Eco-friendly practices might be thought of as applicable to software devlopment. Just as we are concerned with the various biota and climate of the planet, we should be concerned with the over-all health of the software eco-system. I am specifically concerned with threats to free and open source software from patents and copyright concerns.

Actually, I just needed a title and domain, so I thought I might ride the currently fashionable "Green" movement. Please forgive my impetuousness.