Just Barebones

Benchmarking a system command

2011-02-06T14:03:00.000-08:00


time for((n=0;n<100;n++)); do convert CIMG0022.jpg -rotate 90 CIMG0022_90.jpg; done

real 0m42.438s
user 0m33.934s
sys 0m8.424s

Avg run time: ~ 424ms

Getting started with node.js

2011-01-31T14:19:00.000-08:00

By now everyone has more or less heard of node.js.

Node.js is an exciting project that takes the super fast V8 javascript engine open sourced by Google and wraps it with an event oriented framework that allows it to process concurrent requests blazingly fast. Event oriented (using one of epoll, kequeue, /dev/poll, select) concurrency model is considered to be much superior to thread based concurrency model - point in case nginx vs Apache comparison.

Listed below are some resources that I found very useful to get started on node.js.

Main node.js page
Introducuction to node.js. A must read to understand the architectural decisions behind node.js
Installation
This page contains instructions to install node.js and npm (NodeJS Package Manager).
If you plan to use node-inspector for debugging, install v 0.3.0 or later.
I had trouble installing npm when I installed node in /usr/local. The npm install script complains loudly when invoked as sudo - rightly so as packages can run arbitrary scripts.
Best approach I have found is to install node.js in a local directory as shown in this gist (node-and-npm-in-30-seconds.sh) and then install npm.
Npm
Npm is NodeJs Package manager. Some links of interest : intro to npm, docs and a faq. Also here is a list of npm packages.
Express
To simplify setting up a web server using node.js, use the Sinatra inspired Express framework. It is built on top of Connect. I came across an interesting thread regarding a node.js module called Nitrode which points out high memory usage of Connect when streaming large files.
You can generate a skeleton app using express <appname>
Very useful - details on the skeleton generated using the command above (scroll down half-way)
node-inspector
Needs node.js > 0.3.0
Read up on node-inspector at https://github.com/dannycoates/node-inspector and https://github.com/past/node-inspector
npm install node-inspector

Thread programming ref reshers - ruby style

2009-06-22T22:58:00.000-07:00

A quick link pointer - differences b/w ruby mutex and monitors

Some quick tips:
Monitors are re-entrant - meaning you can nest calls to methods that synchronize access to some data
Mutex is not re-entrant

Ruby blocks - explicit vs implicit arguments

2009-05-26T00:41:00.000-07:00

I wanted to pass an implicit block argument to a call to super. Looks like the most efficient way is to use

def on_ready
unless paused?
super { yield }
end
end

-----------------------

From http://www.ruby-forum.com/topic/71221

require 'benchmark'

def outer11(&bl)
inner1(&bl)
end

def outer12(&bl)
inner2(&bl)
end

def outer21
inner1 {yield}
end

def outer22
inner2 {yield}
end

def inner1(&bl)
bl.call
end

def inner2
yield
end

n = 100000

Benchmark.bmbm(10) do |rpt|
rpt.report("outer11") do
n.times {outer11{}}
end

rpt.report("outer12") do
n.times {outer12{}}
end

rpt.report("outer21") do
n.times {outer21{}}
end

rpt.report("outer22") do
n.times {outer22{}}
end
end

__END__

Output:

Rehearsal ---------------------------------------------
outer11 0.890000 0.010000 0.900000 ( 0.894500)
outer12 0.370000 0.000000 0.370000 ( 0.364880)
outer21 0.770000 0.000000 0.770000 ( 0.776638)
outer22 0.170000 0.000000 0.170000 ( 0.163393)
------------------------------------ total: 2.210000sec

user system total real
outer11 0.490000 0.000000 0.490000 ( 0.491969)
outer12 0.400000 0.000000 0.400000 ( 0.396264)
outer21 0.760000 0.000000 0.760000 ( 0.764508)
outer22 0.160000 0.000000 0.160000 ( 0.161982)

CookierStore::CookieOverflow

2009-03-22T23:53:00.001-07:00


/!\ FAILSAFE /!\  Sun Mar 22 23:51:44 -0700 2009
  Status: 500 Internal Server Error
  ActionController::Session::CookieStore::CookieOverflow

This error is typically due to the user trying to store > 4K of data in the session.

Another source of error is if you try to store $! in flash[:error]


begin
  ....
rescue 
  flash[:error] = $!
end

This tries to store the ruby exception object in the flash (which in turn is stored in the cookie) and you will start getting the cookie overflow errors.

The solution is simple


begin
  ....
rescue 
  flash[:error] = $!.to_s
end

ORA-12516: TNS:listener could not find available handler with matching protocol stack

2008-11-04T14:29:00.000-08:00

We use Oracle Database 10g Express Edition Release 10.2.0.1.0 for our RoR development and were hitting the TNS:listener error all the time. This made concurrent development a real pain and we couldn't leave staging servers and sqldeveloper sessions up.

Google searches turned up really scant information - mainly advices to increase the PROECESSES limit. I am not a DBA, so wasn't sure what needed to be done.
In any case, the following solution worked like a charm and I hope others who have this issue find this post useful.

source /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/bin/oracle_env.csh
setenv ORACLE_SID XE

And then execute the following steps to increase the number processes and sessions:
1) Connect to the database using the sys or system user:
sqlplus SYSTEM/<syspwd>@//<dbhost>/xe
2) To confirm the PROCESSES and SESSIONS values, run the following script:
 sql> col name format A30
 Sql> col value format A30
 sql> select name, value from v$parameter where name in ('processes','sessions');
3) SQL> alter system set processes=300 scope=spfile;
4) SQL> alter system set sessions=300 scope=spfile;
5) Bounce the database to reflect the changes
6) To confirm new values run the script (2).

Building Custom Deployment directories using sprinkle

2008-10-09T11:04:00.000-07:00

At work we deploy multiple RoR applications on each server. Managing the environments for all these applications is a huge headache. We have applications in production using Rails 1.2.5 all the way till Rails 2.1.1. In addition to rails, other gems need updates routinely as well. This breaks applications that have been running smoothly for the past year or so.
Further, new IT policy changes have taken away developer's sudo access on these servers, which makes developing cumbersome and difficult - having to go through IT for every environment update request.

So I set upon the task of developing an easy way to deploy each application in its own environment.
Following are the goals I wanted to achieve:

Each application runs in its own deployment environment. By environment I mean a complete directory structure that has its own ruby and gem installation, along with any additional software such as database, web server, Imagemagick, reverse proxy etc.
The environment should be easy to duplicate for staging and development, thus keeping the production environment isolated until QA is done and the application is ready to be deployed in production
The environment should be easy to build - fire off a script and go get a coffee while the environment is being built and configured.

I gave Fiveruns install a try. It is a great product, but the stack is not extensible - I cannot add additional software components to the install process.

I came upon Marcus Crafter's sprinkle which allows easy provisioning of new VPS systems. It is an awesome piece of code that makes setting up new systems a breeze.

Sprinkle is a new prototype tool that you can use to provision your servers/slices. Its declarative policy/state based approach for specifying how a remote system should be provisioned with intelligent logic to support dependencies, multiple installer types and remote installation is really compelling.

The ruby based DSL used to describe the provisioning setup is easy to understand and maintain.

However, sprinkle in its current form provisioned the entire system to a specific deployment environment - not what I had in mind.
So I created a fork of the sprinkle codebase on github and added the capability to build local deployment directories to sprinkle.
Here is a log of the functionality I added:

bin/sprinkle - updated to accept -d option and create a environment source file at the end
sprinkle.gemspec - added actors/local.rb which was missing
lib/sprinkle/actors/local.rb - updated to set env vars before calling the system command (sets PATH and LD_LIB to the deploy dir), also raises an exeception if a command fails
lib/sprinkle/installers/gem.rb - by default doesn't build ri and rdoc anymore - need to specify build_docs as a gem option
lib/sprinkle/installers/source.rb - custom_install is now an array of commands, added custom_stage , cmd to override default stages, source now accepts a path as a valid option to copy the archive from.
lib/sprinkle/verifiers/ruby.rb - removed sudo while verifying a gem
Added has_successful_execution verifier to verify installations

To get started with this new functionality:

Install the sprinkle gem.
- git clone git://github.com/cpatil/sprinkle.git
- cd sprinkle
- gem build sprinkle.gemspec
- gem install sprinkle.gemspec
Create a sprinkle recipe (download)
Run sprinkle and go get a coffee
- sprinkle -c - v -d deploy_dir -s build_deployment_env.rb
Source the env.
- source deploy_dir/source_env.tcsh
Test the paths
- which ruby
- which gem
- gem list

Now cd to your rails application dir and start up the server - Voila - custom a la carte deployment environments :)

Njoy!

Rails 2.0.2, restful authentication and db:create

2008-05-25T10:16:00.000-07:00

Rails 2.0.2, restful authentication and rake db:create don't seem to play nice :(

rake db:create gives me

rake aborted!
Unknown database 'xxxx_devel'

(See full trace by running task with --trace)

Using rake --trace db:create pointed me to the following line in config/environment.rb:

config.active_record.observers = :user_observer

which I had added as part of configuration setup for restful_authentication.

I commented out this line and rake db:create ran fine. I will investigate more later, but this got me going for now.

rbatis patch for custom (non default) resultmaps

2008-05-19T12:42:00.000-07:00

I had to use the following patch to make custom resultmaps (i.e. non :default resultmaps) work with rbatis:

vendor/plugins/rbatis/lib/rbatis.rb (line 406)

def statement(statement_type, name = statement_type, params = {}, &proc)
...
statement.connection_provider = self
if statement.respond_to?(:resultmap=) && statement.resultmap.nil?
statement.resultmap = resultmaps[:default]
else
statement.resultmap = resultmaps[statement.resultmap]
end
statement.validate
...
end

rbatis is pretty slick, especially when you have to work with legacy databases and schemas.

Oracle Query optimization

2008-05-13T09:06:00.000-07:00

We are working on a RoR application that needs to crunch data across multiple tables (> 7 million records) from a huge Oracle database. The sql queries were initially taking a long time, upto 5 minutes for some queries.
Following are some links and tips to optimize the sql queries - specific to Oracle.

I added the following snippet to the environment.rb which gives me some stats for the queries:

# only run this code in development
if ENV["RAILS_ENV"] == "development"

# modify MysqlAdapter to track transfer stats
class ActiveRecord::ConnectionAdapters::OracleAdapter
@@stats_queries = @@stats_bytes = @@stats_rows = 0

def self.get_stats
{ :queries => @@stats_queries,
:rows => @@stats_rows,
:bytes => @@stats_bytes }
end

def self.reset_stats
@@stats_queries = @@stats_bytes = @@stats_rows = 0
end

def select_with_stats(sql, name)
bytes = 0
rows = select_old(sql, name)
rows.each do |row|
row.each do |key, value|
bytes += key.length
bytes += value.length if value && value.respond_to?('length')
end
end
@@stats_queries += 1
@@stats_rows += rows.length
@@stats_bytes += bytes
@logger.info sprintf("%d rows, %.1fk", rows.length, bytes.to_f / 1024)
rows
end

alias :select_old :select
alias :select :select_with_stats
end

# modify ActionController to reset/print stats for each request
class ActionController::Base
def perform_action_reset
ActiveRecord::ConnectionAdapters::OracleAdapter::reset_stats
perform_action_old
end

alias :perform_action_old :perform_action
alias :perform_action :perform_action_reset

def active_record_runtime(runtime)
stats = ActiveRecord::ConnectionAdapters::OracleAdapter::get_stats
"#{super} #{sprintf("%.1fk", stats[:bytes].to_f / 1024)}"
end
end

# trim blob logging
#class ActiveRecord::ConnectionAdapters::OracleAdapter
#def format_log_entry(message, dump = nil)
#if dump
#dump = dump.gsub(/x.([^.]+)./) do |blob|
#(blob.length > 32) ? "x.#{$1[0,32]}. (#{blob.length} bytes)." : $0
#end
#end
#super
#end
#end

end

Check out this link on Oracle.com and download the Oracle version of query_analyzer. This plugin automatically runs an explain plan query for the select query.
Analyzing
plan_table_output
--------------------------------------------------------------------

--------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost |
--------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 31 | 8598 |
| 1 | SORT UNIQUE | | 1 | 31 | 8598 |
| 2 | TABLE ACCESS FULL | XXX | 128 | 3968 | 8583 |
--------------------------------------------------------------------
First level of optimization: the output of the explain plan shows you which tables columns in the WHERE clause cause a FULL table access. These columns are targets for optimizations by dropping indexes on them. This will result in a explain plan output as shown below. Note the INDEX RANGE SCAN bit.
----------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost |
----------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 25 | 675 | 332 |
| 1 | TABLE ACCESS BY INDEX ROWID| XX_XXXXX | 25 | 675 | 332 |
| 2 | INDEX RANGE SCAN | XX_X_INDEX1 | 430 | | 4 |
----------------------------------------------------------------------------
However, being too aggressive in creating indexes is not recommended. An index is essentially a seperate file on the filesystem and every time the table data is altered, each of the index files need to be updated.
Second level of optimization: Read this link to understand Index selectivity. Here are the steps I took to gather statistics and compute selectivity:

Download and install SQL Developer from oracle.com and connect to the database.
Right click on Table -> Statistics -> Gather statistics
Alternately, you can run the following query:
analyze table XXXX compute statistics;
select column_name, num_distinct from user_tab_columns where table_name='XXXXX'
Note the number of distinct values of the data in the column you are interested in. Alternately run a sql query such as
select count(distinct ) from XXXXX
Run a sql query to get number of records in the table:
select count(*) from XXXXX
Index selectivity = Distinct Column values / Num rows in table
A larger value for Index selectivity is good. This means there are fewer rows corresponding to the column's each distinct value.

In my case, due to a large number of records in the table and the num distinct values of the columns being small, the index selectivity was less than 0.01. So even after adding indexs on the columns in the WHERE clause, the query was slow. In fact it made loading data into the database slower.
However if the query is returning only the indexed column's data in the SELECT clause, the data can be returned from the index file directly (the table data is not referred) and that indeed may result in a faster query.
Another option is to consider a Bitmap index instead of a B-Tree index. Bitmap indexes are useful when the column to be indexed has low cardinality, e.g. gender or marital status. Bitmap indexes work well with Data warehousing kind of apps which B-Tree indexes (typical) are more common with OLTP systems. So consider the Bitmap index characteristics and impact on your application carefully before deciding on using Bitmap indexes. Especially the fact that inserting data into tables with bitmap indexes is a lot slower than tables with B-Tree indexes.
Third level of optimization: Composite indexes. Sometimes indexes with lower selectivity can be combined into a composite index to create an overall index with a higher selectivity. Oracle also supports Index Skip scan which allows a composite index to be used even when all the columns specified in the composite index are not present in the SQL WHERE clause.
Note that using LIKE '%xxx' and other terms in the WHERE clause can cause FULL Table scans. However LIKE 'xxx%' can potentially use an index. There are several options as listed here and here.

Links:

Announcing YUIRails

2008-01-18T22:30:00.001-08:00

Update 2/24/08 - I have uploaded new versions of yuirails.js and yuirails-min.js.
The new version adds file upload capability (YUI + YuiRails + attachment_fu) and fixes a few misc bugs. The file upload capability works with single as well as multiple file uploads (ajax). I will be posting on this in coming weeks. Let me know if there is interest and I will try to put it up sooner. The demo with the new js files is here and is otherwise unchanged.

I am using YUI with Rails at work to develop a complex web application using the treeview, panel, connection, and various other widgets. I used to miss being able to use link_to_remote and remote_form_for without having to include prototype.js (which at the latest version is about 124KB.
So over the Christmas break, I sat down and coded up YUIRails. The intent of this library is to provide a thin layer of logic that glues RJS - PrototypeHelper with YUI connection manager. In its present form, it provides partial support for RJS Prototypehelper, limited to Element methods and Ajax.Request and Ajax.Updater. No support is provided for Scriptaculous effects.

To use this library, simply drop it in public/javascripts. In its uncompressed form it is at 18K, but a minimized version is only 6K. If there is enough interest, I will look into making the code tighter.

Following are a few restrictions in the usage:
a) I have stayed away from extending Dom elements with Element.methods. As a result you can't call $('foo').hide(); Rather you will have to use Element.hide('foo');
b) Following Element methods are supported:
- visible
- toggle
- hide
- show
- remove
- update
- replace

As a demo, a simple rails application is included in the distribution. You will need to create a MYSQL database test_yuirails_development owned by yuirails/testing


create database test_yuirails_development;
GRANT SELECT,INSERT,UPDATE,DELETE,CREATE,DROP,ALTER,INDEX ON test_yuirails_development.* TO 'yuirails'@'localhost' IDENTIFIED BY 'testing';

and then

rake db:migrate

demo.tgz
yuirails.js - 18K
yuirails-min.js - 6.5K

I just setup the demo on a server. Check it out.

Enjoy.

Quick ActionMailer tips

2007-12-25T14:25:00.000-08:00

To include your helpers (defined in application_helper.rb), do a
helper :application
in the mailer class defined in app/models.

To be able to use url_for and restful route helpers in the ActionMailer model, do an include ActionController::UrlWriter and set default_url_options[:host] to your host.
If you want to use html_escape (h) in html email templates, feel free. html_escape is defined in ERB, so it is already available.

To add a Return-Path header
http://www.spacebabies.nl/2007/02/16/return-path-in-rails-actionmailer/

To disble ActionMailer from printing in the production log,
config/environments/production.rb
ActionMailer::Base.logger = nil

I want to send an email to multiple users, but not show other email addresses in the To line. For now I am simply looping over the emails and sending a seperate email
by calling Mailer.deliver_ method for each email addresses. I can easily imagine this to be a configuration option to ActionMailer so it can do this looping itself.

This is a good read on the topic.

Auto reverting Migrations

2007-12-17T21:56:00.000-08:00

I often end up writing some code in self.up that craps out and it is painful to reverse any database changes that may have been committed. I often resort to revert to an earlier version, but that is an extra step I would rather not have to do.

I make sure all the extra code is located after any database changes I want to make and then wrap the whole shebang in a begin-rescue block:


class FixCategoriesOrder < ActiveRecord::Migration
  def self.up
    begin
      add_column :categories, :sort_order, :integer
      # ... other database changes

      # Code 
      [["Immunology", 1],
        ["Biochemistry", 2],
        ["Biotechnology", 3],
        ["Cell Biology", 4],
        ["Genomics", 5],
        ["Plant Biology", 6],
        ["Developmental Biology", 7],
        ["Ecology", 8],
        ["Other", 100]].each do |ar|
        puts "Processing #{ar[0]} - #{ar[1]}"
        Category.find_by_name(ar[0]).update_attribute(:sort_order, ar[1])
      end
    rescue
      self.down
      raise $!
    end

  end

  def self.down
    remove_column :categories, :sort_order
  end
end

Njoy

link_to helper, popup option and IE window.open

2007-12-17T19:12:00.000-08:00

Ah, the amount of time spent on this could have really been well spent elsewhere!

If you trying to use link_to as below:


  link_to "View Image", { :action => "view" }, :popup => ['new_window_name', 'height=300,width=600']
  # => View Image

Be very careful when you specify 'new_window_name'.

It shouldn't have any embedded spaces in the string.

Else IE will barf and throw a runtime exception! Firefox works fine though.

has_and_belongs_to_many (HABTM) and MySQL Duplicate entry error

2007-12-16T13:32:00.001-08:00

A quick post:
If you use migrations to create a join table for a HABTM relationship, you should tell rails not to automatically add an id column in the migration (Agile book - 271).
Otherwise you will get a

Mysql::Error: Duplicate entry '#' for key #

error.


create_table :authors_books, :id => false do |t|
   t.column :author_id, :integer, :null => false
   t.column :book_id,   :integer, :null => false
end

Installing Oracle 10g Express Edition on Linux White box - AMD 64 (x86_64)

2007-12-11T16:34:00.000-08:00

I had difficulty installing Oracle 10g express edition server on 64 bit White box linux earlier. So I had installed it on my WIndows XP box to use it with my RoR application.

Today my windows box crashed and I hunkered down and figured out how to install Oracle on my linux box. There are a couple of tricky steps that could trip you up. I am documenting the steps here hoping it will help someone out.

Download Oracle 10g Express Edition rpm from the Oracle website - oracle-xe-10.2.0.1-1.0.i386.rpm (221136869 bytes)
** Don't use sudo to install the rpm. Log in as root and then run
rpm -Uvh oracle-xe-10.2.0.1-1.0.i386.rpm
Otherwise you will get the following error when you try to run sqlplus to connect to the db.
```
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux Error: 2: No such file or directory
```
Also, the web server component of the Oracle server never starts if you use sudo to install the rpm.
Run configure to configure the server.

** If you get

oracleXE: error while loading shared libraries: libaio.so.1: cannot open shared object file: No such file or directory

then you need to install the 32 bit version of libaio library

sudo rpm -Uvh http://fedora.osmirror.nl/core/development/i386/os/Fedora/RPMS/libaio-0.3.106-2.2.i386.rpm

Source the env before you run sqlplus

source /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/bin/oracle_env.csh

Look in /usr/lib/oracle/xe/app/oracle/product/10.2.0/server/network/log/listener.log if the listener or the web server (127.0.0.1:8080/apex) doesn't start. It may contain possible reasons.
Since the web server is bound to 127.0.0.1, to connect to the oracle web server from a remote machine, you will need to use putty (or similar) to create a ssh tunnel.

ActiveRecord find - Eager loading - :include - deep hierarchy of associations

2007-12-07T22:25:00.000-08:00

While including a deep hierarchy of associations, I found a strange issue. Here is a contrived bit of code to describe the issue:


Post.find(:all, :include => [{ :comments => [:replies, { :author => :gravatar }]  } ])

A comment has_many replies which I want to pull in via eager loading. At the same time I also want to pull in the author and gravatar pictures.

The issue here is you can only have one argument that is a key-value pair and should be the last one. So the following doesn't seem to work:


Post.find(:all, :include => [{ :comments => [ { :replies => :author}, { :author => :gravatar }]  } ])

nor does this


Post.find(:all, :include => [{ :comments => [{ :author => :gravatar }, :replies]  } ])

I will post an update when I figure out how to get the author for each reply eager loaded as well.

acts_as_state_machine enhancements

2007-11-29T18:15:00.000-08:00

Download the plugin enhancement here. Drop it into the lib directory and do a require in the model.

A project I am working on required me to use a state machine to manage the state of a model. After downloading acts_as_state_machine plugin and I set it up as mentioned here.

Though I liked the way the plugin sets up the state machine, it seemed to lack some functionality for my usage:

To be able to define an override event that would allow the user to select the state the model should go to.
While performing the transition from one state to another state, perform some operation that is specific to the triad - event, : from, :to

Let me show you what I mean by extending the example mentioned here.


%> script/generate model Person shirt_color:string trouser_type:string status:string
%> rake db:migrate

%> emacs -nw app/models/person.rb

require 'acts_as_state_machine_overrides'

class Person < ActiveRecord::Base

acts_as_state_machine :initial => :sleeping, :column => 'status'
state :sleeping
state :showering
state :working
state :dating

event :shower do
transitions :from => :sleeping, :to => :showering
transitions :from => :working, :to => :showering
transitions :from => :dating, :to => :showering
end

event :work do
transitions :from => :showering, :to => :working
# Going to work before showering?  Stinky.
transitions :from => :sleeping, :to => :working
end

event :date do
transitions :from => :showering, :to => :dating
end

event :sleep do
transitions :from => :showering, :to => :sleeping
transitions :from => :working, :to => :sleeping
transitions :from => :dating, :to => :sleeping
end

event :wakeup do
transitions :from => :sleeping,   :to => [:showering, :working]
end

event :dress do
transitions  :from => :showering,   :to => [:working, :dating],
:on_transition => :wear_clothes
end

def wear_clothes(shirt_color, trouser_type)
self.shirt_color = shirt_color
self.trouser_type = trouser_type
end

end

Ok, I put in a lot of stuff in there to chew on. Lets discuss it one by one.


event :wakeup do
transitions :from => :sleeping,   :to => [:showering, :working]
end

I can wake up and either go take a shower, or go sit on my computer and start working. This implies I have two transitions in the event wakeup:

a) from sleeping to showering
b) from sleeping to working

If I express this as


event :wakeup do
transitions   :from => :sleeping,    :to => :showering
transitions   :from => :sleeping,    :to => :working
end

%> script/console
>> p = Person.new({:shirt_color => 'red', :trouser_type => 'dress pants'})
>> p.save!
>> p.current_state
=> :sleeping
>> p.wakeup!
=> true
>> p.current_state
=> :showering

acts_as_state_machine simply picks up the first transition in this case. There is no way for me to wake up and start working, unless I create a new event for this transition and invoke it.

If you notice, these two transitions were already present in the example, but were fired when the user called work! or shower!. And that makes complete sense for this particular example where for all incoming transitions into a state are modeled as the event - i.e. all transitions that end up in the state :working are bundled in the event :work. I call this bottom up approach, where you create events based on incoming transitions for a state. In complex state machines however, you are not always free to choose the event based on the :to state, rather the events are modeled after real life actions that users perform. This is more like the top down approach, where you are creating events based on outgoing transitions from a state. Does that make sense? In any case, continuing on with this post.

So I extended acts_as_state_machine plugin to accept an array for the :to argument and allow the user to specify the next state to transition to when the event is fired.


event :wakeup do
transitions   :from => :sleeping,    :to => [:working, :showering]
end

%> script/console
>> p = Person.find(:all).last
>> p.set_initial_state
>> p.current_state
=> :sleeping
>> p.wakeup!(:next_state => 'working')
=> true
>> p.current_state
=> :working
>> p.set_initial_state
=> "sleeping"
>> p.wakeup!(:next_state => 'showering')
=> true
>> p.current_state
=> :showering

So now the user can be queried for the desired :to state.

Did you notice the extension to the event method. It now accepts the an optional hash argument as its last argument. Yup, I said - as the last argument- . But more on that in a minute.

So the new method signature is now event_name!(*args, [:next_state => ...]).

Next, we consider the issue of performing some work during the transition. The existing acts_as_state_machine plugin assumes any work that needs to be done would be done as you enter, after and exit each state. But with our new addition above of multiple possible :to states, there might be some common work that needs to be done that is not specific to the final :to state.
Case in point, after I take a shower, I need to wear clothes before either going to work or on a date. The act of wearing clothes does not fit in either going to work or going on a date.

I know what the counter-argument is going to be - :dress should be a state in itself and there should be a transition going from :showering to :dress and then transitions from :dress to :working and :dress to :dating. And then wearing the clothes should be done in the :dress state.


state :dress , :enter => :wear_clothes

event :dress do
transitions :from => :showering, :to => :dress
end
event :work do
transitions :from => :dress, :to => :working
...
end
event :work do
transitions :from => :dress, :to => :working
...
end
event :date do
transition :from => :dress, :to => :dating
...

Good point! I don't really have an answer to that yet, just a gut feeling that creating a virtual state for every little action that needs to be performed may lead to an unnecessarily complex state machine. If the work that needs to be performed is not related to the domain of the model which is being considered, it will introduce unrelated states in model's state machine.
I know from a purist pov all such logic should reside in the controller. But then you have the model domain logic creeping out into the controller where some actions may need to be performed based on the current state and the next state.
In any case, I implemented the on_transition feature for my project and here it is:

Similar to the :guard option for a transition, one can specify a :on_transition argument to the transitions call. The callback can be a symbol or a Proc. You can also specify arguments to the callback by specifying them when you fire the event. Oh, and the callback can also return a value back as shown below:


event :dress do
transitions  :from => :showering,   :to => [:working, :dating],
:on_transition => :wear_clothes
end

def wear_clothes(shirt_color, trouser_type)
self.shirt_color = shirt_color
self.trouser_type = trouser_type
id
end

%> script/console
>> p.current_state
=> :showering
>> success, ret_val = p.dress!("blue", "jeans", :next_state => :working)
=> [true, 10001]
>> p.current_state
=> :working
>> p.shirt_color
=> "blue"

The on_transition callback is called after the guard callback. The return value of the on_transition callback in no way affects the transition. It is simply considered to be a side effect of the trasition.

Oh and one more thing, the :from argument in the transitions call can be an array as well (this was part of the original acts_as_state_machine plugin). So you could do :


event :sleep do
transitions :from => [:showering, :working, :dating], :to => :sleeping, :guard => :brush_teeth
end

With the above enhancements, you can now do


event :dress do
transitions :from => [:showering, :sleeping], :to => [:working, :dating], :on_transition => :wear_clothes

Update: I forgot to mention these enhancements also include a method called next_events_for_current_state that I picked up from here, and fixed it to return the event names rather than the next states.

Note: I haven't tested this rigorously, so if you do find something amiss, drop me a note.

Njoy.

IE issue - style tag embedded in ajax response not parsed

2007-11-16T13:07:00.001-08:00

While working on a recent project, my rails application was returning a html snippet with an embedded style tag in response to an ajax request. Firefox displayed the updated html content perfectly, but IE refused to play nice.

This issue is documented here and a solution has also been provided. I use YUI so I updated the code to use YAHOO.env.ua for browser detection.

applyStyles: function(rawHTML) {
if (YAHOO.env.ua.gecko > 0) return;
var headEl = null; // lazy-load

// find all styles in the string
var styleFragRegex = '<style[^>]*>([\u0001-\uFFFF]*?)</style>';
var matchAll = new RegExp(styleFragRegex, 'img');
var matchOne = new RegExp(styleFragRegex, 'im');
var styles = (rawHTML.match(matchAll) || []).map(function(tagMatch) {
return (tagMatch.match(matchOne) || ['', ''])[1];
});

// add all found style blocks to the HEAD element.
for (i = 0; i < styles.length; i++) {
if (!headEl) {
headEl = document.getElementsByTagName('head')[0];
if (!headEl){
return;
}
}
var newStyleEl = document.createElement('style');
newStyleEl.type = 'text/css';
if (YAHOO.env.ua.ie > 0){
newStyleEl.styleSheet.cssText = styles[i];
} else {
var cssDefinitionsEl = document.createTextNode(styles[i]);
newStyleEl.appendChild(cssDefinitionsEl);
}
headEl.appendChild(newStyleEl);
}
}

Njoy.

attachement_fu - size column

2007-11-08T18:05:00.000-08:00

While using attachement_fu migration , I ran into the issue of 'size' being a reserved word in Oracle and kept getting the following error while running the migration:

OCIError: ORA-00904: : invalid identifier:

I got around it by naming the database column to 'asize' instead of 'size' in my database migration script and then adding an alias_attribute (>= rails 1.2) to my model as

 alias_attribute :size, :asize

Hope this helps someone.

Stepping through your rails action code

2007-10-25T18:30:00.002-07:00

For a long time I couldn't figure out what I could do with breakpoint. Other than running console like operations and inspecting instance variables, I couldn't seem to do much. So I relied mostly on logger.warn and console kind of debugging. Of course, I didn't have to debug much since I write a lot of tests ;)

I come from perl world which has an awesome debugger available. I wanted to be able to step through the code and inspect my local variables in a loop and break on conditionals etc.

Recently I found ruby-debug and it seems to do all the things I have been wanting to do.

The debugger drops me in the action and much to my delight I could inspect loop variables and break on conditionals. Inspecting variables is easy using the command "p".
I ran irb at the rdb prompt and it dropped me into a script/console like environment. The scope is still at the action. Much much better than where breakpoint drops me (lib/breakpoint).

Check it out. I know it will make my life a lot more easier now.

Njoy.

ruby group_by

2007-10-18T23:14:00.000-07:00

I often need to convert an array of objects into a hash, indexed by some attribute of the objects.

For e.g. for a task list, each task is an object consisting of
- type (e.g grocery, auto)
- name (e.g. Safeway, Lucky, OSH)
- notes

So a task list may look like

[
[ :type => :grocery, :name => :safeway, :notes => "buy some milk" ],
[ :type => :grocery, :name => :lucky :notes => "buy halloween candy" ],
[ :type => :auto, :name => :costco :notes => "buy tires for the acura" ]
]

For display purposes I would like to organize the tasks by their type and then by names.

Updated: Rails makes it so easy :)

@tasks = {}
TaskList.find(:all).group_by(&:type) do |type, subtasks|
@tasks[type] = subtasks.group_by(&:name)
end

escape_javascript goodness

2007-10-10T21:49:00.000-07:00

I was trying to print some javascript in application .rhtml and wanted to use a link_to helper to print a link.

var user = <%= current_user.json_user_data %>;
$('welcome_p').innerHTML = "Welcome " + user.login + '! ( <%= link_to("Sign out", session_path(), :method => :delete, :class => "wlcmLnk")) -%> )';

This resulted in an error as link_to expanded to

$('welcome_p').innerHTML = "Welcome " + ud.login + '! ( <a href="/session" class="wlcmLnk" onclick="var f = document.createElement('form'); f.style.display = 'none'; this.parentNode.appendChild(f); f.method = 'POST'; f.action = this.href;var m = document.createElement('input'); m.setAttribute('type', 'hidden'); m.setAttribute('name', '_method'); m.setAttribute('value', 'delete'); f.appendChild(m);f.submit();return false;">Sign out</a> )';

The single quotes are not escaped :(

But not to worry. escape_javascript will do the needful. It escapes carrier returns and single and double quotes.

$('welcome_p').innerHTML = "Welcome " + ud.login + '! ( <a href=\"/session\" class=\"wlcmLnk\" onclick=\"var f = document.createElement(\'form\'); f.style.display = \'none\'; this.parentNode.appendChild(f); f.method = \'POST\'; f.action = this.href;var m = document.createElement(\'input\'); m.setAttribute(\'type\', \'hidden\'); m.setAttribute(\'name\', \'_method\'); m.setAttribute(\'value\', \'delete\'); f.appendChild(m);f.submit();return false;\">Sign out</a> )';

Njoy!

(re)Positionable Modalbox 1.5.5.1 updates

2007-10-09T21:12:00.000-07:00

Download :
modalbox.js
modalbox.css
(let me know if the above links don't work. I am still trying to figure out the best way to allow file downloads from blogspot).

A quick post to describe my changes to modalbox.js:

- (re)Positionable
A user can position Modalbox by specifying the CSS position properties (top, left, right, bottom)
as
Modalbox.show(content, {title: 'my positioned modalbox', top: '200px', left: '100px'});
and then ...
Modalbox.show(content, {title:'my re-positioned modalbox', top: '100px', right: '100px'});

If the CSS position properties are not specified, modalbox pops up in the default position.
If the left and right properties are not specified, the modalbox pops up in the horizontal center.

- enable_rjs
Modalbox treats any content returned by a http request to be html + script. However this craps out with Rails rjs. enable_rjs enables the rjs support so you can do (a bit contrived, but you get the idea)

respond_to do |format|
format.html {
...
}
format.js {
new_form = render_to_string(:partial => 'new', :layout => false)
render :update do |page|
page.call 'Modalbox._insertContent', new_form, { :top => '50px', :right => '100px'}.to_json
page.call 'Modalbox._putContent'
end
}
end

The invocation is
Modalbox.show(content, { enable_rjs: true});

- scroll bars disappear in IE 6
This is described in detail here

Please note that Modalbox behaves like a singleton class. So if you set an option for a content, you need to explicitly reset that option to disable it for subsequent content.
For e.g.
Modalbox.show('/some_rjs_action', {enable_rjs: true});
and then ...
Modalbox.show('/some_action_that_returns_html', {enable_rjs: false});

Let me know if you find any issues.

Njoy!

Copy model data between databases

2007-10-05T00:22:00.000-07:00

Update: I have modified the task to use save_with_valdation(false) and made it available
as a download here.

------------------

This is an update on the post I made yesterday about how easy it is to copy model data between production and development databases.

The problem with the approach mentioned here is that the id of the original model data is not preserved. As a result, if you copy models with relationships defined using foreign keys, the relationships are not maintained.

I found that instead of using [model].create, I can use [model].new and then set the id explicitly.

model_data.each do |m|
new = model_data.new m.attributes
new.id = m.id
new.save!
end

I wrote up a rake task to do something like

rake db:cloneModelData MODEL=User,Account BATCH=Y

And the task is

namespace :db do
desc "Clone the data associated with a model from the production database to the development database.\n\n" +
" # If the development database contains any data related to the model, it is purged before the copy is made.\n" +
" # Specify the model class name on the command line as MODEL=X,Y,Z (no spaces)\n" +
" # Source database as SOURCE=db (production by default)\n" +
" # Destination database as DESTINATION=db (development by default)\n\n" +
" # To enable batch mode BATCH=Y (By default, the task queries the user to view the yaml and \n" +
" # write it to a file (just in case-call me chicken if you wish)\n" +
" # e.g. rake db:cloneModelData MODEL=User,Account,Post BATCH=Y\n"
task :cloneModelData => :environment do

batchmode = ENV['BATCH'] ? ENV['BATCH'] =~ /^y$/i : false
sourcedb = ENV['SOURCE'] || 'production'
puts "Connecting to source database '#{sourcedb}'"
ActiveRecord::Base.establish_connection(sourcedb)

unless ENV['MODEL']
print "Please specify the model class name(s) seperated by commas: "
model_names = $stdin.gets.strip
else
model_names = ENV['MODEL'].strip
end

data = {}
model_names.split(/,/).each do |model_name|

begin
klass = Kernel.const_get model_name
rescue
raise "Hmm unable to get a Class object corresponding to #{model_name}. Check if you specified the correct model name: #{$!}"
end

unless klass.superclass == ActiveRecord::Base
raise "#{model_name}.superclass (#{klass.superclass}) is not ActiveRecord::Base"
end

puts "\nCollecting data for Model Class - #{model_name}"

# Collect all the data from te production database
data[model_name] = klass.find(:all)
puts "Found #{data[model_name].length} records in the #{sourcedb} database"

unless batchmode
print "\nDo you want to see a yaml dump? [y|N]: "
if $stdin.gets.strip =~ /^y/i
puts data[model_name].to_yaml
end
print "\nDo you want to direct the yaml dump to a file? [Y|n]: "
unless $stdin.gets.strip =~ /^n/i
print "\nPlease specify the file name : "
File.open($stdin.gets.strip, 'w') do |f|
f.write data[model_name].to_yaml
end
end
end

end

destdb = ENV['DESTINATION'] || 'development'
puts "\n--------------------------------------\n\nEstablising connection with the '#{destdb}' database\n\n"

ActiveRecord::Base.establish_connection(destdb)

puts "Now deleting model related data from the #{destdb} db\n"

data.keys.each do |model_name|

klass = Kernel.const_get model_name

olddata = klass.find(:all)

puts "\nFound #{olddata.length} #{model_name} records in the #{destdb} database"
unless batchmode
print "Do you want to see a yaml dump? [y|N]: "
if $stdin.gets.strip =~ /^y/i
puts olddata.to_yaml
end
end

puts "Deleting old data"
olddata.each {|f| f.destroy}

puts "Now copying the #{sourcedb} model data into the #{destdb} database"
print "Copying #{model_name} records: "
i = 0
data[model_name].each do |d|
begin
print "#{ i+=1}, "

obj = klass.new d.attributes
obj.id = d.id
obj.save!

#klass.reflect_on_all_associations.each { |a|
# puts classname + " -> " + a.name.to_s.camelize.singularize + " [label="+a.macro.to_s+"]"
#}

rescue
raise "Error saving model data: #{$!}"
end
end
puts "\nDone copying #{model_name}\n"
end
puts "\nDone."

end
end

Njoy!

Copy data between databases using script/console

2007-10-04T00:44:00.000-07:00

A quick blog about how easy it is to copy data between databases using script/console.
For e.g. I copy data from production server to the development server below:

% script/console production -s
> projects = Project.find(:all)
> ActiveRecord::Base.establish_connection(:development)
> projects.each {|p| Project.create p.attributes}
> CtrlD

IE 6 issues with Modalbox 1.5.5 + Protoculous 1.0.2

2007-09-28T22:01:00.000-07:00

Update

I had two issues with Modalbox 1.5.5. on IE-6, and this post describes how I fixed them.
a) IE 6 hangs - page related issue
b) IE 6 vertical scrollbar dissappears when Modalbox is displayed - Modalbox issue

I wanted to use Ajax popus for a user signin-signup flow on a website I am building. Since I was already using prototype and effects from prototculous 1.0.2, I looked around for prototype powered popups. I found Modalbox and immediately liked its capabilities. The effects are configurable, and the content to be displayed can be specified as a url, plain html, a dom node or a javascript html object. The Interface api is elegant in its simplicity.
Thanks to Wildbit's team for releasing such a useful product to open source.

The integration with my code was a snap. I fired up the server and tested the code in Firefox. It worked beautifully.
But IE 6/SP2 was another story.
As soon as I clicked on the link to show the Modalbox popup, IE 6 froze.

IE 6/SP2 had worked fine (except that it complains about a js error on page load) on modalbox's home page, so the problem must be my website's page.
If I turned off modalbox transitions, the popup worked, so I figured effects.js was having some problem with the page css.

On digging through, I zeroed in on the padding property that was defined on the body element. It seemed to be causing the IE 6 hang:

body {
margin: auto;
padding: 15px 0px 25px 0px;
}

On removing this property, IE 6 worked fine. Weird huhn!

But another nagging problem remained.
On IE 6, whenever the modalbox overlay appeared, the vertical scrollbar on my web page would disappear. On canceling the popup, the scrollbar would reappear. If you visit modalbox's home page, you will see what I am talking about. It was distracting to me, so I set out to fix it.

On digging through, I found the following function in modalbox.js:

// For IE browsers -- IE requires height to 100% and overflow hidden (taken from lightbox)
_prepareIE: function(height, overflow){
var body = document.getElementsByTagName('body')[0];
body.style.height = height;
body.style.overflow = overflow;

var html = document.getElementsByTagName('html')[0];
html.style.height = height;
html.style.overflow = overflow;
},

The code is setting the body height to 100% which caused the scrollbar to disappear.

I commented out the function, but then found that the MB_overlay div didn't cover the entire screen height.
So I wrote up some code to calculate the screen height and width in pixels and set the div size explictly.

To summarize, I removed all references which set height (and width) to 100% on the body, html and MB_overlay elements. Instead I calculate the screen size and set MB_overlay size accordingly.

The files can be downloaded from here

modalbox.js
modalbox.css

This fixed the scrollbar issue for me.
Hope this is of help to others.

Arts patch for assert_rjs :replace_html bug

2007-09-20T15:19:00.000-07:00

You can use the slick Rails Arts plugin to test your rjs renders. But the plugin breaks while testing assert_rjs :replace_html

If you have the following construct in your action method:

render :update do |page|
page['elemid'].replace_html some_string
end

The Rails Javascript helper replaces the replace_html call with the following call:

$('elemid').update(some_string)

This breaks the arts plugin which only expects
Element.update('elemid', some_string)

So here is a simple patch to make your assert_rjs :replace_html to work.

53,54c53,55
<> #assert_match Regexp.new("Element.update(.*#{div}.*,.*#{content.source}.*);"),
> assert_match /(\$$['"]#{div}['"]$.update$|Element.update\(.*#{div}.*,)['"].*#{content.source}.*['"]$;/, #'
> @response.body
56c57,59
<> #assert_response_contains("Element.update(\"#{div}\", #{content});",
>
> assert_response_contains(["Element.update(\"#{div}\", #{content});", %Q!$("#{div}").update(#{content});!],
63c66,67
<> #assert_match Regexp.new("Element.update(.*#{div}.*,.*?);"), @response.body
> assert_match /(\$$['"]#{div}['"]$.update$['"]|Element.update\(.*#{div}.*,['"]).*?$;/, @response.body
98c102,107
<> if str.class == Array
> strA = str
> else
> strA = [str]
> end
> assert strA.detect {|s| @response.body.to_s.index(s) != nil }, message
133c142
<> end

RoR, Oracle, sqlplus and life

2007-09-06T16:51:00.001-07:00

Update: Read this if you want to use Oracle 10g Express edition on Linux 64 bit White box

This took me some time to find, so I will document it for everyone's benefit.

At work, I needed to use an Oracle database for my RoR apps.

Here is a quick summary of steps to install the Ruby oci driver etc:

First download Oracle Instantclient and install it to say /opt/oracle/10.2.0.3
This makes connecting to Oracle a snap. Very useful indeed.
setenv LD_LIBRARY_PATH /opt/oracle/10.2.0.3
setenv ORACLE_HOME /opt/oracle/10.2.0.3
setenv NLS_LANG AMERICAN_AMERICA.UTF8

# install oci driver
wget http://rubyforge.org/frs/download.php/22320/ruby-oci8-1.0.0-rc3.tar.gz
tar xvzf ruby-oci8-1.0.0-rc3.tar.gz
cd ruby-oci8-1.0.0-rc3
ruby setup.rb config -- --with-instant-client=/opt/oracle/10.2.0.3
make
make install
test it

%> irb
irb(main):001:0> require 'oci8'
=> true
irb(main):002:0> exit
Specify the following in your database.yml
production:
adapter: oci
host: //dbHost:1521/oracle_instance
username: xxxx
password: xxxx
To use sqlplus to connect to the Oracle db

sqlplus username/password@//dbHost/Oracle_db_instance

or (if you don't want the pwd to be visible in ps listings)

sqlplus /NOLOG
SQL*Plus: Release 10.2.0.3.0 - Production on Thu Sep 6 17:08:28 2007

Copyright (c) 1982, 2006, Oracle. All Rights Reserved.

SQL> connect username/password@//dbHost/Oracle_db_instance

No need to muck with any tnsnames.ora.

netstat

2007-09-06T16:45:00.000-07:00

A quick blog after a brief hiatus:

While trying to debug random lockups of our Oracle database server, I found a cool command to monitor the connections being served by a machine.

I already knew about netstat and netstat -c which gives the user a continuous display of the connections.

But I found another way to view the connections in real time using the watch command:

watch -d "netstat -toupe 2>/dev/null"

Pretty cool!!

Rails - About your application's environment

2007-08-15T14:37:00.000-07:00

I set up a rails instance yesterday on my linux box and started it up in the development mode as usual.
From my windows box I accessed the main page (default with rails) of the application. I wanted to verify the environment that rails was loading. So I clicked on the "About your application's environment" link.

I got an error message -
For security purposes, this information is only available to local requests.

Hmm, I knew I had started my server in the development mode and by default this mode considers all requests local.

I doubled checked config/environments/development.rb and found the setting to be right...


# Show full error reports and disable caching
config.action_controller.consider_all_requests_local = true

Hmm what is going on?

From development.log I figured the controller for this request to be rails/info with the action name as properties.


%> grep -r "def properties" /usr/local/lib/ruby/gems/1.8/gems /*
/usr/local/lib/ruby/gems/1.8/gems/rails-1.1.6/builtin/rails_info/rails/info_controller.rb:  def properties

Ah! So I checked out info_controller.rb


class Rails::InfoController < ActionController::Base
  def properties
    if local_request?
      render :inline => Rails::Info.to_html
    else
      render :text => 'For security purposes, this information is only available to local requests.
', :status => 500
    end
  end
end

Tracing local_request? led me to


%> grep -r 'def local_request?' /usr/local/lib/ruby/gems/1.8/gems/*
/usr/local/lib/ruby/gems/1.8/gems/actionpack-1.13.3/lib/action_controller/rescue.rb:      def local_request? #:doc:
%>

Looking at local_request?


      def local_request? #:doc:
        [request.remote_addr, request.remote_ip] == ["127.0.0.1"] * 2
      end

it is clear that local_request? will return false in this case since I am accessing the page from a different machine.
As I searched for local_request? again, I found this snippet in rescue.rb itself:


      def rescue_action(exception)
        log_error(exception) if logger
        erase_results if performed?

        if consider_all_requests_local || local_request?
          rescue_action_locally(exception)
        else
          rescue_action_in_public(exception)
        end
      end

Ah ha! we need to add consider_all_requests_local to our properties action as well. So I popped open app/controller/application.rb and added the following snippet


class Rails::InfoController < ActionController::Base
  def properties
    if consider_all_requests_local || local_request?
      render :inline => Rails::Info.to_html
    else
      render :text => 'For security purposes, this information is only available to local requests.
', :status => 500
    end
  end
end


class ApplicationController <  ActionController::Base
  # Pick a unique cookie name to distinguish our session data from others'
  session :session_key => '_ptracker_session_id'
end

A server restart and I got the output I wanted:


Ruby version 1.8.6 (x86_64-linux)
RubyGems version 0.9.0
Rails version 1.2.3
Active Record version 1.15.3
Action Pack version 1.13.3
Action Web Service version 1.2.3
Action Mailer version 1.3.3
Active Support version 1.4.2
Application root /home/xxxx/ror
Environment development
Database adapter mysql

Hmm maybe I should submit a patch!

ActiveDirectory (LDAP) integration with Rails - Update

2007-08-02T15:34:00.000-07:00

Update: Replacing balance

So I managed to put together something in place:


                             <----->           [ balance   <-----> Mongrel cluster ]
  (ISAPI rewrite proxy)                                    (backend server - Vmware m/c)

The IIS proxy takes of single-sign on and LDAP authentication. I used ISAPI rewrite proxy to rewrite requests to a virutal machine backend that hosts the actual rails cluster.
The rewrite rules look like:


ISAPI/httpd.ini:
RewriteCond URL (.*)
RewriteHeader X-Forwarded-URL: .* $1

RewriteCond Host: rails-backend(?:-dev)?(?:\.domain.com)?
RewriteProxy (.*) http\://rails-backend:8081$1 [I,U,A]

On the backend side, rails-backend is a VMware guest that runs a Mongrel cluster of three Rails apps behind a load balancer balance (see the update above though).


balance 8081 cadweb:8000 cadweb:8001 cadweb:8002 %

# for debug use following
# balance -d -f -p 8081 cadweb:8000 cadweb:8001 cadweb:8002 %

The % at the end means to use a hash based routing. So the same client gets routed back to the same server for every request. Update - see the top of the post
One thing I wished balance would do is filter incoming connections. In this case I would have like balance to restrict all incoming connections to the IIS proxy. I had to use iptables to do it.


iptables -A INPUT -p tcp -s iis-proxy --dport 8081 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp --dport 8081 -m state --state NEW,ESTABLISHED -j DROP

A big issue with this solution is getting the list of access lists a user has rights to. And then use that information in rails somehow to limit which page can be shown to the user.

csh - executing a unix cmd and checking for its exit status

2007-08-02T14:10:00.000-07:00

Everyone knows how to check the exit status of a command invoked via csh/tcsh


%> cat mycmd
#!/bin/csh -f
exit 0;

%> cat wrapper.csh
#!/bin/csh -f

mycmd
if ($status) then
   echo "error executing mycmd"
   exit 1
endif
echo "successful execution of mycmd"

%>

But there is another easier way if you don't care about the actual value of the exit status.


%> cat easier_wrapper.csh
#!/bin/csh -f

if ! { mycmd } then
   echo "error executing mycmd"
   exit 1
endif
echo "successful execution of mycmd"

%>

Notice the use of curly braces around mycmd.

linux tidbit - Finding the working directory of a process

2007-07-25T18:28:00.000-07:00

If you are working on multiple Ruby on rails apps, you will have at least 2-3 instances running on your machine at the same time. It is a chore to keep straight the port numbers with project directories (hmm maybe I should write a plugin for this :~))

I was in a similar situation where I couldn't recall why a RoR instance was running on one of the ports.

So I did a ps -auxww to get the process id of the process and then


cp>  cd /proc/[process_id]/cwd
cp>  pwd
/home/cpatil/....

Pretty cool!
/proc is a virtual filesystem and allows you to peek inside the OS kernel at runtime and find information about the processes running on the machine. Google for it, a one time read is a must for all linux users, young and old.
It is quite funky the way it is set up. For e.g. most files are 0 bytes and you need to do a cat to read it. Stuff like that.

Rails: Calling another action

2007-07-24T02:37:00.002-07:00

It is interesting how Rails behaves when one action calls another as part of processing a request.

Here is my test:


app/controllers/test_controller.rb
def action1
@foo = "action1"
end

def action2
@foo = "action2"
end

app/view/test/action1.rhtml
<h1>action1.rhtml</h1>
<h2><%= @foo %></h2>

app/view/test/action2.rhtml
<h1>action2.rhtml</h1>
<h2><%= @foo %></h2>

Here is what I observed when I call action2 from action1 like so:


def action1
@foo = "action1"
action2
end

def action2
@foo = "action1"
end

If none of the actions have a render explicitly defined, then calling action2 from action1 will simply execute the logic defined in the action2 method. The template that will be rendered will be action1.rhtml
If both of the actions have a render explictly defined, then calling action2 from action1 is a multi-render error. It doesn't matter if you use render :template or render :action or just render.
If only action2 has an explicit render defined, then
- action2 method is executed
- if action2 calls a default template render, then action1.rhtml will be used as the template
- if action2 calls a specific template, say render :action => 'action2', then action2.rhtml is used.
Note that it is a multi-render error in this case if action1 also has an explicit render defined in it.

Rails caching

2007-07-19T19:02:00.000-07:00

Options :
Page caching
Action caching
Fragment Caching

Read these:
Real World Rails series
Content Caching in Rails

For fragment caching, use timed_fragment_cache which allows the cached content to be expired automatically based on a timeout.

I was thinking of allowing my users to request an explicit rebuild of the page by forcing the cache to expire before the timeout. This could be done by providing an action which will simply expire the fragment and redirect back to the original action.
However in the event that two users click on the rebuild request within say 1 minute of each other, I want the subsequent build request to be ignored automatically. I will take a stab at modifying the timed_fragment_cache plugin for doing this.

SQL Query optimization

2007-07-15T23:58:00.000-07:00

To begin optimization, select the SQL query that
- returns max number of rows
- has most number of executions
- most I/O

Optimizations:
- The basic tenet is to avoid full table scans. Add an index to the fields mentioned in the where clause. Indexs are created as an additional file typically that the database can consult to filter the table based on the values of the corresponding field. Be careful, don't add to many indexes as every index requires an additional change to the database whenever data is inserted in the table. The indexs are typically stored as BTree indexes.

Use EXPLAIN Select query... to examine if the database is using the indexes correctly.

If you do any kind of operation on a field in the where clause, the full table is scanned since the field value for each record needs to be computed to be compared. As a result the index on such a field is a waste. Try using the field as is.

- Joins should be ordered so that the table that has the max rows to be processed is filtered the most. The order is also imp. If you are joining A, B and C where C is 10 rows and A and B are 100000000 rows each, then it makes sense to join B and C first instead of joinging A and C first.
Apply max filtering to table that has max rows.

Other Database trivia:
1NF - normalization in which repeated fields (score1, team1, score2, team2) are moved into their own table. One to many relationship.
2NF - normalization in which repeated values (publisher and publisher address for a list of books) are moved into their own table. Many to Many relationship.
3NF - TO be written.

Some HTML Links

2007-07-07T22:22:00.000-07:00

These were interesting links that I collected while doing some HTML UI design.

Extra spaces after HTML form - solution
Clearing spaces beneath floated elements - solution
Centering content using CSS - solution
HTML Tables - MUST READ

Setting up Ubuntu 7.04 (Feisty) as vmware host with a wireless card

2007-06-25T23:22:00.000-07:00

Phew, after fighting with this setup for 3 days, I have managed to bring up Bridged networking to a wireless interface in a Ubuntu Feisty server.

I am listing a short summary of my wanderings here so other users who are stuck at various places in the process can pick it up from there. There are a whole bunch of solutions out there which unfortunately trample over each other.

I also want to thank all the users who contributed on this topic all over the internet. Awesome!!

For the final solution mix, scroll down to the end of the post.

-- My setup

I have an old HP DL-140 Proliant server sitting in my garage. I wanted to use it as a vmware host to run some development oses. Since the server is situated in the garage with no ethernet wiring, I decided to use a cheap D-Link DWLG510 802.11G card.

I have a Hardware firewall/router that DHCP's for my entire Home LAN and my wireless access point is set up in the bridge mode to connect my wireless network to the ethernet network. This allows me to get a single network topology, allowing easy access to machines and their data.

I wanted to set up the Ubuntu server so that the Guest VM's could bridge to the host's wireless interface and use DHCP to acquire dynamic IP address from the hardware router.

[VMware Guest] <--> (vmnet 0 bridge) <--> [VMware host/Ubuntu] <--> (Wireless NIC ath0) <--> [Wireless AP] <--> (ethernet) <--> [HW Router/DHCP] <--> WAN

I wanted to be able to run multiple web servers on these Guest VMs and reach any server using the IP address (or a dns name) directly instead of using natted ports.

So over the weekend I installed Ubuntu 6.10 server which I later updated to 7.04 (linux kernel 2.6.20) via apt-get.


%> lspci
00:00.0 Host bridge: Broadcom GCNB-LE Host Bridge (rev 32)
00:00.1 Host bridge: Broadcom GCNB-LE Host Bridge
00:03.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
00:0f.0 ISA bridge: Broadcom CSB6 South Bridge (rev a0)
00:0f.1 IDE interface: Broadcom CSB6 RAID/IDE Controller (rev a0)
00:0f.2 USB Controller: Broadcom CSB6 OHCI USB Controller (rev 05)
00:0f.3 Host bridge: Broadcom GCLE-2 Host Bridge
00:10.0 Host bridge: Broadcom CIOB-E I/O Bridge with Gigabit Ethernet (rev 12)
00:10.2 Host bridge: Broadcom CIOB-E I/O Bridge with Gigabit Ethernet (rev 12)
01:06.0 Ethernet controller: Atheros Communications, Inc. AR5005G 802.11abg NIC (rev 01)
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 02)
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704 Gigabit Ethernet (rev 02)

Setting this up with Ubuntu is tricky. I tried a few of the solutions and ended up updating the 6.10 to 7.04 in the process. I finally found this script which worked great and I was able to use my wireless interface as my primary network interface. If you need further instructions with setting this up, drop me a comment and I will try to help.
Note that the madwifi code needs to be patched eventually for vmware bridging to wireless NICs to work. See below.

Next up was the VMware server. I installed a couple of Ubuntu packages listed here as dependencies for the server.

I then tried to run the vmware-install.pl included in VMware-server-1.0.3-44356.tar.gz to install the VMware host. The script failed while trying to build vmmon.


%> cd vmware-server-distrib 
%> sudo vmware-install.pl
...
Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-config0/vmmon-only'
make -C /lib/modules/2.6.20-16-386/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.20-16-386'
  CC [M]  /tmp/vmware-config0/vmmon-only/linux/driver.o
In file included from /tmp/vmware-config0/vmmon-only/linux/driver.c:80:
/tmp/vmware-config0/vmmon-only/./include/compat_kernel.h:21: error: expected declaration specifiers or â...â before âcompat_exitâ
/tmp/vmware-config0/vmmon-only/./include/compat_kernel.h:21: error: expected declaration specifiers or â...â before âexit_codeâ
/tmp/vmware-config0/vmmon-only/./include/compat_kernel.h:21: warning: type defaults to âintâ in declaration of â_syscall1â
make[2]: *** [/tmp/vmware-config0/vmmon-only/linux/driver.o] Error 1
make[1]: *** [_module_/tmp/vmware-config0/vmmon-only] Error 2
make[1]: Leaving directory `/usr/src/linux-headers-2.6.20-16-386'
make: *** [vmmon.ko] Error 2
make: Leaving directory `/tmp/vmware-config0/vmmon-only'
Unable to build the vmmon module.

I used vmware-any-any-update fix to resolve this issue.
Later however I found that vmware-any-any update causes problems with madwifi patch to allow vmware bridging to wireless interfaces on the host. See below how to fix the vmmon compile problem.

Note: if you get vmware server is not configured message when you run vmware, simply delete the /etc/vmware/not_configured file and rerun vmware.

I set up the networking to bridge vmnet0 to ath0 (my Ubuntu server's wireless NIC). I skipped Host only and NAT portions.

I happily bought up my CentOS development VM. But unfortunately the network didn't work. So it was back to googling.

I saw various posts on how it worked for some people by changing /etc/vmware/locations file (make sure VNET_0_INTERFACE is set to ath0), but my version of the file had it already set to that.

Then I came across posts where folks had patched there madwifi drivers. Though my setup wasn't reporting the problem, I tried it anyways, but no dice.
Eventually I found that this patch works, but the vmware-any-any update seems to break it in some way.

Some posts claimed that since wireless frames are different than ethernet frames, the bridging of Guest-eth0 to Host-ath0 is not possible. Soon after I read that VMware server supports bridging to wireless NICs, but it just didn't work for me.

Explored other approaches such as Host only networking, NAT etc to resolve the issue. Spent a day in reading up on iptables. But couldn't figure out how to get a single network topology using my setup. If any networking experts could comment, I would really appreciate it.

I jumped back to vmware bridge solutions and re-read the madwifi ticket. One user warned against installing vmware-any-any patches. So I completely cleaned up my vmware server install and performed the following steps:

Solution Mix:
Note: do this before installing vmware.

1) Apply the madwifi patch described by xgerligand@free.fr. make clean, make and make install and then reboot.

2) Now untar vmware-server-distrib and apply the patch described here by turn1200 to fix the vmmon problem that occurs when you run vmware-install.pl without using the vmware-any-any patch. Recording the commands here...


cd vmware-server-distrib/lib/modules/source/
tar -xvf vmmon.tar
vi vmmon-only/include/compat_kernel.h

change the line that says:
static inline _syscall1(int, compat_exit, int, exit_code);
to the following by removing two commas:
static inline _syscall1(int compat_exit, int exit_code);

sudo tar -cvf vmmon.tar vmmon-only/

3) Now run sudo vmware-install.pl and vmware installation should complete successfully.

Hopefully this will be of help to others.

ActiveDirectory (LDAP) integration with Rails

2007-06-20T23:56:00.000-07:00

Updated:
The solution that I went with. Read this article completely first so you are aware of the landscape.
EOM

I am currently exploring options to allow my rails apps to authenticate against an ActiveDirectory server. Active Directory is Windows 2000/2003 implementation of LDAP.

This post rounds up the options available and is my brain dump on the topic:

Authentication built into Rails:
When you want to code in the authentication mechanism within the rails app itself, you can use one of the following modules. This may be of interest when the rails app needs to know more details about the user and user's group memberships etc for security purposes and restrict parts of the website depending on user.

Net:LDAP
Recommended on rails wiki since it is pure ruby and has no other external dependencies. There is some code on the rails wiki that allows one to derive groups the user belongs to. I haven't tried it.
ruby/ldap
This post describes using ruby/ldap to allows authentication against a AD server. It has sample code to get the groups a user is assigned to.
ActiveLDAP
The rails wiki has more info. Looks like ruby/ldap is recommended over this library.

Authentication done via Web Server:
The idea here is to have the web server perform the authentication and only forward the request to the Rails app if the user is authenticated. The user info (username, groups) etc may be passed to the Rails app using HTTP variables or cgi params. I will write about this more later as I come to that step.

A big reason why one may wish to use authentication done by the web server is to support single sign-on. Microsoft IE (and firefox) allow the automatic authentication using the current Windows user information. This is called Integrated Windows Authentication, formerly called NTLM. However this requirement limits the web server that can be used as our front-end.

Here is a list of possible combinations:
Windows based

IIS + Mongrel + ISAPI Rewrite - more info here. The problem here is rails caching doesn't work with this setup.
IIS + Mongrel + lighttpd + Mongrel + ISAPI Rewrite - more info here. A better solution since static files and cached fragments can be served by the lighttpd quickly and faster.
Apache + SCGI + mod_auth_sspi - here . mod_auth_sspi adds NTLM/LDAP authentication to apache.
However it is not known if it supports groups.

Linux

Apache + mod_auth_ldap + mod_ntlm - reported to be sorta unstable.

Hybrid

IIS + ISAPI Rewrite on Windows, proxying http requests to a lighttpd + Mongrel setup on a Linux box.
This seems to be the ideal solution. The Lighttpd + Mongrel cluster is a proven setup and can be scaled by simply adding more rails instances. lighttpd can easily load balance these rails clusters. And static and cached fragments can be served quickly by the lighttpd.

More to come.

Rails - render :action vs render :template

2007-06-14T20:42:00.000-07:00

render :action vs render :template

One can use render :action to render the template associated with another action.

However, when the user does


def action1
   @foo = "action1"
   render :action => "action2"
end

def action2
  @foo = "action2"
   render
end

the user should realize the method associated with action2 is not exectued when action1 calls render :action => "action2". This is simply the equivalent of render :template => "my_controller/action2".

This can get confusing when action2 in turn uses a non-default template to render itself.


def action1
   @foo = "action1"
   render :action => "action2"
end

def action2
  @foo = "action2"
   render  :template => "some/other"
end

In this case, when action1 is executed, rails will return an error complaining "views/my_controller/action2.rhtml" was not found.

Rails scripts/breakpointer questions

2007-06-13T17:18:00.000-07:00

NOTES:

Things of interest : local_variables, instance_variables, caller, methods
Keep a close eye on the prompt. A * means breakpointer (irb really) expects you to enter more ruby code since it senses that the code entered so far is not fully valid yet.
breakpointer (irb really) will by default print the return value to the irb console. You can explicitly return false to disable printing of large datasets.
```
breakpointer> @copy = @huge_dataset; false
```
For regular irb (and script/console) setting conf.return_format allows you to limit what is printed, but conf object doesn't seem to exist in breakpointer's irb session.

How do you print out variable values?
puts doesn't work :) Update: It actually prints the output along with the script/server output.

You can use @logger.info etc which will print the info to development.log

Local variables
I put a breakpoint in a helper method and tried to view some local variables defined in the helper method, but it didn't work.


breakpoint> local_variables
=> ["id", "block", "_"]

(note that I cleaned the irb prompt and replaced it with breakpoint for blogging)
The reason is breakpoint drops me in the breakpoint.rb's breakpoint method.


breakpoint> caller
=> ["/usr/local/lib/ruby/gems/1.8/gems/rails-1.1.6/lib/breakpoint.rb:512:in `breakpoint'", "/usr/local/lib/ruby/gems/1.8/gems/rails-1.1.6/lib/breakpoint.rb:512:in `breakpoint'", "script/../...

Now to figure out how we pop the stack back up so we end up in the method. Anyone know?

Printing Huge datasets
Suppose I am debugging some calculations I am doing on a hude dataset in breakpointer:


breakpoint> @huge_dataset

This will print the entire dataset. Ctrl-C will stop the display.


breakpoint> @huge_dataset;

This doesn't print the dataset.

So how is one to print the dataset. One solution is to use to_yaml.


breakpoint>  fh = File.open("/tmp/foo", "w");
breakpoint>  fh.puts(@huge_dataset.to_yaml)
=> nil
breakpoint>

I would really like to print the value in the breakpointer's irb console though.
Is there a special console object that I can target?

Update: Another way to debug your rails app - ruby-debug

Rails script/console questions

2007-06-13T16:46:00.000-07:00

How to invoke a method defined in a helper module in script/console?

How to get the list of classes loaded by underlying irb in script/console?

Extending Tcl's package include directories at run-time

2007-06-11T14:58:00.000-07:00

A quick one here:
Say you are debugging a tcl package and have a local copy to debug.

To make the tcl script pick up the local copy of the package:

- If the tcl script is editable, append to auto_path as


set auto_path "./src $::auto_path"
package require Foo

Note that you would need to create a pkgIndex.tcl in the directory where your local copy exists.

- If the tcl script is not-editable, use TCLLIBPATH env var. Haven't tried it though.

o&o

Perl - Variable Interpolation for single quoted strings

2007-05-17T15:49:00.000-07:00

A colleague was writing a perl script to print a string with embedded newlines in it. The string originated as a multi-line environment variable value i.e.


csh%>   setenv Foo "a b c\nd e f"
csh %>  echo $Foo
a b c
d e f
csh %>

csh %> perl -e 'print $ENV{Foo} . "\n";'
a b c\nd e f

The value of $ENV{Foo} was being interpreted as a singly quoted string and the \n was being automatically backslashed internally whenever variable interpolation took place.


index($ENV{Foo}, '\n') ;  #  returned 5
index($ENV{Foo}, "\\n") ;  #  returned 5

There are a couple of ways to solve this


% perl -e '$ENV{Foo} =~ s/\\n/\n/g; print $ENV{Foo} . "\n";'
or
% perl -e 'eval "\$f = \"" . $ENV{Foo} . "\""; print $f . "\n";'
  ( or in a script  use: eval '$f = "'. $ENV{Foo} . '"'; )

I was hoping there are better ways to do this. The first solution uses the regular expression engine to match every character in the string with the pattern. The second solution uses eval!

Is there a perl hack that will force variable interpolation on a singly quoted string? I thought of unpack but didn't have enough time to look into it.

First Post

2007-05-15T16:31:00.000-07:00

Hey there!
Welcome to my blog.
I plan on posting tech stuff like Javascript, ROR, HTML, CSS, Perl/Tcl and stuff that interests me!

So stick around if this interests you!

BTW, this is sort of continuation from http://rail101.blogspot.com.
In the transition to Google, blogger.com somehow mucked my account. I am unable to access all my earlier blogs.

Ruby version	1.8.6 (x86_64-linux)
RubyGems version	0.9.0
Rails version	1.2.3
Active Record version	1.15.3
Action Pack version	1.13.3
Action Web Service version	1.2.3
Action Mailer version	1.3.3
Active Support version	1.4.2
Application root	/home/xxxx/ror
Environment	development
Database adapter	mysql