Sahi V3 2010-06-10 released

Sahi V3 (2010-06-10) is now available for download. https://sourceforge.net/projects/sahi/

There have been significant improvements and bugfixes in this release.

• API _under ( http://sahi.co.in/w/browser-accessor-apis ) has been added to locate elements physically under another.
  
• 
  API _byXPath has been added to help users from Selenium and other tools move to Sahi.
  
• 
  API _row and _option have been modified to be in sync with other APIs. Existing scripts may need to be modified if you use these APIs.
  

Below is the changelog:


10 Jun 2010
-----------

* Bugfixes 
Fixed data truncation bug introduced in 30 Apr build 
Fixed getText bug which returned ab on FF and a b on IE for a<br>b 
Fixed for &nbsp; in select option text 
Fixed window.opener behaviour for link clicks 
Fixed parsing error for $a == $b 

* Features added 
Added Shiretoko as variant of Firefox 
_byXPath added. 
   For browsers without XPath support, 
   download the javascript file from http://coderepos.org/share/wiki/JavaScript-XPath 
   and save the contents in sahi/htdocs/spr /javascript-xpath.js 
_under added as a positional relation 
   Lets identify elements under another element eg. _cell(0, _under(_tableHeader("Status"))) 
5xx errors are displayed on screen too. 
Time taken by each test is displayed in logs. 
XHR redirects are excluded from injection 
Added Driver.setControllerMode. Takes sahi, java, ruby 
_option brought in sync with other APIs. 
   NOTE: older _option(selElement, value) will not work any more 
   Replace with _option(value, _in(selElement)) 
_row brought in sync with other APIs. 
   NOTE: older _row(tablElement, rowNumber) will not work any more 
   Replace with _row(rowNumber, _in(tablElement))

Ruby Sahi with Cucumber

What is Cucumber?

Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable domain-specific language and serves as documentation, automated tests and development-aid - all rolled into one format.
- From http://cukes.info/

Follow the steps below to get started with Ruby Sahi and Cucumber.

1. Install Java


2. Install Ruby


3. Install cucumber:
   

   gem install cucumber



4. Install Sahi proxy: Download Sahi from sourceforge and unzip to some location. (say D:\sahi)


5. Start Sahi:
   

   
   cd D:\sahi\userdata\bin
   start_sahi.bat
   

   
   


6. Install Sahi Ruby client:

   gem install sahi



7. Create a file D:\test\login.feature, add the content below and save it.
   

   
   Feature: Login
     In order to access the system
     As a user
     I want to be able to login
     
     Scenario: Login with valid credentials
       Given I am not logged in
       When I try to login with "test" and "secret"
       Then I should be logged in
       
     Scenario: Login with invalid credentials
       Given I am not logged in
       When I try to login with "test" and "wrongpassword"
       Then I should not be logged in
       And I should be shown error message "Invalid username or password"
   

   
   


8. Run this feature:
   

   
   cd D:\test\
   cucumber login.feature
   

   
   There will be a lot of messages with hints on implementing the right steps.
   


9. Implement the steps:
   
   Create a file D:\test\login.rb, add the content below and save it
   

   
   require  'sahi'
   
   def init_browser()
     #Use the correct paths from your system
     userdata_dir = "D:/sahi/userdata"
     browser_path = "C:\\Program Files\\Mozilla Firefox\\firefox.exe"
     browser_options = "-profile #{userdata_dir}/browser/ff/profiles/sahi0 -no-remote"
     return Sahi::Browser.new(browser_path, browser_options)
   end
   
   #open the browser at the start
   browser = init_browser()
   browser.open
   
   #close the browser on exit
   at_exit do
     browser.close
   end
   
   Given /^I am not logged in$/ do
     browser.navigate_to("http://sahi.co.in/demo/training/index.htm")
   end
   
   When /^I try to login with "([^\"]*)" and "([^\"]*)"$/ do |username, password|
     browser.textbox("user").value = username
     browser.password("password").value = password
     browser.submit("Login").click
   end
   
   Then /^I should be logged in$/ do
     if !browser.button("Logout").exists?()
       raise "Not logged in"
     end
   end
   
   Then /^I should not be logged in$/ do
     if !browser.submit("Login").exists?()
       raise "Logged in"
     end
   end
   
   Then /^I should be shown error message "([^\"]*)"$/ do |msg|
     value = browser.div("errorMessage").text() 
     if value != msg
       raise "Incorrect message: #{value}" 
     end
   end
   

   
   


10. Run and watch the tests complete successfully
    

    
    cd D:\test\
    cucumber login.feature
    

    
    


11. Done!

Sahi API _under added

Continuing with our tradition of innovation for simplicity, Tyto adds another wonderful API to Sahi.

NOTE: _under will be available in Sahi's next release

The problem:

Let us take the example of a dynamically generated grid. The example we use here is available at http://www.zkoss.org/zkdemo/userguide/#g7

We wish to assert the value of "Received" column for "Style Guide for ZK 3.5 released". If we bring up the controller and CTRL-hover over that element, what we see is

_div("2008/11/14 13:23:07")

which is not useful as a finder.

Looking at the alternatives listed, we notice that none of them can really help.

To fix this, we shall try using _near.

1) We put

_div(0, _near(_div("Style Guide for ZK 3.5 released")))

into the Evaluate Expression box, and click Highlight. This highlights the "Style Guide" element itself.

2) We experiment with the index passed as the first parameter, and using Highlight, pinpoint on the correct accessor as
_div(4, _near(_div("Style Guide for ZK 3.5 released")))

But this will not make a good accessor.
Why?
Because, this uses an index which seems like it would change when another column is added before the Received column.

What we really want, is that element, which is UNDER _div("Received").

Introducing the _under API

_under(element):

_under is a POSITIONAL marker. What it means is that it checks for coordinate based alignment under a particular element within a specific threshold.
So, in our case, it will look for a div which is roughly positioned underneath _div("Received")

Here is the final accessor:

_div(0, _near(_div("Style Guide for ZK 3.5 released")), _under(_div("Received")))

Note how this accessor is independent of the order of the rows and the columns.

And then the testers lived happily ever after ...

Notes:

This grid is not a simple table but actually composed of 2 tables, one for the header and one for the contents. So we could have approaced this problem using _cell(_table(2), "Style Guide for ZK 3.5 released", 4), but again the 4 would trip us later if the order/number of columns changes.

_under(el) can be passed as a last parameter to any Sahi API.

NOTE: _under will be available in Sahi's next release

Choosing the right web automation tool or web testing tool

Web automation is a little trickier than most other automation because there are many combinations of browsers and operating systems and they are fast evolving too.

What do you look for before you choose a tool for web automation?

The answer may actually depend on what your organization specializes in. If you are a product company which ships applications meant only for Internet Explorer, you need not consider multi-browser support or Linux support. But if you develop outward facing web applications, you may need to test the application on multiple browsers.

Here are some factors you should consider before choosing a testing tool.

The tool:

1. Should be techincally sound
   
   
   
   1. Should be able to identify elements/record on all browsers.
   2. Should handle complexities like HTTPS, Frames, IFrames, AJAX, dynamic ids.
   3. Should not need tinkering with source code of tool
   4. Should not require hard coded waits
   
   


2. Should save time and effort for teams
   
   
   
   1. Ramp up time should be minimal. Users should get productive within an hour.
   2. Complexities like AJAX, dynamic ids, object identification etc. should be handled by the tool instead of passing it on to testers. *
   3. Should run reliably across browsers and operating systems to reduce re-runs and debugging effort.
   4. Should not be dependent on knowledge of various other tools/technologies.
   5. Should need minimal maintenance of scripts/code
   
   


3. Should work with existing teams instead of requiring a drastic overhaul
   
   
   
   1. Should not require your teams to change from testers to developer testers, but let them easily pick up some scripting knowledge and get functional.
   2. Should not require expertise in various peripheral technologies like Java, Junit, TestNG, XPath, Firebug, Browser DOM etc. to just get started.
   
   


4. Should require minimal stakeholders
   
   
   
   1. Should not need developer involvement for modification to application in the name of “testability”. Dynamic ids, elements without ids, etc. should be handled well by the tool. *
   
   


5. Should be easy to scale testing teams
   
   
   
   1. Should be easy to hire and add more members to your testing team. This requires the tool to be simple to use.
   2. Should be able to move the teams across projects and products. This means that the tool needs to be sound enough to work with various technologies and frameworks.
   
   


6. Should have authoritative support available


7. Should be cost effective. The following need to be considered:
   
   
   
   1. Cost of acquiring the tool
   2. Cost of employing capable testers who can use the tool
   3. Cost of maintaining test infrastructure
   4. Cost of authoritative support
      
   
   
   Too often, especially with open source tools, the amount of money wasted in man hours due to limitations of the tool, incompatibility with existing expertise of team, lack of support etc. far outweighs the cost of acquiring alternative commercial tools. (Developers and testers with not much business experience invariably think that their time is not a cost to their company, and do not mind spending a week on an effort which should have lasted a day, thus wasting 25% of a month’s salary for a tool which may cost 10%)
   

* It is possible to just use the Sahi Controller and identify various elements reliably. Because tools like Selenium cannot record across frames, iframes, the tester is forced to learn to use Firebug to figure out what ID or XPath to use, add a line of selectFrame etc. These are very tool specific. While learning to use Firebug is an awesome skill to have, it should not be required at each step of the automation process. Adding conditional waits with knowledge of DOM is an unnecessary effort put on the tester, which can be handled by intelligent tools. Same goes for making developers add custom id generators for handling dynamic ids.

Sahi vs. Selenium

Though Sahi is aimed at non-programmer testers and Selenium at programmers, we get a lot of queries on their differences. Here is a brief document which compares Selenium and Sahi.

To those who read this:

1. If you are a Selenium fan, be open minded and verify the claims for yourself or contact us for clarification. And again, please be open minded. You may save a lot of time for yourself and your team.
2. This document is biased towards Sahi because we built Sahi. We believe what we state is true, but if you have evidence to prove otherwise, please do contact us at support@sahi.co.in.
3. When we say "Not sure" or "?" it means we do not have enough information because of lack of research on our part. Please verify for yourself.
4. If you want a comparison between other tools and Sahi, we would be happy to discuss it.
5. If you are another tool developer/supporter, please let us know how it compares and let the world benefit from alternatives.

Sahi vs. Selenium: Comparison document

Recorder

Selenium	Sahi
Works only on Firefox	Works on all browsers (IE, FF, Chrome, Safari, Opera)
Has trouble recording IFrames, Frames and popup windows	Can record on IFrames, Frames and popup windows
For Frames and IFrames, need to explicitly selectFrame	Implicit smart identification of elements even across Frames and IFrames
Uses XPath for identification of elements if id or name not present	Uses various algorithms to uniquely identify elements in a simple human recognizable way

Programming Language support

Selenium	Sahi
Java, Ruby, Perl, Python, C# (and may be more).	Sahi Script, Java, Ruby Sahi Script has the syntax of javascript but can interoperate with any Java code directly from script. The Java/Ruby drivers are available since Sahi V3
Needs language bridges for each new feature. For example, needs java bridge to invoke Flash via ExternalAPI.	Sahi Script can directly invoke anything exposed by javascript.

Ease of use

Selenium	Sahi
Easy to start with because of Selenium IDE which is a firefox plugin. Estimated start time less than 5 minutes	More difficult than Selenium to start because it needs installation of Sahi. Estimated start time 10-30 minutes, depending on Java installation etc.
Deep learning curve when the need is felt to move from Selenium IDE to Selenium RC.	There is only one mode of operation for Sahi. Extremely simple to learn and use for testers
Knowledge of programming language required	Can achieve most automation with just functions and variables. Sahi has inbuilt APIs for most other complex tasks like data driven testing
Needs JUnit style of programming	Can choose your own style
Uses XPath based identification for elements in complex html structures or those with dynamic ids. css selectors and javascript may also be used.	Has nearness APIs like _in and _near which can help show nearness of elements. Eg. _image(“delete.gif”, _near(“username 4”))
Needs waits for making AJAX work	No waits needed in 90% cases
Supports parallel execution	Inbuilt parallel execution. Needs only one parameter change

Stability of scripts and ease of maintenance

	Selenium	Sahi
Smart DOM Relations resilient to UI changes	No Dependent on XPath Difficult for testers to understand and debug	Yes Does not use XPaths. Uses _near and _in
Implicit waits for page load and AJAX: 1) Saves time 2) Keeps scripts simple 3) Reduces random failures	No Explicit waits needed.	Yes
Ease of adoption by a team of testers	Needs testers to know TestNG/Junit, XPaths, HTML structures, Frames IFrame knowledge, Javascript for AJAX conditional waits	Sahi abstracts out all these for the tester.

Dependency on other tools

Selenium – Java (Others need something similar)	Sahi
Needs JUnit (and optionally eclipse) to run tests	No additional tools required. Tests run from the Sahi Controller/command line/ant
Non persistent reporting. Needs TestNG or something similar for that	Persistent HTML reporting which can be shared via URL or file

Stability of product and number of releases

Selenium	Sahi
Started 2004(?) in ThoughtWorks	Started 2005 in ThoughtWorks
Version 1 took 5 years, Version 2 planned mid-2010. Moving away from original architecture to WebDriver based architecture	Current release: Version 3 Number of stable releases in 2009: 7

Footprint

Selenium	Sahi
RC: 10.5 MB, Grid 15 MB	less than 2 .5 MB with source
Not sure	Runtime ~ 50MB for 3 parallel threads

Reporting

Selenium	Sahi
Needs external tools to create readable reports	Inbuilt HTML reports with click through to relevant portion of script

Others

	Selenium	Sahi
Build tool integration (ant, batch files)	Yes	Yes
Multiple OS support	Yes	Yes
Version Controllable Scripts/Code	Yes	Yes
HTTPS support/redirects	Not sure	Yes
401 Authentication, Windows/NTLM Authentication dialogs	Not sure	Yes
External proxy tunneling	Yes	Yes
In built APIs for data driven testing	No	Yes
Works only with browsers	Yes	Yes
Needs privileged modes on browsers for operation. (Privileged is bad)	Yes	No
Extensible on future browsers	Depends on finding a way to use privileged mode on that browser	Yes. Very little dependency on type of browser.
Editor support	Has good editors in various languages	Editor support for javascript is not as good as for Java.

Support available

	Selenium	Sahi
Free support via Forums	Yes	Yes
Paid support available	Yes	Yes
Authoritative training available	?	Yes

Presentation at XP Goa day

Narayan Raman presented on Functional Testing of Web Applications using Sahi at the XP Goa Day in Goa University.

Goa XP day

View more presentations from narayanraman.

As part of a presentation , we did a small demo on record and playback of a script using Sahi, then refactored the code to be maintainable.

The site under test is available here: http://sahi.co.in/demo/training
The first cut from the recorder came out to be this:

_setValue(_textbox("user"), "test");
_setValue(_password("password"), "secret");
_click(_submit("Login"));
_setValue(_textbox("q"), "2");
_setValue(_textbox("q[1]"), "1");
_setValue(_textbox("q[2]"), "1");
_click(_button("Add"));
_assertExists(_textbox("total"));
_assert(_isVisible(_textbox("total")));
_assertEqual("1150", _textbox("total").value);
_click(_button("Logout"));

This was then refactored into 2 scripts, one containing functions and the other invoking it:

// goa3_included.sah
function login($username, $password){
 _setValue(_textbox("user"), $username);
 _setValue(_password("password"), $password);
 _click(_submit("Login"));
}

function addBooks($numJava, $numRuby, $numPython){
 _setValue(_textbox("q"), $numJava);
 _setValue(_textbox("q[1]"), $numRuby);
 _setValue(_textbox("q[2]"), $numPython);
 _click(_button("Add"));
}

function verifyTotal($total){
 _assertEqual($total, _textbox("total").value); 
}

function logout(){
 _click(_button("Logout"));
}

// goa3.sah
_include("goa3_included.sah");

login("test", "secret");
addBooks(2, 1, 1);
verifyTotal(1150);
logout();

The next step was to modify

function addBooks($numJava, $numRuby, $numPython){
 _setValue(_textbox("q"), $numJava);
 _setValue(_textbox("q[1]"), $numRuby);
 _setValue(_textbox("q[2]"), $numPython);
 _click(_button("Add"));
}

such that identifiers "q", "q[1]" and "q[2]" become more meaningful and are independent of their order. Using the _near API, the function becomes:

function addBooks($numJava, $numRuby, $numPython){
 _setValue(_textbox("q", _near(_cell("Core Java"))), $numJava);
 _setValue(_textbox("q", _near(_cell("Ruby for Rails"))), $numRuby);
 _setValue(_textbox("q", _near(_cell("Python Cookbook"))), $numPython);
 _click(_button("Add"));
}

We then data drive the whole test by wrapping the various steps into a single function "addAndVerify", build a 2 dimensional array of values and then invoke "addAndVerify" for each row of values using _dataDrive

// club the functionality into a single function
function addAndVerify($numJava, $numRuby, $numPython, $total){
 login("test", "secret");
 addBooks($numJava, $numRuby, $numPython);
 verifyTotal($total);
 logout();
}

// build a 2D array
var $data = [
[2, 1, 1, 1150],
[3, 2, 1, 1650],
[1, 1, 1, 850]
]

// automatically invoke addAndVerify for each row in $data. 
_dataDrive(addAndVerify, $data);

We concluded the talk with an enthusiastic Q & A session.
Thank you Goa University for being a great host!

ThoughtWorks Studios' Twist 2.0 with Sahi

ThoughtWorks announces Twist 2.0 availability from 31st March 2010:
http://www.prnewswire.com/news-releases/thoughtworks-studios-new-twist-20-provides-collaborative-agile-test-management-85809582.html

Tyto Software has been collaborating with ThoughtWorks Studios to integrate Sahi with Twist and results will be visible in Twist 2.0.

"Twist 2.0 has added Sahi as an additional option for web testing. The main benefit of Sahi is that it abstracts out most difficulties that testers face while automating web applications. Its features include an excellent recorder, platform and browser independence, no XPaths, no waits and multi-threaded playback. In addition, it allows you to identify UI components within the application as you record test scenarios." Announcing Twist 2.0: Available for download on March 31