Sahi vs. Selenium

Though Sahi is aimed at non-programmer testers and Selenium at programmers, we get a lot of queries on their differences. Here is a brief document which compares Selenium and Sahi.

To those who read this:

1. If you are a Selenium fan, be open minded and verify the claims for yourself or contact us for clarification. And again, please be open minded. You may save a lot of time for yourself and your team.
2. This document is biased towards Sahi because we built Sahi. We believe what we state is true, but if you have evidence to prove otherwise, please do contact us at support@sahi.co.in.
3. When we say "Not sure" or "?" it means we do not have enough information because of lack of research on our part. Please verify for yourself.
4. If you want a comparison between other tools and Sahi, we would be happy to discuss it.
5. If you are another tool developer/supporter, please let us know how it compares and let the world benefit from alternatives.

Sahi vs. Selenium: Comparison document

Recorder

Selenium	Sahi
Works only on Firefox	Works on all browsers (IE, FF, Chrome, Safari, Opera)
Has trouble recording IFrames, Frames and popup windows	Can record on IFrames, Frames and popup windows
For Frames and IFrames, need to explicitly selectFrame	Implicit smart identification of elements even across Frames and IFrames
Uses XPath for identification of elements if id or name not present	Uses various algorithms to uniquely identify elements in a simple human recognizable way

Programming Language support

Selenium	Sahi
Java, Ruby, Perl, Python, C# (and may be more).	Sahi Script, Java, Ruby Sahi Script has the syntax of javascript but can interoperate with any Java code directly from script. The Java/Ruby drivers are available since Sahi V3
Needs language bridges for each new feature. For example, needs java bridge to invoke Flash via ExternalAPI.	Sahi Script can directly invoke anything exposed by javascript.

Ease of use

Selenium	Sahi
Easy to start with because of Selenium IDE which is a firefox plugin. Estimated start time less than 5 minutes	More difficult than Selenium to start because it needs installation of Sahi. Estimated start time 10-30 minutes, depending on Java installation etc.
Deep learning curve when the need is felt to move from Selenium IDE to Selenium RC.	There is only one mode of operation for Sahi. Extremely simple to learn and use for testers
Knowledge of programming language required	Can achieve most automation with just functions and variables. Sahi has inbuilt APIs for most other complex tasks like data driven testing
Needs JUnit style of programming	Can choose your own style
Uses XPath based identification for elements in complex html structures or those with dynamic ids. css selectors and javascript may also be used.	Has nearness APIs like _in and _near which can help show nearness of elements. Eg. _image(“delete.gif”, _near(“username 4”))
Needs waits for making AJAX work	No waits needed in 90% cases
Supports parallel execution	Inbuilt parallel execution. Needs only one parameter change

Stability of scripts and ease of maintenance

	Selenium	Sahi
Smart DOM Relations resilient to UI changes	No Dependent on XPath Difficult for testers to understand and debug	Yes Does not use XPaths. Uses _near and _in
Implicit waits for page load and AJAX: 1) Saves time 2) Keeps scripts simple 3) Reduces random failures	No Explicit waits needed.	Yes
Ease of adoption by a team of testers	Needs testers to know TestNG/Junit, XPaths, HTML structures, Frames IFrame knowledge, Javascript for AJAX conditional waits	Sahi abstracts out all these for the tester.

Dependency on other tools

Selenium – Java (Others need something similar)	Sahi
Needs JUnit (and optionally eclipse) to run tests	No additional tools required. Tests run from the Sahi Controller/command line/ant
Non persistent reporting. Needs TestNG or something similar for that	Persistent HTML reporting which can be shared via URL or file

Stability of product and number of releases

Selenium	Sahi
Started 2004(?) in ThoughtWorks	Started 2005 in ThoughtWorks
Version 1 took 5 years, Version 2 planned mid-2010. Moving away from original architecture to WebDriver based architecture	Current release: Version 3 Number of stable releases in 2009: 7

Footprint

Selenium	Sahi
RC: 10.5 MB, Grid 15 MB	less than 2 .5 MB with source
Not sure	Runtime ~ 50MB for 3 parallel threads

Reporting

Selenium	Sahi
Needs external tools to create readable reports	Inbuilt HTML reports with click through to relevant portion of script

Others

	Selenium	Sahi
Build tool integration (ant, batch files)	Yes	Yes
Multiple OS support	Yes	Yes
Version Controllable Scripts/Code	Yes	Yes
HTTPS support/redirects	Not sure	Yes
401 Authentication, Windows/NTLM Authentication dialogs	Not sure	Yes
External proxy tunneling	Yes	Yes
In built APIs for data driven testing	No	Yes
Works only with browsers	Yes	Yes
Needs privileged modes on browsers for operation. (Privileged is bad)	Yes	No
Extensible on future browsers	Depends on finding a way to use privileged mode on that browser	Yes. Very little dependency on type of browser.
Editor support	Has good editors in various languages	Editor support for javascript is not as good as for Java.

Support available

	Selenium	Sahi
Free support via Forums	Yes	Yes
Paid support available	Yes	Yes
Authoritative training available	?	Yes

Presentation at XP Goa day

Narayan Raman presented on Functional Testing of Web Applications using Sahi at the XP Goa Day in Goa University.

Goa XP day

View more presentations from narayanraman.

As part of a presentation , we did a small demo on record and playback of a script using Sahi, then refactored the code to be maintainable.

The site under test is available here: http://sahi.co.in/demo/training
The first cut from the recorder came out to be this:

_setValue(_textbox("user"), "test");
_setValue(_password("password"), "secret");
_click(_submit("Login"));
_setValue(_textbox("q"), "2");
_setValue(_textbox("q[1]"), "1");
_setValue(_textbox("q[2]"), "1");
_click(_button("Add"));
_assertExists(_textbox("total"));
_assert(_isVisible(_textbox("total")));
_assertEqual("1150", _textbox("total").value);
_click(_button("Logout"));

This was then refactored into 2 scripts, one containing functions and the other invoking it:

// goa3_included.sah
function login($username, $password){
 _setValue(_textbox("user"), $username);
 _setValue(_password("password"), $password);
 _click(_submit("Login"));
}

function addBooks($numJava, $numRuby, $numPython){
 _setValue(_textbox("q"), $numJava);
 _setValue(_textbox("q[1]"), $numRuby);
 _setValue(_textbox("q[2]"), $numPython);
 _click(_button("Add"));
}

function verifyTotal($total){
 _assertEqual($total, _textbox("total").value); 
}

function logout(){
 _click(_button("Logout"));
}

// goa3.sah
_include("goa3_included.sah");

login("test", "secret");
addBooks(2, 1, 1);
verifyTotal(1150);
logout();

The next step was to modify

function addBooks($numJava, $numRuby, $numPython){
 _setValue(_textbox("q"), $numJava);
 _setValue(_textbox("q[1]"), $numRuby);
 _setValue(_textbox("q[2]"), $numPython);
 _click(_button("Add"));
}

such that identifiers "q", "q[1]" and "q[2]" become more meaningful and are independent of their order. Using the _near API, the function becomes:

function addBooks($numJava, $numRuby, $numPython){
 _setValue(_textbox("q", _near(_cell("Core Java"))), $numJava);
 _setValue(_textbox("q", _near(_cell("Ruby for Rails"))), $numRuby);
 _setValue(_textbox("q", _near(_cell("Python Cookbook"))), $numPython);
 _click(_button("Add"));
}

We then data drive the whole test by wrapping the various steps into a single function "addAndVerify", build a 2 dimensional array of values and then invoke "addAndVerify" for each row of values using _dataDrive

// club the functionality into a single function
function addAndVerify($numJava, $numRuby, $numPython, $total){
 login("test", "secret");
 addBooks($numJava, $numRuby, $numPython);
 verifyTotal($total);
 logout();
}

// build a 2D array
var $data = [
[2, 1, 1, 1150],
[3, 2, 1, 1650],
[1, 1, 1, 850]
]

// automatically invoke addAndVerify for each row in $data. 
_dataDrive(addAndVerify, $data);

We concluded the talk with an enthusiastic Q & A session.
Thank you Goa University for being a great host!