| Share

Saturday, April 24, 2010

Sahi vs. Selenium

Though Sahi is aimed at non-programmer testers and Selenium at programmers, we get a lot of queries on their differences. Here is a brief document which compares Selenium and Sahi.

To those who read this:
  1. If you are a Selenium fan, be open minded and verify the claims for yourself or contact us for clarification. And again, please be open minded. You may save a lot of time for yourself and your team.
  2. This document is biased towards Sahi because we built Sahi. We believe what we state is true, but if you have evidence to prove otherwise, please do contact us at support@sahi.co.in.
  3. When we say "Not sure" or "?" it means we do not have enough information because of lack of research on our part. Please verify for yourself.
  4. If you want a comparison between other tools and Sahi, we would be happy to discuss it.
  5. If you are another tool developer/supporter, please let us know how it compares and let the world benefit from alternatives.

Sahi vs. Selenium: Comparison document

Recorder

SeleniumSahi
Works only on FirefoxWorks on all browsers (IE, FF, Chrome, Safari, Opera)
Has trouble recording IFrames, Frames and popup windowsCan record on IFrames, Frames and popup windows
For Frames and IFrames, need to explicitly selectFrameImplicit smart identification of elements even across Frames and IFrames
Uses XPath for identification of elements if id or name not presentUses various algorithms to uniquely identify elements in a simple human recognizable way

Programming Language support

SeleniumSahi
Java, Ruby, Perl, Python, C# (and may be more).Sahi Script, Java, Ruby Sahi Script has the syntax of javascript but can interoperate with any Java code directly from script. The Java/Ruby drivers are available since Sahi V3
Needs language bridges for each new feature. For example, needs java bridge to invoke Flash via ExternalAPI.Sahi Script can directly invoke anything exposed by javascript.

Ease of use

SeleniumSahi
Easy to start with because of Selenium IDE which is a firefox plugin. Estimated start time less than 5 minutesMore difficult than Selenium to start because it needs installation of Sahi. Estimated start time 10-30 minutes, depending on Java installation etc.
Deep learning curve when the need is felt to move from Selenium IDE to Selenium RC.There is only one mode of operation for Sahi. Extremely simple to learn and use for testers
Knowledge of programming language requiredCan achieve most automation with just functions and variables. Sahi has inbuilt APIs for most other complex tasks like data driven testing
Needs JUnit style of programmingCan choose your own style
Uses XPath based identification for elements in complex html structures or those with dynamic ids. css selectors and javascript may also be used.Has nearness APIs like _in and _near which can help show nearness of elements. Eg. _image(“delete.gif”, _near(“username 4”))
Needs waits for making AJAX workNo waits needed in 90% cases
Supports parallel executionInbuilt parallel execution. Needs only one parameter change


Stability of scripts and ease of maintenance

SeleniumSahi
Smart DOM Relations resilient to UI changesNo
Dependent on XPath
Difficult for testers to understand and debug
Yes
Does not use XPaths.
Uses _near and _in
Implicit waits for page load and AJAX:
1) Saves time
2) Keeps scripts simple
3) Reduces random failures
No
Explicit waits needed.
Yes
Ease of adoption by a team of testersNeeds testers to know TestNG/Junit, XPaths, HTML structures, Frames IFrame knowledge, Javascript for AJAX conditional waits Sahi abstracts out all these for the tester.

Dependency on other tools

Selenium – Java (Others need something similar)Sahi
Needs JUnit (and optionally eclipse) to run testsNo additional tools required. Tests run from the Sahi Controller/command line/ant
Non persistent reporting. Needs TestNG or something similar for thatPersistent HTML reporting which can be shared via URL or file

Stability of product and number of releases

SeleniumSahi
Started 2004(?) in ThoughtWorksStarted 2005 in ThoughtWorks
Version 1 took 5 years, Version 2 planned mid-2010. Moving away from original architecture to WebDriver based architectureCurrent release: Version 3 Number of stable releases in 2009: 7

Footprint

SeleniumSahi
RC: 10.5 MB, Grid 15 MBless than 2 .5 MB with source
Not sureRuntime ~ 50MB for 3 parallel threads

Reporting

SeleniumSahi
Needs external tools to create readable reportsInbuilt HTML reports with click through to relevant portion of script

Others

SeleniumSahi
Build tool integration (ant, batch files)YesYes
Multiple OS supportYesYes
Version Controllable Scripts/CodeYesYes
HTTPS support/redirectsNot sureYes
401 Authentication, Windows/NTLM Authentication dialogsNot sureYes
External proxy tunnelingYesYes
In built APIs for data driven testingNoYes
Works only with browsersYesYes
Needs privileged modes on browsers for operation. (Privileged is bad)YesNo
Extensible on future browsersDepends on finding a way to use privileged mode on that browserYes. Very little dependency on type of browser.
Editor supportHas good editors in various languagesEditor support for javascript is not as good as for Java.

Support available

SeleniumSahi
Free support via ForumsYesYes
Paid support availableYesYes
Authoritative training available?Yes

14 comments:

felix said...

How about stability?

Even selenium scores badly against webdriver...

V. Narayan Raman said...

Hi Felix,
Stability is something that comes up again and again. Thanks for bringing up that point.

The cause for instability generally is due to
1) Timeouts: when the server does not respond within time, the tests break. The tool should have the intelligence to wait for the necessary time. Sahi monitors page loads and AJAX and waits till activity has subsided. Selenium requires explicit waits or waits with conditions. The conditional wait is preferred but it complicates the script and demands more from the tester.
2) Scripts very dependent on UI changes. In the absence of ids, Selenium is very dependent on XPaths, which is a hassle for testers. If the tester does not tweak the XPath, the tests will be brittle. Tweaking XPath for lots of UI elements is not a trivial task and can take up valuable tester time.

zawistol said...

Raman,
What would you recommend for Flex testing, we have a few folks using QTP with Flex add in.
Lech

V. Narayan Raman said...

Hi Lech,

We are adding support for Flex too. It will follow the same principles of Sahi so it should be simple. If your requirement is urgent, you could check out FlexMonkey. http://www.gorillalogic.com/flexmonkey

Regards,
Narayan

David Burns said...

Selenium does use XPath but does not rely on it. Selenium has a number of different locator techniques from using IDs or Names to CSS or XPath. You can even use JavaScript if you wanted.

Selenium, like Sahi, can invoke External APIs as long as you can access them via JavaScript. It something that is widely used by Developers/Testers around the world.

Selenium 2 has implicit waits. Its something that Selenium has been lacking and was one of the things that we needed to do and have now done. Users can use the implicit wait but these will still timeout after a while because something not loading quickly can show a bug, its not that the test is brittle.

Everything else looks correct from the Selenium point of view as far as I can see.

V. Narayan Raman said...

Hi David,

Thanks for stopping by and sharing your knowledge. I have updated the post with your information.

Selenium 2 is still in alpha so I am not adding in implicit waits yet :)

Thanks,
Narayan

psylence said...

Does Sahi handles subdomains while testing? As i understood, selenium does not..

V. Narayan Raman said...

Hi psylence,

Yes Sahi works with subdomains. You may need to modify sahi/htdocs/spr/domainfix.js to set document.domain to the base domain.

psylence said...

Raman, thanks for your answer!

Btw, i have problems running Sahi on MacOS X. I've gooled and found this one: http://sahi.co.in/forums/viewtopic.php?id=36 but it not helped(

Maybe you can give some tips, thanks.

V. Narayan Raman said...

Hi psylence,

If you post your problem on the Sahi Forums (http://sahi.co.in/forums) or email us at support@sahi.co.in, we will be able to guide you.

psylence said...

Raman, thanks! I"m already managed to make it running finally.

halperinko said...

Are there any plans for SAHI to support legacy Windows and/or Java applications not under a web browser?

V. Narayan Raman said...

Hi Halperinko,

Thanks for the interest.

Currently we are concentrating only on the web. We do not have plans for windows/Java apps right now.

You could have a look at White for Windows Apps, Frankenstein for Swing apps and
SWTBot
for SWT apps.

Thanks,
Narayan

Dina Hart said...

I was given the opportunity to look for automated testing tools and had a few to compare. Eventually I ended up with Selenium or Sahi. I happily chose Sahi, because of these reasons :

1. Quick Start. The product I was testing was way established in terms of functionalities. I needed something quick to build all the test cases around it. Sahi made it easier for startup and I learnt the code very quickly.

2. Java integration. There are a lot of things you can accomplish using Java. I love the Sahi integration with Java, which allows you to test the 'untestable'.

3. Support and Documentation. There are a lot of Sahi resources in documents and forums. I usually find all my answers there. The support team also responds quickly.

4. Browser support. The flexibility of recording on different browsers is a definite plus point. Yes, you can do without it, but it saves you a lot of time if you do use it.

Using Sahi, Loving Sahi, would recommend it!