Few days back I got a chance to play with Sauce Labs,thought I would share my experience as there is very little material floating around related to Sauce Labs
To give a little background,I already had a small set of Selenium RC tests in Java and my goal was to see what’s Sauce Labs all about.I had this vague idea that it runs your tests on remote machines,but it was hard to picture what happens to your TestNG results or what happens when there is an error and some other questions related to continuous integration,scheduling etc.
I tried looking up the internet for a demo video or a tutorial but sadly there’s none around which is kind of surprising given how popular Selenium itself is.So anyway I figured the next best option would be try out myself.Here’s a quick summary of experience
Day 1 (half day)
Signup was a breeze and after that I started looking for clues on what to do next.The product I was looking to evaluate was Sauce On Demand.The Sauce Labs documentation is pretty thin right now (the forum also had no entries).Anyway I figured,I had to email support to get first 100 free trial minutes .I got the response pretty quick and I had 100 nice trial meeting sitting in my account but still no idea on what to do.
Day 2
Finally I tried to do some copy/paste with the documentation provided(On Demand documentation) and a link with a Java example.What confused me though was different URL’s and port number provided in the documentation vs the Java generated example.Next hour or so I tried debugging the issue by playing around with port numbers.Even mucking around with my framework as I was not sure what is sufficient for Sauce On Demand.Finally I emailed support for help and got a quick reply with some suggestions.By the time I had also figured out the correct combination of things(in case you are interested,it’s ondemand.saucelabs.com /port 4444).Finally I was able to run my tests on Sauce on demand and understood that sauce labs only acts as a machine running the tests and rest of heavy lifting is still done my your test driver /machine.e.g collecting test framework reports etc.On the Sauce Labs side you can view the video of the test run and associated selenium logs.I figured this should be pretty useful for what I want to know and called it a day.
Day 3
The above exercise was just one test,I was tempted to run the whole suite on the Labs and see how it does.I strangely started getting firefox launcher errors(which I should say was independent of Sauce Labs),I trolled around forums and finally I tweeted for help.I got a tweetback from Labs support to make use of my 2 free help tickets to solve the issue.Around same time ,I had a colleague asking how to programatically download videos/logs for post-processing of test runs.Again looked around and saw that Labs documentation suggested to use “selenium.getSessionID’.The stable release of selenium server did not support this method and I again emailed to support with this two issues.
Day 4
I got a response from help at saucelabs for both my issues.Basically for my Firefox launcher issue,they suggested me to use Sauce RC(free) and lo and behold my error was gone.For the 2nd issue they sent me a patched version of selenium server jar to use along with the patched file so that I can get sessionID to download the logs and videos.(As a side note,I am not sure how Sauce RC is going to evolve since Sauce Labs has an interest by Selenium founder (twitter:@hugs),it would be interesting to know why certain patches are available as part of Sauce RC but not part of open source selenium stable release)
So overall,I was pretty happy with my experience.The documentation and knowledgebase,forum activity is pretty light at this moment which makes the ramping up part difficult but is more than made up by some excellent support.However life would be much easier if there were some demo video’s/tutorials to get started.
Some issues that will stop me from going all aboard on Sauce Labs train
1)To give access to an internal system there are some additional steps to secure connection betn Labs and your QA environment.I didn’t try it,but seems like a hassle
2)Variety of browsers are supported but only Windows 2003/Linux OS is supported.I would be much more comfortable if different flavors of OS were offered too.I can understand the logic that browser issues can be caught regardless of which OS it’s run,but as a QA person,I feel satisfied if I run the test on a proper configuration.
3)Our product is not a standalone web application.It interacts with a Java applet as well as allows exporting to excel etc.I would have to give up on this tests as I have no control over what software goes on the Sauce Labs machine.
I would defintely suggest Sauce Labs if you are already a Selenium shop and a Web only application.If you are spending too much money in buying new machines and maintenance to run your automated tests,I’d recommend give Sauce Labs a try.I am already a fan of Sauce RC and have been using it for my local scripting(Specially since it got rid of the “Firefox launcher error”).I would see if my colleague can also give his thoughts on Sauce Labs,we pretty much evaluated at the same time.Would be nice to know his experience as well.
Here’s the sauce labs evaluation post from my colleague for his thoughts on Sauce Labs
Follow us on @apexon
