Selenium With Java: Google Search

In this tutorial, we will be exploring the basics of how to use Selenium with Java. We will use Selenium to open Google, search, and click a URL.

The code is available on GitHub.

What Is Selenium?

Selenium automates web browsers. That’s really it.

Selenium enables us to emulate user interaction with a web page. There are two Selenium products we can use: Selenium WebDriver and Selenium IDE. We will be using WebDriver.

What is WebDriver? WebDriver is an official W3C Specification, and in essence, it is a way of interacting with a web browser. Previously, with Selenium RC, Selenium would operate with the browser by injecting JavaScript to interact with elements. With the adoption of the WebDriver specification, companies like Google, Mozilla, and Microsoft release their browser with the ability to be controlled by a hook, that Selenium can tap into. This hook enables Selenium to interact with the web browser in the same way that humans do.

We will be using Google Chrome — therefore, it is required that we download the chromedriver.

After downloading the driver, we need to execute the file.

On Macs, we can simply do this, for instance:

./chromedriver

pom.xml

I use Spring Tool Suite and created a new Spring Starter project, which wasn’t necessary, but I tend to like Spring. So Selenium is managed by the Spring Boot Starter Parent actually. The version is 2.53.1.

<!-- typical pom beginning-->
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.10.RELEASE</version>
        <relativePath /> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>

        <dependency>
            <groupId>com.h2database</groupId>
            <artifactId>h2</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
<!-- typical pom ending-->

Open Chrome and Search

For this step, we will be establishing the connection to the chromedriver, opening the browser, and searching for “Selenium”.

The port we target for our localhost is 9515 because the chromedriver runs on the local server’s port 9515.

RemoteWebDriver implements WebDriver and WebDriver’s goal is to supply an object-oriented API that provides support for modern advanced web-app testing problems. So, we can tell based on these facts that RemoteWebDriver is the implementation that allows the use of a remote browser. The benefits include separation of where the tests run from where the browser is and the ability to test with browsers not available on the current OS. The cons include the fact that we need an external servlet container to be running and there may be latency if an exception is thrown.

// create a Chrome Web Driver
URL local = new URL("http://localhost:9515");
WebDriver driver = new RemoteWebDriver(local, DesiredCapabilities.chrome());
// open the browser and go to open google.com
driver.get("https://www.google.com"); 

driver.findElement(By.id("lst-ib")).sendKeys("Selenium");
driver.findElement(By.name("btnK")).click();
driver.manage().window().maximize();

Get Pages and Click

WebDriver gives us the methods findElement and findElements method to locate element(s) on a web page. These methods accept a By object as a parameter. By has methods to locate elements within a document with the help of a locator value. Selenium has documented their API well.

Once we understand how Selenium is used to identify elements, it is easy to read any of the the driver.findElements(By…) methods. But we need to know how to write them too. Using a browser, like Chrome, we can right-click (or the equivalent) to Inspect an element to get its HTML/CSS information. Also, we can “View Source” to get more complete information as well.

To demonstrate how to scroll on a web page, we perform jse.executeScript(“window.scrollBy(0,250)”, “”).

Like the name suggests, JavaScriptExecutor executes JavaScript. JavaScriptExecutor is an interface provided through Selenium WebDriver. It provides two methods, “executescript” and “executeAsyncScript”, to run JavaScript on the selected window or current page.

With the code below, it may be possible to create a more comprehensive bot to search Google and click URLs for several pages.

// get the number of pages
   int size = driver.findElements(By.cssSelector("[valign='top'] > td")).size();
   for(int j = 1 ; j < size ; j++) {
       if (j > 1) {// we don't need to navigate to the first page
           driver.findElement(By.cssSelector("[aria-label='Page " + j + "']")).click(); // navigate to page number j
       }

       String pagesearch = driver.getCurrentUrl();

       List<WebElement> findElements = driver.findElements(By.xpath("//*[@id='rso']//h3/a"));
       System.out.println(findElements.size());

       for(int i=0;i<findElements.size();i++){
           findElements= driver.findElements(By.xpath("//*[@id='rso']//h3/a"));                
           findElements.get(i).click(); 

           driver.navigate().to(pagesearch);
           JavascriptExecutor jse = (JavascriptExecutor) driver;
           //Scroll vertically downward by 250 pixels
           jse.executeScript("window.scrollBy(0,250)", "");
       }
   }

Conclusion

This was a basic introduction to Selenium with Java. As we discovered, in Selenium Webdriver, locators like XPath, CSS, etc. are used to identify and perform operations on a web page. It is also possible to execute arbitrary JavaScript.

The complete code can be found on GitHub.