DZone
Web Dev Zone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
  • Refcardz
  • Trend Reports
  • Webinars
  • Zones
  • |
    • Agile
    • AI
    • Big Data
    • Cloud
    • Database
    • DevOps
    • Integration
    • IoT
    • Java
    • Microservices
    • Open Source
    • Performance
    • Security
    • Web Dev
DZone > Web Dev Zone > Retrieving URLs in Parallel With CURL and PHP

Retrieving URLs in Parallel With CURL and PHP

Mats Lindh user avatar by
Mats Lindh
·
Nov. 15, 11 · Web Dev Zone · Interview
Like (0)
Save
Tweet
6.27K Views

Join the DZone community and get the full member experience.

Join For Free

As we’ve recently added support for querying Solr servers in parallel, one of the things we added was a simple class to allow us to query several servers at the same time. The CURL library (which has a PHP extension) even provides an abstraction layer for doing the nitty gritty work for you, as long as you keep track of the resources. The code beneath is based on examples in the documentation and a few tweaks of my own.

The code beneath is licensed under a MIT license. You can also download the file (gzipped).



    class Footo_Content_Retrieve_HTTP_CURLParallel
    {
        /**
         * Fetch a collection of URLs in parallell using cURL. The results are
         * returned as an associative array, with the URLs as the key and the
         * content of the URLs as the value.
         *
         * @param array<string> $addresses An array of URLs to fetch.
         * @return array<string> The content of each URL that we've been asked to fetch.
         **/
        public function retrieve($addresses)
        {
            $multiHandle = curl_multi_init();
            $handles = array();
            $results = array();
     
            foreach($addresses as $url)
            {
                $handle = curl_init($url);
                $handles[$url] = $handle;
     
                curl_setopt_array($handle, array(
                    CURLOPT_HEADER => false,
                    CURLOPT_RETURNTRANSFER => true,
                ));
     
                curl_multi_add_handle($multiHandle, $handle);
            }
     
            //execute the handles
            $result = CURLM_CALL_MULTI_PERFORM;
            $running = false;
     
            // set up and make any requests..
            while ($result == CURLM_CALL_MULTI_PERFORM)
            {
                $result = curl_multi_exec($multiHandle, $running);
            }
     
            // wait until data arrives on all sockets
            while($running && ($result == CURLM_OK))
            {
                if (curl_multi_select($multiHandle) > -1)
                {
                    $result = CURLM_CALL_MULTI_PERFORM;
     
                    // while we need to process sockets
                    while ($result == CURLM_CALL_MULTI_PERFORM)
                    {
                        $result = curl_multi_exec($multiHandle, $running);
                    }
                }
            }
     
            // clean up
            foreach($handles as $url => $handle)
            {
                $results[$url] = curl_multi_getcontent($handle);
     
                curl_multi_remove_handle($multiHandle, $handle);
                curl_close($handle);
            }
     
            curl_multi_close($multiHandle);
     
            return $results;
        }
    }

 

Download the file.

Source: http://e-mats.org/2010/01/retrieving-urls-in-parallel-with-curl-and-php/
CURL PHP

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Why to Implement GitOps into Your Kubernetes CI/CD Pipelines
  • How BDD Works Well With EDA
  • Basic Convolutional Neural Network Architectures
  • An Overview of 3 Java Embedded Databases

Comments

Web Dev Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • MVB Program
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends:

DZone.com is powered by 

AnswerHub logo