Over a million developers have joined DZone.
{{announcement.body}}
{{announcement.title}}

Python: Streaming/Appending to a File

DZone's Guide to

Python: Streaming/Appending to a File

· Web Dev Zone
Free Resource

Tips, tricks and tools for creating your own data-driven app, brought to you in partnership with Qlik.

I’ve been playing around with Twitter’s API (via the tweepy library) and due to the rate limiting it imposes I wanted to stream results to a CSV file rather than waiting until my whole program had finished.

I wrote the following program to simulate what I was trying to do:

import csv
import time
 
with open("rows.csv", "a") as file:
    writer = csv.writer(file, delimiter = ",")
 
    end = time.time() + 10
    while True:
        if time.time() > end:
            break
        else:
            writer.writerow(["mark", "123"])
            time.sleep(1)

The program will run for 10 seconds and append one line to ‘rows.csv’ once a second. Although I have used the ‘a’ flag in my call to ‘open’ if I poll that file before the 10 seconds is up it’s empty:

$ date && wc -l rows.csv
Mon  9 Mar 2015 22:54:27 GMT
       0 rows.csv
 
$ date && wc -l rows.csv
Mon  9 Mar 2015 22:54:31 GMT
       0 rows.csv
 
$ date && wc -l rows.csv
Mon  9 Mar 2015 22:54:34 GMT
       0 rows.csv
 
$ date && wc -l rows.csv
Mon  9 Mar 2015 22:54:43 GMT
      10 rows.csv

I thought the flushing of the file was completely controlled by the with block but lucky for me there’s actually a flush() function which allows me to force writes to the file whenever I want.

Here’s the new and improved sample program:

import csv
import time
 
with open("rows.csv", "a") as file:
    writer = csv.writer(file, delimiter = ",")
 
    end = time.time() + 10
    while True:
        if time.time() > end:
            break
        else:
            writer.writerow(["mark", "123"])
            time.sleep(1)
            file.flush()

And if we poll the file while the program’s running:

$ date && wc -l rows.csv
Mon  9 Mar 2015 22:57:36 GMT
      14 rows.csv
 
$ date && wc -l rows.csv
Mon  9 Mar 2015 22:57:37 GMT
      15 rows.csv
 
$ date && wc -l rows.csv
Mon  9 Mar 2015 22:57:40 GMT
      18 rows.csv
 
$ date && wc -l rows.csv
Mon  9 Mar 2015 22:57:45 GMT
      20 rows.csv

Much easier than I expected – I ♥ python!

Explore data-driven apps with less coding and query writing, brought to you in partnership with Qlik.

Topics:

Published at DZone with permission of Mark Needham, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}