Splitting CSV Files in Python

In this article, see a code snippet that splits CSV files in Python.

Zehra Can

Oct. 08, 20 · Code Snippet

Likes (5)

Comment

Save

71.5K Views

Sometimes it is necessary to split big files into small ones. In this brief article, I will share a small script which is written in Python. Maybe you can develop it further. However, if the file size is bigger you should be careful using loops in your code.

Let's assume your input file name input.csv. The code will split it into new files with names input_1.csv, ..., input_10.csv.,...

     Python 
   
          x 
         
import pandas as pd
#csv file name to be read in 
in_csv = 'input.csv'
#get the number of lines of the csv file to be read
number_lines = sum(1 for row in (open(in_csv)))
#size of rows of data to write to the csv, 
#you can change the row size according to your need
rowsize = 500
#start looping through data writing it to a new file for each set
for i in range(1,number_lines,rowsize):
    df = pd.read_csv(in_csv,
          header=None,
          nrows = rowsize,#number of rows to read at each loop
          skiprows = i)#skip rows that have been read
    #csv to write data to a new file with indexed name. input_1.csv etc.
    out_csv = 'input' + str(i) + '.csv'
    df.to_csv(out_csv,
          index=False,
          header=False,
          mode='a',#append data to csv file
          chunksize=rowsize)#size of data to append for each loop

Python (language) CSV

Opinions expressed by DZone contributors are their own.

Splitting CSV Files in Python

Partner Resources