Over a million developers have joined DZone.

Building a Directory Structure Index in Python

· Web Dev Zone

Start coding today to experience the powerful engine that drives data application’s development, brought to you in partnership with Qlik.

I’m working through examples in “Natural Language Processing with Python” (read my review) and found that the corpus I have to work with is large enough to require special performance tuning exercises.

If you have a large enough directory structure, it becomes difficult to walk with os.walk – for instance any failure in longer scripts require starting from scratch. This is a common issue in larger systems – typically they manage file listings through a relational database, and directory storage is obfuscated in some way.

In this environment it takes at an hour for Windows to count the files, and python seems to take longer.

It’s worth generating a list of the files in advance – this lists which PDFs and HTML documents exist, and for which a text extract has been generated (see a Node.JS approach here). This supports a few uses, including generating missing text renditions.

import os
import re
import datetime
print datetime.datetime.now()
pdf_idx = open('pdfs.idx', 'w')
rendition_idx = open('txts.idx', 'w')
html_idx = open('htmls.idx', 'w')
xml_idx = open('xmls.idx', 'w')
for root, dirs, files in os.walk('.'):
  for f in files:
    if f.endswith(".pdf"):
      rend = root + os.sep + f + ".textrendition.txt"
          with open(rend):
              rendition_idx.write(rend + "\n")
      except IOError:
      pdf_idx.write(root + os.sep + f + "\n")
    if f.endswith(".html") or f.endswith('.htm'):
      html_idx.write(root + os.sep + f + "\n")
    if f.endswith(".xml"):
      xml_idx.write(root + os.sep + f + "\n")
print datetime.datetime.now()

What this allows is quite useful – you can read the file quickly, and select a random subset for training and test data for NLP algorithms. This could also be done by storing all the names in a database, but this is probably the simplest and fastest for my current needs.

rendition_idx = open('txts1.idx', 'r')
files = [f[:-1] for f in rendition_idx]
>>> files[:4]
import random
contents = [open(f[2:]).read() for f in files[:100]]

Create data driven applications in Qlik’s free and easy to use coding environment, brought to you in partnership with Qlik.


Published at DZone with permission of Gary Sieling, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

The best of DZone straight to your inbox.

Please provide a valid email address.

Thanks for subscribing!

Awesome! Check your inbox to verify your email so you can start receiving the latest in tech news and resources.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}