Text Pre-processing Basics with Pandas

Home//Text Pre-processing Basics with Pandas

In this post, we’ll take a look at the data provided in Kaggle’s Home Depot Product Search Relevance challenge to demonstrate some techniques that may be helpful in getting started with feature generation for text data. Dealing with text data is considerably different than numerical data, so there are a few basic approaches that are an excellent place to start. As always, before we start creating features we’ll need to clean and massage the data!

In the Home Depot challenge, we have a few files which provide attributes and descriptions of each of the products on their website. The idea is to figure out how relevant a particular search term(s) is to a product. As with most data science problems there are a LOT of different ways we could approach this. One exceptionally simple way would be to calculate the percentage of the search terms that are found in the product title and product description, normalized into the range. That would be a solid baseline/first submission for this task. We’re going to skip that, and move straight to an approach called TF-IDF. We’ll go into that in detail in the next post, but that approach attempts to determine the relevance of different words based on their frequency within each document (or product definition in our case) to the frequency within all documents.

The data for the Home Depot challenge is separated into several different files which we will want to combine into a single “document” for each product:

  1. attributes.csv – Contains each attribute/value pair for each product in the catalog
  2. product_descriptions.csv – Contains a single text description for each product in the catalog
  3. test/train.csv – Contains the product title and search terms used to build the model, along with the relevance for the training data (we won’t be dealing with this in this post, but there’s useful info in there as well)

Let’s start with the attributes file. There are multiple entries per product, one per attribute. The structure of this file is: product_uid, name, value. We could naively just concatenate all the values together into a single string to build the “document” for each product. However, if we take a close look at the attribute types, we can divide them into 3 main categories:

  • general: general descriptions, typically the bullet points on the product info page
  • flags: Some attributes only have yes/no values (‘MildewResistant’, ‘Waterproof’, ‘Caulkless’)
  • measurements: Some attributes are explicitly measurements (‘Number of Pieces’, ‘Size of Opening’, ‘Product Width’)

attributes file sample

The values of flags won’t be very useful in a concatenated string. We’d just have a lot of “yes” and “no” values within each product without any information about what those represent. So instead, let’s replace that value with the feature name, and concatenate “Non” to the front if the original value is “no”:

flag attributes

The measurement-type attributes are similarly unhelpful without some modification. For example, if the value is simply a number, that doesn’t really help to identify what measurement is being specified. For example, if the attribute name is “Number of Panels” simply including the value of “2” doesn’t really provide useful information for a feature unless we incorporate the attribute name into the value: “2” -> “2 Panels”

measurement attributes

As you can see, this isn’t perfectly clean (cases like the above “Number of Faucet Handles” attribute pops up here and there) but this is definitely better than the alternative. Also in this example, this just provides extra weight in TF-IDF for ‘Handles’ which may be very useful as most products don’t have handles. For this data, the “Number of” attributes are one case of a measurement attribute, but there are many others (height, width, size, etc) that we’ll handle a bit differently, just with simple concatentation: “8” -> “A/C Coverage Area (sq. ft.) 8”

measurement attributes 2

At this point we have a dataframe of modestly munged text data that can be turned into features. To be sure, this barely scratches the surface of what could be done for preprocessing, but it at least ensures that most of the data we have is available for the feature generator. A couple ideas of other things we could have done:

  • In ‘Number of’ attributes, change value of “0” or “None” to “No” so it is closer to what a search term may be: “0 Bulbs Required -> “No Bulbs Required”
  • Convert all measurement abbreviations to full words: “(sq. ft.)” -> “square feet”
  • Dealing with apostrophes, ampersands, degree symbols, and other punctuation and other symbols

Rather than have multiple entries for each product (the ultimate level of detail we’ll eventually predict about) we must combine all the information we have into a single “document”. We’ll first do this for the attributes we’ve worked on, and then we’ll bring in the raw product descriptions and combine those as well

product attributes

product descriptions

product documents

And now we’re ready for TF-IDF, in a post to follow

A Jupyter Notebook containing the full script can be found on Github: https://github.com/UltravioletAnalytics/text-features

2017-01-30T13:47:13+00:00 May 10th, 2016|4 Comments
  • pedrov1

    hello
    can you please explain the meaning of df ??
    because df is not defined at the begining

    • Sautrik Joardar

      It refers to a pandas dataframe object named ‘df’

    • Hi pedrov1! Sorry for the confusion, I didn’t include all of the code in this post, but the link at the bottom will take you to the entirety of the code used!

      • pedrov1

        okay thank you for the clarification