spaCy Introduction: Linguistic Feature Extraction

Getting Started with spaCy

In this blog, we cover spaCy and the NLP linguistic features it provides. We will work through tokenization, part-of-speech tagging, named entity recognition, dependency parsing, and visualization using displaCy.

spaCy is a free, open-source library for advanced Natural Language Processing (NLP) in Python. It is designed for production use and helps us build applications that process and understand large volumes of text. It is written in Cython and built for information extraction, natural language understanding, and pre-processing text for deep learning.

Linguistic Features in spaCy

Processing raw text well is hard. Most words are rare, and words that look very different can mean almost the same thing.

That is exactly what spaCy is designed to handle. We put in raw text and get back a Doc object with many linguistic tags attached.

spaCy covers tokenization, lemmatization, part-of-speech (POS) tagging, named entity recognition, dependency parsing, sentence segmentation, word vectors, and text cleaning.

Setup

pip install -U spacy
pip install -U spacy-lookups-data
python -m spacy download en_core_web_sm

Once we have downloaded and installed a model, we load it via spacy.load(). spaCy includes several pretrained models. The default model for the English language is en_core_web_sm.

The nlp object is a language instance of the spaCy model. It holds all the parts and data needed to process text.

import spacy
nlp = spacy.load('en_core_web_sm')

Tokenization

Tokenization is the task of splitting a text into small units called tokens. The input is a unicode text and the output is a Doc object.

A Doc is a sequence of Token objects. Each Doc consists of individual tokens that we can iterate over.

doc = nlp("Apple isn't looking at buyig U.K. startup for $1 billion")
for token in doc:
    print(token.text)

Apple
is
n't
looking
at
buyig
U.K.
startup
for
$
1
billion

Lemmatization

Lemmatization reduces a word to its base or origin form. This reduced form is called a lemma.

For example, organizes, organized, and organizing are all forms of organize. Here, organize is the lemma.

Lemmatization brings the different forms of a word together, so we can treat them as one item. It also cleans up the text for later steps.

doc

Apple isn't looking at buyig U.K. startup for $1 billion

for token in doc:
    print(token.text, token.lemma_)

Apple Apple
is be
n't not
looking look
at at
buyig buyig
U.K. U.K.
startup startup
for for
$ $
1 1
billion billion

Part-of-speech tagging

Part-of-speech tagging assigns a POS tag to each token depending on its usage in the sentence.

for token in doc:
    print(f'{token.text:{15}} {token.lemma_:{15}} {token.pos_:{10}} {token.is_stop}')

Apple           Apple           PROPN      False
is              be              AUX        True
n't             not             PART       True
looking         look            VERB       False
at              at              ADP        True
buyig           buyig           NOUN       False
U.K.            U.K.            PROPN      False
startup         startup         NOUN       False
for             for             ADP        True
$               $               SYM        False
1               1               NUM        False
billion         billion         NUM        False

Dependency Parsing

Dependency parsing finds the grammar structure of a sentence. It links headwords to the words that depend on them. The head of a sentence has no link of its own and is called the root. The verb is usually the head. All other words link back to it.

Noun chunks are flat phrases that have a noun as their head. To get the noun chunks in a document, iterate over Doc.noun_chunks.

for chunk in doc.noun_chunks:
    print(f'{chunk.text:{30}} {chunk.root.text:{15}} {chunk.root.dep_}')

Apple                          Apple           nsubj
buyig U.K. startup             startup         pobj

Named Entity Recognition

Named Entity Recognition (NER) finds named entities in raw text and sorts them into set groups. These include people, organizations, places, money, percentages, and time.

We use it to build tags for a set of documents and improve keyword search. Named entities are on the ents property of a Doc.

doc

Apple isn't looking at buyig U.K. startup for $1 billion

for ent in doc.ents:
    print(ent.text, ent.label_)

Apple ORG
U.K. GPE
$1 billion MONEY

Sentence Segmentation

Sentence Segmentation finds where each sentence starts and ends, and splits the text into clear units. spaCy uses the dependency parse to set the sentence breaks. The sents property extracts sentences from a Doc.

doc

Apple isn't looking at buyig U.K. startup for $1 billion

for sent in doc.sents:
    print(sent)

Apple isn't looking at buyig U.K. startup for $1 billion

doc1 = nlp("Welcome to KGP Talkie. Thanks for watching. Please like and subscribe")
for sent in doc1.sents:
    print(sent)

Welcome to KGP Talkie.
Thanks for watching.
Please like and subscribe

doc1 = nlp("Welcome to.*.KGP Talkie.*.Thanks for watching")
for sent in doc1.sents:
    print(sent)

Welcome to.*.KGP Talkie.*.Thanks for watching

When delimiters are present, sentence segmentation may fail to detect boundaries correctly. In that case, write custom rules based on the delimiter pattern.

Here is an example where ... is used as the delimiter.

def set_rule(doc):
    for token in doc[:-1]:
        if token.text == '...':
            doc[token.i + 1].is_sent_start = True
    return doc

nlp.add_pipe(set_rule, before = 'parser')

text = 'Welcome to KGP Talkie...Thanks...Like and Subscribe!'
doc = nlp(text)
for sent in doc.sents:
    print(sent)

Welcome to KGP Talkie...
Thanks...
Like and Subscribe!

for token in doc:
    print(token.text)

Welcome
to
KGP
Talkie
...
Thanks
...
Like
and
Subscribe
!

Visualization

spaCy includes a built-in visualizer called displaCy. We can use it to visualize a dependency parse or named entities in a browser or Jupyter notebook.

Pass a Doc or a list of Doc objects to displaCy and call displacy.serve to run the web server, or displacy.render to generate the raw markup.

from spacy import displacy

doc

Welcome to KGP Talkie...Thanks...Like and Subscribe!

Visualizing the dependency parse

The dependency visualizer, dep, shows part-of-speech tags and syntactic dependencies.

displacy.render(doc, style='dep')

The argument options accepts a dictionary of settings to customize the layout.

displacy.render(doc, style='dep', options={'compact':True, 'distance': 100})

Visualizing the entity recognizer

The entity visualizer, ent, highlights named entities and their labels in a text.

doc = nlp("Apple isn't looking at buyig U.K. startup for $1 billion")

displacy.render(doc, style='ent')

Conclusion

spaCy is a production-grade NLP library for Python. It covers tokenization, lemmatization, POS tagging, named entity recognition, dependency parsing, sentence segmentation, and visualization through displaCy.