Natural Language Processing (NLP) Cheat Sheet

Natural Language Processing (NLP) is a branch of AI focused on enabling machines to understand, interpret, and generate human language. It combines computational linguistics with AI and machine learning to process and analyze large amounts of natural language data. ### **1. Common NLP Tasks** - **Tokenization**: Breaking text into words, sentences, or smaller units (tokens). - **POS Tagging**: Assigning parts of speech (nouns, verbs, adjectives, etc.) to each word in a sentence. - **Named Entity Recognition (NER)**: Identifying and classifying proper nouns (people, places, organizations) in text. - **Sentiment Analysis**: Determining the emotional tone (positive, negative, or neutral) within a text. - **Machine Translation**: Automatically translating text from one language to another. - **Text Classification**: Categorizing text into predefined labels (e.g., spam vs. non-spam emails). - **Text Summarization**: Producing concise summaries of long texts. - **Question Answering (QA)**: Automatically answering questions based on text input. - **Speech Recognition**: Converting spoken language into text. ### **2. Key NLP Concepts** - **Tokens**: Basic units of text, usually words or subwords (e.g., "dogs" becomes "dog" + "s"). - **Corpus**: A large collection of text used for training NLP models. - **Vocabulary**: A set of known words or tokens from the corpus used for NLP tasks. - **Stopwords**: Commonly used words (e.g., "is," "the," "and") that are often removed from text because they add little meaning. - **Lemmatization**: Reducing words to their base form (e.g., "running" to "run"). - **Stemming**: Cutting off suffixes to reduce words to their root form (e.g., "running" becomes "run"). - **N-grams**: A contiguous sequence of n tokens (e.g., "big data" is a 2-gram or bi-gram). ### **3. Key NLP Techniques** - **Bag of Words (BoW)**: A model that represents text as a collection of words without considering word order. - **TF-IDF (Term Frequency-Inverse Document Frequency)**: A statistic that reflects how important a word is in a document relative to a corpus. - **Word Embeddings**: Dense vector representations of words in continuous space where semantically similar words are closer together (e.g., Word2Vec, GloVe). - **Language Models**: Models trained to predict the next word in a sequence (e.g., GPT, BERT). They can be used for text generation, translation, and many other tasks. ### **4. NLP Pipeline** 1. **Text Preprocessing**: - Lowercasing - Removing punctuation, special characters, and stopwords - Tokenization - Stemming/Lemmatization 2. **Feature Extraction**: - Bag of Words - TF-IDF - Word Embeddings (Word2Vec, GloVe) 3. **Modeling**: - Supervised learning (text classification) - Unsupervised learning (clustering, topic modeling) 4. **Evaluation**: - Accuracy, Precision, Recall, F1-score - Confusion matrix - BLEU score for machine translation - Perplexity for language models ### **5. Preprocessing in NLP** - **Tokenization**: Breaking text into individual words or phrases. E.g., `"I love NLP"` becomes `[I, love, NLP]`. - **Stopwords Removal**: Removing common words that carry little meaning, like "the," "is," "in." - **Lowercasing**: Converting all words to lowercase to reduce vocabulary size. E.g., `"NLP"` becomes `"nlp"`. - **Punctuation Removal**: Stripping punctuation to focus on words. E.g., `"NLP is fun!"` becomes `"NLP is fun"`. - **Stemming**: Cutting off suffixes to get the base word. E.g., `"fishing"` becomes `"fish"`. - **Lemmatization**: Reducing words to their dictionary form. E.g., `"am," "are," "is"` become `"be"`. ### **6. Word Embeddings** - **Word2Vec**: Maps words into continuous vector space where similar words have closer representations. Comes in two types: - **CBOW (Continuous Bag of Words)**: Predicts the current word based on surrounding words. - **Skip-gram**: Predicts surrounding words from the current word. - **GloVe (Global Vectors for Word Representation)**: Combines matrix factorization techniques with word co-occurrence to create word vectors. - **FastText**: An extension of Word2Vec that considers subwords, useful for capturing morphological information (e.g., "play," "player," "played"). ### **7. Language Models** - **Recurrent Neural Networks (RNNs)**: Useful for sequential data like text, as they maintain "memory" of previous inputs. - **Long Short-Term Memory (LSTM)**: A type of RNN that overcomes the vanishing gradient problem, allowing better handling of long-term dependencies in text. - **Transformers**: A neural network architecture that processes text in parallel and has revolutionized NLP with models like BERT and GPT. - **BERT (Bidirectional Encoder Representations from Transformers)**: Pre-trained on massive text data, BERT uses both left and right context to understand word meaning. - **GPT (Generative Pretrained Transformer)**: Primarily used for text generation. GPT models can generate coherent and contextually relevant text. ### **8. Popular NLP Algorithms** - **Naive Bayes**: A probabilistic classifier based on Bayes’ Theorem, often used for text classification (e.g., spam detection). - **Support Vector Machines (SVMs)**: A supervised machine learning algorithm that can be used for text classification. - **Logistic Regression**: A classification algorithm often used for binary tasks (e.g., positive vs. negative sentiment). - **K-Means Clustering**: An unsupervised learning algorithm that groups data into clusters based on similarity. ### **9. Sentiment Analysis** - **Lexicon-based**: Relies on a predefined list of words with positive, negative, or neutral sentiments. - **Machine Learning-based**: Trained models that learn to identify sentiment based on labeled data. Example algorithms include Naive Bayes, SVM, and logistic regression. ### **10. Named Entity Recognition (NER)** - **NER**: Identifies entities like people, organizations, locations, dates, etc., in text. Example entities: - `"Barack Obama"` → Person - `"Google"` → Organization - `"New York"` → Location ### **11. Evaluation Metrics for NLP** - **Accuracy**: Percentage of correctly classified instances. - **Precision**: How many of the predicted positives were actual positives. - **Recall**: How many actual positives were correctly identified. - **F1-Score**: The harmonic mean of precision and recall, balancing the two. - **BLEU (Bilingual Evaluation Understudy)**: A score for evaluating the quality of machine-translated text compared to human translations. - **Perplexity**: A metric used to evaluate language models, indicating how well the model predicts a sample. ### **12. Key NLP Libraries and Tools** - **NLTK (Natural Language Toolkit)**: A powerful Python library for text processing and linguistic analysis. - **spaCy**: An industrial-strength NLP library for Python with support for tokenization, POS tagging, dependency parsing, and named entity recognition. - **Gensim**: A Python library focused on topic modeling and document similarity using techniques like Word2Vec and LDA (Latent Dirichlet Allocation). - **Hugging Face Transformers**: Provides easy-to-use interfaces for state-of-the-art pre-trained models like BERT, GPT, and others for a wide range of NLP tasks. - **StanfordNLP**: A suite of NLP tools developed by Stanford University for a wide range of languages and NLP tasks. ### **14. Real-World Applications of NLP** - **Chatbots and Virtual Assistants**: AI-powered systems like Siri, Alexa, and Google Assistant rely heavily on NLP for voice recognition and response generation. - **Text Summarization**: Automatically creating concise summaries of long articles or reports. - **Sentiment Analysis**: Used in social media monitoring, customer feedback, and market analysis to gauge public opinion. - **Machine Translation**: Systems like Google Translate rely on NLP to convert text between languages. - **Speech Recognition**: Converting spoken language into text, as seen in transcription software and voice-controlled applications. - **Document Classification**: Automatically categorizing legal documents, emails, or academic papers into predefined categories. This **Natural Language Processing Cheat Sheet** provides a concise overview of NLP’s essential concepts, techniques, and tools. It can serve as a quick reference guide for beginners and practitioners looking to deepen their understanding of language processing.