docx_parser.py

Overview

The docx_parser.py file provides functionality for parsing Microsoft Word .docx documents, specifically focusing on extracting and structuring textual and tabular data. It is designed to process paragraphs and tables within a DOCX file, classify and organize table contents intelligently based on detected data types, and return structured outputs that can be used for downstream Natural Language Processing (NLP) or data analysis tasks.

At its core, the file defines a single class, RAGFlowDocxParser, which encapsulates the parsing logic, including:

• Reading DOCX files from a file path or binary stream.

• Extracting paragraphs with page-aware segmentation.

• Extracting tables and converting their content into structured lists of strings.

• Classifying content blocks within tables to enhance interpretation and composition of the data.

This parser is particularly tailored for documents that may contain mixed content types such as dates, numbers, categories, English text, named entities, and others, and it applies heuristics and tokenization (via a RAG tokenizer) to classify and organize the data accordingly.

Classes and Methods

Class: RAGFlowDocxParser

This class handles the parsing of DOCX files and the extraction of paragraph and table content.

Method: __extract_table_content(self, tb)

Purpose:
Extracts content from a given DOCX table object (tb), converting it into a structured format.

Parameters:

• tb (docx.table.Table): A table object from the DOCX document.

Returns:

• list[str]: A list of strings representing the composed content of the table.

Description:
It iterates over each row and cell of the table, extracting the raw text, then converts the collected data into a pandas DataFrame before passing it to the __compose_table_content method for further processing and classification.

Method: __compose_table_content(self, df)

Purpose:
Processes a pandas DataFrame representation of a table to classify and organize its contents into a clean, readable list format.

Parameters:

• df (pandas.DataFrame): DataFrame containing the raw table data.

Returns:

• list[str]: A list of strings where each string represents a processed row or combined rows from the original table, formatted with header context when applicable.

Usage:
This method applies several heuristics and regex-based rules to classify the content of each cell in the DataFrame into one of several block types, using the nested helper function blockType. It uses these classifications to infer headers and data rows, concatenating headers with their corresponding cell values to generate a semicolon-separated string list of table content. The method handles different table structures, including tables dominated by numerical data or textual data, and distinguishes between headers and content rows accordingly.

Nested Function: blockType(b)

Purpose:
Classifies a block of text into one of several types based on regex patterns and token analysis.

Parameters:

• b (str): A string block (text cell content).

Returns:

• str: A code representing the type of block detected. Possible return types include:

  • "Dt": Date formats

  • "Nu": Numeric data

  • "Ca": Category-like codes

  • "En": English words

  • "NE": Named entities or complex tokens

  • "Sg": Single character

  • "Tx": Short text (moderate token count)

  • "Lx": Long text (larger token count)

  • "Nr": Named entity tagged by RAG tokenizer as "nr"

  • "Ot": Other/unknown types

Implementation Details:

• Uses regex to detect specific patterns such as dates, quarters, years, numbers, and categorized codes.

• Uses rag_tokenizer from an external NLP module for tokenization and tagging.

• Determines the block type based on the frequency and length of tokens.

Method: __call__(self, fnm, from_page=0, to_page=100000000)

Purpose:
Enables the class instance to be called as a function to parse the DOCX document and return its content.

Parameters:

• fnm (str or bytes-like object): Path to the DOCX file as a string or a binary stream representing the file content.

• from_page (int, optional): The starting page number (zero-based) for content extraction (default 0).

• to_page (int, optional): The ending page number (exclusive) for content extraction (default a very large number).

Returns:

• Tuple:

  • list[tuple[str, str]]: A list of tuples, each containing the paragraph text and its style name.

  • list[list[str]]: A list of lists where each inner list contains the processed string content from a table.

Usage Example:

parser = RAGFlowDocxParser()
paragraphs, tables = parser("example.docx", from_page=0, to_page=5)

for para, style in paragraphs:
    print(f"[{style}] {para}")

for table_content in tables:
    for line in table_content:
        print(line)

Implementation Details:

• Supports reading from file path or binary stream (e.g., downloaded file in memory).

• Iterates through paragraphs and their runs to accumulate text, respecting page breaks indicated by the presence of lastRenderedPageBreak in the run's XML.

• Collects paragraphs within the requested page range.

• Extracts and processes all tables in the document using __extract_table_content.

Important Implementation Details and Algorithms

• Page Break Detection:
  The parser detects page breaks by inspecting the XML of each run for the existence of the lastRenderedPageBreak tag. This allows page-aware segmentation of paragraphs, which is critical for processing multi-page documents selectively.

• Table Content Classification:
  The parser uses a combination of regex patterns and NLP tokenization to classify each cell's content type. This classification helps to infer headers and data rows in tables that do not explicitly mark headers. The algorithm then reconstructs table rows by prefixing cell values with their associated headers derived from preceding rows.

• Tokenization and Tagging:
  The external rag_tokenizer module is used to tokenize strings and tag tokens with entity types (e.g., "nr" for named entities). This enriches the classification logic, enabling more nuanced detection of content types in tables.

• Flexible Input Handling:
  The parser can accept either a file path or a binary stream, making it versatile for different data ingestion pipelines (e.g., local files or files downloaded in memory).

Interaction with Other System Components

• Dependency on rag.nlp.rag_tokenizer:
  The file imports and relies on the rag_tokenizer for tokenization and tagging of text blocks. This indicates that the parser is part of a larger NLP system where RAG (Retrieval-Augmented Generation) or similar techniques are employed.

• Integration with Data Processing Pipelines:
  Output from this parser (paragraph and table content) is likely used as input for downstream NLP tasks such as entity recognition, summarization, or knowledge extraction.

• Use of pandas:
  The intermediate representation of tables as DataFrames suggests that the system might further process or analyze tabular data using Pandas or export it to other formats.

Visual Diagram

classDiagram
    class RAGFlowDocxParser {
        -doc: Document
        -__extract_table_content(tb)
        -__compose_table_content(df)
        -__call__(fnm, from_page=0, to_page=100000000)
    }
    RAGFlowDocxParser : +__call__()
    RAGFlowDocxParser : -__extract_table_content()
    RAGFlowDocxParser : -__compose_table_content()

Summary

The docx_parser.py file implements a specialized DOCX document parser via the RAGFlowDocxParser class, capable of extracting paragraph text and intelligently interpreting table data into structured formats. Its main strengths lie in its page-aware paragraph segmentation, regex and NLP-based table content classification, and flexible input handling. This parser forms an essential part of the InfiniFlow project's document ingestion and NLP preprocessing pipeline.

If you require further details or additional examples for specific methods or integration instructions, please let me know!