pdstools.decision_analyzer.data_read_utils
==========================================

.. py:module:: pdstools.decision_analyzer.data_read_utils


Functions
---------

.. autoapisummary::

   pdstools.decision_analyzer.data_read_utils.read_nested_zip_files
   pdstools.decision_analyzer.data_read_utils.read_gzipped_data
   pdstools.decision_analyzer.data_read_utils.read_gzips_with_zip_extension
   pdstools.decision_analyzer.data_read_utils.read_data
   pdstools.decision_analyzer.data_read_utils.get_da_data_path
   pdstools.decision_analyzer.data_read_utils.validate_columns


Module Contents
---------------

.. py:function:: read_nested_zip_files(file_buffer) -> polars.DataFrame

   Reads a zip file buffer (uploaded from Streamlit) that contains .zip files,
   which are in fact gzipped ndjson files. Extracts, reads, and concatenates
   them into a single Polars DataFrame.

   :param file_buffer: The uploaded zip file buffer from Streamlit.
   :type file_buffer: UploadedFile

   :returns: A concatenated Polars DataFrame containing the data from all gzipped ndjson files.
   :rtype: pl.DataFrame


.. py:function:: read_gzipped_data(data: io.BytesIO) -> Optional[polars.DataFrame]

   Reads gzipped ndjson data from a BytesIO object and returns a Polars DataFrame.

   :param data: The gzipped ndjson data.
   :type data: BytesIO

   :returns: The Polars DataFrame containing the data, or None if reading fails.
   :rtype: Optional[pl.DataFrame]


.. py:function:: read_gzips_with_zip_extension(path: str) -> polars.DataFrame

   Iterates over all files with a .zip extension in the given directory, treats them
   as gzipped ndjson files, reads, and concatenates them into a single Polars DataFrame.

   :param path: The path to the directory containing the .zip files.
   :type path: str

   :returns: A concatenated Polars DataFrame containing the data from all gzipped ndjson files.
   :rtype: pl.DataFrame


.. py:function:: read_data(path)

.. py:function:: get_da_data_path()

.. py:function:: validate_columns(df: polars.LazyFrame, extract_type: Dict[str, pdstools.decision_analyzer.table_definition.TableConfig])