Файловый менеджер - Редактировать - /home/digitalm/venv/lib/python3.7/site-packages/pandas/io/__pycache__/parquet.cpython-37.pyc
Назад
B �5�gKB � @ sf d Z ddlmZ ddlZddlZddlmZmZ ddlm Z ddl m Z m Z ddl mZ ddlmZ dd lmZ dd lmZmZmZ dd lmZ dd lmZ dd lm Z m Z m Z m Z m Z ddd�dd�Z!d2ddddddd�dd �Z"G d d� d�Z#G d d � d e#�Z$G d d!� d!e#�Z%eej&d" d#�d3d&d'dd(d)dd*d+d,�d-d.��Z'eej&d" d#�d4dddd/�d0d1��Z(dS )5z parquet compat � )� annotationsN)�Any�AnyStr)�catch_warnings)�FilePathOrBuffer�StorageOptions)�import_optional_dependency)�AbstractMethodError)�doc)� DataFrame� MultiIndex� get_option)�generic)�Version)� IOHandles� get_handle� is_fsspec_url�is_url�stringify_path�str�BaseImpl)�engine�returnc C s� | dkrt d�} | dkrzttg}d}xF|D ]>}y|� S tk rf } z|dt|� 7 }W dd}~X Y q*X q*W td|� ���| dkr�t� S | dkr�t� S td ��dS ) zreturn our implementation�autozio.parquet.engine� z - Nz�Unable to find a usable engine; tried using: 'pyarrow', 'fastparquet'. A suitable version of pyarrow or fastparquet is required for parquet support. Trying to import the above resulted in these errors:�pyarrow� fastparquetz.engine must be one of 'pyarrow', 'fastparquet')r � PyArrowImpl�FastParquetImpl� ImportErrorr � ValueError)r Zengine_classesZ error_msgsZ engine_class�err� r" �G/home/digitalm-up/venv/lib/python3.7/site-packages/pandas/io/parquet.py� get_engine% s" & r$ �rbFr r r �boolz.tuple[FilePathOrBuffer, IOHandles | None, Any])�path�fs�storage_options�mode�is_dirr c C s� t | �}t|�r:|dkr:td�}|jj|f|p0i �\}}n |rVt|�rN|dkrVtd��d}|s�|s�t|t�r�t j � |�s�t ||d|d�}d}|j }|||fS )zFile handling for PyArrow.N�fsspecr% z8storage_options passed with buffer, or non-supported URLF)�is_textr) )r r r �coreZ url_to_fsr r � isinstancer �osr' �isdirr �handle)r' r( r) r* r+ �path_or_handler, �handlesr" r" r# �_get_path_or_handleG s" r5 c @ s6 e Zd Zedd�dd��Zdd�dd�Zd dd �ZdS ) r r )�dfc C sx t | t�std��t | jt�r>tdd� | jjD ��sRtd��n| jjdkrRtd��tdd� | jj D ��}|sttd��d S ) Nz+to_parquet only supports IO with DataFramesc s s | ]}|j d kV qdS )> �string�emptyN)� inferred_type)�.0�xr" r" r# � <genexpr>w s z.BaseImpl.validate_dataframe.<locals>.<genexpr>z� parquet must have string column names for all values in each level of the MultiIndex > r7 r8 z%parquet must have string column namesc s s | ]}|d k rt |t�V qd S )N)r/ r )r: �namer" r" r# r< � s z!Index level names must be strings) r/ r r �columnsr �allZlevelsr9 �index�names)r6 Z valid_namesr" r" r# �validate_dataframen s zBaseImpl.validate_dataframec K s t | ��d S )N)r )�selfr6 r' � compression�kwargsr" r" r# �write� s zBaseImpl.writeNc K s t | ��d S )N)r )rC r' r> rE r" r" r# �read� s z BaseImpl.read)N)�__name__� __module__� __qualname__� staticmethodrB rF rG r" r" r" r# r m s c @ s> e Zd Zdd� Zdddddd d d �d d �Zdd d�dd�ZdS )r c C s& t ddd� dd l}dd l}|| _d S )Nr z(pyarrow is required for parquet support.)�extrar )r Zpyarrow.parquetZ pandas.core.arrays._arrow_utils�api)rC r �pandasr" r" r# �__init__� s zPyArrowImpl.__init__�snappyNr zFilePathOrBuffer[AnyStr]z str | Nonez bool | Noner zlist[str] | None)r6 r' rD r@ r) �partition_colsc K s� | � |� d|�dd �i}|d k r*||d< | jjj|f|�} t||�dd �|d|d k d�\} } |d< zH|d k r�| jjj| | f||d�|�� n | jjj| | fd|i|�� W d | d k r�| � � X d S )NZschemaZpreserve_index� filesystem�wb)r) r* r+ )rD rQ rD ) rB �poprM �TableZ from_pandasr5 �parquetZwrite_to_datasetZ write_table�close) rC r6 r' rD r@ r) rQ rE Zfrom_pandas_kwargs�tabler3 r4 r" r" r# rF � s. zPyArrowImpl.writeF)r) c K sD d|d< i }|r�dd l }| j�� |�� | j�� |�� | j�� |�� | j�� |� � | j� � |� � | j� � |� � | j�� |�� | j�� |�� | j�� |�� | j�� |�� i }|j|d< td�} | dkr�d|d< t||�dd �|d d �\} } |d< z>| jjj| fd |i|��j f |�} | dk�r(| j dd d �} | S | d k �r>| � � X d S )NTZuse_pandas_metadatar Z types_mapperzmode.data_manager�arrayZ split_blocksrR r% )r) r* r> F)�copy) rN rM Zint8Z Int8DtypeZint16Z Int16DtypeZint32Z Int32DtypeZint64Z Int64DtypeZuint8Z UInt8DtypeZuint16Z UInt16DtypeZuint32Z UInt32DtypeZuint64Z UInt64DtypeZbool_Z BooleanDtyper7 Z StringDtype�getr r5 rT rV Z read_table� to_pandasZ _as_managerrW ) rC r' r> �use_nullable_dtypesr) rE Zto_pandas_kwargs�pd�mapping�managerr3 r4 �resultr" r" r# rG � s@ zPyArrowImpl.read)rP NNN)NFN)rH rI rJ rO rF rG r" r" r" r# r � s ( r c @ s6 e Zd Zdd� Zd ddd�dd �Zddd �d d �ZdS )r c C s t ddd�}|| _d S )Nr z,fastparquet is required for parquet support.)rL )r rM )rC r r" r" r# rO � s zFastParquetImpl.__init__rP Nr r )r6 r) c s� | � |� d|kr$|d k r$td��nd|kr6|�d�}|d k rFd|d< t|�}t|�rrtd�� � �fdd�|d< n �r~td ��td d ��$ | jj||f|||d �|�� W d Q R X d S ) N� partition_onzYCannot use both partition_on and partition_cols. Use partition_cols for partitioning dataZhiveZ file_schemer, c s � j | df�pi �� � S )NrS )�open)r' �_)r, r) r" r# �<lambda># s z'FastParquetImpl.write.<locals>.<lambda>� open_withz?storage_options passed with file object or non-fsspec file pathT)�record)rD Z write_indexrb ) rB r rT r r r r rM rF )rC r6 r' rD r@ rQ r) rE r" )r, r) r# rF s. zFastParquetImpl.write)r) c s� i }|� dd�}t| jj�td�kr,d|d< |r8td��t|�}d }t|�r�td�� t| jj�td�kr�� j|df�pxi �j |d < qć �fd d �|d < n,t |t �r�t j �|�s�t|dd�d �}|j}| jj|f|�}|jf d|i|��} |d k r�|�� | S )Nr] Fz0.7.1Z pandas_nullszNThe 'use_nullable_dtypes' argument is not supported for the fastparquet enginer, z0.6.1r% r( c s � j | df�pi �� � S )Nr% )rc )r' rd )r, r) r" r# re L s z&FastParquetImpl.read.<locals>.<lambda>rf )r- r) r> )rT r rM � __version__r r r r rc r( r/ r r0 r' r1 r r2 Z ParquetFiler\ rW ) rC r' r> r) rE Zparquet_kwargsr] r4 Z parquet_filera r" )r, r) r# rG 5 s0 zFastParquetImpl.read)rP NNN)NN)rH rI rJ rO rF rG r" r" r" r# r � s ,r r) )r) r rP r zFilePathOrBuffer | Nonez str | Nonez bool | Nonezlist[str] | Nonez bytes | None)r6 r' r rD r@ r) rQ r c K sr t |t�r|g}t|�}|dkr(t�� n|} |j| | f||||d�|�� |dkrjt | tj�sbt�| �� S dS dS )a Write a DataFrame to the parquet format. Parameters ---------- df : DataFrame path : str or file-like object, default None If a string, it will be used as Root Directory path when writing a partitioned dataset. By file-like object, we refer to objects with a write() method, such as a file handle (e.g. via builtin open function) or io.BytesIO. The engine fastparquet does not accept file-like objects. If path is None, a bytes object is returned. .. versionchanged:: 1.2.0 engine : {{'auto', 'pyarrow', 'fastparquet'}}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. compression : {{'snappy', 'gzip', 'brotli', None}}, default 'snappy' Name of the compression to use. Use ``None`` for no compression. index : bool, default None If ``True``, include the dataframe's index(es) in the file output. If ``False``, they will not be written to the file. If ``None``, similar to ``True`` the dataframe's index(es) will be saved. However, instead of being saved as values, the RangeIndex will be stored as a range in the metadata so it doesn't require much space and is faster. Other indexes will be included as columns in the file output. partition_cols : str or list, optional, default None Column names by which to partition the dataset. Columns are partitioned in the order they are given. Must be None if path is not a string. {storage_options} .. versionadded:: 1.2.0 kwargs Additional keyword arguments passed to the engine Returns ------- bytes if no path argument is provided else None N)rD r@ rQ r) )r/ r r$ �io�BytesIOrF �AssertionError�getvalue) r6 r' r rD r@ r) rQ rE �implZ path_or_bufr" r" r# � to_parqueta s 9 rn )r r) r] c K s" t |�}|j| f|||d�|��S )a Load a parquet object from the file path, returning a DataFrame. Parameters ---------- path : str, path object or file-like object Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, gs, and file. For file URLs, a host is expected. A local file could be: ``file://localhost/path/to/table.parquet``. A file URL can also be a path to a directory that contains multiple partitioned parquet files. Both pyarrow and fastparquet support paths to directories as well as file URLs. A directory path could be: ``file://localhost/path/to/tables`` or ``s3://bucket/partition_dir`` If you want to pass in a path object, pandas accepts any ``os.PathLike``. By file-like object, we refer to objects with a ``read()`` method, such as a file handle (e.g. via builtin ``open`` function) or ``StringIO``. engine : {{'auto', 'pyarrow', 'fastparquet'}}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. columns : list, default=None If not None, only these columns will be read from the file. {storage_options} .. versionadded:: 1.3.0 use_nullable_dtypes : bool, default False If True, use dtypes that use ``pd.NA`` as missing value indicator for the resulting DataFrame. (only applicable for the ``pyarrow`` engine) As new dtypes are added that support ``pd.NA`` in the future, the output with this option will change to use those dtypes. Note: this is an experimental option, and behaviour (e.g. additional support dtypes) may change without notice. .. versionadded:: 1.2.0 **kwargs Any additional kwargs are passed to the engine. Returns ------- DataFrame )r> r) r] )r$ rG )r' r r> r) r] rE rm r" r" r# � read_parquet� s <ro )Nr% F)Nr rP NNN)r NNF))�__doc__� __future__r ri r0 �typingr r �warningsr Zpandas._typingr r Zpandas.compat._optionalr Z pandas.errorsr Zpandas.util._decoratorsr rN r r r Z pandas.corer Zpandas.util.versionr Zpandas.io.commonr r r r r r$ r5 r r r Z _shared_docsrn ro r" r" r" r# �<module> s@ % !$ig H
| ver. 1.4 |
Github
|
.
| PHP 8.3.23 | Генерация страницы: 0.05 |
proxy
|
phpinfo
|
Настройка