Файловый менеджер - Редактировать - /home/digitalm/venv/lib/python3.7/site-packages/pandas/io/sas/__pycache__/sas7bdat.cpython-37.pyc
Назад
B �5�gbu � @ s d Z ddlmZ ddlmZ ddlmZmZ ddlZddlm Z m Z m Z ddl Z ddlmZmZ ddlZddlmZmZ dd lmZ dd lmZ ddlm m m Z dd l m Z d d d�dd�Z dd dd�dd�Z!G dd� d�Z"G dd� d�Z#G dd� de ej$�Z%dS )a� Read SAS7BDAT files Based on code written by Jared Hobbs: https://bitbucket.org/jaredhobbs/sas7bdat See also: https://github.com/BioStatMatt/sas7bdat Partial documentation of the file format: https://cran.r-project.org/package=sas7bdat/vignettes/sas7bdat.pdf Reference for binary data compression: http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/CUJ/1992/9210/ross/ross.htm � )� annotations)�abc)�datetime� timedeltaN)�IO�Any�cast)�EmptyDataError�OutOfBoundsDatetime)� DataFrame�isna)� get_handle)�Parser)� ReaderBase�float�str)� sas_datetime�unitc C sV t | �rtjS |dkr,tddd�t| d� S |dkrJtddd�t| d� S td��d S )N�si� � )�seconds�d)�dayszunit must be 'd' or 's')r �pdZNaTr r � ValueError)r r � r �L/home/digitalm-up/venv/lib/python3.7/site-packages/pandas/io/sas/sas7bdat.py�_parse_datetime1 s r z pd.Series)� sas_datetimesr �returnc C sD yt j| |dd�S tk r> | jt|d�}tt j|�}|S X dS )a� Convert to Timestamp if possible, otherwise to datetime.datetime. SAS float64 lacks precision for more than ms resolution so the fit to datetime.datetime is ok. Parameters ---------- sas_datetimes : {Series, Sequence[float]} Dates or datetimes in SAS unit : {str} "d" if the floats represent dates, "s" for datetimes Returns ------- Series Series of datetime64 dtype or datetime.datetime. z 1960-01-01)r �origin)r N)r � to_datetimer �applyr r ZSeries)r r Zs_seriesr r r �_convert_datetimes? s r# c @ sB e Zd ZU ded< ded< ded< ded< ddddd�dd�Zd S ) �_SubheaderPointer�int�offset�length� compression�ptype)r& r' r( r) c C s || _ || _|| _|| _d S )N)r& r' r( r) )�selfr& r' r( r) r r r �__init___ s z_SubheaderPointer.__init__N)�__name__� __module__� __qualname__�__annotations__r+ r r r r r$ Y s r$ c @ sV e Zd ZU ded< ded< ded< ded< ded< ded < ddddddd �d d �Zd S )�_Columnr% �col_idz str | bytes�name�label�format�bytes�ctyper' )r1 r2 r3 r4 r6 r' c C s( || _ || _|| _|| _|| _|| _d S )N)r1 r2 r3 r4 r6 r' )r* r1 r2 r3 r4 r6 r' r r r r+ n s z_Column.__init__N)r, r- r. r/ r+ r r r r r0 f s r0 c @ s� e Zd ZU dZded< ded< dTdd �Zd d �d d �Zd d �dd�Zd d �dd�Zdd �dd�Z dd �dd�Z dd� Z ddd�dd�Z dddd �d d �Z ddd �d d!�Zdd �d"d#�Zd$d �d%d&�Zd'd(� Zdd �d)d*�Zd+dd,�d-d.�Zddd/d0�d1d2�Zdd+d3�d4d5�Zdd/dd6�d7d8�Zdddd9�d:d;�Zdddd9�d<d=�Zdddd9�d>d?�Zdddd9�d@dA�Zdddd9�dBdC�Zdddd9�dDdE�Z dddd9�dFdG�Z dddd9�dHdI�Z dUdJdKdL�dMdN�Z dOdP� Z dQd �dRdS�Z!dS )V�SAS7BDATReadera Read SAS files in SAS7BDAT format. Parameters ---------- path_or_buf : path name or buffer Name of SAS file or file-like object pointing to SAS file contents. index : column identifier, defaults to None Column to use as index. convert_dates : bool, defaults to True Attempt to convert dates to Pandas datetime values. Note that some rarely used SAS date formats may be unsupported. blank_missing : bool, defaults to True Convert empty strings to missing values (SAS uses blanks to indicate missing character variables). chunksize : int, defaults to None Return SAS7BDATReader object for iterations, returns chunks with given number of lines. encoding : string, defaults to None String encoding. convert_text : bool, defaults to True If False, text variables are left as raw bytes. convert_header_text : bool, defaults to True If False, header text, including column names, are left as raw bytes. r% � _int_lengthz bytes | None� _cached_pageNTc C s� || _ || _|| _|| _|| _|| _|| _d| _d| _g | _ g | _ g | _ g | _ g | _ d | _g | _g | _g | _d| _d| _d| _t|ddd�| _ttt | jj�| _y| �� | � � W n t k r� | � � � Y nX d S )Nzlatin-1� r �rbF)Zis_text) �index� convert_dates� blank_missing� chunksize�encoding� convert_text�convert_header_text�default_encodingr( �column_names_strings� column_names�column_formats�columns�%_current_page_data_subheader_pointersr9 �_column_data_lengths�_column_data_offsets� _column_types�_current_row_in_file_indexZ_current_row_on_page_indexr �handlesr r r �handle� _path_or_buf�_get_properties�_parse_metadata� Exception�close) r* Z path_or_bufr< r= r> r? r@ rA rB r r r r+ � s: zSAS7BDATReader.__init__z np.ndarray)r c C s t j| jt jd�S )z5Return a numpy int64 array of the column data lengths)�dtype)�np�asarrayrI �int64)r* r r r �column_data_lengths� s z"SAS7BDATReader.column_data_lengthsc C s t j| jt jd�S )z0Return a numpy int64 array of the column offsets)rT )rU rV rJ rW )r* r r r �column_data_offsets� s z"SAS7BDATReader.column_data_offsetsc C s t j| jt �d�d�S )zj Returns a numpy character array of the column types: s (string) or d (double) ZS1)rT )rU rV rK rT )r* r r r � column_types� s zSAS7BDATReader.column_types�Nonec C s | j �� d S )N)rM rS )r* r r r rS � s zSAS7BDATReader.closec C s | j �d� tt| j �d��| _| jdttj�� tjkrBt d��d\}}| � tj tj �}|tj kr�tj}d| _d| _tj| _tj| _n d| _tj| _tj| _d| _| � tjtj�}|tjkr�tj}|| }| � tjtj�}|d kr�d | _ nd | _ | � tj tj �d }|tj k�r tj | | _ nd |� d �| _ | � tj!tj"�}|dk�rPd| _#n|dk�rbd| _#nd| _#| � tj$tj%�}|�&d�| _'| j(�r�| j'�)| j*�p�| j+�| _'| � tj,tj-�}|�&d�| _.| j(�r�| j.�)| j*�p�| j+�| _.t/ddd�}| �0tj1| tj2�}|t3j4|dd� | _5| �0tj6| tj7�}|t3j4|dd� | _8| �9tj:| tj;�| _<tt| j �| j<d ��}| j|7 _t| j�| j<k�r�t d��| �9tj=| tj>�| _?| �9tj@| tjA�| _B| � tjC| tjD�}|�&d�| _E| j(�r�| jE�)| j*�p�| j+�| _E| � tjF| tjG�}|�&d�| _H| j(�r>| jH�)| j*�p8| j+�| _H| � tjI| tjJ�}|�&d�| _K| j(�r~| jK�)| j*�px| j+�| _K| � tjL| tjM�}|�&d�}t|�dk�r�|�)| j*�p�| j+�| _Nn@| � tjO| tjP�}|�&d�| _N| j(�r| jN�)| j*�p�| j+�| _Nd S )Nr i z'magic number mismatch (not a SAS file?))r r T� F� � �<�>zunknown (code=�)� 1�unix� 2�windows�unknowns i� r r )r z*The SAS7BDAT file appears to be truncated.)QrO �seekr r5 �readr9 �len�const�magicr � _read_bytesZalign_1_offsetZalign_1_lengthZu64_byte_checker_valueZ align_2_value�U64r8 Zpage_bit_offset_x64�_page_bit_offsetZ subheader_pointer_length_x64�_subheader_pointer_lengthZpage_bit_offset_x86Z subheader_pointer_length_x86Zalign_2_offsetZalign_2_lengthZalign_1_checker_valueZendianness_offsetZendianness_length� byte_orderZencoding_offsetZencoding_lengthZencoding_names� file_encodingZplatform_offsetZplatform_length�platformZdataset_offsetZdataset_length�rstripr2 rB �decoder@ rC Zfile_type_offsetZfile_type_length� file_typer � _read_floatZdate_created_offsetZdate_created_lengthr Z to_timedeltaZ date_createdZdate_modified_offsetZdate_modified_lengthZ date_modified� _read_intZheader_size_offsetZheader_size_length� header_lengthZpage_size_offsetZpage_size_length� _page_lengthZpage_count_offsetZpage_count_lengthZ _page_countZsas_release_offsetZsas_release_lengthZ sas_releaseZsas_server_type_offsetZsas_server_type_lengthZ server_typeZos_version_number_offsetZos_version_number_length� os_versionZos_name_offsetZos_name_length�os_nameZos_maker_offsetZos_maker_length)r* Zalign1Zalign2�bufZ total_align�epoch�xr r r rP � s� z SAS7BDATReader._get_propertiesc C s* | j | jp dd�}|d kr&| �� t�|S )Nr )�nrows)rh r? rS � StopIteration)r* �dar r r �__next__g s zSAS7BDATReader.__next__)r& �widthc C sJ |dkr| � � td��| �||�}|dkr0dnd}t�| j| |�d S )N)r] r\ zinvalid float widthr] �fr r )rS r rl �struct�unpackrp )r* r& r� r| �fdr r r rv o s zSAS7BDATReader._read_float)r& r� r c C sP |dkr| � � td��| �||�}ddddd�| }t�| j| |�d }|S )N)r � r] r\ zinvalid int width�b�h�l�qr )rS r rl r� r� rp )r* r& r� r| �itZivr r r rw x s zSAS7BDATReader._read_int)r& r' c C s� | j d krX| j�|� | j�|�}t|�|k rT| �� d|d�d|d�d�}t|��|S || t| j �krz| �� td��| j ||| � S d S )NzUnable to read r z bytes from file position �.z The cached page is too small.)r9 rO rg rh ri rS r )r* r& r' r| �msgr r r rl � s zSAS7BDATReader._read_bytesc C sX d}xN|sRt t| j�| j��| _t| j�dkr0P t| j�| jkrHtd��| �� }qW d S )NFr z2Failed to read a meta data page from the SAS file.) r r5 rO rh ry r9 ri r �_process_page_meta)r* �doner r r rQ � s z SAS7BDATReader._parse_metadata�boolc C sZ | � � tjtjgtj }| j|kr,| �� | jtj@ }| jtjk}t|pV|pV| j g k�S )N) �_read_page_headerrj �page_meta_typeZ page_amd_type�page_mix_types�_current_page_type�_process_page_metadata�page_data_typer� rH )r* �pt� is_data_pageZ is_mix_pager r r r� � s z!SAS7BDATReader._process_page_metac C sX | j }tj| }| �|tj�| _tj| }| �|tj�| _tj | }| �|tj �| _ d S )N) rn rj Zpage_type_offsetrw Zpage_type_lengthr� Zblock_count_offsetZblock_count_lengthZ_current_page_block_countZsubheader_count_offsetZsubheader_count_length� _current_page_subheaders_count)r* � bit_offsetZtxr r r r� � s z SAS7BDATReader._read_page_headerc C st | j }xht| j�D ]Z}| �tj| |�}|jdkr4q|jtjkrBq| � |j �}| � ||j|j �}| � ||� qW d S )Nr )rn �ranger� �_process_subheader_pointersrj Zsubheader_pointers_offsetr' r( Ztruncated_subheader_id�_read_subheader_signaturer& �_get_subheader_indexr) �_process_subheader)r* r� �i�pointer�subheader_signature�subheader_indexr r r r� � s z%SAS7BDATReader._process_page_metadatar5 )� signaturer c C s` t j�|�}|d kr\|t jkp$|dk}|t jk}| jdkrL|rL|rLt jj}n| �� t d��|S )Nr r: zUnknown subheader signature) rj Z subheader_signature_to_index�getZcompressed_subheader_idZcompressed_subheader_typer( �SASIndex�data_subheader_indexrS r )r* r� r( r) r< �f1�f2r r r r� � s z#SAS7BDATReader._get_subheader_indexr$ )r& �subheader_pointer_indexr c C st | j }||| }| �|| j�}|| j7 }| �|| j�}|| j7 }| �|d�}|d7 }| �|d�}t||||�} | S )Nr )ro rw r8 r$ ) r* r& r� Zsubheader_pointer_lengthZ total_offsetZsubheader_offsetZsubheader_lengthZsubheader_compressionZsubheader_typer~ r r r r� � s z*SAS7BDATReader._process_subheader_pointers)r& r c C s | � || j�}|S )N)rl r8 )r* r& r� r r r r� � s z(SAS7BDATReader._read_subheader_signature)r� r� r c C s� |j }|j}|tjjkr | j}n�|tjjkr4| j}n�|tjjkrH| j }n�|tjj kr\| j }nt|tjj krp| j }n`|tjjkr�| j}nL|tjjkr�| j}n8|tjjkr�| j}n$|tjjkr�| j�|� d S td��|||� d S )Nzunknown subheader index)r& r' rj r� Zrow_size_index�_process_rowsize_subheaderZcolumn_size_index� _process_columnsize_subheaderZcolumn_text_index� _process_columntext_subheaderZcolumn_name_index� _process_columnname_subheaderZcolumn_attributes_index�#_process_columnattributes_subheaderZformat_and_label_index�_process_format_subheaderZcolumn_list_index� _process_columnlist_subheaderZsubheader_counts_index�_process_subheader_countsr� rH �appendr )r* r� r� r& r' � processorr r r r� � s. z!SAS7BDATReader._process_subheader)r& r' r c C s� | j }|}|}| jr&|d7 }|d7 }n|d7 }|d7 }| �|tj| |�| _| �|tj| |�| _| �|tj| |�| _ | �|tj | |�| _ tj | }| �|| |�| _ | �|d�| _| �|d�| _d S )Ni� i� ib iz r� )r8 rm rw rj Z row_length_offset_multiplierZ row_lengthZrow_count_offset_multiplier� row_countZcol_count_p1_multiplier� col_count_p1Zcol_count_p2_multiplier� col_count_p2Z'row_count_on_mix_page_offset_multiplierZ_mix_page_row_count�_lcs�_lcp)r* r& r' �int_lenZ lcs_offsetZ lcp_offsetZmxr r r r� s( z)SAS7BDATReader._process_rowsize_subheaderc C sT | j }||7 }| �||�| _| j| j | jkrPtd| j� d| j� d| j� d�� d S )Nz Warning: column count mismatch (z + z != z) )r8 rw � column_countr� r� �print)r* r& r' r� r r r r� ' s z,SAS7BDATReader._process_columnsize_subheaderc C s d S )Nr )r* r& r' r r r r� 2 s z(SAS7BDATReader._process_subheader_countsc C s� || j 7 }| �|tj�}| �||�}|d|� �d�}|}| jrR|�| jpN| j �}| j � |� t | j �dk�r�d}xtj D ]}||krz|}qzW || _|| j 8 }|d } | jr�| d7 } | �| | j�}|�d�}|dk�rd| _|d } | jr�| d7 } | �| | j�}|d| j� | _n�|tjk�rV|d } | j�r6| d7 } | �| | j�}|d| j� | _nH| jdk�r�d| _|d } | j�r�| d7 } | �| | j�}|d| j� | _| j�r�t| d ��r�| j�| j�p�| j �| _d S ) Nr s r r: � r] � � �( � creator_proc)r8 rw rj Ztext_block_size_lengthrl rs rB rt r@ rC rD r� ri Zcompression_literalsr( rm r� r� r� Zrle_compression�hasattr) r* r& r' Ztext_block_sizer| Z cname_raw�cnameZcompression_literal�clZoffset1r r r r� 5 sX z,SAS7BDATReader._process_columntext_subheaderc C s� | j }||7 }|d| d d }x�t|�D ]�}|tj|d tj }|tj|d tj }|tj|d tj }| �|tj�} | �|tj �} | �|tj �} | j | } | j � | | | | � � q,W d S )Nr� � r\ r )r8 r� rj Zcolumn_name_pointer_lengthZ!column_name_text_subheader_offsetZcolumn_name_offset_offsetZcolumn_name_length_offsetrw Z!column_name_text_subheader_lengthZcolumn_name_offset_lengthZcolumn_name_length_lengthrD rE r� ) r* r& r' r� Zcolumn_name_pointers_countr� Ztext_subheaderZcol_name_offsetZcol_name_length�idx� col_offsetZcol_lenZname_strr r r r� i s z,SAS7BDATReader._process_columnname_subheaderc C s� | j }|d| d |d }x�t|�D ]�}|| tj ||d }|d| tj ||d }|d| tj ||d }| �||�} | j�| � | �|tj �} | j �| � | �|tj �} | j �| dkr�dnd� q(W d S )Nr� r� r\ r � d� s) r8 r� rj Zcolumn_data_offset_offsetZcolumn_data_length_offsetZcolumn_type_offsetrw rJ r� Zcolumn_data_length_lengthrI Zcolumn_type_lengthrK ) r* r& r' r� Z column_attributes_vectors_countr� Zcol_data_offsetZ col_data_lenZ col_typesr~ r r r r� � s z2SAS7BDATReader._process_columnattributes_subheaderc C s d S )Nr )r* r& r' r r r r� � s z,SAS7BDATReader._process_columnlist_subheaderc C sl | j }|tj d| }|tj d| }|tj d| }|tj d| }|tj d| }|tj d| } | �|tj �} t | t | j �d �} | �|tj �} | �|tj�} | �|tj�}t |t | j �d �}| �|tj�}| �| tj�}| j | }|||| � }| j | }|| | | � }t | j�}t|| j| ||| j| | j| �}| j�|� | j�|� d S )N� r )r8 rj Z)column_format_text_subheader_index_offsetZcolumn_format_offset_offsetZcolumn_format_length_offsetZ(column_label_text_subheader_index_offsetZcolumn_label_offset_offsetZcolumn_label_length_offsetrw Z)column_format_text_subheader_index_length�minri rD Zcolumn_format_offset_lengthZcolumn_format_length_lengthZ(column_label_text_subheader_index_lengthZcolumn_label_offset_lengthZcolumn_label_length_lengthrG r0 rE rK rI rF r� )r* r& r' r� Ztext_subheader_formatZcol_format_offsetZcol_format_lenZtext_subheader_labelZcol_label_offsetZ col_label_lenr~ Z format_idxZ format_startZ format_lenZ label_idxZ label_startZ label_lenZ label_namesZ column_labelZ format_namesZ column_formatZcurrent_column_number�colr r r r� � s@ z(SAS7BDATReader._process_format_subheaderz int | NonezDataFrame | None)r r c C s� |d kr| j d k r| j }n|d kr(| j}t| j�dkrF| �� td��| j| jkrVd S | j| j }||krn|}| j�d�}| j�d�}tj ||ft d�| _ tj |d| ftj d�| _d| _t| �}|�|� | �� }| jd k r�|�| j�}|S )Nr z No columns to parse from filer� r� )rT r\ )r? r� ri rK rS r rL �countrU �empty�object� _string_chunkZzerosZuint8� _byte_chunk�_current_row_in_chunk_indexr rh �_chunk_to_dataframer< Z set_index)r* r �m�nd�ns�p�rsltr r r rh � s. zSAS7BDATReader.readc C s� g | _ tt| j�| j��| _t| j�dkr.dS t| j�| jkrl| �� dt| j�d�d| jd�d�}t |��| � � | j }|t j kr�| �� |t j@ }t j gt j }|s�| j |kr�| �� S dS )Nr Tz-failed to read complete page from file (read r z of z bytes)F)rH r r5 rO rh ry r9 ri rS r r� r� rj r� r� r� r� �_read_next_page)r* r� Z page_typer� r� r r r r� � s" z SAS7BDATReader._read_next_pager c C s� | j }| j}t|| |�}t|d�}d\}}�xft| j�D �]V}| j| }| j| dkr�| j|d d �f j| j d d�||< t j || t j d�||< | j r�| j| tjkr�t|| d�||< n"| j| tjkr�t|| d�||< |d7 }q:| j| dk�rv| j|d d �f ||< | j�rB| jd k �rB|| j�| j�p:| j�||< | j�rl|| j�� d k} t j|j | |f< |d7 }q:| � � t d | j| � ���q:W |S ) N)r< )r r r� r )rT r r r� r zunknown column type ) r� rL r� r r� rE rK r� �viewrp rU rV Zfloat64r= rF rj Zsas_date_formatsr# Zsas_datetime_formatsr� rA r@ r rt rC r> ri �nan�locrS r ) r* �nr� Zixr� �jsZjb�jr2 �iir r r r� s8 $ z"SAS7BDATReader._chunk_to_dataframe)NTTNNTT)N)"r, r- r. �__doc__r/ r+ rX rY rZ rS rP r� rv rw rl rQ r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� rh r� r� r r r r r7 � sN ' 4 1"r7 )&r� � __future__r � collectionsr r r r� �typingr r r �numpyrU Z pandas.errorsr r Zpandasr r r Zpandas.io.commonr Zpandas.io.sas._sasr Zpandas.io.sas.sas_constants�io�sasZ sas_constantsrj Zpandas.io.sas.sasreaderr r r# r$ r0 �Iteratorr7 r r r r �<module> s$
| ver. 1.4 |
Github
|
.
| PHP 8.3.23 | Генерация страницы: 0.05 |
proxy
|
phpinfo
|
Настройка