Файловый менеджер - Редактировать - /home/digitalm/venv/lib/python3.7/site-packages/pandas/io/__pycache__/stata.cpython-37.pyc
Назад
B �5�g�� � @ s� d Z ddlmZ ddlmZ ddlZddlmZ ddlZddl Z ddl Z ddl m Z m Z mZmZmZ ddlZddlmZ ddlZddlmZ dd lmZ dd lmZm Z m Z m Z dd l m Z m!Z! dd l"m#Z#m$Z$m%Z% dd l&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z. ddl/m0Z0 ddl1m2Z2 ddl3m4Z4 ddl5m6Z6 ddl7m8Z8 dZ9dZ:dZ;dZ<de0j=d � �Z>dZ?dZ@de:� d e;� d e<� d e?� d e>� d e@� d � ZAd e:� d e;� d �ZBd!e:� d e;� d e<� d e>� d"e@� d � ZCd#d$d%d&d'd(d)d*d+g ZDe�d,d-d-�ZEd.d/�d0d1�ZFd.d2d.d3�d4d5�ZGd6ZHG d7d8� d8eI�ZJd9ZKG d:d;� d;eI�ZLd<ZMG d=d>� d>eI�ZNd?ZOG d@dA� dAeI�ZPdBZQdCdCdD�dEdF�ZRG dGdH� dH�ZSG dIdJ� dJ�ZTG dKdL� dL�ZUG dMdN� dNeUejV�ZWe eA�d�dRdSdSdTdSdSdUdSdVdSdWdXdYdZ� d[d\��ZXd2d2d]�d^d_�ZYd`dad`db�dcdd�ZZd2dedf�dgdh�Z[didjdidk�dldm�Z\ded.dadn�dodp�Z]d�d.dadSd2dr�dsdt�Z^e!e0j=d du�G dvdw� dweU��Z_ded.dSdadx�dydz�Z`d{dad|db�d}d~�ZaG dd�� d��ZbG d�d�� d�e_�ZcG d�d�� d�ec�ZddS )�a� Module contains tools for processing Stata files into DataFrames The StataReader below was originally written by Joe Presbrey as part of PyDTA. It has been extended and improved by Skipper Seabold from the Statsmodels project who also developed the StataWriter and was finally added to pandas in a once again improved version. You can find more information on http://presbrey.mit.edu/PyDTA and https://www.statsmodels.org/devel/ � )� annotations)�abcN)�BytesIO)�Any�AnyStr�Hashable�Sequence�cast)� relativedelta)� infer_dtype)�max_len_string_array)�Buffer�CompressionOptions�FilePathOrBuffer�StorageOptions)�Appender�doc)� ensure_object�is_categorical_dtype�is_datetime64_dtype)� Categorical� DatetimeIndex�NaT� Timestamp�concat�isna� to_datetime� to_timedelta)�generic)� DataFrame)�Index)�Series)� get_handlez�Version of given Stata file is {version}. pandas supports importing versions 105, 108, 111 (Stata 7SE), 113 (Stata 8/9), 114 (Stata 10/11), 115 (Stata 12), 117 (Stata 13), 118 (Stata 14/15/16),and 119 (Stata 15/16, over 32,767 variables).z�convert_dates : bool, default True Convert date variables to DataFrame time values. convert_categoricals : bool, default True Read value labels and convert columns to Categorical/Factor variables.a index_col : str, optional Column to set as index. convert_missing : bool, default False Flag indicating whether to convert missing values to their Stata representations. If False, missing values are replaced with nan. If True, columns containing missing values are returned with object data types and missing values are represented by StataMissingValue objects. preserve_dtypes : bool, default True Preserve Stata datatypes. If False, numeric data are upcast to pandas default types for foreign data (float64 or int64). columns : list or None Columns to retain. Columns will be returned in the given order. None returns all columns. order_categoricals : bool, default True Flag indicating whether converted categorical data are ordered.zzchunksize : int, default None Return StataReader object for iterations, returns chunks with given number of lines.aH compression : str or dict, default None If string, specifies compression mode. If dict, value at key 'method' specifies compression mode. Compression mode must be one of {'infer', 'gzip', 'bz2', 'zip', 'xz', None}. If compression mode is 'infer' and `filepath_or_buffer` is path-like, then detect compression from the following extensions: '.gz', '.bz2', '.zip', or '.xz' (otherwise no compression). If dict and compression mode is one of {'zip', 'gzip', 'bz2'}, or inferred as one of the above, other entries passed as additional compression options. �storage_optionsz=iterator : bool, default False Return StataReader object.z�Notes ----- Categorical variables read through an iterator may not have the same categories and dtype. This occurs when a variable stored in a DTA file is associated to an incomplete set of value labels that only label a strict subset of the values.a> Read Stata file into DataFrame. Parameters ---------- filepath_or_buffer : str, path object or file-like object Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. A local file could be: ``file://localhost/path/to/table.dta``. If you want to pass in a path object, pandas accepts any ``os.PathLike``. By file-like object, we refer to objects with a ``read()`` method, such as a file handle (e.g. via builtin ``open`` function) or ``StringIO``. � z� Returns ------- DataFrame or StataReader See Also -------- io.stata.StataReader : Low-level reader for Stata data files. DataFrame.to_stata: Export Stata data files. z� Examples -------- Read a Stata dta file: >>> df = pd.read_stata('filename.dta') Read a Stata dta file in 10,000 line chunks: >>> itr = pd.read_stata('filename.dta', chunksize=10000) >>> for chunk in itr: ... do_something(chunk) z�Reads observations from Stata file, converting them into a dataframe Parameters ---------- nrows : int Number of lines to read from data file, if None read whole file. z Returns ------- DataFrame z�Class for reading Stata dta files. Parameters ---------- path_or_buf : path (string), buffer or path object string, path object (pathlib.Path or py._path.local.LocalPath) or object implementing a binary read() functions. z z%tcz%tCz%tdz%dz%twz%tmz%tqz%thz%tyi� � r! )�returnc sf t jjt jj ��t jt�ddd� j� t jt�ddd� j��d d d �� d d d �dd���fdd � }dd���fd d � }dd�� ���fd d � }t�| �}d}|�� r�d}t | �}d||< | � tj �} |� d�r�t }| } ||| d�} �n^|� d��r*t�d� t | td�} |�r&t| |< | S |� d��rNt }| } ||| d�} �n|� d��r�t j| d } | d d } || | �} n�|� d��r�t j| d } | d d } || | �} n�|� d ��r�t j| d } | d d d }|| |�} nl|� d ��r t j| d! } | d! d" d } || | �} n6|� d#��rD| } t�| �}|| |�} ntd$|� d%���|�rbt| |< | S )&a Convert from SIF to datetime. https://www.stata.com/help.cgi?datetime Parameters ---------- dates : Series The Stata Internal Format date to convert to datetime according to fmt fmt : str The format to convert to. Can be, tc, td, tw, tm, tq, th, ty Returns Returns ------- converted : Series The converted dates Examples -------- >>> dates = pd.Series([52]) >>> _stata_elapsed_date_to_datetime_vec(dates , "%tw") 0 1961-01-01 dtype: datetime64[ns] Notes ----- datetime/c - tc milliseconds since 01jan1960 00:00:00.000, assuming 86,400 s/day datetime/C - tC - NOT IMPLEMENTED milliseconds since 01jan1960 00:00:00.000, adjusted for leap seconds date - td days since 01jan1960 (01jan1960 = 0) weekly date - tw weeks since 1960w1 This assumes 52 weeks in a year, then adds 7 * remainder of the weeks. The datetime value is the start of the week in terms of days in the year, not ISO calendar weeks. monthly date - tm months since 1960m1 quarterly date - tq quarters since 1960q1 half-yearly date - th half-years since 1960h1 yearly date - ty years since 0000 i� r% � i i� r! )r& c sX | � � � k r,| �� �kr,td| | dd�S t| dd�}tdd� t| |�D �|d�S dS ) z� Convert year and month to datetimes, using pandas vectorized versions when the date range falls within the range supported by pandas. Otherwise it falls back to a slower but more robust method using datetime. �d z%Y%m)�format�indexNc S s g | ]\}}t � ||d ��qS )r% )�datetime)�.0�y�m� r/ �E/home/digitalm-up/venv/lib/python3.7/site-packages/pandas/io/stata.py� <listcomp> s zX_stata_elapsed_date_to_datetime_vec.<locals>.convert_year_month_safe.<locals>.<listcomp>)r* )�max�minr �getattrr! �zip)�year�monthr* )�MAX_YEAR�MIN_YEARr/ r0 �convert_year_month_safe s zD_stata_elapsed_date_to_datetime_vec.<locals>.convert_year_month_safec sd | � � � d k r4| �� �kr4t| dd�t|dd� S t| dd�}dd � t| |�D �}t||d �S dS ) z{ Converts year (e.g. 1999) and days since the start of the year to a datetime or datetime64 Series r% z%Y)r) �d)�unitr* Nc S s, g | ]$\}}t � |d d �tt|�d� �qS )r% )�days)r+ r �int)r, r- r; r/ r/ r0 r1 $ s zW_stata_elapsed_date_to_datetime_vec.<locals>.convert_year_days_safe.<locals>.<listcomp>)r* )r2 r3 r r r4 r5 r! )r6 r= r* �value)r8 r9 r/ r0 �convert_year_days_safe s zC_stata_elapsed_date_to_datetime_vec.<locals>.convert_year_days_safec s� t |dd�}|dkrL|�� �ks,|�� �k r�� fdd�|D �}t||d�S nH|dkr�|�� �ksl|�� �k r�� fdd�|D �}t||d�S ntd ��t� �� t||d �}� | S ) z� Convert base dates and deltas to datetimes, using pandas vectorized versions if the deltas satisfy restrictions required to be expressed as dates in pandas. r* Nr; c s g | ]}� t t|�d � �qS ))r= )r r> )r, r; )�baser/ r0 r1 2 s zS_stata_elapsed_date_to_datetime_vec.<locals>.convert_delta_safe.<locals>.<listcomp>)r* �msc s" g | ]}� t t|�d d� �qS )i� )� microseconds)r r> )r, r; )rA r/ r0 r1 7 s zformat not understood)r< )r4 r2 r3 r! � ValueErrorr r )rA Zdeltasr< r* �values)� MAX_DAY_DELTA� MAX_MS_DELTA� MIN_DAY_DELTA� MIN_MS_DELTA)rA r0 �convert_delta_safe) s z?_stata_elapsed_date_to_datetime_vec.<locals>.convert_delta_safeFTg �?)z%tc�tcrB )z%tC�tCz9Encountered %tC format. Leaving in Stata Internal Format.)�dtype)z%td�tdz%dr; r; )z%tw�tw�4 � )z%tm�tm� )z%tq�tq� � )z%th�th� � )z%ty�tyz Date fmt z not understood)r r3 r6 r2 r+ r= �np�isnan�anyr! �astype�int64� startswith� stata_epoch�warnings�warn�objectr Z ones_likerD )�dates�fmtr: r@ rJ Zbad_locsZhas_bad_valuesZdata_colrA rB � conv_datesr= r6 r7 Z quarter_monthZ first_monthr/ )rF rG r8 rH rI r9 r0 �#_stata_elapsed_date_to_datetime_vec� sj . rh �str)re rf r& c s� | j �d� � d �d"� ��fdd� }t| �}| j �|�� r`t| �} t| �rXtt�| |< nt| |< |dkr�|| dd�}|jd }�n<|d kr�t� d � | }�n"|d kr�|| dd�}|j� }�n |d kr�|| ddd �}d|j tj |j d }n�|dk�r"|| dd�}d|j tj |j d }n�|dk�rX|| dd�}d|j tj |j d d }nf|dk�r�|| dd�}d|j tj |j dk� t� }n.|dk�r�|| dd�}|j }ntd|� d ���t|tjd �}t�d d �d }|||< t|�d!�S )#aX Convert from datetime to SIF. https://www.stata.com/help.cgi?datetime Parameters ---------- dates : Series Series or array containing datetime.datetime or datetime64[ns] to convert to the Stata Internal Format given by fmt fmt : str The format to convert to. Can be, tc, td, tw, tm, tq, th, ty l �"R: i� Fc sR i }t | j�r�|r0| t }|j�tj�d |d< |s8|rXt| �}|jj |d< |jj |d< |r�| �tj�t |d dd��tj� }|� |d< n�t | dd �d k�r>|r�| jt }d d d ��fdd� }t� |�} | |�|d< |�r| �dd� �} | jd |d< | j|d d |d< |�rFddd �dd�} t� | �} | | �|d< ntd��t|�d�S )Ni� �deltar6 r7 z%Y)r) r= F)�skipnar+ zdatetime.timedelta�float)�xr& c s � | j d| j | j S )Ni@B )r= �secondsrC )rm )� US_PER_DAYr/ r0 �f� s zC_datetime_to_stata_elapsed_vec.<locals>.parse_dates_safe.<locals>.fc S s d| j | j S )Nr( )r6 r7 )rm r/ r/ r0 �<lambda>� � zJ_datetime_to_stata_elapsed_vec.<locals>.parse_dates_safe.<locals>.<lambda>r( zdatetime.datetimer> c S s | t � | jdd� jS )Nr% )r+ r6 r= )rm r/ r/ r0 �g� s zC_datetime_to_stata_elapsed_vec.<locals>.parse_dates_safe.<locals>.gzZColumns containing dates must contain either datetime64, datetime.datetime or null values.)r* )r rM ra �_values�viewr[ r_ r �_datar6 r7 r r Z vectorize�applyrD r ) re rj r6 r= r; Z time_deltaZ date_indexZ days_in_nsrp �vZ year_monthrs )� NS_PER_DAYro r* r/ r0 �parse_dates_safe� s> z8_datetime_to_stata_elapsed_vec.<locals>.parse_dates_safe)z%tcrK T)rj )z%tCrL z'Stata Internal Format tC not supported.)z%tdrN )z%twrO )r6 r= rP rQ )z%tmrR )r6 rS r% )z%tqrT rU rV )z%thrW rX rY )z%tyrZ zFormat z! is not a known Stata date format)rM z<ds �r )r* )FFF)r* r r] r! r r ra rj rb rc r6 r= r7 r^ r> rD r[ �float64�struct�unpack)re rf rz Zbad_locr; rg � missing_valuer/ )ry ro r* r0 � _datetime_to_stata_elapsed_vecy sP , " r z� Fixed width strings in Stata .dta files are limited to 244 (or fewer) characters. Column '{0}' does not satisfy this restriction. Use the 'version=117' parameter to write the newer (Stata 13 and later) format. c @ s e Zd ZdS )�PossiblePrecisionLossN)�__name__� __module__� __qualname__r/ r/ r/ r0 r� � s r� z� Column converted from {0} to {1}, and some data are outside of the lossless conversion range. This may result in a loss of precision in the saved data. c @ s e Zd ZdS )�ValueLabelTypeMismatchN)r� r� r� r/ r/ r/ r0 r� � s r� z� Stata value labels (pandas categories) must be strings. Column {0} contains non-string labels which will be converted to strings. Please check that the Stata data file created has not lost information due to duplicate labels. c @ s e Zd ZdS )�InvalidColumnNameN)r� r� r� r/ r/ r/ r0 r� � s r� a; Not all pandas column names were valid Stata variable names. The following replacements have been made: {0} If this is not what you expect, please make sure you have Stata-compliant column names in your DataFrame (strings only, max 32 characters, only alphanumerics and underscores, no Stata reserved words) c @ s e Zd ZdS )� CategoricalConversionWarningN)r� r� r� r/ r/ r/ r0 r� s r� a� One or more series with value labels are not fully labeled. Reading this dataset with an iterator results in categorical variable with different categories. This occurs since it is not possible to know all possible values until the entire dataset has been read. To avoid this warning, you can either read dataset without an iterator, or manually convert categorical data by ``convert_categoricals`` to False and then accessing the variable labels through the value_labels method of the reader. r )�datar& c C s� d}t jt jt jft jt jt jft jt jt jft jt jt jff}t � dd�d }t � dd�d }�xF| D �]<}| | j }x�|D ]z}||d kr~| | � � t � |d �j kr�|d }n|d }|d t jkr�| | � � d kr�t�d d �}| | �|�| |< q~W |t jk�rB| | � � d k�s,| | �� d k �r�| | �t j�| |< qh|t jk�r�| | � � dk�sr| | �� dk �r�| | �t j�| |< qh|t jk�r| | � � dk�r�| | �� dk�r�| | �t j�| |< nD| | �t j�| |< | | � � d k�s| | �� dk�r�t�dd �}qh|t jt jfkrh| | � � }t �|��rLtd|� d���|t jk�rx||k�rx| | �t j�| |< qh|t jkrh||krhtd|� d|� d|� d���qhW |�r�t�|t� | S )a- Checks the dtypes of the columns of a pandas DataFrame for compatibility with the data types and ranges supported by Stata, and converts if necessary. Parameters ---------- data : DataFrame The DataFrame to check and convert Notes ----- Numeric columns in Stata must be one of int8, int16, int32, float32 or float64, with some additional value restrictions. int8 and int16 columns are checked for violations of the value restrictions and upcast if needed. int64 data is not usable in Stata, and so it is downcast to int32 whenever the value are in the int32 range, and sidecast to float64 when larger than this range. If the int64 values are outside of the range of those perfectly representable as float64 values, a warning is raised. bool columns are cast to int8. uint columns are converted to int of the same size if there is no loss in precision, otherwise are upcast to a larger type. uint64 is currently not supported since it is concerted to object in a DataFrame. � z<fs ���~r z<ds �������r% rX l �uint64r{ r( i����i� i���i�i �l���� r_ zColumn zO has a maximum value of infinity which is outside the range supported by Stata.z has a maximum value (z() outside the range supported by Stata (�))r[ Zbool_�int8�uint8�int16Zuint16�int32Zuint32r_ r| r} rM r2 Ziinfo�precision_loss_docr) r^ r3 r{ �float32�isinfrD rb rc r� ) r� �wsZconversion_data� float32_max� float64_max�colrM Zc_datar? r/ r/ r0 �_cast_to_stata_types sX $ $ $$ r� c @ s2 e Zd ZdZd ddd�dd�Zddd �d d �Zd S )�StataValueLabelz� Parse a categorical column and prepare formatted output Parameters ---------- catarray : Series Categorical Series to encode encoding : {"latin-1", "utf-8"} Encoding to use for value labels. �latin-1r! ri )�catarray�encodingc C sZ |dkrt d��|j| _|| _|jj}ttt� t |��|��| _ | j j dd� d� d| _ g | _d| _g }g }x�| j D ]�}|d }t|t�s�t|�}t�t�|j�t� |�|�}|�| j � | j t |�d 7 _ |�|d � | j�|� | jd7 _qtW | j dk�rt d ��tj|tjd �| _tj|tjd �| _ d d | j d | j | j | _ d S ) N)zlatin-1zutf-8z%Only latin-1 and utf-8 are supported.c S s | d S )Nr r/ )rm r/ r/ r0 rq � rr z*StataValueLabel.__init__.<locals>.<lambda>)�keyr r% i } zaStata value labels for a single variable must have a combined length less than 32,000 characters.)rM � rU ) rD �name�labname� _encoding�cat� categories�listr5 r[ �arange�len� value_labels�sort�text_len�txt�n� isinstanceri rb rc �value_label_mismatch_docr) r� �encode�append�arrayr� �off�val)�selfr� r� r� �offsetsrE �vl�categoryr/ r/ r0 �__init__� s> zStataValueLabel.__init__�bytes)� byteorderr& c C s0 | j }t� }d}|�t�|d | j�� t| j�dd� �|�}|dkrLdnd}t ||d �}|�|� x"t d�D ]}|�t�d |�� qrW |�t�|d | j �� |�t�|d | j �� x$| j D ]}|�t�|d |�� q�W x$| jD ]} |�t�|d | �� q�W x | jD ]} |�| | � �qW |�� S ) a! Generate the binary representation of the value labels. Parameters ---------- byteorder : str Byte order of the output Returns ------- value_label : bytes Bytes containing the formatted value label � �iN� )zutf-8�utf8� r% rV �c)r� r �writer| �packr� ri r� r� � _pad_bytes�ranger� r� r� r� r� �getvalue) r� r� r� �bio� null_byter� Zlab_lenr� �offsetr? �textr/ r/ r0 �generate_value_label� s& z$StataValueLabel.generate_value_labelN)r� )r� r� r� �__doc__r� r� r/ r/ r/ r0 r� t s ,r� c @ s e Zd ZU dZi Zded< dZx@eD ]8Zdee< x*edd�D ] Z de de � ee e < q<W q$W d Z e � d d �d Zxped�D ]dZ e � d e �d Zdee< e d kr�ee e de �7 < e � d e �d e��d e Ze �d e�Z q~W dZe � dd�d Zxted�D ]hZ e � de�d Zdee< e d k�rBee e de �7 < e � de �de��d e Ze �de�Z�qW ddde � d e �d e � de�d d�Zdd�dd�Zedd�d d ��Zedd�d d ��Zdd�d d!�Zdd�d"d#�Zd$d%d&�d'd(�Zed)dd*�d+d,��Z d-S ).�StataMissingValuea� An observation's missing value. Parameters ---------- value : {int, float} The Stata missing value code Notes ----- More information: <https://www.stata.com/help.cgi?missing> Integer missing values make the code '.', '.a', ..., '.z' to the ranges 101 ... 127 (for int8), 32741 ... 32767 (for int16) and 2147483621 ... 2147483647 (for int32). Missing values for floating point data types are more complex but the pattern is simple to discern from the following table. np.float32 missing values (float in Stata) 0000007f . 0008007f .a 0010007f .b ... 00c0007f .x 00c8007f .y 00d0007f .z np.float64 missing values (double in Stata) 000000000000e07f . 000000000001e07f .a 000000000002e07f .b ... 000000000018e07f .x 000000000019e07f .y 00000000001ae07f .z zdict[float, str]�MISSING_VALUES)�e i� i��.r% � �` s z<is r z<fs ��qs z<dr� i� i�)r� r� r� r� r{ z int | float)r? c C s. || _ |dk rt|�nt|�}| j| | _d S )Nl )�_valuer> rl r� �_str)r� r? r/ r/ r0 r� + s zStataMissingValue.__init__ri )r& c C s | j S )z� The Stata representation of the missing value: '.', '.a'..'.z' Returns ------- str The representation of the missing value. )r� )r� r/ r/ r0 �string1 s zStataMissingValue.stringc C s | j S )z� The binary representation of the missing value. Returns ------- {int, float} The binary representation of the missing value. )r� )r� r/ r/ r0 r? = s zStataMissingValue.valuec C s | j S )N)r� )r� r/ r/ r0 �__str__I s zStataMissingValue.__str__c C s t | �� d| � d�S )N�(r� )�type)r� r/ r/ r0 �__repr__L s zStataMissingValue.__repr__r �bool)�otherr& c C s& t |t| ��o$| j|jko$| j|jkS )N)r� r� r� r? )r� r� r/ r/ r0 �__eq__O s zStataMissingValue.__eq__znp.dtype)rM r& c C sz |t jkr| jd }n`|t jkr,| jd }nJ|t jkrB| jd }n4|t jkrX| jd }n |t jkrn| jd }ntd��|S )Nr� r� r� r� r{ zUnsupported dtype)r[ r� �BASE_MISSING_VALUESr� r� r� r{ rD )�clsrM r? r/ r/ r0 �get_base_missing_valueV s z(StataMissingValue.get_base_missing_valueN) r� r� r� r� r� �__annotations__�bases�br� r� �chrZ float32_baser| r} � incrementr� r� Z int_valueZ float64_baser� r� �propertyr� r? r� r� r� � classmethodr� r/ r/ r/ r0 r� � sJ # r� c @ s e Zd Zdd� ZdS )� StataParserc C s� t tttdd�dd� tdd�D ���dt�tj�fdt�tj�fdt�tj�fdt�tj �fd t�tj �fg �| _ t�tj �t�tj �t�tj �t�tj�t�tj�t�tj�d �| _ ttd��td � | _d d ddddd �| _d}d}d}d}dddt� t�d|�d �t� t�d|�d �ft� t�d|�d �t� t�d|�d �fd �| _ddddd d �| _d d d t� t�dd!�d �t� t�dd"�d �d �| _d#d$d%d&d'd(d)�| _d*| _d S )+Nr% � c S s g | ]}t �d t|� ��qS )�a)r[ rM ri )r, r� r/ r/ r0 r1 � s z(StataParser.__init__.<locals>.<listcomp>� � � � � )i � i�� i�� i�� i�� i�� Zbhlfd�Qr; rp �l�hr� s ����s ���~s �������s �������)i����r( )i���i� )i �i�z<fr z<d)r� r� r� rp r; )�b �i �l �f r( r� i� i�s s ��i1�i2�i4Zf4Zf8�u8)r� r� r� rp r; r� )<Z aggregater� �boolean�break�byte�caseZcatch�classZ colvector�complex�const�continue�defaultZdelegate�delete�do�double�elseZ eltypedef�end�enum�explicitZexportZexternalrl �forZfriend�function�globalZgoto�if�inliner> �local�longZNULLZpragmaZ protected�quadZ rowvector�shortZtypedef�typenameZvirtualZ_allZ_NZ_skip�_b�_pizstr#�inZ_predZstrLZ_coefZ_rc�usingZ_consZ_se�withZ_n)�dictr� r5 r� r[ rM r� r� r� r� r{ � DTYPE_MAPr� � DTYPE_MAP_XML�TYPE_MAP� TYPE_MAP_XMLr| r} � VALID_RANGE�OLD_TYPE_MAPPINGr� �NUMPY_TYPE_MAP�RESERVED_WORDS)r� Z float32_minr� Z float64_minr� r/ r/ r0 r� r sb " zStataParser.__init__N)r� r� r� r� r/ r/ r/ r0 r� q s r� c s* e Zd ZeZdeddddddddd d d d � � fd d� Zd d�dd�Zdd�dd�Zdd�dd�Zdd�dd�Z dd�dd�Z dd�dd �Z d d d �d d!�Z d"d�d#d$�Z d"d�d%d&�Zd"d�d'd(�Zd"d�d)d*�Zd d�d+d,�Zd-d�d.d/�Zd-d�d0d1�Zd d�d2d3�Zd4dd5�d6d7�Zd8d�d9d:�Zd;d d<�d=d>�Zd4d-d?�d@dA�Zdd�dBdC�Zdd�dDdE�ZdFd�dGdH�Zdfd dFdI�dJdK�Z e e �dgd dLdLddLdLddLdFdM� dNdO��Z dFddFdP�dQdR�Z dFdFdS�dTdU�Z!dFdVdFdW�dXdY�Z"dFdZdVddFd[�d\d]�Z#e$d-d�d^d_��Z%d`d�dadb�Z&dZd�dcdd�Z'� Z(S )h� StataReaderTNF�inferr r� z str | NonezSequence[str] | Nonez int | Noner r ) � path_or_buf� convert_dates�convert_categoricals� index_col�convert_missing�preserve_dtypes�columns�order_categoricals� chunksize� compressionr# c s� t � �� g | _|| _|| _|| _|| _|| _|| _|| _ d| _ | | _ d| _ | j d kr^d| _ nt | t�rp| dkrxtd��d| _d| _d| _d| _d| _d| _d | _d| _ttj�| _t |d| d| d��} t | j � � �| _ W d Q R X | �!� | �"� d S )Nr� Fr% r z.chunksize must be a positive integer when set.�rb)r# �is_textr* )#�superr� � col_sizes�_convert_dates�_convert_categoricals� _index_col�_convert_missing�_preserve_dtypes�_columns�_order_categoricalsr� � _chunksize�_using_iteratorr� r> rD Z_has_string_dataZ_missing_values�_can_read_value_labels�_column_selector_set�_value_labels_read� _data_read�_dtype� _lines_read�_set_endianness�sysr� �_native_byteorderr" r �handle�readr! � _read_header� _setup_dtype) r� r! r" r# r$ r% r& r' r( r) r* r# �handles)� __class__r/ r0 r� sF zStataReader.__init__)r& c C s | S )zenter context managerr/ )r� r/ r/ r0 � __enter__S s zStataReader.__enter__�Nonec C s | � � dS )zexit context managerN)�close)r� �exc_type� exc_value� tracebackr/ r/ r0 �__exit__W s zStataReader.__exit__c C s | j �� dS )z close the handle if its openN)r! rI )r� r/ r/ r0 rI [ s zStataReader.closec C s | j dk rd| _nd| _dS )zC Set string encoding which depends on file version �v zlatin-1zutf-8N)�format_versionr� )r� r/ r/ r0 � _set_encoding_ s zStataReader._set_encodingc sh � j �d�}t�d|�d dkr*� �� n � �|� tdd� � jD ��dk� _� fdd�� jD �� _ d S )Nr% r� r � <c S s g | ]}t |�tkr|�qS r/ )r� r> )r, rm r/ r/ r0 r1 o s z,StataReader._read_header.<locals>.<listcomp>c s g | ]}� � |��qS r/ )� _calcsize)r, �typ)r� r/ r0 r1 r s ) r! rB r| r} �_read_new_header�_read_old_headerr� �typlistZhas_string_datar. )r� � first_charr/ )r� r0 rC h s zStataReader._read_headerc C s | j �d� t| j �d��| _| jdkr:ttj| jd���| �� | j �d� | j �d�dkrbdpdd| _| j �d � | jd kr�d nd }| jd kr�d nd}t � | j| | j �|��d | _ | j �d� | � � | _ | j �d� | �� | _| j �d� | �� | _| j �d� | j �d� | j �d� t � | jd | j �d��d d | _t � | jd | j �d��d d | _t � | jd | j �d��d d | _t � | jd | j �d��d d | _t � | jd | j �d��d d | _| �� | _| j �d� t � | jd | j �d��d d | _t � | jd | j �d��d d | _t � | jd | j �d��d d | _| � | j�\| _ | _ | j � | j� | � � | _!| j � | j� t � | jd| j d | j �d | j d ��d d � | _"| j � | j� | �#� | _$| j � | j� | �%� | _&| j � | j� | �'� | _(d S ) Nr� rV )�u rN �w )�version� s MSF�>�<� rN �H�IrX rU r rQ � � � r� r� � � � rY � r� r% ���))r! rB r> rO rD �_version_errorr) rP r� r| r} �nvar� _get_nobs�nobs�_get_data_label� _data_label�_get_time_stamp� time_stampZ_seek_vartypesZ_seek_varnamesZ_seek_sortlistZ _seek_formats�_seek_value_label_names�_get_seek_variable_labelsZ_seek_variable_labels� data_location� seek_strls�seek_value_labels� _get_dtypesrV �dtyplist�seek� _get_varlist�varlist�srtlist� _get_fmtlist�fmtlist� _get_lbllist�lbllist�_get_variable_labels�_variable_labels)r� � nvar_typeZ nvar_sizer/ r/ r0 rT t s\ $$$$$ $$$ z StataReader._read_new_headerr> z,tuple[list[int | str], list[str | np.dtype]])� seek_vartypesr& c sx �j �|� �fdd�t�j�D �}ddd��fdd� � � fdd�|D �}dd d��fd d � ��fd d�|D �}||fS ) Nc s* g | ]"}t �� jd � j�d��d �qS )r_ rX r )r| r} r� r! rB )r, �_)r� r/ r0 r1 � s z+StataReader._get_dtypes.<locals>.<listcomp>r> z int | str)rS r& c sP | dkr | S y � j | S tk rJ } ztd| � d��|�W d d }~X Y nX d S )Ni� z cannot convert stata types [�])r �KeyErrorrD )rS �err)r� r/ r0 rp � s z"StataReader._get_dtypes.<locals>.fc s g | ] }� |��qS r/ r/ )r, rm )rp r/ r0 r1 � s zstr | np.dtypec sT | dkrt | �S y � j| S tk rN } ztd| � d��|�W d d }~X Y nX d S )Ni� z cannot convert stata dtype [r� )ri r r� rD )rS r� )r� r/ r0 rs � s z"StataReader._get_dtypes.<locals>.gc s g | ] }� |��qS r/ r/ )r, rm )rs r/ r0 r1 � s )r! rx r� rj )r� r� Z raw_typlistrV rw r/ )rp rs r� r0 rv � s zStataReader._get_dtypesz list[str]c s, �j dk rdnd� � �fdd�t�j�D �S )NrN �! � c s g | ]}�� �j�� ���qS r/ )�_decoder! rB )r, r� )r� r� r/ r0 r1 � s z,StataReader._get_varlist.<locals>.<listcomp>)rO r� rj )r� r/ )r� r� r0 ry � s zStataReader._get_varlistc sN �j dkrd� n$�j dkr d� n�j dkr0d� nd� � �fdd �t�j�D �S ) NrN �9 �q �1 �h rS rQ c s g | ]}�� �j�� ���qS r/ )r� r! rB )r, r� )r� r� r/ r0 r1 � s z,StataReader._get_fmtlist.<locals>.<listcomp>)rO r� rj )r� r/ )r� r� r0 r| � s zStataReader._get_fmtlistc s> �j dkrd� n�j dkr d� nd� � �fdd�t�j�D �S )NrN r� r� r� rf c s g | ]}�� �j�� ���qS r/ )r� r! rB )r, r� )r� r� r/ r0 r1 � s z,StataReader._get_lbllist.<locals>.<listcomp>)rO r� rj )r� r/ )r� r� r0 r~ � s zStataReader._get_lbllistc sd � j dkr$� fdd�t� j�D �}n<� j dkrH� fdd�t� j�D �}n� fdd�t� j�D �}|S )NrN c s g | ]}� � � j�d ���qS )iA )r� r! rB )r, r� )r� r/ r0 r1 s z4StataReader._get_variable_labels.<locals>.<listcomp>r� c s g | ]}� � � j�d ���qS )�Q )r� r! rB )r, r� )r� r/ r0 r1 s c s g | ]}� � � j�d ���qS )r� )r� r! rB )r, r� )r� r/ r0 r1 s )rO r� rj )r� Zvlblistr/ )r� r0 r� � s z StataReader._get_variable_labelsc C sJ | j dkr(t�| jd | j�d��d S t�| jd | j�d��d S d S )NrN r� r� r r` rU )rO r| r} r� r! rB )r� r/ r/ r0 rk s zStataReader._get_nobsri c C s� | j dkr:t�| jd | j�d��d }| �| j�|��S | j dkrnt�d| j�d��d }| �| j�|��S | j dkr�| �| j�d ��S | �| j�d ��S d S ) NrN r_ rX r rX r� r% r� r� r� )rO r| r} r� r! rB r� )r� �strlenr/ r/ r0 rm s zStataReader._get_data_labelc C s� | j dkr4t�d| j�d��d }| j�|��d�S | j dkrht�d| j�d��d }| �| j�|��S | j dkr�| �| j�d��S t� �d S ) NrN r� r% r zutf-8rX r� � )rO r| r} r! rB �decoder� rD )r� r� r/ r/ r0 ro s zStataReader._get_time_stampc C sd | j dkr.| j�d� | jd| j d d S | j dkrZt�| jd | j�d��d d S t� �d S ) NrX r� r� � � rN r� r ) rO r! rB rq rj r| r} r� rD )r� r/ r/ r0 rr , s "z%StataReader._get_seek_variable_labelsr� )rW r& c sF t �d|�d � _� jdkr.ttj� jd���� �� t �d� j�d��d dkrVdpXd� _ t �d� j�d��d � _ � j�d� t �� j d � j�d ��d � _ � � � � _ � �� � _� �� � _� jd kr� fd d �t� j �D �}n^� j�� j �}tj|tjd �}g }x:|D ]2}|� jk�r.|�� j| � n|�|d � �q W y� fdd �|D �� _W nJ tk �r� } z*d�dd� |D ��}td|� d��|�W d d }~X Y nX y� fdd �|D �� _W nJ tk �r } z*d�dd� |D ��}td|� d��|�W d d }~X Y nX � jd k�r2� fdd �t� j �D �� _n� fdd �t� j �D �� _t �� j d� j d � j�d � j d ��d d� � _ � � � � _ � � � � _ � �!� � _"� jd k�r6x�t �� j d � j�d��d } � jd k�r�t �� j d � j�d ��d } n t �� j d � j�d ��d } | dk�r$P � j�| � �q�W � j�#� � _$d S ) Nr� r )r� r� r� �o r� �r �s )rZ r% r\ r] r_ rX r� c s g | ]}t � j�d ���qS )r% )�ordr! rB )r, r� )r� r/ r0 r1 L s z0StataReader._read_old_header.<locals>.<listcomp>)rM � c s g | ]}� j | �qS r/ )r )r, rS )r� r/ r0 r1 X s �,c s s | ]}t |�V qd S )N)ri )r, rm r/ r/ r0 � <genexpr>Z s z/StataReader._read_old_header.<locals>.<genexpr>z cannot convert stata types [r� c s g | ]}� j | �qS r/ )r )r, rS )r� r/ r0 r1 ] s c s s | ]}t |�V qd S )N)ri )r, rm r/ r/ r0 r� _ s z cannot convert stata dtypes [c s g | ]}� � � j�d ���qS )r� )r� r! rB )r, r� )r� r/ r0 r1 d s c s g | ]}� � � j�d ���qS )rf )r� r! rB )r, r� )r� r/ r0 r1 h s r� rh r� r� rU )%r| r} rO rD ri r) rP r! rB r� Zfiletyperj rk rl rm rn ro rp r� r[ � frombufferr� r r� rV �joinrw rz r{ r| r} r~ r r� r� �tellrs ) r� rW rV �bufZtyplistb�tpr� Z invalid_typesZinvalid_dtypesZ data_type�data_lenr/ )r� r0 rU 8 sp & $$ z StataReader._read_old_headerznp.dtypec C s� | j dk r| j S g }xlt| j�D ]^\}}|| jkr`tt|�}|�dt|� | j| j| f� q |�dt|� dt|� f� q W t� |�| _ | j S )z"Map between numpy and state dtypesN�s�S) r< � enumeraterV r r ri r� r� r[ rM )r� �dtypesr� rS r/ r/ r0 rD � s $" zStataReader._setup_dtypez int | str)rf r& c C s t |t�r|S t�| j| �S )N)r� r> r| �calcsizer� )r� rf r/ r/ r0 rR � s zStataReader._calcsize)r� r& c C sX |� d�d }y |�| j�S tk rR | j}d|� d�}t�|t� |�d�S X d S )Nr� r z@ One or more strings in the dta file could not be decoded using z�, and so the fallback encoding of latin-1 is being used. This can happen when a file has been incorrectly encoded by Stata or some other software. You should verify the string values returned are correct.zlatin-1)� partitionr� r� �UnicodeDecodeErrorrb rc �UnicodeWarning)r� r� r� �msgr/ r/ r0 r� � s zStataReader._decodec C s | j r d S | jdkr$d| _ i | _d S | jdkr>| j�| j� n.| jd k sLt�| j| jj }| j�| j | � d| _ i | _�x�| jdkr�| j� d�dkr�P | j� d�}|s�P | jdkr�| � | j� d��}n| � | j� d��}| j� d � t �| jd | j� d��d }t �| jd | j� d��d }tj| j� d| �| jd |d �}tj| j� d| �| jd |d �}t�|�}|| }|| }| j� |�} i | j|< xTt|�D ]H} | |d k �r�|| d n|} | � | || | � �| j| || < �q�W | jdkr|| j� d� q|W d| _ d S )Nr� TrX � s </valrU r� r� rV r` r r� )rM �countr% rY )r: rO �value_label_dictr! rx ru r< �AssertionErrorrl �itemsizers rB r� r| r} r� r[ r� Zargsortr� ) r� r� Zslengthr� r� Ztxtlenr� r� �iir� r� r r/ r/ r0 �_read_value_labels� sR * z StataReader._read_value_labelsc C sJ | j �| j� ddi| _�x*| j �d�dkr.P | jdkrXt�| jd | j �d��d }nr| j �d �}| jd krrd nd}| jd kr�|d|� |d d | � }n |d|� |d | d � }t�d|�d }t�d| j �d��d }t�| jd | j �d ��d }| j �|�}|dk�r,|dd� � | j �}nt |�}|| jt |�< q W d S )N�0r� rV s GSOrX r� r� r rS rN rX r] rU �Br% r` � rh ) r! rx rt �GSOrB rO r| r} r� r� r� ri )r� Zv_or� Zv_sizerS �length�vaZ decoded_var/ r/ r0 � _read_strls� s( zStataReader._read_strlsr c C s d| _ | j| jd�S )NT)�nrows)r7 rB r6 )r� r/ r/ r0 �__next__ s zStataReader.__next__)�sizer& c C s |dkr| j }| j|d�S )a Reads lines from Stata file and returns as dataframe Parameters ---------- size : int, defaults to None Number of lines to read. If None, reads whole file. Returns ------- DataFrame N)r� )r6 rB )r� r� r/ r/ r0 � get_chunk s zStataReader.get_chunkz bool | None) r� r" r# r$ r% r& r' r( r& c s� | j dkr2|d kr2d| _d| _| �� t| jd�S |d kr@| j}|d krN| j}|d kr\| j}|d krj| j }|d krx| j }|d kr�| j }|d kr�| j }|d kr�| j }| j dkr�| js�d| _| �� | jd k s�t�| j} | j | j | j } || j } t| | �} | dk�r |�r| �� | �� t�| j| j } | j�| j| � t|| j | j �} tj| j� | �| | d�}| j| 7 _| j| j k�r�d| _d| _| j | j k�r�|� � � � }|�r�| �� t!|�dk�r�t| jd�}nt�"|�}| j|_#|d k�r t�$| j| | j�}|�%|�}|d k �rHy| �&||�}W n t'k �rF | �� � Y nX x@t(|| j)�D ]0\}}t*|�t+k�rV|| j,| j-dd�||< �qVW | �.|�}t�/dd� | j0D ��d }|j1}d }g }x�|D ]�}| j0| d k �r�|j#| }|| j2} | t�2t3�k�r0| | j0| k�r0d}|�4|t5|| || j0| �f� n|�4||| f� �q�W |�r\t�6t7|��}~| �8||�}|�r�d d d �d d�� t�/� fdd�| j9D ��d }xV|D ]N}|j#| }y t:|| | j9| �||< W n t'k �r� | �� � Y nX �q�W |�r | j dk�r | �;|| j<| j=|�}|�s�g }d }x�|D ]�}|| j2} | t�2tj>�t�2tj?�fk�rnt�2tj@�} d}n8| t�2tjA�t�2tjB�t�2tjC�fk�r�t�2tjD�} d}|�4||| �E| �f� �q0W |�r�t�6t7|��}|d k �r�|�%|�F|��}|S )Nr T)r' rX )rM r� )Z convert_dtypec S s g | ] }|d k �qS )Nr/ )r, Zdtypr/ r/ r0 r1 � s z$StataReader.read.<locals>.<listcomp>Fri r� )rm r& c s t � fdd�tD ��S )Nc 3 s | ]}� � |�V qd S )N)r` )r, rf )rm r/ r0 r� � s z;StataReader.read.<locals>.any_startswith.<locals>.<genexpr>)r] � _date_formats)rm r/ )rm r0 �any_startswith� s z(StataReader.read.<locals>.any_startswithc s g | ] }� |��qS r/ r/ )r, rm )r� r/ r0 r1 � s r� )Grl r8 r; rI r rz r/ r0 r2 r3 r4 r5 r1 rO r: r� r< r� r= r� r3 r� � StopIterationr! rx rs r[ r� rB r� r@ �byteswap� newbyteorderr� Z from_recordsr' r� Z set_index�_do_select_columnsrD r5 rV r� r> rw r� � _insert_strls�whererw r* rM rd r� r! � from_dictr �_do_convert_missingr} rh �_do_convert_categoricalsr� r Zfloat16r� r{ r� r� r� r_ r^ �pop)r� r� r" r# r$ r% r& r' r( rM Z max_read_len�read_lenr� Z read_linesr� Zixr� rS Zcols_Zrequires_type_conversion�data_formattedr� �colsZ retyped_data�convertr/ )r� r0 rB s� zStataReader.read)r� r% r& c C sX i }�xt |�D �]\}}| j| }|| jkr0qtt|�}| j| \}}|| } t�| |k | |k�} | �� snq|r�t�t� | ��d } tj | | dd�\} } t | t d�}xft | �D ]&\}}t |�}| | |k }||j|< q�W n2| j}|tjtjfkr�tj}t | |d�}tj|| < |||< qW |�rT|j}t|�}t|j|jdd�|gdd�}|| }|S )Nr T)Zreturn_inverse)rM r% )Zaxis)r� rV r r ri r[ � logical_orr] ZnonzeroZasarray�uniquer! rd r� �ilocrM r� r{ �nanr' r r Zdrop)r� r� r% Z replacementsr� Zcolnamerf ZnminZnmaxZseries�missingZ missing_locZumissingZ umissing_loc� replacement�jZumr~ �locrM r' Zreplacement_dfZreplacedr/ r/ r0 r� � s@ z StataReader._do_convert_missing)r� r& c sp t � d�rt� j�dkr |S xNt� j�D ]@\}}|dkr:q(� fdd�|jd d �|f D �|jd d �|f< q(W |S )Nr� r r� c s g | ]}� j t|� �qS r/ )r� ri )r, �k)r� r/ r0 r1 � s z-StataReader._insert_strls.<locals>.<listcomp>)�hasattrr� r� r� rV r� )r� r� r� rS r/ )r� r0 r� � s 2zStataReader._insert_strlsz Sequence[str])r� r' r& c C s� | j s�t|�}t|�t|�kr&td��|�|j�}|rRd�t|��}td|� ���g }g }g }g } xX|D ]P} |j�| �} |� | j | � |� | j | � |� | j | � | � | j | � qhW || _ || _ || _ | | _ d| _ || S )Nz"columns contains duplicate entriesz, z<The following columns were not found in the Stata data set: T)r9 �setr� rD � differencer' r� r� Zget_locr� rw rV r} r ) r� r� r' Z column_set� unmatchedZjoinedrw rV r} r r� r� r/ r/ r0 r� � s2 z StataReader._do_select_columnsz!dict[str, dict[float | int, str]])r� r� r r( r& c C s� t |�� �}g }�xzt||�D �]j\}}||k�rx|| } t�t | �� ��} || } | �| �} | jrr| �� rr| } n| jr�t� t t � d} t | | |d�}| dkr�g }x>|j D ]&}|| kr�|�| | � q�|�|� q�W n t | �� �}y ||_ W nl tk �rX } zLt|��� }t |j|dk �}dd�|� }d|� d|� d�}t|�|�W dd}~X Y nX t||jd�}|�||f� q |�||| f� q W t�t|��}|S ) zC Converts categorical columns to Categorical type. N)r� Zorderedr% zQ-------------------------------------------------------------------------------- r$ z Value labels for column a are not unique. These cannot be converted to pandas categoricals. Either read the file with `convert_categoricals` set to False or use the low level interface in `StataReader` to separately read the values and the value_labels. The repeated labels are: )r* )r� �keysr5 r[ r� �isinr7 �allrb rc � categorical_conversion_warningr� r r� r� rE rD r! Z value_countsr* r� r r� r )r� r� r� r r( r� Zcat_converted_datar� �labelr� r� �columnZ key_matchesZinitial_categoriesZcat_datar� r� r� �vcZ repeated_catsZrepeatsr� Z cat_seriesr/ r/ r0 r� sH z$StataReader._do_convert_categoricalsc C s | j S )z2 Return data label of Stata file. )rn )r� r/ r/ r0 � data_labeld s zStataReader.data_labelzdict[str, str]c C s t t| j| j��S )z� Return variable labels as a dict, associating each variable name with corresponding label. Returns ------- dict )r r5 rz r� )r� r/ r/ r0 �variable_labelsk s zStataReader.variable_labelsc C s | j s| �� | jS )z� Return a dict, associating each variable name a dict, associating each value its corresponding label. Returns ------- dict )r: r� r� )r� r/ r/ r0 r� v s zStataReader.value_labels) TTNFTNTNr N)N)NNNNNNNN))r� r� r� �_stata_reader_docr� r� rG rM rI rP rC rT rv ry r| r~ r� rk rm ro rr rU rD rR r� r� r� r� r� r �_read_method_docrB r� r� r� r� r� r� r� r� � __classcell__r/ r/ )rF r0 r sf (1 M" V6 " ) !J r TFr r r� z str | NonezSequence[str] | Nonez int | Noner r zDataFrame | StataReader) �filepath_or_bufferr" r# r$ r% r& r' r( r) �iteratorr* r# r& c C sB t | ||||||||| | d� } | s&|r*| S | � | �� S Q R X d S )N) r" r# r$ r% r& r' r( r) r# r* )r rB ) r� r" r# r$ r% r& r' r( r) r� r* r# �readerr/ r/ r0 � read_stata� s r� )� endiannessr& c C s4 | � � dkrdS | � � dkr dS td| � d���d S )N)r] �littler] )r\ �bigr\ z Endianness z not understood)�lowerrD )r� r/ r/ r0 r> � s r> r r> )r� r� r& c C s2 t | t�r | d|t| � S | d|t| � S )zQ Take a char string and pads it with null bytes until it's length chars. r� � )r� r� r� )r� r� r/ r/ r0 r� � s r� znp.dtype)rf r& c C s( | dkrt �t j�S td| � d���dS )zK Convert from one of the stata date formats to a type in TYPE_MAP. )rK z%tcrN z%tdrO z%twrR z%tmrT z%tqrW z%thrZ z%tyzFormat z not implementedN)r[ rM r{ �NotImplementedError)rf r/ r/ r0 � _convert_datetime_to_stata_type� s r� r zlist[Hashable])r" rz r& c C sz i }xp| D ]h}| | � d�s,d| | | |< ||krN|�|�|�| | i� q t|t�s`td��|�|| | i� q W |S )N�%z0convert_dates key must be a column or an integer)r` �updater* r� r> rD )r" rz Znew_dictr� r/ r/ r0 �_maybe_convert_to_int_keys� s r� )rM r� r& c C s~ | j tjkr$tt|j��}t|d�S | tjkr2dS | tjkr@dS | tj krNdS | tj kr\dS | tj krjdS t d| � d���d S ) a� Convert dtype types to stata types. Returns the byte of the given ordinal. See TYPE_MAP and comments for an explanation. This is also explained in the dta spec. 1 - 244 are strings of this length Pandas Stata 251 - for int8 byte 252 - for int16 int 253 - for int32 long 254 - for float32 float 255 - for double double If there are dates to convert, then dtype will already have the correct type inserted. r% r� r� r� r� r� z Data type z not supported.N) r� r[ �object_r r rt r2 r{ r� r� r� r� r� )rM r� r� r/ r/ r0 �_dtype_to_stata_type� s r� r� )r� � dta_version� force_strlr& c C s� |dk rd}n d}|rdS | j tjkrntt|j��}||krX|dkrHdS tt�|j ���dt t |d�� d S | tj kr|dS | tj kr�d S | tjkr�d S | tjks�| tjkr�d S td | � d ���dS )a� Map numpy dtype to stata's default format for this type. Not terribly important since users can change this in Stata. Semantics are object -> "%DDs" where DD is the length of the string. If not a string, raise ValueError float64 -> "%10.0g" float32 -> "%9.0g" int64 -> "%9.0g" int32 -> "%12.0g" int16 -> "%8.0g" int8 -> "%8.0g" strl -> "%9s" rX �� i� z%9sr� r% r� z%10.0gz%9.0gz%12.0gz%8.0gz Data type z not supported.N)r� r[ r� r r rt rD � excessive_string_length_errorr) r� ri r2 r{ r� r� r� r� r� )rM r� r� r� � max_str_lenr� r/ r/ r0 �_dtype_to_default_stata_fmt s* r� )r# c s e Zd ZdZdZdZd\ddd d d d d d d dd� � fdd� Zddd�dd�Zddd�dd�Zddd�d d �Z ddd�d d �Z dd �d!d"�Z ddd#�d$d%�Z ddd�d&d'�Z d(dd)�d*d+�Zddd�d,d-�Zdd �d.d/�Zdd �d0d1�Zdd �d2d3�Zdd �d4d5�Zdd �d6d7�Zdd �d8d9�Zdd �d:d;�Zdd �d<d=�Zdd �d>d?�Zd]d d dd@�dAdB�Zdd �dCdD�Zdd �dEdF�Zdd �dGdH�Z dd �dIdJ�Z dd �dKdL�Z dd �dMdN�Z ddd�dOdP�Z dQd �dRdS�Z!dQddT�dUdV�Z"e#dddW�dXdY��Z$dddW�dZd[�Z%� Z&S )^� StataWritera� A class for writing Stata binary dta files Parameters ---------- fname : path (string), buffer or path object string, path object (pathlib.Path or py._path.local.LocalPath) or object implementing a binary write() functions. If using a buffer then the buffer will not be automatically closed after the file is written. data : DataFrame Input to save convert_dates : dict Dictionary mapping columns containing datetime types to stata internal format to use when writing the dates. Options are 'tc', 'td', 'tm', 'tw', 'th', 'tq', 'ty'. Column can be either an integer or a name. Datetime columns that do not have a conversion type specified will be converted to 'tc'. Raises NotImplementedError if a datetime column has timezone information write_index : bool Write the index to Stata dataset. byteorder : str Can be ">", "<", "little", or "big". default is `sys.byteorder` time_stamp : datetime A datetime to use as file creation date. Default is the current time data_label : str A label for the data set. Must be 80 characters or smaller. variable_labels : dict Dictionary containing columns as keys and variable labels as values. Each label must be 80 characters or smaller. compression : str or dict, default 'infer' For on-the-fly compression of the output dta. If string, specifies compression mode. If dict, value at key 'method' specifies compression mode. Compression mode must be one of {{'infer', 'gzip', 'bz2', 'zip', 'xz', None}}. If compression mode is 'infer' and `fname` is path-like, then detect compression from the following extensions: '.gz', '.bz2', '.zip', or '.xz' (otherwise no compression). If dict and compression mode is one of {{'zip', 'gzip', 'bz2'}}, or inferred as one of the above, other entries passed as additional compression options. .. versionadded:: 1.1.0 {storage_options} .. versionadded:: 1.2.0 Returns ------- writer : StataWriter instance The StataWriter instance has a write_file method, which will write the file to the given `fname`. Raises ------ NotImplementedError * If datetimes contain timezone information ValueError * Columns listed in convert_dates are neither datetime64[ns] or datetime.datetime * Column dtype is not representable in Stata * Column listed in convert_dates is not in DataFrame * Categorical label contains more than 32,000 characters Examples -------- >>> data = pd.DataFrame([[1.0, 1]], columns=['a', 'b']) >>> writer = StataWriter('./data_file.dta', data) >>> writer.write_file() Directly write a zip file >>> compression = {{"method": "zip", "archive_name": "data_file.dta"}} >>> writer = StataWriter('./data_file.zip', data, compression=compression) >>> writer.write_file() Save a DataFrame with dates >>> from datetime import datetime >>> data = pd.DataFrame([[datetime(2000,1,1)]], columns=['date']) >>> writer = StataWriter('./date_data_file.dta', data, {{'date' : 'tw'}}) >>> writer.write_file() r� zlatin-1NTr r r zdict[Hashable, str] | Noner� z str | Nonezdatetime.datetime | Noner r ) �fnamer� r" � write_indexr� rp r� r� r* r# c s� t � �� |d kri n|| _|| _|| _|| _|| _| | _d | _| � |� | | _ |d kr^t j }t |�| _|| _tjtjtjd�| _i | _d S )N)r� r� r� )r- r� r/ � _write_index� _time_stamprn r� � _compression� _output_file�_prepare_pandasr# r? r� r> � _byteorder�_fnamer[ r� r� r� Ztype_converters�_converted_names) r� r� r� r" r� r� rp r� r� r* r# )rF r/ r0 r� � s zStataWriter.__init__ri rH )�to_writer& c C s | j j�|�| j�� dS )zS Helper to call encode before writing to file for Python 3 compat. N)rE rA r� r� r� )r� r r/ r/ r0 �_write� s zStataWriter._writer� )r? r& c C s | j j�|� dS )z? Helper to assert file is open before writing. N)rE rA r� )r� r? r/ r/ r0 � _write_bytes� s zStataWriter._write_bytes)r� r& c s* � fdd�� D �}|| _ g | _t|�s*� S tj}g }x�t� |�D ]�\}}|�rt� | | jd�}| j�|� � | j j j }|t j kr�td��� | j j j�� } | �� ||�kr�|t jkr�t j}n|t jkr�t j}nt j}t j| |d�} ||�| | dk< |�|| f� q@|�|� | f� q@W t�t|��S )z� Check for categorical columns, retain categorical information for Stata file and convert categorical data to int c s g | ]}t � | j��qS r/ )r rM )r, r� )r� r/ r0 r1 � s z5StataWriter._prepare_categoricals.<locals>.<listcomp>)r� zCIt is not possible to export int64-based categorical data to Stata.)rM rh )� _is_col_cat� _value_labelsr] r� r� r5 r� r� r� r� �codesrM r[ r_ rD rt �copyr2 r� r� r� r{ r� r r� r ) r� r� Zis_catr� r� r� Z col_is_catZsvlrM rE r/ )r� r0 �_prepare_categoricals� s6 z!StataWriter._prepare_categoricalsc C s^ xX|D ]P}|| j }|tjtjfkr|tjkr:| jd }n | jd }|| �|�||< qW |S )z� Checks floating point data columns for nans, and replaces these with the generic Stata for missing value (.) rp r; )rM r[ r� r{ r� �fillna)r� r� r� rM r� r/ r/ r0 � _replace_nans� s zStataWriter._replace_nans)r& c C s dS )z No-op, forward compatibilityNr/ )r� r/ r/ r0 �_update_strl_names� s z StataWriter._update_strl_names)r� r& c C sV xP|D ]H}|dk s|dkr|dk s*|dkr|dk s:|dkr|dkr|� |d�}qW |S )a� Validate variable names for Stata export. Parameters ---------- name : str Variable name Returns ------- str The validated name with invalid characters replaced with underscores. Notes ----- Stata 114 and 117 support ascii characters in a-z, A-Z, 0-9 and _. �A�Zr� �zr� �9r� )�replace)r� r� r� r/ r/ r0 �_validate_variable_name s z#StataWriter._validate_variable_namec C s� i }t |j�}|dd� }d}x�t|�D ]�\}}|}t|t�sFt|�}| �|�}|| jkrbd| }d|d krzdkr�n nd| }|dtt|�d�� }||ks�x@|� |�dkr�dt|� | }|dtt|�d�� }|d7 }q�W |||< |||< q(W t |�|_| j �rHx:t ||�D ],\} } | | k�r| j | | j | < | j | = �qW |�r�g } x.|� � D ]"\}}|� d|� �} | �| � �q\W t�d �| ��} t�| t� || _| �� |S ) a� Checks column names to ensure that they are valid Stata column names. This includes checks for: * Non-string names * Stata keywords * Variables that start with numbers * Variables with names that are too long When an illegal variable name is detected, it is converted, and if dates are exported, the variable name is propagated to the date conversion dictionary Nr r� r� r r� r% z -> z )r� r' r� r� ri r r r3 r� r� r r/ r5 �itemsr� �invalid_name_docr) r� rb rc r� r r )r� r� Zconverted_namesr' Zoriginal_columnsZduplicate_var_idr� r� � orig_namer� �oZconversion_warningr� r� r/ r/ r0 �_check_column_names sJ z StataWriter._check_column_namesr! )r� r& c C sV g | _ g | _xD|�� D ]8\}}| j �t|| j| �� | j�t|| j| �� qW d S )N)r} rV r r� r� r� r� )r� r� r� rM r/ r/ r0 �_set_formats_and_typesd s z"StataWriter._set_formats_and_typesc C s, |� � }| jr$|�� }t|t�r$|}| �|�}t|�}| �|�}| �|�}|j \| _ | _ || _ |j �� | _|j}x.|D ]&}|| jkr�qvt|| �rvd| j|< qvW t| j| j�| _x*| jD ] }t| j| �}t�|�||< q�W | �� | �|� | jd k �r(x,| jD ]"}t|t��r| j| | j|< �qW d S )NrK )r r� Z reset_indexr� r r r� r r �shaperl rj r� r' �tolistrz r� r/ r r� r� r[ rM �_encode_stringsr r> r} )r� r� �tempr� r� r� �new_typer/ r/ r0 r k s: zStataWriter._prepare_pandasc C s� | j }t| dg �}x�t| j�D ]�\}}||ks ||kr8q | j| }|j}|jtjkr t|dd�}|dks�t |�dks�|j }t d|� d���| j| j � | j�}tt|j��| jkr || j|< q W dS ) z� Encode strings in dta-specific encoding Do not encode columns marked for date conversion or for strL conversion. The strL converter independently handles conversion and also accepts empty string arrays. � _convert_strlT)rk r� r zColumn `a ` cannot be exported. Only string-like object arrays containing all strings or a mix of strings and None can be exported. Object arrays containing only null values are prohibited. Other object types cannot be exported and must first be converted to one of the supported types.N)r/ r4 r� r� rM r� r[ r� r r� r� rD ri r� r� r r rt �_max_string_length) r� r" � convert_strlr� r� r� rM Zinferred_dtype�encodedr/ r/ r0 r � s" zStataWriter._encode_stringsc C s� t | jd| jd| jd���n| _| jjd d k rV| jjt� | _| j_| jj � | jj� y�| j | j | j d� | �� | �� | �� | �� | �� | �� | �� | �� | �� | �� }| �|� | �� | �� | �� | �� | � � W n� t k �r� } zt| j� � t | jt t!j"f��rnt!j#�$| j��rnyt!�%| j� W n, t&k �rl t'�(d| j� d�t)� Y nX |�W d d }~X Y nX W d Q R X d S )N�wbF)r* r, r# �method)r� rp z!This save was not successful but z. could not be deleted. This file is not valid.)*r" r r r# rE r* rA r r Zcreated_handlesr� � _write_headerrn r � _write_map�_write_variable_types�_write_varnames�_write_sortlist�_write_formats�_write_value_label_names�_write_variable_labels�_write_expansion_fields�_write_characteristics� _prepare_data� _write_data� _write_strls�_write_value_labels�_write_file_close_tag�_close� ExceptionrI r� ri �os�PathLike�path�isfile�unlink�OSErrorrb rc �ResourceWarning)r� �records�excr/ r/ r0 � write_file� sP zStataWriter.write_filec C sF | j dk rBt| jjt�s t�| jj| j }| j_| jj�|�� � dS )z� Close the file if it was created by the writer. If a buffer or file-like object was passed in, for example a GzipFile, then leave this file open for the caller to close. N)r r� rE rA r r� r� r� )r� r� r/ r/ r0 r8 � s zStataWriter._closec C s dS )zNo-op, future compatibilityNr/ )r� r/ r/ r0 r* s zStataWriter._write_mapc C s dS )zNo-op, future compatibilityNr/ )r� r/ r/ r0 r7 s z!StataWriter._write_file_close_tagc C s dS )zNo-op, future compatibilityNr/ )r� r/ r/ r0 r2 s z"StataWriter._write_characteristicsc C s dS )zNo-op, future compatibilityNr/ )r� r/ r/ r0 r5 s zStataWriter._write_strlsc C s | � tdd�� dS )z"Write 5 zeros for expansion fieldsr� r� N)r r� )r� r/ r/ r0 r1 s z#StataWriter._write_expansion_fieldsc C s& x | j D ]}| �|�| j�� qW d S )N)r r r� r )r� r� r/ r/ r0 r6 s z StataWriter._write_value_labels)r� rp r& c C sH | j }| �t�dd�� | �|dkr(dp*d� | �d� | �d� | �t�|d | j�d d� � | �t�|d | j�d d � � |d kr�| �| �td d ��� n | �| �t|d d � d ��� |d kr�t j � � }nt |t j �s�t d ��ddddddddddddg }dd� t |�D �}|�d �||j |�d � }| �| �|�� d S ) Nr� r� r\ ��r� r� rX r� rU r� �P z"time_stamp should be datetime type�Jan�Feb�Mar�Apr�May�Jun�Jul�Aug�Sep�Oct�Nov�Decc S s i | ]\}}||d �qS )r% r/ )r, r� r7 r/ r/ r0 � <dictcomp>M s z-StataWriter._write_header.<locals>.<dictcomp>z%d z %Y %H:%M)r r r| r� r rj rl �_null_terminate_bytesr� r+ �nowr� rD r� �strftimer7 )r� r� rp r� �months� month_lookup�tsr/ r/ r0 r) s< zStataWriter._write_headerc C s& x | j D ]}| �t�d|�� qW d S )Nr� )rV r r| r� )r� rS r/ r/ r0 r+ U s z!StataWriter._write_variable_typesc C s: x4| j D ]*}| �|�}t|d d� d�}| �|� qW d S )Nr� r� )rz �_null_terminate_strr� r )r� r� r/ r/ r0 r, Y s zStataWriter._write_varnamesc C s" t dd| jd �}| �|� d S )Nr� rX r% )r� rj r )r� r{ r/ r/ r0 r- a s zStataWriter._write_sortlistc C s$ x | j D ]}| �t|d�� qW d S )Nr� )r} r r� )r� rf r/ r/ r0 r. f s zStataWriter._write_formatsc C sd x^t | j�D ]P}| j| rL| j| }| �|�}t|d d� d�}| �|� q | �tdd�� q W d S )Nr� r� r� )r� rj r rz rZ r� r )r� r� r� r/ r/ r0 r/ k s z$StataWriter._write_value_label_namesc C s� t dd�}| jd kr6x t| j�D ]}| �|� q W d S xp| jD ]f}|| jkr�| j| }t|�dkrjtd��tdd� |D ��}|s�td��| �t |d�� q>| �|� q>W d S )Nr� r� rF z.Variable labels must be 80 characters or fewerc s s | ]}t |�d k V qdS )� N)r� )r, r� r/ r/ r0 r� � s z5StataWriter._write_variable_labels.<locals>.<genexpr>zKVariable labels must contain only characters that can be encoded in Latin-1) r� r� r� rj r r� r� rD r� )r� �blankr� r� r� Z is_latin1r/ r/ r0 r0 w s z"StataWriter._write_variable_labelsc C s |S )zNo-op, future compatibilityr/ )r� r� r/ r/ r0 �_convert_strls� s zStataWriter._convert_strlsz np.recarrayc C s | j }| j}| j}| jd k rRx4t|�D ](\}}||kr&t|| | j| �||< q&W | �|�}i }| jtt j �k}x�t|�D ]�\}}|| }|| j kr�|| � d�j t|fd�||< d|� �} | ||< || �| �||< qz|| j} |s�| �| j�} | ||< qzW |jd|d�S )Nr� )�argsr� F)r* Z column_dtypes)r� rV r/ r� r r} r] r r> r? r� r$ r rw r� r^ rM r� Z to_records) r� r� rV r" r� r� r� Znative_byteorderrS �styperM r/ r/ r0 r3 � s. zStataWriter._prepare_data)rA r& c C s | � |�� � d S )N)r �tobytes)r� rA r/ r/ r0 r4 � s zStataWriter._write_data)r� r& c C s | d7 } | S )Nr� r/ )r� r/ r/ r0 rZ � s z StataWriter._null_terminate_strc C s | � |��| j�S )N)rZ r� r� )r� r� r/ r/ r0 rT � s z!StataWriter._null_terminate_bytes)NTNNNNr N)NN)'r� r� r� r� r$ r� r� r r r r r r r r r r rC r8 r* r7 r2 r5 r1 r6 r) r+ r, r- r. r/ r0 r] r3 r4 � staticmethodrZ rT r� r/ r/ )rF r0 r� > sT Q &* E7%2 4 r� )rM r� r� r& c C s� |rdS | j tjkr<tt|j��}t|d�}|dkr8|S dS | tjkrJdS | tjkrXdS | tj krfdS | tj krtdS | tj kr�dS t d | � d ���d S ) a Converts dtype types to stata types. Returns the byte of the given ordinal. See TYPE_MAP and comments for an explanation. This is also explained in the dta spec. 1 - 2045 are strings of this length Pandas Stata 32768 - for object strL 65526 - for int8 byte 65527 - for int16 int 65528 - for int32 long 65529 - for float32 float 65530 - for double double If there are dates to convert, then dtype will already have the correct type inserted. i � r% i� i�� i�� i�� i�� i�� z Data type z not supported.N) r� r[ r� r r rt r2 r{ r� r� r� r� r� )rM r� r� r� r/ r/ r0 �_dtype_to_stata_type_117� s& rb z str | bytesr� c C s( t | t�rt| d�} | d|t| � S )zU Takes a bytes instance and pads it with null bytes until it's length chars. zutf-8r� )r� ri r� r� )r� r� r/ r/ r0 �_pad_bytes_new� s rc c @ sT e Zd ZdZdddddd�d d �Zd dd �d d�Zdd�dd�Zddd�dd�ZdS )�StataStrLWritera� Converter for Stata StrLs Stata StrLs map 8 byte values to strings which are stored using a dictionary-like format where strings are keyed to two values. Parameters ---------- df : DataFrame DataFrame to convert columns : Sequence[str] List of columns names to convert to StrL version : int, optional dta version. Currently supports 117, 118 and 119 byteorder : str, optional Can be ">", "<", "little", or "big". default is `sys.byteorder` Notes ----- Supports creation of the StrL block of a dta file for dta versions 117, 118 and 119. These differ in how the GSO is stored. 118 and 119 store the GSO lookup value as a uint32 and a uint64, while 117 uses two uint32s. 118 and 119 also encode all strings as unicode which is required by the format. 117 uses 'latin-1' a fixed width encoding that extends the 7-bit ascii table with an additional 128 characters. rX Nr z Sequence[str]r> z str | None)�dfr' rZ r� c C s� |dkrt d��|| _|| _|| _ddi| _|d kr:tj}t|�| _d}d}d| _ |dkrjd }d}d | _ n|d krxd }nd }ddd| | _ || _ || _ d S )N)rX rN rY z,Only dta versions 117, 118 and 119 supportedr� )r r r` r� zutf-8rX rU zlatin-1rN rY r� rX r� ) rD Z_dta_verre r' � _gso_tabler? r� r> r r� �_o_offet� _gso_o_type� _gso_v_type)r� re r' rZ r� Z gso_v_typeZ gso_o_typeZo_sizer/ r/ r0 r� s, zStataStrLWriter.__init__ztuple[int, int])r� r& c C s |\}}|| j | S )N)rg )r� r� rx r r/ r/ r0 � _convert_key: s z StataStrLWriter._convert_keyz,tuple[dict[str, tuple[int, int]], DataFrame])r& c s | j }| j}t|j�� || j }� fdd�| jD �}tj|jtjd�}x�t|� � �D ]|\}\}}xnt|�D ]b\} \} } || } | dkr�dn| } |� | d�} | dkr�| d |d f} | || < | � | �||| f< qjW qTW x*t| j�D ] \}} |dd�|f || < q�W ||fS )a� Generates the GSO lookup table for the DataFrame Returns ------- gso_table : dict Ordered dictionary using the string found as keys and their lookup position (v,o) as values gso_df : DataFrame DataFrame where strl columns have been converted to (v,o) values Notes ----- Modifies the DataFrame in-place. The DataFrame returned encodes the (v,o) values as uint64s. The encoding depends on the dta version, and can be expressed as enc = v + o * 2 ** (o_size * 8) so that v is stored in the lower bits and o is in the upper bits. o_size is * 117: 4 * 118: 6 * 119: 5 c s g | ]}|� � |�f�qS r/ )r* )r, r� )r' r/ r0 r1 _ s z2StataStrLWriter.generate_table.<locals>.<listcomp>)rM Nr� r% ) rf re r� r' r[ �emptyr r� r� Ziterrows�getrj )r� � gso_tableZgso_df�selectedZ col_indexr� r �idx�rowr� r� rx r� r� r� r/ )r' r0 �generate_table> s$ z StataStrLWriter.generate_tablezdict[str, tuple[int, int]]r� )rm r& c C s� t � }tdd�}t�| jd d�}t�| jd d�}| j| j }| j| j }| jd }x�|�� D ]�\} } | dkrrq`| \} } |�|� |�t�|| �� |�t�|| �� |�|� t| d�} |�t�|t | �d �� |�| � |�|� q`W |� � S ) a� Generates the binary blob of GSOs that is written to the dta file. Parameters ---------- gso_table : dict Ordered dictionary (str, vo) Returns ------- gso : bytes Binary content of dta file to be placed between strl tags Notes ----- Output format depends on dta version. 117 uses two uint32s to express v and o while 118+ uses a uint32 for v and a uint64 for o. r� �asciir� r� r r` )r r zutf-8r% ) r r� r| r� r ri rh r r� r� r� )r� rm r� ZgsoZgso_type�nullZv_typeZo_typeZlen_typeZstrlZvorx r Z utf8_stringr/ r/ r0 � generate_blobq s( z StataStrLWriter.generate_blob)rX N)r� r� r� r� r� rj rq rt r/ r/ r/ r0 rd � s 3rd c sb e Zd ZdZdZdZdDddd d d d d d d ddd� � fdd� Zedddd�dd��Zddd�dd �Z dEd d dd �d d �Z dd �d!d"�Z dd �d#d$�Z dd �d%d&�Z dd �d'd(�Zdd �d)d*�Zdd �d+d,�Zdd �d-d.�Zdd �d/d0�Zdd �d1d2�Zdd �d3d4�Zdd �d5d6�Zdd �d7d8�Zdd �d9d:�Zdd �d;d<�Zddd=�d>d?�Zd@ddA�dBdC�Z� ZS )F�StataWriter117a� A class for writing Stata binary dta files in Stata 13 format (117) Parameters ---------- fname : path (string), buffer or path object string, path object (pathlib.Path or py._path.local.LocalPath) or object implementing a binary write() functions. If using a buffer then the buffer will not be automatically closed after the file is written. data : DataFrame Input to save convert_dates : dict Dictionary mapping columns containing datetime types to stata internal format to use when writing the dates. Options are 'tc', 'td', 'tm', 'tw', 'th', 'tq', 'ty'. Column can be either an integer or a name. Datetime columns that do not have a conversion type specified will be converted to 'tc'. Raises NotImplementedError if a datetime column has timezone information write_index : bool Write the index to Stata dataset. byteorder : str Can be ">", "<", "little", or "big". default is `sys.byteorder` time_stamp : datetime A datetime to use as file creation date. Default is the current time data_label : str A label for the data set. Must be 80 characters or smaller. variable_labels : dict Dictionary containing columns as keys and variable labels as values. Each label must be 80 characters or smaller. convert_strl : list List of columns names to convert to Stata StrL format. Columns with more than 2045 characters are automatically written as StrL. Smaller columns can be converted by including the column name. Using StrLs can reduce output file size when strings are longer than 8 characters, and either frequently repeated or sparse. compression : str or dict, default 'infer' For on-the-fly compression of the output dta. If string, specifies compression mode. If dict, value at key 'method' specifies compression mode. Compression mode must be one of {'infer', 'gzip', 'bz2', 'zip', 'xz', None}. If compression mode is 'infer' and `fname` is path-like, then detect compression from the following extensions: '.gz', '.bz2', '.zip', or '.xz' (otherwise no compression). If dict and compression mode is one of {'zip', 'gzip', 'bz2'}, or inferred as one of the above, other entries passed as additional compression options. .. versionadded:: 1.1.0 Returns ------- writer : StataWriter117 instance The StataWriter117 instance has a write_file method, which will write the file to the given `fname`. Raises ------ NotImplementedError * If datetimes contain timezone information ValueError * Columns listed in convert_dates are neither datetime64[ns] or datetime.datetime * Column dtype is not representable in Stata * Column listed in convert_dates is not in DataFrame * Categorical label contains more than 32,000 characters Examples -------- >>> from pandas.io.stata import StataWriter117 >>> data = pd.DataFrame([[1.0, 1, 'a']], columns=['a', 'b', 'c']) >>> writer = StataWriter117('./data_file.dta', data) >>> writer.write_file() Directly write a zip file >>> compression = {"method": "zip", "archive_name": "data_file.dta"} >>> writer = StataWriter117('./data_file.zip', data, compression=compression) >>> writer.write_file() Or with long strings stored in strl format >>> data = pd.DataFrame([['A relatively long string'], [''], ['']], ... columns=['strls']) >>> writer = StataWriter117('./data_file_with_long_strings.dta', data, ... convert_strl=['strls']) >>> writer.write_file() i� rX NTr r r zdict[Hashable, str] | Noner� z str | Nonezdatetime.datetime | NonezSequence[Hashable] | Noner r ) r� r� r" r� r� rp r� r� r% r* r# c sJ g | _ | d k r| j �| � t� j||||||||| | d� i | _d| _d S )N)r� rp r� r� r* r# rr )r# �extendr- r� �_map� _strl_blob) r� r� r� r" r� r� rp r� r� r% r* r# )rF r/ r0 r� s zStataWriter117.__init__z str | bytesri r� )r� �tagr& c C s< t | t�rt| d�} td| d d�| td| d d� S )z Surround val with <tag></tag>zutf-8r] r\ z</)r� ri r� )r� ry r/ r/ r0 �_tag, s zStataWriter117._tagrH )ry r& c C s& | j jdk st�| j j�� | j|< dS )z.Update map location for tag with file positionN)rE rA r� r� rw )r� ry r/ r/ r0 � _update_map3 s zStataWriter117._update_map)r� rp r& c C s� | j }| �tdd�� t� }|�| �tt| j�d�d�� |�| �|dkrNdpPdd�� | jdkrfd nd }|�| �t� || | j �d �� | jd kr�d nd }|�| �t� || | j �d�� |dk r�|dd� nd}|� | j �}| jd kr�dnd } t� || t|��} | | }|�| �|d�� |dk�r6tj�� }nt|tj��sLtd��dddddddd d d d d g } d!d"� t| �D �} |�d#�| |j |�d$� } d%t| d� }|�| �|d&�� | �| �|�� d'�� dS )(zWrite the file headerz <stata_dta>zutf-8�releaser\ ZMSFZLSFr� rN r_ r` �KrX r� �NNrF r� r� r� z"time_stamp should be datetime typerG rH rI rJ rK rL rM rN rO rP rQ rR c S s i | ]\}}||d �qS )r% r/ )r, r� r7 r/ r/ r0 rS g s z0StataWriter117._write_header.<locals>.<dictcomp>z%d z %Y %H:%M� � timestamp�header)r r r� r r� rz ri � _dta_versionr| r� rj rl r� r� r� r+ rU r� rD r� rV r7 r� )r� r� rp r� r� r� Z nobs_sizer� Z encoded_labelZ label_sizeZ label_lenrW rX rY Zstata_tsr/ r/ r0 r) 8 sH z StataWriter117._write_header)r& c C s� | j s2d| jj�� ddddddddddddd�| _ | jj�| j d � t� }x*| j �� D ] }|�t� | j d |�� qXW | � | � |� � d�� dS )z� Called twice during file write. The first populates the values in the map with 0s. The second call writes the final map locations when all blocks have been written. r )Z stata_data�map�variable_types�varnames�sortlist�formats�value_label_namesr� �characteristicsr� �strlsr� �stata_data_closez end-of-filer� r� N)rw rE rA r� rx r rE r� r| r� r r rz r� )r� r� r� r/ r/ r0 r* r s( zStataWriter117._write_mapc C sR | � d� t� }x&| jD ] }|�t�| jd |�� qW | �| �|� � d�� d S )Nr� r_ ) r{ r rV r� r| r� r r rz r� )r� r� rS r/ r/ r0 r+ � s z$StataWriter117._write_variable_typesc C s~ | � d� t� }| jdkr dnd}x@| jD ]6}| �|�}t|d d� �| j�|d �}|�|� q*W | � | � |� � d�� d S )Nr� rX r� r� r% ) r{ r r� rz rZ rc r� r� r� r rz r� )r� r� Zvn_lenr� r/ r/ r0 r, � s z StataWriter117._write_varnamesc C s@ | � d� | jdk rdnd}| �| �d| | jd d�� d S )Nr� rY rX rU r� r% )r{ r� r rz rj )r� Z sort_sizer/ r/ r0 r- � s z StataWriter117._write_sortlistc C sd | � d� t� }| jdkr dnd}x&| jD ] }|�t|�| j�|�� q*W | �| � |� � d�� d S )Nr� rX r� r� ) r{ r r� r} r� rc r� r� r rz r� )r� r� Zfmt_lenrf r/ r/ r0 r. � s z StataWriter117._write_formatsc C s� | � d� t� }| jdkr dnd}x\t| j�D ]N}d}| j| rJ| j| }| �|�}t|d d� � | j �|d �}|� |� q.W | � | � |�� d�� d S )Nr� rX r� r� r� r% )r{ r r� r� rj r rz rZ rc r� r� r� r rz r� )r� r� �vl_lenr� r� � encoded_namer/ r/ r0 r/ � s z'StataWriter117._write_value_label_namesc C s, | � d� t� }| jdkr dnd}td|d �}| jd krrx t| j�D ]}|�|� qFW | �| � |� � d�� d S x�| j D ]�}|| jk�r| j| }t |�dkr�t d��y|�| j�}W n4 tk r� } zt d| j� ��|�W d d }~X Y nX |�t||d �� qz|�|� qzW | �| � |� � d�� d S ) Nr� rX rF i@ r� r% z.Variable labels must be 80 characters or fewerzDVariable labels must contain only characters that can be encoded in )r{ r r� rc r� r� rj r� r rz r� r� r� rD r� r� �UnicodeEncodeError) r� r� r� r\ r� r� r� r& r� r/ r/ r0 r0 � s. z%StataWriter117._write_variable_labelsc C s | � d� | �| �dd�� d S )Nr� rr )r{ r rz )r� r/ r/ r0 r2 � s z%StataWriter117._write_characteristicsc C s0 | � d� | �d� | �|�� � | �d� d S )Nr� s <data>s </data>)r{ r r` )r� rA r/ r/ r0 r4 � s zStataWriter117._write_datac C s" | � d� | �| �| jd�� d S )Nr� )r{ r rz rx )r� r/ r/ r0 r5 � s zStataWriter117._write_strlsc C s dS )zNo-op in dta 117+Nr/ )r� r/ r/ r0 r1 � s z&StataWriter117._write_expansion_fieldsc C s\ | � d� t� }x0| jD ]&}|�| j�}| �|d�}|�|� qW | �| �|�� d�� d S )Nr� Zlbl) r{ r r r� r rz r� r r� )r� r� r� Zlabr/ r/ r0 r6 � s z"StataWriter117._write_value_labelsc C s( | � d� | �tdd�� | � d� d S )Nr� z </stata_dta>zutf-8z end-of-file)r{ r r� )r� r/ r/ r0 r7 � s z$StataWriter117._write_file_close_tagc C s< x6| j �� D ](\}}|| jkr | j�|�}|| j|< q W dS )z� Update column names for conversion to strl if they might have been changed to comply with Stata naming rules N)r r r# r* )r� �orig�newro r/ r/ r0 r � s z!StataWriter117._update_strl_names)r� r& c sJ � fdd�t |�D �}|rFt||� jd�}|�� \}}|}|�|�� _|S )zg Convert columns to StrLs if either very large or in the convert_strl variable c s, g | ]$\}}� j | d ks$|� jkr|�qS )i � )rV r# )r, r� r� )r� r/ r0 r1 s z1StataWriter117._convert_strls.<locals>.<listcomp>)rZ )r� rd r� rq rt rx )r� r� Z convert_colsZssw�tabZnew_datar/ )r� r0 r] s z StataWriter117._convert_strlsr! )r� r& c C sn g | _ g | _x\|�� D ]P\}}|| jk}t|| j| | j|d�}| j�|� | j �t|| j| |�� qW d S )N)r� r� ) rV r} r r# r� r� r� r� rb )r� r� r� rM r� rf r/ r/ r0 r s z%StataWriter117._set_formats_and_types) NTNNNNNr N)NN) r� r� r� r� r$ r� r� ra rz r{ r) r* r+ r, r- r. r/ r0 r2 r4 r5 r1 r6 r7 r r] r r� r/ r/ )rF r0 ru � s@ T ( 7 ru c sR e Zd ZdZdZddddd d d d dd d ddd� � fdd� Zddd�dd�Z� ZS )�StataWriterUTF8ub Stata binary dta file writing in Stata 15 (118) and 16 (119) formats DTA 118 and 119 format files support unicode string data (both fixed and strL) format. Unicode is also supported in value labels, variable labels and the dataset label. Format 119 is automatically used if the file contains more than 32,767 variables. .. versionadded:: 1.0.0 Parameters ---------- fname : path (string), buffer or path object string, path object (pathlib.Path or py._path.local.LocalPath) or object implementing a binary write() functions. If using a buffer then the buffer will not be automatically closed after the file is written. data : DataFrame Input to save convert_dates : dict, default None Dictionary mapping columns containing datetime types to stata internal format to use when writing the dates. Options are 'tc', 'td', 'tm', 'tw', 'th', 'tq', 'ty'. Column can be either an integer or a name. Datetime columns that do not have a conversion type specified will be converted to 'tc'. Raises NotImplementedError if a datetime column has timezone information write_index : bool, default True Write the index to Stata dataset. byteorder : str, default None Can be ">", "<", "little", or "big". default is `sys.byteorder` time_stamp : datetime, default None A datetime to use as file creation date. Default is the current time data_label : str, default None A label for the data set. Must be 80 characters or smaller. variable_labels : dict, default None Dictionary containing columns as keys and variable labels as values. Each label must be 80 characters or smaller. convert_strl : list, default None List of columns names to convert to Stata StrL format. Columns with more than 2045 characters are automatically written as StrL. Smaller columns can be converted by including the column name. Using StrLs can reduce output file size when strings are longer than 8 characters, and either frequently repeated or sparse. version : int, default None The dta version to use. By default, uses the size of data to determine the version. 118 is used if data.shape[1] <= 32767, and 119 is used for storing larger DataFrames. compression : str or dict, default 'infer' For on-the-fly compression of the output dta. If string, specifies compression mode. If dict, value at key 'method' specifies compression mode. Compression mode must be one of {'infer', 'gzip', 'bz2', 'zip', 'xz', None}. If compression mode is 'infer' and `fname` is path-like, then detect compression from the following extensions: '.gz', '.bz2', '.zip', or '.xz' (otherwise no compression). If dict and compression mode is one of {'zip', 'gzip', 'bz2'}, or inferred as one of the above, other entries passed as additional compression options. .. versionadded:: 1.1.0 Returns ------- StataWriterUTF8 The instance has a write_file method, which will write the file to the given `fname`. Raises ------ NotImplementedError * If datetimes contain timezone information ValueError * Columns listed in convert_dates are neither datetime64[ns] or datetime.datetime * Column dtype is not representable in Stata * Column listed in convert_dates is not in DataFrame * Categorical label contains more than 32,000 characters Examples -------- Using Unicode data and column names >>> from pandas.io.stata import StataWriterUTF8 >>> data = pd.DataFrame([[1.0, 1, 'ᴬ']], columns=['a', 'β', 'ĉ']) >>> writer = StataWriterUTF8('./data_file.dta', data) >>> writer.write_file() Directly write a zip file >>> compression = {"method": "zip", "archive_name": "data_file.dta"} >>> writer = StataWriterUTF8('./data_file.zip', data, compression=compression) >>> writer.write_file() Or with long strings stored in strl format >>> data = pd.DataFrame([['ᴀ relatively long ŝtring'], [''], ['']], ... columns=['strls']) >>> writer = StataWriterUTF8('./data_file_with_long_strings.dta', data, ... convert_strl=['strls']) >>> writer.write_file() zutf-8NTr r r zdict[Hashable, str] | Noner� z str | Nonezdatetime.datetime | NonezSequence[Hashable] | Nonez int | Noner r ) r� r� r" r� r� rp r� r� r% rZ r* r# c s| | d kr |j d dkrdnd} n0| dkr2td��n | dkrP|j d dkrPtd��t� j||||||||| | | d� | | _d S ) Nr% i� rN rY )rN rY z"version must be either 118 or 119.zKYou must use version 119 for data sets containing more than32,767 variables) r" r� r� rp r� r� r% r* r# )r rD r- r� r� ) r� r� r� r" r� r� rp r� r� r% rZ r* r# )rF r/ r0 r� � s( zStataWriterUTF8.__init__ri )r� r& c C s~ xx|D ]p}t |�dk rN|dk s&|dkrN|dk s6|dkrN|dk sF|dkrN|dksjdt |� krfd k rn q|�|d�}qW |S ) a� Validate variable names for Stata export. Parameters ---------- name : str Variable name Returns ------- str The validated name with invalid characters replaced with underscores. Notes ----- Stata 118+ support most unicode characters. The only limitation is in the ascii range where the characters supported are a-z, A-Z, 0-9 and _. r� r r r� r r� r r� r[ )r� r )r� r� r� r/ r/ r0 r � s z'StataWriterUTF8._validate_variable_name) NTNNNNNNr N)r� r� r� r� r� r� r r� r/ r/ )rF r0 r� , s b * r� ) TTNFTNTNFr N)r� F)er� � __future__r � collectionsr r+ �ior r: r| r? �typingr r r r r rb Zdateutil.relativedeltar �numpyr[ Zpandas._libs.libr Zpandas._libs.writersr Zpandas._typingr r r r Zpandas.util._decoratorsr r Zpandas.core.dtypes.commonr r r Zpandasr r r r r r r r Z pandas.corer Zpandas.core.framer Zpandas.core.indexes.baser Zpandas.core.seriesr! Zpandas.io.commonr" ri Z _statafile_processing_params1Z _statafile_processing_params2Z_chunksize_paramsZ _shared_docsZ_compression_paramsZ_iterator_paramsZ _reader_notesZ_read_stata_docr� r� r� ra rh r r� �Warningr� r� r� r� r� r r� r� r� r� r� r� �Iteratorr r� r> r� r� r� r� r� r� rb rc rd ru r� r/ r/ r/ r0 �<module> s� ( -*$ #k Ym # x * /, 4 6 }
| ver. 1.4 |
Github
|
.
| PHP 8.3.23 | Генерация страницы: 0.05 |
proxy
|
phpinfo
|
Настройка