Файловый менеджер - Редактировать - /home/digitalm/venv/lib/python3.7/site-packages/pandas/io/__pycache__/pytables.cpython-37.pyc
Назад
B �5�g�� � @ s� d Z ddlmZ ddlmZ ddlZddlmZmZ ddl Z ddl Z ddl Z ddl m Z ddlmZmZmZmZmZmZ ddlZddlZddlmZmZ dd lm Z m Z dd l m Z dd l!m"Z"m#Z#m$Z$m%Z%m&Z& dd l'm(Z( dd l)m*Z* ddl+m,Z, ddl-m.Z. ddl/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8m9Z9 ddl:m;Z; ddl<m=Z=m>Z>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZF ddlGmHZHmIZImJZJ ddlKmL mMZN ddlOmPZPmQZQ ddlRmSZS ddlTmUZU ddlVmWZWmXZX ddlYmZZZ ddl[m\Z\m]Z] e�r ddl^m_Z_m`Z`maZa ddlVmbZb d Zcd Zdd d � Zed d!� Zfd"d#� ZgePZhd$d%�d&d'�ZiG d(d)� d)ej�ZkG d*d+� d+ej�ZlG d,d-� d-em�Znd.ZoG d/d0� d0em�Zpd1ZqG d2d3� d3em�Zrd4Zsd5Ztd6d6d7d7d8�Zue=dgiZvd9Zwd:Zxe�yd;��8 ejzd<d=ewej{d>� ejzd?dexe�|d6d7dg�d>� W dQ R X da}d=a~d@dA� Zd�dEdFdEdGdHdIdHdIdJdKdLdEdEdM� dNdO�Z�d�dEdEdGdGdGdQ�dRdS�Z�dTdTdIdU�dVdW�Z�G dXdY� dY�Z�G dZd[� d[�Z�G d\d]� d]�Z�G d^d_� d_e��Z�G d`da� dae��Z�G dbdc� dce��Z�G ddde� dee��Z�G dfdg� dg�Z�G dhdi� die��Z�G djdk� dke��Z�G dldm� dme��Z�G dndo� doe��Z�G dpdq� dqe��Z�G drds� dse��Z�G dtdu� due��Z�G dvdw� dwe��Z�G dxdy� dye��Z�G dzd{� d{e��Z�G d|d}� d}e��Z�G d~d� de��Z�d�d�d$d�d�d��d�d��Z�d�d�d��d�d��Z�d�d�d�dId�d��d�d��Z�dEd�dEdEd]d��d�d��Z�dEdEdEd�d��d�d��Z�dEd�d�d��d�d��Z�d�dEdEd�d��d�d��Z�d�dEdEd�d��d�d��Z�d�dEdEdEd��d�d��Z�dEdEdEd��d�d��Z�dEdId��d�d��Z�dEd�dEd��d�d��Z�dEdEd��d�d��Z�d�d��d�d��Z�G d�d�� d��Z�dS )�zY High level interface to PyTables for reading and writing pandas data structures to disk � )� annotations)�suppressN)�date�tzinfo)�dedent)� TYPE_CHECKING�Any�Callable�Hashable�Sequence�cast)�config� get_option)�lib�writers)� timezones)� ArrayLike�DtypeArg� FrameOrSeries�FrameOrSeriesUnion�Shape)�import_optional_dependency)� patch_pickle)�PerformanceWarning)�cache_readonly) � ensure_object�is_categorical_dtype�is_complex_dtype�is_datetime64_dtype�is_datetime64tz_dtype�is_extension_array_dtype� is_list_like�is_string_dtype�is_timedelta64_dtype�needs_i8_conversion)�array_equivalent) � DataFrame� DatetimeIndex�Index� Int64Index� MultiIndex� PeriodIndex�Series�TimedeltaIndex�concat�isna)� Categorical� DatetimeArray� PeriodArray)� PyTablesExpr�maybe_expression)� extract_array)� ensure_index)� ArrayManager� BlockManager)�stringify_path)�adjoin� pprint_thing)�Col�File�Node)�Blockz0.15.2�UTF-8c C s t | tj�r| �d�} | S )z(if we have bytes, decode them to unicodezUTF-8)� isinstance�np�bytes_�decode)�s� rF �H/home/digitalm-up/venv/lib/python3.7/site-packages/pandas/io/pytables.py�_ensure_decodedu s rH c C s | d kr t } | S )N)�_default_encoding)�encodingrF rF rG �_ensure_encoding| s rK c C s t | t�rt| �} | S )z� Ensure that an index / column name is a str (python 3); otherwise they may be np.string dtype. Non-string dtypes are passed through unchanged. https://github.com/pandas-dev/pandas/issues/13492 )rA �str)�namerF rF rG � _ensure_str� s rN �int)� scope_levelc sV |d � t | ttf�r*� fdd�| D �} nt| �r>t| � d�} | dksNt| �rR| S dS )z� Ensure that the where is a Term or a list of Term. This makes sure that we are capturing the scope of variables that are passed create the terms here with a frame_level=2 (we are 2 levels down) � c s0 g | ](}|d k rt |�r(t|� d d�n|�qS )NrQ )rP )r4 �Term)�.0�term)�levelrF rG � <listcomp>� s z _ensure_term.<locals>.<listcomp>)rP N)rA �list�tupler4 rR �len)�whererP rF )rU rG � _ensure_term� s r[ c @ s e Zd ZdS )�PossibleDataLossErrorN)�__name__� __module__� __qualname__rF rF rF rG r\ � s r\ c @ s e Zd ZdS )�ClosedFileErrorN)r] r^ r_ rF rF rF rG r` � s r` c @ s e Zd ZdS )�IncompatibilityWarningN)r] r^ r_ rF rF rF rG ra � s ra z� where criteria is being ignored as this version [%s] is too old (or not-defined), read the file in and write it out to a new file to upgrade (with the copy_to method) c @ s e Zd ZdS )�AttributeConflictWarningN)r] r^ r_ rF rF rF rG rb � s rb zu the [%s] attribute of the existing index is [%s] which conflicts with the new [%s], resetting the attribute to None c @ s e Zd ZdS )�DuplicateWarningN)r] r^ r_ rF rF rF rG rc � s rc z; duplicate entries in table, taking most recently appended z� your performance may suffer as PyTables will pickle object types that it cannot map directly to c-types [inferred_type->%s,key->%s] [items->%s] �fixed�table)�frd �tre z; : boolean drop ALL nan rows when appending to a table z~ : format default format writing format, if None, then put will default to 'fixed' and append will default to 'table' zio.hdfZ dropna_tableF)� validator�default_formatc C s8 t d kr4dd l} | a tt�� | jjdkaW d Q R X t S )Nr �strict)� _table_mod�tablesr �AttributeError�fileZ_FILE_OPEN_POLICY�!_table_file_open_policy_is_strict)rl rF rF rG �_tables� s rp �aTrj rL r z int | Nonez str | None�boolzint | dict[str, int] | Nonez bool | Nonezbool | list[str] | None) �key�value�mode� complevel�complib�append�format�index� min_itemsize�dropna� data_columns�errorsrJ c s� |r$� ��������� f dd�}n � ��������� f dd�}t | �} t| t�rzt| |||d��}||� W dQ R X n|| � dS )z+store this object, close it if we opened itc s | j �� ������ ��d� S )N)ry rz r{ �nan_repr| r} r~ rJ )rx )�store) r} r| rJ r~ ry rz rs r{ r rt rF rG �<lambda> s zto_hdf.<locals>.<lambda>c s | j �� ����� ���d� S )N)ry rz r{ r r} r~ rJ r| )�put)r� ) r} r| rJ r~ ry rz rs r{ r rt rF rG r� ( s )ru rv rw N)r9 rA rL �HDFStore)� path_or_bufrs rt ru rv rw rx ry rz r{ r r| r} r~ rJ rf r� rF ) r} r| rJ r~ ry rz rs r{ r rt rG �to_hdf s r� �r)ru r~ �start�stop� chunksizec K s� |dkrt d|� d���|dk r,t|dd�}t| t�rN| jsDtd��| } d} nvt| �} t| t�shtd ��yt j � | �} W n t t fk r� d} Y nX | s�t d | � d ���t| f||d �| ��} d } yz|dk�r&| �� }t|�dkr�t d��|d }x*|dd� D ]}t||��st d���qW |j}| j||||||| | d�S t t tfk �r� t| t��s~tt�� | �� W dQ R X � Y nX dS )a� Read from the store, close it if we opened it. Retrieve pandas object stored in file, optionally based on where criteria. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- path_or_buf : str, path object, pandas.HDFStore Any valid string path is acceptable. Only supports the local file system, remote URLs and file-like objects are not supported. If you want to pass in a path object, pandas accepts any ``os.PathLike``. Alternatively, pandas accepts an open :class:`pandas.HDFStore` object. key : object, optional The group identifier in the store. Can be omitted if the HDF file contains a single pandas object. mode : {'r', 'r+', 'a'}, default 'r' Mode to use when opening the file. Ignored if path_or_buf is a :class:`pandas.HDFStore`. Default is 'r'. errors : str, default 'strict' Specifies how encoding and decoding errors are to be handled. See the errors argument for :func:`open` for a full list of options. where : list, optional A list of Term (or convertible) objects. start : int, optional Row number to start selection. stop : int, optional Row number to stop selection. columns : list, optional A list of columns names to return. iterator : bool, optional Return an iterator object. chunksize : int, optional Number of rows to include in an iteration when using an iterator. **kwargs Additional keyword arguments passed to HDFStore. Returns ------- item : object The selected object. Return type depends on the object stored. See Also -------- DataFrame.to_hdf : Write a HDF file from a DataFrame. HDFStore : Low-level access to HDF files. Examples -------- >>> df = pd.DataFrame([[1, 1.0, 'a']], columns=['x', 'y', 'z']) >>> df.to_hdf('./store.h5', 'data') >>> reread = pd.read_hdf('./store.h5') )r� zr+rq zmode zG is not allowed while performing a read. Allowed modes are r, r+ and a.NrQ )rP z&The HDFStore must be open for reading.Fz5Support for generic buffers has not been implemented.zFile z does not exist)ru r~ Tr z]Dataset(s) incompatible with Pandas data types, not table, or no datasets found in HDF5 file.z?key must be provided when HDF5 file contains multiple datasets.)rZ r� r� �columns�iteratorr� � auto_close)� ValueErrorr[ rA r� �is_open�OSErrorr9 rL �NotImplementedError�os�path�exists� TypeError�FileNotFoundError�groupsrY �_is_metadata_of� _v_pathname�select�KeyErrorr rm �close)r� rs ru r~ rZ r� r� r� r� r� �kwargsr� r� r� r� Zcandidate_only_groupZgroup_to_checkrF rF rG �read_hdf? s` O r� r> )�group� parent_group�returnc C sJ | j |j krdS | }x0|j dkrD|j}||kr<|jdkr<dS |j}qW dS )zDCheck if a given group is a metadata group for a given parent_group.FrQ �metaT)Z_v_depthZ _v_parent�_v_name)r� r� �current�parentrF rF rG r� � s r� c @ s� e Zd ZU dZded< ded< ded< ded < d�dd dd�dd�Zdd� Zedd� �Zedd� �Z dd�dd�Z dd�dd�Z dd�d d �Z dd �d d �Z ddd!�d"d#�Zdd$�d%d&�Zdd$�d'd(�Zd)d*� Zd+d,� Zd�dd.d/�d0d1�Zd2d3� Zd4d5� ZeZd�dd6�d7d8�Zd9d:� Zedd$�d;d<��Zd�dd=�d>d?�Zdd�d@dA�Zd�dddB�dCdD�Z d�dd d dE�dFdG�Z d�ddd d dH�dIdJ�Z d�ddK�dLdM�Z d�ddPd dQdRddddS�dTdU�Z d�dd�dVdW�Z!d�ddPd dQdXdRddY�dZd[�Z"d�d\d]�d^d_�Z#d�dd d`da�dbdc�Z$ddde� Z%d�dgdh�Z&ddid!�djdk�Z'ddld!�dmdn�Z(d�dd ddp�dqdr�Z)dd$�dsdt�Z*dudv� Z+dddw�dxdy�Z,d�d{dddld|�d}d~�Z-d�ddPd dQddd�d�d��Z.d�d��d�d��Z/ddd�d��d�d��Z0dd�d!�d�d��Z1d S )�r� aa Dict-like IO interface for storing pandas objects in PyTables. Either Fixed or Table format. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- path : str File path to HDF5 file. mode : {'a', 'w', 'r', 'r+'}, default 'a' ``'r'`` Read-only; no data can be modified. ``'w'`` Write; a new file is created (an existing file with the same name would be deleted). ``'a'`` Append; an existing file is opened for reading and writing, and if the file does not exist it is created. ``'r+'`` It is similar to ``'a'``, but the file must already exist. complevel : int, 0-9, default None Specifies a compression level for data. A value of 0 or None disables compression. complib : {'zlib', 'lzo', 'bzip2', 'blosc'}, default 'zlib' Specifies the compression library to be used. As of v0.20.2 these additional compressors for Blosc are supported (default if no compressor specified: 'blosc:blosclz'): {'blosc:blosclz', 'blosc:lz4', 'blosc:lz4hc', 'blosc:snappy', 'blosc:zlib', 'blosc:zstd'}. Specifying a compression library which is not available issues a ValueError. fletcher32 : bool, default False If applying compression use the fletcher32 checksum. **kwargs These parameters will be passed to the PyTables open_file method. Examples -------- >>> bar = pd.DataFrame(np.random.randn(10, 4)) >>> store = pd.HDFStore('test.h5') >>> store['foo'] = bar # write to HDF5 >>> bar = store['foo'] # retrieve >>> store.close() **Create or load HDF5 file in-memory** When passing the `driver` option to the PyTables open_file method through **kwargs, the HDF5 file is loaded or created in-memory and will only be written when closed: >>> bar = pd.DataFrame(np.random.randn(10, 4)) >>> store = pd.HDFStore('test.h5', driver='H5FD_CORE') >>> store['foo'] = bar >>> store.close() # only now, data is written to disk z File | None�_handlerL �_moderO � _complevelrr � _fletcher32rq NFz int | None)ru rv � fletcher32c K s� d|krt d��td�}|d k r@||jjkr@t d|jj� d���|d krX|d k rX|jj}t|�| _|d krnd}|| _d | _|r�|nd| _ || _ || _ d | _ | j f d|i|�� d S ) Nry z-format is not a defined argument for HDFStorerl zcomplib only supports z compression.rq r ru )r� r �filtersZ all_complibsZdefault_complibr9 �_pathr� r� r� �_complibr� �_filters�open)�selfr� ru rv rw r� r� rl rF rF rG �__init__/ s$ zHDFStore.__init__c C s | j S )N)r� )r� rF rF rG � __fspath__Q s zHDFStore.__fspath__c C s | � � | jdk st�| jjS )zreturn the root nodeN)�_check_if_openr� �AssertionError�root)r� rF rF rG r� T s z HDFStore.rootc C s | j S )N)r� )r� rF rF rG �filename[ s zHDFStore.filename)rs c C s | � |�S )N)�get)r� rs rF rF rG � __getitem___ s zHDFStore.__getitem__c C s | � ||� d S )N)r� )r� rs rt rF rF rG � __setitem__b s zHDFStore.__setitem__c C s | � |�S )N)�remove)r� rs rF rF rG � __delitem__e s zHDFStore.__delitem__)rM c C sD y | � |�S ttfk r" Y nX tdt| �j� d|� d���dS )z$allow attribute access to get stores�'z' object has no attribute 'N)r� r� r` rm �typer] )r� rM rF rF rG � __getattr__h s zHDFStore.__getattr__)rs r� c C s8 | � |�}|dk r4|j}||ks0|dd� |kr4dS dS )zx check for existence of this key can match the exact pathname or the pathnm w/o the leading '/' NrQ TF)�get_noder� )r� rs �noderM rF rF rG � __contains__r s zHDFStore.__contains__)r� c C s t | �� �S )N)rY r� )r� rF rF rG �__len__~ s zHDFStore.__len__c C s t | j�}t| �� d|� d�S )Nz File path: � )r; r� r� )r� �pstrrF rF rG �__repr__� s zHDFStore.__repr__c C s | S )NrF )r� rF rF rG � __enter__� s zHDFStore.__enter__c C s | � � d S )N)r� )r� �exc_type� exc_value� tracebackrF rF rG �__exit__� s zHDFStore.__exit__�pandasz list[str])�includer� c C s^ |dkrdd� | � � D �S |dkrJ| jdk s0t�dd� | jjddd �D �S td |� d ���dS ) a# Return a list of keys corresponding to objects stored in HDFStore. Parameters ---------- include : str, default 'pandas' When kind equals 'pandas' return pandas objects. When kind equals 'native' return native HDF5 Table objects. .. versionadded:: 1.1.0 Returns ------- list List of ABSOLUTE path-names (e.g. have the leading '/'). Raises ------ raises ValueError if kind has an illegal value r� c S s g | ] }|j �qS rF )r� )rS �nrF rF rG rV � s z!HDFStore.keys.<locals>.<listcomp>�nativeNc S s g | ] }|j �qS rF )r� )rS r� rF rF rG rV � s �/�Table)� classnamez8`include` should be either 'pandas' or 'native' but is 'r� )r� r� r� Z walk_nodesr� )r� r� rF rF rG �keys� s z HDFStore.keysc C s t | �� �S )N)�iterr� )r� rF rF rG �__iter__� s zHDFStore.__iter__c c s" x | � � D ]}|j|fV q W dS )z' iterate on key->group N)r� r� )r� �grF rF rG �items� s zHDFStore.items)ru c K s� t � }| j|krR| jdkr$|dkr$n(|dkrL| jrLtd| j� d| j� d���|| _| jr`| �� | jr�| jdkr�t � j| j| j| j d�| _ t r�| jr�d }t |��|j | j| jf|�| _d S ) a9 Open the file in the specified mode Parameters ---------- mode : {'a', 'w', 'r', 'r+'}, default 'a' See HDFStore docstring or tables.open_file for info about modes **kwargs These parameters will be passed to the PyTables open_file method. )rq �w)r� zr+)r� zRe-opening the file [z ] with mode [z ] will delete the current file!r )r� zGCannot open HDF5 file, which is already opened, even in read-only mode.N)rp r� r� r\ r� r� r� �Filtersr� r� r� ro r� � open_filer� )r� ru r� rl �msgrF rF rG r� � s$ z HDFStore.openc C s | j dk r| j �� d| _ dS )z0 Close the PyTables file handle N)r� r� )r� rF rF rG r� � s zHDFStore.closec C s | j dkrdS t| j j�S )zF return a boolean indicating whether the file is open NF)r� rr Zisopen)r� rF rF rG r� � s zHDFStore.is_open)�fsyncc C s@ | j dk r<| j �� |r<tt�� t�| j �� � W dQ R X dS )a� Force all buffered modifications to be written to disk. Parameters ---------- fsync : bool (default False) call ``os.fsync()`` on the file handle to force writing to disk. Notes ----- Without ``fsync=True``, flushing may not guarantee that the OS writes to disk. With fsync, the operation will block until the OS claims the file has been written; however, other caching layers may still interfere. N)r� �flushr r� r� r� �fileno)r� r� rF rF rG r� � s zHDFStore.flushc C s> t � �. | �|�}|dkr*td|� d���| �|�S Q R X dS )z� Retrieve pandas object stored in file. Parameters ---------- key : str Returns ------- object Same type as object stored in file. NzNo object named z in the file)r r� r� � _read_group)r� rs r� rF rF rG r� s z HDFStore.get)rs r� c st | � |�} | dkr"td|� d���t|dd�}| �| ����� � �fdd�} t| �| |�j|||||d� } | �� S ) a� Retrieve pandas object stored in file, optionally based on where criteria. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- key : str Object being retrieved from file. where : list or None List of Term (or convertible) objects, optional. start : int or None Row number to start selection. stop : int, default None Row number to stop selection. columns : list or None A list of columns that if not None, will limit the return columns. iterator : bool or False Returns an iterator. chunksize : int or None Number or rows to include in iteration, return an iterator. auto_close : bool or False Should automatically close the store when finished. Returns ------- object Retrieved object from file. NzNo object named z in the filerQ )rP c s �j | ||� d�S )N)r� r� rZ r� )�read)�_start�_stop�_where)r� rE rF rG �funcZ s z HDFStore.select.<locals>.func)rZ �nrowsr� r� r� r� r� )r� r� r[ �_create_storer� infer_axes� TableIteratorr� � get_result) r� rs rZ r� r� r� r� r� r� r� r� �itrF )r� rE rG r� " s& . zHDFStore.select)rs r� r� c C s8 t |dd�}| �|�}t|t�s(td��|j|||d�S )a� return the selection as an Index .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- key : str where : list of Term (or convertible) objects, optional start : integer (defaults to None), row number to start selection stop : integer (defaults to None), row number to stop selection rQ )rP z&can only read_coordinates with a table)rZ r� r� )r[ � get_storerrA r� r� �read_coordinates)r� rs rZ r� r� �tblrF rF rG �select_as_coordinatesm s z HDFStore.select_as_coordinates)rs �columnr� r� c C s, | � |�}t|t�s td��|j|||d�S )a~ return a single column from the table. This is generally only useful to select an indexable .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- key : str column : str The column of interest. start : int or None, default None stop : int or None, default None Raises ------ raises KeyError if the column is not found (or key is not a valid store) raises ValueError if the column can not be extracted individually (it is part of a data block) z!can only read_column with a table)r� r� r� )r� rA r� r� � read_column)r� rs r� r� r� r� rF rF rG � select_column� s # zHDFStore.select_column)r� c s~ t |dd�}t|ttf�r.t|�dkr.|d }t|t�rR�j||�||||| d�S t|ttf�shtd��t|�sxtd��|dkr�|d }�fdd �|D ���� |�} d} xxt � | |fgt �|��D ]\\} } | dkr�t d | � d ���| js�td | j� d ���| dk�r | j} q�| j| kr�td��q�W dd � �D �}tdd� |D ��d � � ��fdd�}t�| ||| ||||| d� }|jdd�S )a� Retrieve pandas objects from multiple tables. .. warning:: Pandas uses PyTables for reading and writing HDF5 files, which allows serializing object-dtype data with pickle when using the "fixed" format. Loading pickled data received from untrusted sources can be unsafe. See: https://docs.python.org/3/library/pickle.html for more. Parameters ---------- keys : a list of the tables selector : the table to apply the where criteria (defaults to keys[0] if not supplied) columns : the columns I want back start : integer (defaults to None), row number to start selection stop : integer (defaults to None), row number to stop selection iterator : bool, return an iterator, default False chunksize : nrows to include in iteration, return an iterator auto_close : bool, default False Should automatically close the store when finished. Raises ------ raises KeyError if keys or selector is not found or keys is empty raises TypeError if keys is not a list or tuple raises ValueError if the tables are not ALL THE SAME DIMENSIONS rQ )rP r )rs rZ r� r� r� r� r� r� zkeys must be a list/tuplez keys must have a non-zero lengthNc s g | ]}� � |��qS rF )r� )rS �k)r� rF rG rV � s z/HDFStore.select_as_multiple.<locals>.<listcomp>zInvalid table [�]zobject [z>] is not a table, and cannot be used in all select as multiplez,all tables must have exactly the same nrows!c S s g | ]}t |t�r|�qS rF )rA r� )rS �xrF rF rG rV s c S s h | ]}|j d d �qS )r )�non_index_axes)rS rg rF rF rG � <setcomp> s z.HDFStore.select_as_multiple.<locals>.<setcomp>c s* � ���fdd��D �}t |�dd��� S )Nc s g | ]}|j ��� �d ��qS ))rZ r� r� r� )r� )rS rg )r� r� r� r� rF rG rV s z=HDFStore.select_as_multiple.<locals>.func.<locals>.<listcomp>F)�axis�verify_integrity)r. � _consolidate)r� r� r� Zobjs)r� r� �tbls)r� r� r� rG r� s z)HDFStore.select_as_multiple.<locals>.func)rZ r� r� r� r� r� r� T)� coordinates)r[ rA rW rX rY rL r� r� r� r� � itertools�chain�zipr� �is_table�pathnamer� r� r� )r� r� rZ �selectorr� r� r� r� r� r� rE r� rg r� Z_tblsr� r� rF )r� r� r� r� rG �select_as_multiple� s^ + " zHDFStore.select_as_multipleTrj r zint | dict[str, int] | Nonezlist[str] | None)rs rt rv r{ r} r~ � track_timesr| c C sH |dkrt d�pd}| �|�}| j||||||||| | | | | |d� dS )a< Store object in HDFStore. Parameters ---------- key : str value : {Series, DataFrame} format : 'fixed(f)|table(t)', default is 'fixed' Format to use when storing object in HDFStore. Value can be one of: ``'fixed'`` Fixed format. Fast writing/reading. Not-appendable, nor searchable. ``'table'`` Table format. Write as a PyTables Table structure which may perform worse but allow more flexible operations like searching / selecting subsets of the data. append : bool, default False This will force Table format, append the input data to the existing. data_columns : list, default None List of columns to create as data columns, or True to use all columns. See `here <https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#query-via-data-columns>`__. encoding : str, default None Provide an encoding for strings. track_times : bool, default True Parameter is propagated to 'create_table' method of 'PyTables'. If set to False it enables to have the same h5 files (same hashes) independent on creation time. .. versionadded:: 1.1.0 Nzio.hdf.default_formatrd ) ry rz rx rw rv r{ r r} rJ r~ r r| )r �_validate_format�_write_to_group)r� rs rt ry rz rx rw rv r{ r r} rJ r~ r r| rF rF rG r� / s$ 0 z HDFStore.putc C s� t |dd�}y| �|�}W n~ tk r0 � Y nj tk rD � Y nV tk r� } z8|dk rftd�|�| �|�}|dk r�|jdd� dS W dd}~X Y nX t� |||�r�|j jdd� n |j s�td��|j |||d�S dS ) a: Remove pandas object partially by specifying the where condition Parameters ---------- key : str Node to remove or delete rows from where : list of Term (or convertible) objects, optional start : integer (defaults to None), row number to start selection stop : integer (defaults to None), row number to stop selection Returns ------- number of rows removed (or None if not a Table) Raises ------ raises KeyError if key is not a valid store rQ )rP Nz5trying to remove a node with a non-None where clause!T)� recursivez7can only remove with where on objects written as tables)rZ r� r� ) r[ r� r� r� � Exceptionr� r� Z _f_remove�com�all_noner� r� �delete)r� rs rZ r� r� rE �errr� rF rF rG r� s s, zHDFStore.removez bool | None)rs rt rv r{ r| r} r~ c C sl | dk rt d��|dkr td�}|dkr4td�p2d}| �|�}| j||||||||| | | | ||||d� dS )a6 Append to Table in file. Node must already exist and be Table format. Parameters ---------- key : str value : {Series, DataFrame} format : 'table' is the default Format to use when storing object in HDFStore. Value can be one of: ``'table'`` Table format. Write as a PyTables Table structure which may perform worse but allow more flexible operations like searching / selecting subsets of the data. append : bool, default True Append the input data to the existing. data_columns : list of columns, or True, default None List of columns to create as indexed data columns for on-disk queries, or True to use all columns. By default only the axes of the object are indexed. See `here <https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#query-via-data-columns>`__. min_itemsize : dict of columns that specify minimum str sizes nan_rep : str to use as str nan representation chunksize : size to chunk the writing expectedrows : expected TOTAL row size of this table encoding : default None, provide an encoding for str dropna : bool, default False Do not write an ALL nan row to the store settable by the option 'io.hdf.dropna_table'. Notes ----- Does *not* check if data being appended overlaps with existing data in the table, so be careful Nz>columns is not a supported keyword in append, try data_columnszio.hdf.dropna_tablezio.hdf.default_formatre )ry �axesrz rx rw rv r{ r r� � expectedrowsr| r} rJ r~ )r� r r r )r� rs rt ry r rz rx rw rv r� r{ r r� r r| r} rJ r~ rF rF rG rx � s2 8 zHDFStore.append�dict)�dc s� |dk rt d��t|t�s"td��||kr2td��ttt�j��ttt �� � �d }d} g } x<|� � D ]0\} � � dkr�| dk r�td��| } qj| � � � qjW | dk rڈj | } | � t| ��} t| �| ��} | �| �|| < |dkr�|| }|�r2�fdd�|�� D �}t|�}x|D ]}|�|�}�qW �j| �|�d d�}xt|� � D ]h\} � | |k�r^|nd}�j� |d �}|dk �r�� fd d �|� � D �nd}| j| |f||d �|�� �qHW dS )a Append to multiple tables Parameters ---------- d : a dict of table_name to table_columns, None is acceptable as the values of one node (this will get all the remaining columns) value : a pandas object selector : a string that designates the indexable table; all of its columns will be designed as data_columns, unless data_columns is passed, in which case these are used data_columns : list of columns to create as data columns, or True to use all columns dropna : if evaluates to True, drop rows from all tables if any single row in each table has all NaN. Default False. Notes ----- axes parameter is currently not accepted Nztaxes is currently not accepted as a parameter to append_to_multiple; you can create the tables independently insteadzQappend_to_multiple must have a dictionary specified as the way to split the valuez=append_to_multiple requires a selector that is in passed dictr z<append_to_multiple can only have one value in d that is Nonec 3 s | ]}� | j d d�jV qdS )�all)�howN)r| rz )rS �cols)rt rF rG � <genexpr>L s z.HDFStore.append_to_multiple.<locals>.<genexpr>r{ )r� c s i | ]\}}|� kr||�qS rF rF )rS rs rt )�vrF rG � <dictcomp>\ s z/HDFStore.append_to_multiple.<locals>.<dictcomp>)r} r{ )r� rA r r� rW �set�range�ndim� _AXES_MAPr� r� �extendr � differencer( �sorted� get_indexer�take�values�next� intersection�loc�pop�reindexrx )r� r rt r� r} r r| r� r� Z remain_keyZ remain_valuesr� �orderedZorddZidxsZ valid_indexrz r{ �dc�val�filteredrF )r rt rG �append_to_multiple sN & zHDFStore.append_to_multiplez str | None)rs �optlevel�kindc C sB t � | �|�}|dkr dS t|t�s.td��|j|||d� dS )a� Create a pytables index on the table. Parameters ---------- key : str columns : None, bool, or listlike[str] Indicate which columns to create an index on. * False : Do not create any indexes. * True : Create indexes on all columns. * None : Create indexes on all columns. * listlike : Create indexes on the given columns. optlevel : int or None, default None Optimization level, if None, pytables defaults to 6. kind : str or None, default None Kind of index, if None, pytables defaults to "medium". Raises ------ TypeError: raises if the node is not a table Nz1cannot create table index on a Fixed format store)r� r( r) )rp r� rA r� r� � create_index)r� rs r� r( r) rE rF rF rG �create_table_indexb s zHDFStore.create_table_indexc C s< t � | �� | jdk s t�tdk s(t�dd� | j�� D �S )z� Return a list of all the top-level nodes. Each node returned is not a pandas storage object. Returns ------- list List of objects. Nc S sP g | ]H}t |tjj�st|jd d�sHt|dd�sHt |tjj�r|jdkr|�qS )� pandas_typeNre ) rA rk �link�Link�getattr�_v_attrsre r� r� )rS r� rF rF rG rV � s z#HDFStore.groups.<locals>.<listcomp>)rp r� r� r� rk � walk_groups)r� rF rF rG r� � s zHDFStore.groupsr� c c s� t � | �� | jdk s t�tdk s(t�x�| j�|�D ]�}t|jdd�dk rNq6g }g }xP|j� � D ]B}t|jdd�}|dkr�t |tj j �r�|� |j� qb|� |j� qbW |j�d�||fV q6W dS )aS Walk the pytables group hierarchy for pandas objects. This generator will yield the group path, subgroups and pandas object names for each group. Any non-pandas PyTables objects that are not a group will be ignored. The `where` group itself is listed first (preorder), then each of its child groups (following an alphanumerical order) is also traversed, following the same procedure. Parameters ---------- where : str, default "/" Group where to start walking. Yields ------ path : str Full path to a group (without trailing '/'). groups : list Names (strings) of the groups contained in `path`. leaves : list Names (strings) of the pandas objects contained in `path`. Nr, r� )rp r� r� r� rk r1 r/ r0 Z _v_childrenr rA r� �Grouprx r� r� �rstrip)r� rZ r� r� �leaves�childr, rF rF rG �walk� s z HDFStore.walkz Node | Nonec C s~ | � � |�d�sd| }| jdk s(t�tdk s4t�y| j�| j|�}W n tjjk r` dS X t |tj �sztt |���|S )z9return the node with the key or None if it does not existr� N) r� � startswithr� r� rk r� r� � exceptionsZNoSuchNodeErrorrA r> r� )r� rs r� rF rF rG r� � s zHDFStore.get_nodezGenericFixed | Tablec C s8 | � |�}|dkr"td|� d���| �|�}|�� |S )z<return the storer object for a key, raise if not in the fileNzNo object named z in the file)r� r� r� r� )r� rs r� rE rF rF rG r� � s zHDFStore.get_storerr� )� propindexesrv r� c C s� t |||||d�} |dkr&t| �� �}t|ttf�s:|g}x�|D ]�} | �| �} | dk r@| | krl|rl| �| � | �| �} t| t�r�d} |r�dd� | j D �} | j | | | t | dd�| j d� q@| j | | | j d� q@W | S ) a; Copy the existing store to a new file, updating in place. Parameters ---------- propindexes : bool, default True Restore indexes in copied file. keys : list, optional List of keys to include in the copy (defaults to all). overwrite : bool, default True Whether to overwrite (remove and replace) existing nodes in the new store. mode, complib, complevel, fletcher32 same as in HDFStore.__init__ Returns ------- open file handle of the new store )ru rw rv r� NFc S s g | ]}|j r|j�qS rF )� is_indexedrM )rS rq rF rF rG rV s z!HDFStore.copy.<locals>.<listcomp>r} )rz r} rJ )rJ )r� rW r� rA rX r� r� r� r� r rx r/ rJ r� )r� rn ru r9 r� rw rv r� � overwriteZ new_storer� rE �datarz rF rF rG �copy� s2 z HDFStore.copyc C s t | j�}t| �� d|� d�}| j�rt| �� �}t|�r�g }g }x�|D ]�}y<| �|�}|dk r�|�t |j pn|�� |�t |p�d�� W qH t k r� � Y qH t k r� } z(|�|� t |�} |�d| � d�� W dd}~X Y qHX qHW |t d||�7 }n|d7 }n|d 7 }|S ) zg Print detailed information on the store. Returns ------- str z File path: r� Nzinvalid_HDFStore nodez[invalid_HDFStore node: r� � �EmptyzFile is CLOSED) r; r� r� r� r r� rY r� rx r� r� r r: ) r� r� �outputZlkeysr� r r� rE �detailZdstrrF rF rG �info- s. ( z HDFStore.infoc C s | j st| j� d���d S )Nz file is not open!)r� r` r� )r� rF rF rG r� W s zHDFStore._check_if_open)ry r� c C sJ yt |�� }W n4 tk rD } ztd|� d��|�W dd}~X Y nX |S )z validate / deprecate formatsz#invalid HDFStore format specified [r� N)� _FORMAT_MAP�lowerr� r� )r� ry r rF rF rG r [ s $zHDFStore._validate_format�UTF-8zFrameOrSeries | None)rt rJ r~ r� c s �dk r t �ttf�s td��� ��fdd�}tt�jdd��}tt�jdd��}|dkrƈdkr�t� tdk stt �t�dd�s�t �tj j �r�d}d }q�td ��n$t �t�r�d }nd }� dkr�|d 7 }d|k�r&t t d�} y | | } W n. tk �r } z|d�| �W dd} ~ X Y nX | | �||d�S |dk�r��dk �r�|dk�r~t�dd�} | dk �r�| jdk�rld}n| jdk�r�d}nB|dk�r�t�dd�} | dk �r�| jdk�r�d}n| jdk�r�d}ttttttd�} y | | } W n. tk �r } z|d�| �W dd} ~ X Y nX | | �||d�S )z"return a suitable class to operateNz(value must be None, Series, or DataFramec s$ t d| � d�� dt��� d� � ��S )Nz(cannot properly create the storer for: [z ] [group->z,value->z ,format->)r� r� )rg )ry r� rt rF rG �errors s z&HDFStore._create_storer.<locals>.errorr, � table_typere � frame_table� generic_tablezKcannot create a storer if the object is not existing nor a value are passed�series�frame�_table)rJ rK � _STORER_MAP)rJ r~ � series_tablerz rQ �appendable_series�appendable_multiseries�appendable_frame�appendable_multiframe)rI rO rP rQ rR �worm� _TABLE_MAP)rA r, r& r� rH r/ r0 rp rk r� re r� � SeriesFixed� FrameFixedr� �nlevels� GenericTable�AppendableSeriesTable�AppendableMultiSeriesTable�AppendableFrameTable�AppendableMultiFrameTable� WORMTable)r� r� ry rt rJ r~ rF �pt�ttrM �clsr rz rT rF )ry r� rt rG r� e sl zHDFStore._create_storer)rs rt rv r{ r~ r c C s� t |dd �r |dks|r d S | �||�}| j|||||d�}|rr|jrZ|jrb|dkrb|jrbtd��|jsz|�� n|�� |js�|r�td��|j|||||| | | | | |||d� t|t �r�|r�|j |d� d S ) N�emptyre )rJ r~ rd zCan only append to Tablesz0Compression not supported on Fixed format stores) �objr rx rw rv r� r{ r� r r| r r} r )r� ) r/ �_identify_groupr� r� � is_existsr� �set_object_info�writerA r� r* )r� rs rt ry r rz rx rw rv r� r{ r� r r| r r} rJ r~ r r� rE rF rF rG r � s8 zHDFStore._write_to_groupr> )r� c C s | � |�}|�� |�� S )N)r� r� r� )r� r� rE rF rF rG r� � s zHDFStore._read_group)rs rx r� c C sN | � |�}| jdk st�|dk r8|s8| jj|dd� d}|dkrJ| �|�}|S )z@Identify HDF5 group based on key, delete/create group if needed.NT)r )r� r� r� � remove_node�_create_nodes_and_group)r� rs rx r� rF rF rG rc s zHDFStore._identify_groupc C sz | j dk st�|�d�}d}xX|D ]P}t|�s0q"|}|�d�sF|d7 }||7 }| �|�}|dkrn| j �||�}|}q"W |S )z,Create nodes from key and return group name.Nr� )r� r� �splitrY �endswithr� Z create_group)r� rs �pathsr� �p�new_pathr� rF rF rG rh s z HDFStore._create_nodes_and_group)rq NNF)r� )rq )F)NNNNFNF)NNN)NN)NNNNNFNF) NTFNNNNNNrj TF)NNN)NNTTNNNNNNNNNNrj )NNF)NNN)r� )r� TNNNFT)NNrE rj )NTFNNNNNNFNNNrj T)2r] r^ r_ �__doc__�__annotations__r� r� �propertyr� r� r� r� r� r� r� r� r� r� r� r� r� r� � iteritemsr� r� r� r� r� r� r� r� r r� r� rx r' r+ r� r6 r� r� r= rB r� r r� r r� rc rh rF rF rF rG r� � s� @ "- E & t 5= H ] # 0 4* [ +r� c @ sb e Zd ZU dZded< ded< ded< dddd dd d �d d �Zdd� Zdd� Zdd d�dd�ZdS )r� aa Define the iteration interface on a table Parameters ---------- store : HDFStore s : the referred storer func : the function to execute the query where : the where of the query nrows : the rows to iterate on start : the passed start value (default is None) stop : the passed stop value (default is None) iterator : bool, default False Whether to use the default iterator. chunksize : the passed chunking value (default is 100000) auto_close : bool, default False Whether to automatically close the store at the end of iteration. z int | Noner� r� r� zGenericFixed | TablerE NFrr )r� rE r� r� r� c C s� || _ || _|| _|| _| jjrN|d kr,d}|d kr8d}|d krD|}t||�}|| _|| _|| _d | _ |sr| d k r�| d kr~d} t | �| _ nd | _ | | _ d S )Nr i�� ) r� rE r� rZ r� �minr� r� r� r� rO r� r� ) r� r� rE r� rZ r� r� r� r� r� r� rF rF rG r� D s, zTableIterator.__init__c c sz | j }| jd krtd��xT|| jk rlt|| j | j�}| �d d | j||� �}|}|d kst|�sdq|V qW | �� d S )Nz*Cannot iterate until get_result is called.) r� r� r� r� rr r� r� rY r� )r� r� r� rt rF rF rG r� n s zTableIterator.__iter__c C s | j r| j�� d S )N)r� r� r� )r� rF rF rG r� ~ s zTableIterator.close)r� c C s� | j d k r4t| jt�s td��| jj| jd�| _| S |rft| jt�sLtd��| jj| j| j| j d�}n| j}| � | j| j |�}| � � |S )Nz0can only use an iterator or chunksize on a table)rZ z$can only read_coordinates on a table)rZ r� r� ) r� rA rE r� r� r� rZ r� r� r� r� r� )r� r� rZ �resultsrF rF rG r� � s zTableIterator.get_result)NNFNF)F) r] r^ r_ rn ro r� r� r� r� rF rF rF rG r� , s r� c @ sv e Zd ZU dZdZdZdddgZded< ded< dJdd d �d d �Ze dd�dd��Z e dd�dd��Z dd�dd�Z dd�dd�Z ddd�d d �Zdd�d d �Ze dd�d d!��Zd"ddd#�d$d%�Zd&d'� Ze d(d)� �Ze d*d+� �Ze d,d-� �Ze d.d/� �Zd0d1� ZdKd2d3�Zd4d5� Zd6dd7�d8d9�ZdLd:d;�Zdd<�d=d>�Z d?d@� Z dAdB� Z dCdD� Z d6dE�dFdG�Z d6dE�dHdI�Z!d S )M�IndexCola an index column description class Parameters ---------- axis : axis which I reference values : the ndarray like converted values kind : a string description of this type typ : the pytables type pos : the position in the pytables T�freq�tz� index_namerL rM �cnameNz str | None)rM rx c C s� t |t�std��|| _|| _|| _|| _|p0|| _|| _|| _ || _ | | _ | | _ | | _ | | _| | _|| _|d k r|| �|� t | jt�s�t�t | jt�s�t�d S )Nz`name` must be a str.)rA rL r� r r) �typrM rx r� �posru rv rw r# re r� �metadata�set_posr� )r� rM r r) ry rx r� rz ru rv rw r# re r� r{ rF rF rG r� � s( zIndexCol.__init__rO )r� c C s | j jS )N)ry �itemsize)r� rF rF rG r} � s zIndexCol.itemsizec C s | j � d�S )N�_kind)rM )r� rF rF rG � kind_attr� s zIndexCol.kind_attr)rz c C s$ || _ |dk r | jdk r || j_dS )z,set the position of this column in the TableN)rz ry Z_v_pos)r� rz rF rF rG r| � s zIndexCol.set_posc C sF t tt| j| j| j| j| jf��}d�dd� t dddddg|�D ��S ) N�,c s s | ]\}}|� d |� �V qdS )z->NrF )rS rs rt rF rF rG r � s z$IndexCol.__repr__.<locals>.<genexpr>rM rx r� rz r) ) rX �mapr; rM rx r� rz r) �joinr� )r� �temprF rF rG r� � s zIndexCol.__repr__r rr )�otherr� c s t � �fdd�dD ��S )zcompare 2 col itemsc 3 s& | ] }t �|d �t � |d �kV qd S )N)r/ )rS rq )r� r� rF rG r � s z"IndexCol.__eq__.<locals>.<genexpr>)rM rx r� rz )r )r� r� rF )r� r� rG �__eq__� s zIndexCol.__eq__c C s | � |� S )N)r� )r� r� rF rF rG �__ne__� s zIndexCol.__ne__c C s" t | jd�sdS t| jj| j�jS )z%return whether I am an indexed columnr F)�hasattrre r/ r rx r: )r� rF rF rG r: s zIndexCol.is_indexedz np.ndarray)r rJ r~ c C s� t |tj�stt|���|jjdk r.|| j }t| j �}t ||||�}i }t| j �|d< | j dk rpt| j �|d< t }t|j�s�t|j�r�t}y||f|�}W n0 tk r� d|kr�d|d< ||f|�}Y nX t|| j�}||fS )zV Convert the data from this selection to the appropriate pandas type. NrM ru )rA rB �ndarrayr� r� �dtype�fieldsrx rH r) �_maybe_convertrw ru r( r r r' r� �_set_tzrv ) r� r r rJ r~ �val_kindr� �factoryZ new_pd_indexrF rF rG �convert s( zIndexCol.convertc C s | j S )zreturn the values)r )r� rF rF rG � take_data/ s zIndexCol.take_datac C s | j jS )N)re r0 )r� rF rF rG �attrs3 s zIndexCol.attrsc C s | j jS )N)re � description)r� rF rF rG r� 7 s zIndexCol.descriptionc C s t | j| jd�S )z!return my current col descriptionN)r/ r� rx )r� rF rF rG �col; s z IndexCol.colc C s | j S )zreturn my cython values)r )r� rF rF rG �cvalues@ s zIndexCol.cvaluesc C s t | j�S )N)r� r )r� rF rF rG r� E s zIndexCol.__iter__c C sP t | j�dkrLt|t�r$|�| j�}|dk rL| jj|k rLt� j || j d�| _dS )z� maybe set a string col itemsize: min_itemsize can be an integer or a dict with this columns name with an integer size �stringN)r} rz ) rH r) rA r r� rM ry r} rp � StringColrz )r� r{ rF rF rG �maybe_set_sizeH s zIndexCol.maybe_set_sizec C s d S )NrF )r� rF rF rG �validate_namesU s zIndexCol.validate_names�AppendableTable)�handlerrx c C s: |j | _ | �� | �|� | �|� | �|� | �� d S )N)re � validate_col� validate_attr�validate_metadata�write_metadata�set_attr)r� r� rx rF rF rG �validate_and_setX s zIndexCol.validate_and_setc C s^ t | j�dkrZ| j}|dk rZ|dkr*| j}|j|k rTtd|� d| j� d|j� d���|jS dS )z:validate this column: return the compared against itemsizer� Nz#Trying to store a string with len [z] in [z)] column but this column has a limit of [zC]! Consider using min_itemsize to preset the sizes on these columns)rH r) r� r} r� rx )r� r} �crF rF rG r� ` s zIndexCol.validate_col)rx c C sB |r>t | j| jd �}|d k r>|| jkr>td|� d| j� d���d S )Nzincompatible kind in col [z - r� )r/ r� r r) r� )r� rx Z existing_kindrF rF rG r� s s zIndexCol.validate_attrc C s� x�| j D ]�}t| |d�}|�| ji �}|�|�}||kr�|dk r�||kr�|dkr�t|||f }tj|tdd� d||< t | |d� q�t d| j� d|� d|� d|� d � ��q|dk s�|dk r|||< qW dS ) z set/update the info for this indexable with the key/value if there is a conflict raise/warn as needed N)ru rw � )� stacklevelzinvalid info for [z] for [z], existing_value [z ] conflicts with new value [r� ) � _info_fieldsr/ � setdefaultrM r� �attribute_conflict_doc�warnings�warnrb �setattrr� )r� rB rs rt �idxZexisting_value�wsrF rF rG � update_info| s $zIndexCol.update_infoc C s$ |� | j�}|dk r | j�|� dS )z!set my state from the passed infoN)r� rM �__dict__�update)r� rB r� rF rF rG �set_info� s zIndexCol.set_infoc C s t | j| j| j� dS )z set the kind for this columnN)r� r� r r) )r� rF rF rG r� � s zIndexCol.set_attr)r� c C sB | j dkr>| j}|�| j�}|dk r>|dk r>t||�s>td��dS )z:validate that kind=category does not change the categories�categoryNzEcannot append a categorical with different categories to the existing)r� r{ � read_metadatarx r% r� )r� r� Z new_metadataZ cur_metadatarF rF rG r� � s zIndexCol.validate_metadatac C s | j dk r|�| j| j � dS )zset the meta dataN)r{ r� rx )r� r� rF rF rG r� � s zIndexCol.write_metadata) NNNNNNNNNNNNN)N)N)"r] r^ r_ rn �is_an_indexable�is_data_indexabler� ro r� rp r} r r| r� r� r� r: r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� r� rF rF rF rG rt � sV & rt c @ s< e Zd ZdZedd�dd��Zdddd�d d �Zd d � Zd S )�GenericIndexColz:an index which is not represented in the data of the tablerr )r� c C s dS )NFrF )r� rF rF rG r: � s zGenericIndexCol.is_indexedz np.ndarrayrL )r rJ r~ c C s2 t |tj�stt|���tt�t|���}||fS )z� Convert the data from this selection to the appropriate pandas type. Parameters ---------- values : np.ndarray nan_rep : str encoding : str errors : str )rA rB r� r� r� r) �arangerY )r� r r rJ r~ rF rF rG r� � s zGenericIndexCol.convertc C s d S )NrF )r� rF rF rG r� � s zGenericIndexCol.set_attrN)r] r^ r_ rn rp r: r� r� rF rF rF rG r� � s r� c s, e Zd ZdZdZdZddgZd9ddd�� fd d � Zedd �d d ��Z edd �dd��Z dd �dd�Z ddd�dd�Z dd�dd�Z dd � Zedd d �d d ��Zed!d"� �Zedd#d$�d%d&��Zedd d$�d'd(��Zed)d*� �Zed+d,� �Zed-d.� �Zed/d0� �Zd1d2� Zd3ddd4�d5d6�Zd7d8� Z� ZS ):�DataCola3 a data holding column, by definition this is not indexable Parameters ---------- data : the actual data cname : the column name in the table to hold the data (typically values) meta : a string description of the metadata metadata : the actual metadata Frv r# NrL zDtypeArg | None)rM r� c s2 t � j||||||||| | | d� | | _| | _d S )N) rM r r) ry rz rx rv r# re r� r{ )�superr� r� r< )r� rM r r) ry rx rz rv r# re r� r{ r� r< )� __class__rF rG r� � s zDataCol.__init__)r� c C s | j � d�S )N�_dtype)rM )r� rF rF rG � dtype_attr s zDataCol.dtype_attrc C s | j � d�S )N�_meta)rM )r� rF rF rG � meta_attr s zDataCol.meta_attrc C sF t tt| j| j| j| j| jf��}d�dd� t dddddg|�D ��S ) Nr� c s s | ]\}}|� d |� �V qdS )z->NrF )rS rs rt rF rF rG r s z#DataCol.__repr__.<locals>.<genexpr>rM rx r� r) �shape) rX r� r; rM rx r� r) r� r� r� )r� r� rF rF rG r� s zDataCol.__repr__r rr )r� r� c s t � �fdd�dD ��S )zcompare 2 col itemsc 3 s& | ] }t �|d �t � |d �kV qd S )N)r/ )rS rq )r� r� rF rG r s z!DataCol.__eq__.<locals>.<genexpr>)rM rx r� rz )r )r� r� rF )r� r� rG r� s zDataCol.__eq__r )r< c C s@ |d k s t �| jd kst �t|�\}}|| _|| _t|�| _d S )N)r� r� �_get_data_and_dtype_namer< �_dtype_to_kindr) )r� r< � dtype_namerF rF rG �set_data$ s zDataCol.set_datac C s | j S )zreturn the data)r< )r� rF rF rG r� . s zDataCol.take_datar< )r r� c C s� |j }|j}|j}|jdkr&d|jf}t|t�rJ|j}| j||j j d�}ntt |�sZt |�rf| � |�}nXt |�rz| �|�}nDt|�r�t� j||d d�}n&t|�r�| �||�}n| j||j d�}|S )zW Get an appropriately typed and shaped pytables.Col object for values. rQ )r) r )r} r� )r� r} r� r �sizerA r0 �codes� get_atom_datarM r r �get_atom_datetime64r# �get_atom_timedelta64r rp Z ComplexColr" �get_atom_string)r` r r� r} r� r� �atomrF rF rG � _get_atom2 s$ zDataCol._get_atomc C s t � j||d d�S )Nr )r} r� )rp r� )r` r� r} rF rF rG r� R s zDataCol.get_atom_stringz type[Col])r) r� c C sR |� d�r$|dd� }d|� d�}n"|� d�r4d}n|�� }|� d�}tt� |�S )z0return the PyTables column class for this column�uint� NZUIntr< �period�Int64Col)r7 � capitalizer/ rp )r` r) Zk4Zcol_nameZkcaprF rF rG �get_atom_coltypeV s zDataCol.get_atom_coltypec C s | j |d�|d d�S )N)r) r )r� )r� )r` r� r) rF rF rG r� e s zDataCol.get_atom_datac C s t � j|d d�S )Nr )r� )rp r� )r` r� rF rF rG r� i s zDataCol.get_atom_datetime64c C s t � j|d d�S )Nr )r� )rp r� )r` r� rF rF rG r� m s z DataCol.get_atom_timedelta64c C s t | jdd �S )Nr� )r/ r< )r� rF rF rG r� q s z DataCol.shapec C s | j S )zreturn my cython values)r< )r� rF rF rG r� u s zDataCol.cvaluesc C s` |r\t | j| jd�}|dk r2|t| j�kr2td��t | j| jd�}|dk r\|| jkr\td��dS )zAvalidate that we have the same order as the existing & same dtypeNz4appended items do not match existing items in table!z@appended items dtype do not match existing items dtype in table!)r/ r� r rW r r� r� r� )r� rx Zexisting_fieldsZexisting_dtyperF rF rG r� z s zDataCol.validate_attrz np.ndarray)r rJ r~ c C s t |tj�stt|���|jjdk r.|| j }| jdk s<t�| jdkr\t |�\}}t |�}n|}| j}| j }t |tj�s|t�t | j �}| j} | j} | j} |dk s�t�t |�} | dkr�t|| dd�}�n(| dkr�tj|dd�}�n| dk�r8y tjd d � |D �td�}W n. tk �r4 tjd d � |D �td�}Y nX n�|d k�r�| } |�� }| dk�rhtg tjd�} n<t| �}|�� �r�| | } ||d k |�t�� � j 8 < t j || | d�}n8y|j| dd�}W n$ t k �r� |jddd�}Y nX t |�dk�r t!||||d�}| j"|fS )aR Convert the data from this selection to the appropriate pandas type. Parameters ---------- values : np.ndarray nan_rep : encoding : str errors : str Returns ------- index : listlike to become an Index data : ndarraylike to become a column N� datetime64T)�coerce� timedelta64zm8[ns])r� r c S s g | ]}t �|��qS rF )r � fromordinal)rS r rF rF rG rV � s z#DataCol.convert.<locals>.<listcomp>c S s g | ]}t �|��qS rF )r � fromtimestamp)rS r rF rF rG rV � s r� ���)� categoriesr# F)r= �Or� )r rJ r~ )#rA rB r� r� r� r� r� rx ry r� r� r) rH r� r{ r# rv r� �asarray�objectr� �ravelr( Zfloat64r/ �any�astyperO Zcumsum�_valuesr0 Z from_codesr� �_unconvert_string_arrayr )r� r r rJ r~ � convertedr� r) r� r{ r# rv r� r� r� �maskrF rF rG r� � s^ zDataCol.convertc C sH t | j| j| j� t | j| j| j� | jdk s2t�t | j| j| j� dS )z set the data for this columnN) r� r� r r r� r� r� r� r� )r� rF rF rG r� � s zDataCol.set_attr) NNNNNNNNNNNN) r] r^ r_ rn r� r� r� r� rp r� r� r� r� r� r� � classmethodr� r� r� r� r� r� r� r� r� r� r� � __classcell__rF rF )r� rG r� � sB er� c @ sT e Zd ZdZdZdd� Zedd� �Zeddd �d d ��Zed d � �Z edd� �Z dS )�DataIndexableColz+represent a data column that can be indexedTc C s t | j��� std��d S )Nz-cannot have non-object label DataIndexableCol)r( r Z is_objectr� )r� rF rF rG r� � s z DataIndexableCol.validate_namesc C s t � j|d�S )N)r} )rp r� )r` r� r} rF rF rG r� � s z DataIndexableCol.get_atom_stringrL r< )r) r� c C s | j |d�� S )N)r) )r� )r` r� r) rF rF rG r� s z DataIndexableCol.get_atom_datac C s t � �� S )N)rp r� )r` r� rF rF rG r� s z$DataIndexableCol.get_atom_datetime64c C s t � �� S )N)rp r� )r` r� rF rF rG r� s z%DataIndexableCol.get_atom_timedelta64N) r] r^ r_ rn r� r� r� r� r� r� r� rF rF rF rG r� � s r� c @ s e Zd ZdZdS )�GenericDataIndexableColz(represent a generic pytables data columnN)r] r^ r_ rn rF rF rF rG r� s r� c @ s� e Zd ZU dZded< dZded< ded< ded < ded < d ed < d ed< ded< dZdLd d ddd�dd�Zedd�dd��Z edd�dd ��Z ed d � �Z dd�d d �Z d!d"� Z d#d$� Zed%d&� �Zed'd(� �Zed)d*� �Zed+d,� �Zedd�d-d.��Zedd�d/d0��Zed1d2� �Zd3d4� Zd5d6� Zed7d8� �Zedd�d9d:��Zed;d<� �Zd=d>� ZdMd@dA�Z dBdC� Z dNdDdDdE�dFdG�Z dHdI� Z dOdDdDdE�dJdK�Z d?S )P�Fixedz� represent an object in my store facilitate read/write of various types of objects this is an abstract base class Parameters ---------- parent : HDFStore group : Node The group node where the table resides. rL � pandas_kindrd � format_typeztype[FrameOrSeriesUnion]�obj_typerO r rJ r� r� r> r� r~ F�UTF-8rj )r� r� rJ r~ c C sZ t |t�stt|���td k s"t�t |tj�s:tt|���|| _|| _t|�| _ || _ d S )N) rA r� r� r� rk r> r� r� rK rJ r~ )r� r� r� rJ r~ rF rF rG r� , s zFixed.__init__rr )r� c C s* | j d dko(| j d dko(| j d dk S )Nr rQ � � )�version)r� rF rF rG �is_old_version; s zFixed.is_old_versionztuple[int, int, int]c C sb t t| jjdd��}y0tdd� |�d�D ��}t|�dkrB|d }W n tk r\ d}Y nX |S ) zcompute and set our version�pandas_versionNc s s | ]}t |�V qd S )N)rO )rS r� rF rF rG r D s z Fixed.version.<locals>.<genexpr>�.r� )r )r r r )rH r/ r� r0 rX ri rY rm )r� r� rF rF rG r� ? s z Fixed.versionc C s t t| jjdd ��S )Nr, )rH r/ r� r0 )r� rF rF rG r, K s zFixed.pandas_typec C s^ | � � | j}|dk rXt|ttf�rDd�dd� |D ��}d|� d�}| jd�d|� d �S | jS ) z(return a pretty representation of myselfNr� c s s | ]}t |�V qd S )N)r; )rS r� rF rF rG r U s z!Fixed.__repr__.<locals>.<genexpr>�[r� z12.12z (shape->�))r� r� rA rW rX r� r, )r� rE ZjshaperF rF rG r� O s zFixed.__repr__c C s t | j�| j_t t�| j_dS )z set my pandas type & versionN)rL r� r� r, �_versionr� )r� rF rF rG re Z s zFixed.set_object_infoc C s t � | �}|S )N)r= )r� Znew_selfrF rF rG r= _ s z Fixed.copyc C s | j S )N)r� )r� rF rF rG r� c s z Fixed.shapec C s | j jS )N)r� r� )r� rF rF rG r� g s zFixed.pathnamec C s | j jS )N)r� r� )r� rF rF rG r� k s z Fixed._handlec C s | j jS )N)r� r� )r� rF rF rG r� o s zFixed._filtersc C s | j jS )N)r� r� )r� rF rF rG r� s s zFixed._complevelc C s | j jS )N)r� r� )r� rF rF rG r� w s zFixed._fletcher32c C s | j jS )N)r� r0 )r� rF rF rG r� { s z Fixed.attrsc C s dS )zset our object attributesNrF )r� rF rF rG � set_attrs s zFixed.set_attrsc C s dS )zget our object attributesNrF )r� rF rF rG � get_attrs� s zFixed.get_attrsc C s | j S )zreturn my storable)r� )r� rF rF rG �storable� s zFixed.storablec C s dS )NFrF )r� rF rF rG rd � s zFixed.is_existsc C s t | jdd �S )Nr� )r/ r� )r� rF rF rG r� � s z Fixed.nrowsc C s |dkr dS dS )z%validate against an existing storableNTrF )r� r� rF rF rG �validate� s zFixed.validateNc C s dS )z+are we trying to operate on an old version?TrF )r� rZ rF rF rG �validate_version� s zFixed.validate_versionc C s | j }|dkrdS | �� dS )zr infer the axes of my storer return a boolean indicating if we have a valid storer or not NFT)r� r� )r� rE rF rF rG r� � s zFixed.infer_axesz int | None)r� r� c C s t d��d S )Nz>cannot read on an abstract storer: subclasses should implement)r� )r� rZ r� r� r� rF rF rG r� � s z Fixed.readc K s t d��d S )Nz?cannot write on an abstract storer: subclasses should implement)r� )r� r� rF rF rG rf � s z Fixed.writec C s0 t �|||�r$| jj| jdd� dS td��dS )zs support fully deleting the node in its entirety (only) - where specification must be None T)r Nz#cannot delete on an abstract storer)r r r� rg r� r� )r� rZ r� r� rF rF rG r � s z Fixed.delete)r� rj )N)NNNN)NNN)!r] r^ r_ rn ro r� r� r� rp r� r� r, r� re r= r� r� r� r� r� r� r� r� r� r� rd r� r� r� r� r� rf r rF rF rF rG r� sN r� c @ s& e Zd ZU dZedediZdd� e�� D �Zg Z de d< dd �d d �Z d d � Z dd� Z dd� Zedd �dd��Zdd� Zdd� Zdd� Zd9dd d d �d d �Zd:dd d d d!�d"d#�Zdd d$�d%d&�Zdd'd$�d(d)�Zd;dd d d'd!�d*d+�Zd<d,d d d d-�d.d/�Zdd0d1�d2d3�Zd=dd4d5d6�d7d8�ZdS )>� GenericFixedza generified fixed version�datetimer� c C s i | ]\}}||�qS rF rF )rS r� r rF rF rG r � s zGenericFixed.<dictcomp>z list[str]� attributesrL )r� c C s | j �|d�S )N� )�_index_type_mapr� )r� r` rF rF rG �_class_to_alias� s z GenericFixed._class_to_aliasc C s t |t�r|S | j�|t�S )N)rA r� �_reverse_index_mapr� r( )r� �aliasrF rF rG �_alias_to_class� s z GenericFixed._alias_to_classc C s� | � tt|dd���}|tkr.d dd�}|}n |tkrFd dd�}|}n|}i }d|krn|d |d< |tkrnt}d|kr�t|d t�r�|d � d�|d< n |d |d< |tks�t �||fS ) N� index_classr� c S s: t j| j|d�}tj|d d�}|d k r6|�d��|�}|S )N)ru )rM �UTC)r1 � _simple_newr r' � tz_localize� tz_convert)r ru rv Zdta�resultrF rF rG rf � s z*GenericFixed._get_index_factory.<locals>.fc S s t j| |d�}tj|d d�S )N)ru )rM )r2 r r+ )r ru rv ZparrrF rF rG rf � s ru rv zutf-8)NN)NN) r rH r/ r' r+ r( r- rA �bytesrD r� )r� r� r rf r� r� rF rF rG �_get_index_factory� s( z GenericFixed._get_index_factoryc C s$ |dk rt d��|dk r t d��dS )zE raise if any keywords are passed which are not-None Nzqcannot pass a column specification when reading a Fixed format store. this store must be selected in its entiretyzucannot pass a where specification when reading from a Fixed format store. this store must be selected in its entirety)r� )r� r� rZ rF rF rG � validate_read s zGenericFixed.validate_readrr c C s dS )NTrF )r� rF rF rG rd s zGenericFixed.is_existsc C s | j | j_ | j| j_dS )zset our object attributesN)rJ r� r~ )r� rF rF rG r� s zGenericFixed.set_attrsc C sV t t| jdd��| _tt| jdd��| _x(| jD ] }t| |tt| j|d��� q0W dS )zretrieve our attributesrJ Nr~ rj )rK r/ r� rJ rH r~ r� r� )r� r� rF rF rG r� s zGenericFixed.get_attrsc K s | � � d S )N)r� )r� rb r� rF rF rG rf # s zGenericFixed.writeNz int | None)rs r� r� c C s� ddl }t| j|�}|j}t|dd�}t||j�rD|d ||� }nztt|dd��} t|dd�} | dk rxtj| | d�}n |||� }| dkr�t|d d�} t || d d �}n| d kr�tj |d d�}|r�|j S |S dS )z2read an array for the specified node (off of groupr N� transposedF� value_typer� )r� r� rv T)r� r� zm8[ns]) rl r/ r� r0 rA ZVLArrayrH rB ra r� r� �T) r� rs r� r� rl r� r� r �retr� r� rv rF rF rG � read_array& s& zGenericFixed.read_arrayr( )rs r� r� r� c C sh t t| j|� d���}|dkr.| j|||d�S |dkrVt| j|�}| j|||d�}|S td|� ���d S )N�_variety�multi)r� r� �regularz unrecognized index variety: )rH r/ r� �read_multi_indexr� �read_index_noder� )r� rs r� r� Zvarietyr� rz rF rF rG � read_indexH s zGenericFixed.read_index)rs rz c C s� t |t�r,t| j|� d�d� | �||� n�t| j|� d�d� td|| j| j�}| �||j � t | j |�}|j |j _ |j|j _t |ttf�r�| �t|��|j _t |tttf�r�|j|j _t |t�r�|jd k r�t|j�|j _d S )Nr r r rz )rA r* r� r� �write_multi_index�_convert_indexrJ r~ � write_arrayr r/ r� r) r0 rM r' r+ r� r� r r- ru rv �_get_tz)r� rs rz r� r� rF rF rG � write_indexV s zGenericFixed.write_indexr* c C s� t | j|� d�|j� x�tt|j|j|j��D ]�\}\}}}t|�rLt d��|� d|� �}t ||| j | j �}| � ||j� t| j|�} |j| j_|| j_t | j|� d|� �|� |� d|� �} | � | |� q.W d S )N�_nlevelsz=Saving a MultiIndex with an extension dtype is not supported.�_level�_name�_label)r� r� rW � enumerater� �levelsr� �namesr r� r rJ r~ r r r/ r� r) r0 rM ) r� rs rz �i�lev� level_codesrM � level_keyZ conv_levelr� � label_keyrF rF rG r m s " z GenericFixed.write_multi_indexc C s� t | j|� d��}g }g }g }xxt|�D ]l}|� d|� �} t | j| �} | j| ||d�} |�| � |�| j� |� d|� �} | j| ||d�} |�| � q(W t|||dd�S )Nr r )r� r� r T)r r� r r� ) r/ r� r r� r rx rM r r* )r� rs r� r� rW r r� r r! r$ r� r" r% r# rF rF rG r � s z GenericFixed.read_multi_indexr> )r� r� r� r� c C s� |||� }d|j kr>t�|j j�dkr>tj|j j|j jd�}t|j j�}d }d|j krlt|j j �}t|�}|j }| � |�\}} |dkr�|t ||| j | j d�fdti| ��} n |t ||| j | j d�f| �} || _ | S )Nr� r )r� rM r )rJ r~ r� )r0 rB �prodr� ra r rH r) rN rM r �_unconvert_indexrJ r~ r� ) r� r� r� r� r< r) rM r� r� r� rz rF rF rG r � s, z GenericFixed.read_index_noder )rs rt c C sJ t �d|j �}| j�| j||� t| j|�}t|j�|j _ |j |j _ dS )zwrite a 0-len array)rQ N) rB ra r r� � create_arrayr� r/ rL r� r0 r r� )r� rs rt Zarrr� rF rF rG �write_array_empty� s z GenericFixed.write_array_emptyr z Index | None)rs rb r� c C s2 t |dd�}|| jkr&| j�| j|� |jdk}d}t|j�rFtd��|s^t|d�r^|j }d}d }| j d k r�t t �� t � j�|j�}W d Q R X |d k r�|s�| jj| j|||j| j d�}||d d �< n | �||� �nH|jjtjk�rHtj|dd�} |r�n*| d k�rn t| ||f } tj| td d � | j� | j|t � � � �} | � |� n�t |j��r~| j� | j||�!d �� d t"| j|�j#_$n�t%|j��r�| j� | j||j&� t"| j|�} t'|j(�| j#_(d | j#_$n\t)|j��r�| j� | j||�!d �� dt"| j|�j#_$n&|�r | �||� n| j� | j||� |t"| j|�j#_*d S )NT)Z extract_numpyr Fz]Cannot store a category dtype in a HDF5 dataset that uses format="fixed". Use format="table".r )r� )�skipnar� � )r� �i8r� r� )+r5 r� r� rg r� r r� r� r� r r� r r� rp ZAtomZ from_dtypeZ create_carrayr� r) r� rB Zobject_r � infer_dtype�performance_docr� r� r Zcreate_vlarray� ObjectAtomrx r r( �viewr/ r0 r r �asi8r rv r# r ) r� rs rb r� rt Z empty_arrayr r� �ca� inferred_typer� Zvlarrr� rF rF rG r � s` zGenericFixed.write_array)NN)NN)NN)NN)N) r] r^ r_ rn r' r+ r� r� r� r� ro r� r r r rp rd r� r� rf r r r r r r r) r rF rF rF rG r� � s( .# % r� c sN e Zd ZU dZdgZded< edd� �Zd ddd�d d �Z� fd d �Z � Z S )rU rJ rM r c C s, yt | jj�fS ttfk r& d S X d S )N)rY r� r r� rm )r� rF rF rG r� + s zSeriesFixed.shapeNz int | None)r� r� c C s< | � ||� | jd||d�}| jd||d�}t||| jd�S )Nrz )r� r� r )rz rM )r r r r, rM )r� rZ r� r� r� rz r rF rF rG r� 2 s zSeriesFixed.readc s8 t � j|f|� | �d|j� | �d|� |j| j_d S )Nrz r )r� rf r rz r rM r� )r� rb r� )r� rF rG rf > s zSeriesFixed.write)NNNN) r] r^ r_ r� r� ro rp r� r� rf r� rF rF )r� rG rU % s rU c sR e Zd ZU ddgZded< edd�dd��Zdd d d �d d �Z� fd d�Z� Z S )�BlockManagerFixedr �nblocksrO z Shape | None)r� c C s� y�| j }d}xFt| j�D ]8}t| jd|� d��}t|dd �}|d k r||d 7 }qW | jj}t|dd �}|d k r�t|d|d � �}ng }|�|� |S tk r� d S X d S )Nr �block�_itemsr� rQ ) r r r5 r/ r� Z block0_valuesrW rx rm )r� r r� r! r� r� rF rF rG r� J s" zBlockManagerFixed.shapeNz int | None)r� r� c C s | � ||� | �� �d�}g }xJt| j�D ]<}||kr>||fnd\}} | jd|� �|| d�} |�| � q*W |d } g } xht| j�D ]Z}| �d|� d��} | jd|� d�|| d�}| | � | � }t |j ||d d �}| �|� q�W t | �dk�r t | dd �}|j| d d �}|S t |d |d d �S ) Nr )NNr� )r� r� r6 r7 r� rQ )r� rz )r� F)r� r= )r r� Z_get_block_manager_axisr r r rx r5 r r r& r rY r. r" )r� rZ r� r� r� Z select_axisr r! r� r� �axr� �dfs� blk_itemsr �df�outrF rF rG r� e s( zBlockManagerFixed.readc s� t � j|f|� t|jt�r&|�d�}|j}|�� s<|�� }|j| j _x>t |j �D ]0\}}|dkrp|j spt d��| �d|� �|� qRW t|j�| j _xRt |j�D ]D\}}|j�|j�}| jd|� d�|j|d� | �d|� d�|� q�W d S )Nr6 r z/Columns index has to be unique for fixed formatr� r� )r� r7 )r� rf rA �_mgrr7 � _as_managerZis_consolidatedZ consolidater r� r r Z is_uniquer� r rY �blocksr5 r� r �mgr_locsr r )r� rb r� r< r! r8 �blkr: )r� rF rG rf � s zBlockManagerFixed.write)NNNN) r] r^ r_ r� ro rp r� r� rf r� rF rF )r� rG r4 E s r4 c @ s e Zd ZdZeZdS )rV rK N)r] r^ r_ r� r& r� rF rF rF rG rV � s rV c s� e Zd ZU dZdZdZded< ded< dZded < d Zd ed < d ed< ded< ded< ded< ded< d�dddd�� fdd � Z e dd �d d ��Z dd �d d!�Z dd"�d#d$�Z d%d&� Ze d'd �d(d)��Zd*d+d,�d-d.�Ze d/d �d0d1��Ze d'd �d2d3��Ze d4d5� �Ze d6d7� �Ze d8d9� �Ze d:d;� �Ze d<d=� �Ze d/d �d>d?��Ze d'd �d@dA��Ze dBdC� �ZdDd �dEdF�ZdGdH� Z dId �dJdK�Z dddL�dMdN�Z ddOdP�dQdR�Z ddS�dTdU�Z dVdW� Z!dXdY� Z"d�dZd[�Z#d\d]� Z$e%d^d_� �Z&d�d`da�dbdc�Z'd�dddddedf�dgdh�Z(e)d'di�djdk��Z*dldm� Z+d�dnd'do�dpdq�Z,e-dnd'dr�dsdt��Z.d�dudv�dwdx�Z/ddd'dddDdy�dzd{�Z0d�ddddd|�d}d~�Z1d�dddddd�d�d��Z2� Z3S )�r� a� represent a table: facilitate read/write of various types of tables Attrs in Table Node ------------------- These are attributes that are store in the main table node, they are necessary to recreate these tables when read back in. index_axes : a list of tuples of the (original indexing axis and index column) non_index_axes: a list of tuples of the (original index axis and columns on a non-indexing axis) values_axes : a list of the columns which comprise the data of this table data_columns : a list of the columns that we are allowing indexing (these become single columns in values_axes), or True to force all columns nan_rep : the string to use for nan representations for string objects levels : the names of levels metadata : the names of the metadata columns Z wide_tablere rL r� rG rQ zint | list[Hashable]r Tzlist[IndexCol]� index_axeszlist[tuple[int, Any]]r� z list[DataCol]� values_axesrW r} r{ r rB Nrj r� r> )r� r� r~ c sP t � j||||d� |pg | _|p$g | _|p.g | _|p8g | _| pBi | _| | _d S )N)rJ r~ )r� r� rB r� rC r} rB r ) r� r� r� rJ r~ rB r� rC r} rB r )r� rF rG r� � s zTable.__init__)r� c C s | j �d�d S )N�_r )rG ri )r� rF rF rG �table_type_short� s zTable.table_type_shortc C s� | � � t| j�r d�| j�nd}d|� d�}d}| jrZd�dd� | jD ��}d|� d�}d�d d� | jD ��}| jd �|� d | j� d | j � d | j � d|� d|� d� S )z(return a pretty representation of myselfr� r� z,dc->[r� r� c s s | ]}t |�V qd S )N)rL )rS r� rF rF rG r � s z!Table.__repr__.<locals>.<genexpr>r� c s s | ] }|j V qd S )N)rM )rS rq rF rF rG r � s z12.12z (typ->z,nrows->z,ncols->z ,indexers->[r� ) r� rY r} r� r� r� rB r, rE r� �ncols)r� Zjdcr$ �verZjverZ jindex_axesrF rF rG r� � s zTable.__repr__)r� c C s" x | j D ]}||jkr|S qW dS )zreturn the axis for cN)r rM )r� r� rq rF rF rG r� � s zTable.__getitem__c C s� |dkr dS |j | j kr2td|j � d| j � d���x�dD ]�}t| |d�}t||d�}||kr8x@t|�D ]4\}}|| }||krftd|� d|� d|� d���qfW td|� d|� d|� d���q8W dS ) z"validate against an existing tableNz'incompatible table_type with existing [z - r� )rB r� rC zinvalid combination of [z] on appending data [z] vs current table [)rG r� r/ r r� r )r� r� r� �sv�ovr! �saxZoaxrF rF rG r� s zTable.validaterr c C s t | jt�S )z@the levels attribute is 1 or a list in the case of a multi-index)rA r rW )r� rF rF rG �is_multi_index s zTable.is_multi_indexr z tuple[DataFrame, list[Hashable]])rb r� c C sf dd� t |jj�D �}y |�� }W n, tk rN } ztd�|�W dd}~X Y nX t|t�s^t�||fS )ze validate that we can store the multi-index; reset and return the new object c S s& g | ] \}}|d k r|nd|� ��qS )NZlevel_rF )rS r! �lrF rF rG rV - s z-Table.validate_multiindex.<locals>.<listcomp>zBduplicate names/columns in the multi-index when storing as a tableN)r rz r Z reset_indexr� rA r& r� )r� rb r Z reset_objr rF rF rG �validate_multiindex% s zTable.validate_multiindexrO c C s t �dd� | jD ��S )z-based on our axes, compute the expected nrowsc S s g | ]}|j jd �qS )r )r� r� )rS r! rF rF rG rV ; s z(Table.nrows_expected.<locals>.<listcomp>)rB r& rB )r� rF rF rG �nrows_expected8 s zTable.nrows_expectedc C s d| j kS )zhas this table been createdre )r� )r� rF rF rG rd = s zTable.is_existsc C s t | jdd �S )Nre )r/ r� )r� rF rF rG r� B s zTable.storablec C s | j S )z,return the table group (this is my storable))r� )r� rF rF rG re F s z Table.tablec C s | j jS )N)re r� )r� rF rF rG r� K s z Table.dtypec C s | j jS )N)re r� )r� rF rF rG r� O s zTable.descriptionc C s t �| j| j�S )N)r� r� rB rC )r� rF rF rG r S s z Table.axesc C s t dd� | jD ��S )z.the number of total columns in the values axesc s s | ]}t |j�V qd S )N)rY r )rS rq rF rF rG r Z s z Table.ncols.<locals>.<genexpr>)�sumrC )r� rF rF rG rF W s z Table.ncolsc C s dS )NFrF )r� rF rF rG � is_transposed\ s zTable.is_transposedc C s( t t�dd� | jD �dd� | jD ���S )z@return a tuple of my permutated axes, non_indexable at the frontc S s g | ]}t |d ��qS )r )rO )rS rq rF rF rG rV e s z*Table.data_orientation.<locals>.<listcomp>c S s g | ]}t |j��qS rF )rO r� )rS rq rF rF rG rV f s )rX r� r� r� rB )r� rF rF rG �data_orientation` s zTable.data_orientationzdict[str, Any]c sR ddd�� dd� �j D �}� fdd��jD �}�fdd��jD �}t|| | �S )z<return a dict of the kinds allowable columns for this objectrz r� )r rQ c S s g | ]}|j |f�qS rF )rx )rS rq rF rF rG rV p s z$Table.queryables.<locals>.<listcomp>c s g | ]\}}� | d f�qS )NrF )rS r� r )� axis_namesrF rG rV q s c s& g | ] }|j t� j�kr|j|f�qS rF )rM r r} rx )rS r )r� rF rG rV s s )rB r� rC r )r� Zd1Zd2Zd3rF )rR r� rG � queryablesj s zTable.queryablesc C s dd� | j D �S )z return a list of my index colsc S s g | ]}|j |jf�qS rF )r� rx )rS r! rF rF rG rV } s z$Table.index_cols.<locals>.<listcomp>)rB )r� rF rF rG � index_colsz s zTable.index_colsz list[str]c C s dd� | j D �S )z return a list of my values colsc S s g | ] }|j �qS rF )rx )rS r! rF rF rG rV � s z%Table.values_cols.<locals>.<listcomp>)rC )r� rF rF rG � values_cols s zTable.values_cols)rs r� c C s | j j}|� d|� d�S )z)return the metadata pathname for this keyz/meta/z/meta)r� r� )r� rs r� rF rF rG �_get_metadata_path� s zTable._get_metadata_pathz np.ndarray)rs r c C s0 t |�}| jj| �|�|d| j| j| jd� dS )z� Write out a metadata array to the key as a fixed-format Series. Parameters ---------- key : str values : ndarray re )ry rJ r~ r N)r, r� r� rV rJ r~ r )r� rs r rF rF rG r� � s zTable.write_metadata)rs c C s0 t t | jdd�|d�dk r,| j�| �|��S dS )z'return the meta data array for this keyr� N)r/ r� r� r� rV )r� rs rF rF rG r� � s zTable.read_metadatac C sp t | j�| j_| �� | j_| �� | j_| j| j_| j| j_| j| j_| j| j_| j | j_ | j | j_ | j | j_ dS )z set our table type & indexablesN) rL rG r� rT rU r� r} r rJ r~ r rB )r� rF rF rG r� � s zTable.set_attrsc C s� t | jdd�pg | _t | jdd�p$g | _t | jdd�p8i | _t | jdd�| _tt | jdd��| _tt | jdd��| _ t | jd d�p�g | _ d d � | j D �| _ d d � | j D �| _ dS ) zretrieve our attributesr� Nr} rB r rJ r~ rj r c S s g | ]}|j r|�qS rF )r� )rS rq rF rF rG rV � s z#Table.get_attrs.<locals>.<listcomp>c S s g | ]}|j s|�qS rF )r� )rS rq rF rF rG rV � s )r/ r� r� r} rB r rK rJ rH r~ r � indexablesrB rC )r� rF rF rG r� � s zTable.get_attrsc C s\ |dk rX| j d dkrX| j d dkrX| j d dk rXtd�dd� | j D �� }t�|t� dS ) z+are we trying to operate on an old version?Nr rQ r� r� r� c s s | ]}t |�V qd S )N)rL )rS r� rF rF rG r � s z)Table.validate_version.<locals>.<genexpr>)r� �incompatibility_docr� r� r� ra )r� rZ r� rF rF rG r� � s *zTable.validate_versionc C sV |dkr dS t |t�sdS | �� }x.|D ]&}|dkr6q(||kr(td|� d���q(W dS )z� validate the min_itemsize doesn't contain items that are not in the axes this needs data_columns to be defined Nr zmin_itemsize has the key [z%] which is not an axis or data_column)rA r rS r� )r� r{ �qr� rF rF rG �validate_min_itemsize� s zTable.validate_min_itemsizec s� g }�j ��jj�xzt�jj�D ]j\}\}}t�|�}��|�}|dk rLdnd}|� d�}t�|d�} t|||| |�j||d�} |�| � q W t �j ��t |�� � ����fdd��|� �fdd�t�jj �D �� |S ) z/create/cache the indexables if they don't existNr� r~ )rM r� rz r) ry re r� r{ c s� t |t�st�t}|�kr t}t�|�}t|�j�}t�|� d�d �}t�|� d�d �}t|�}�� |�}t�|� d�d �} |||||� | |�j | ||d� } | S )Nr~ r� r� ) rM rx r r) rz ry re r� r{ r� ) rA rL r� r� r� r/ �_maybe_adjust_namer� r� r� re ) r! r� �klassr� �adj_namer r� r) �mdr� rb )�base_posr$ �descr� � table_attrsrF rG rf � s. zTable.indexables.<locals>.fc s g | ]\}}� ||��qS rF rF )rS r! r� )rf rF rG rV ! s z$Table.indexables.<locals>.<listcomp>)r� re r� r rT r/ r� rt rx r r} rY r rU ) r� � _indexablesr! r� rM r� r^ r� r r) � index_colrF )r_ r$ r` rf r� ra rG rW � s0 # zTable.indexablesz str | None)r) c C sV | � � s dS |dkrdS |dks(|dkr8dd� | jD �}t|ttf�sL|g}i }|dk r`||d< |dk rp||d< | j}x�|D ]�}t|j|d�}|dk �r |jr�|j }|j } |j } |dk r�| |kr�|� � n| |d< |dk r�| |kr�|� � n| |d< |j�sN|j �d��rtd ��|jf |� q||| jd d kr|td |� d |� d|� d���q|W dS )aZ Create a pytables index on the specified columns. Parameters ---------- columns : None, bool, or listlike[str] Indicate which columns to create an index on. * False : Do not create any indexes. * True : Create indexes on all columns. * None : Create indexes on all columns. * listlike : Create indexes on the given columns. optlevel : int or None, default None Optimization level, if None, pytables defaults to 6. kind : str or None, default None Kind of index, if None, pytables defaults to "medium". Raises ------ TypeError if trying to create an index on a complex-type column. Notes ----- Cannot index Time64Col or ComplexCol. Pytables must be >= 3.0. NFTc S s g | ]}|j r|j�qS rF )r� rx )rS rq rF rF rG rV H s z&Table.create_index.<locals>.<listcomp>r( r) �complexz�Columns containing complex values can be stored but cannot be indexed when using table format. Either use fixed format, set index=False, or do not include the columns containing complex values to data_columns when initializing the table.r rQ zcolumn z/ is not a data_column. In order to read column z: you must reload the dataframe into HDFStore and include z with the data_columns argument.)r� r rA rX rW re r/ r r: rz r( r) Z remove_indexr� r7 r� r* r� rm ) r� r� r( r) �kwre r� r rz Z cur_optlevelZcur_kindrF rF rG r* % sF zTable.create_indexz int | Nonez!list[tuple[ArrayLike, ArrayLike]])r� r� r� c C s^ t | |||d�}|�� }g }x<| jD ]2}|�| j� |j|| j| j| jd�}|� |� q$W |S )a Create the axes sniffed from the table. Parameters ---------- where : ??? start : int or None, default None stop : int or None, default None Returns ------- List[Tuple[index_values, column_values]] )rZ r� r� )r rJ r~ ) � Selectionr� r r� rB r� r rJ r~ rx ) r� rZ r� r� � selectionr rs rq �resrF rF rG � _read_axesy s zTable._read_axes)r c C s |S )z return the data for this objrF )r` rb r rF rF rG � get_object� s zTable.get_objectc s� t |�s g S |d \}� | j�|i �}|�d�dkrL|rLtd|� d|� ���|dkr^t� �}n |dkrjg }t|t�r�t|��t|�}|��fdd �|� � D �� � fd d �|D �S ) zd take the input data_columns and min_itemize and create a data columns spec r r� r* z"cannot use a multi-index on axis [z] with data_columns TNc s g | ]}|d kr|� kr|�qS )r rF )rS r� )�existing_data_columnsrF rG rV � s z/Table.validate_data_columns.<locals>.<listcomp>c s g | ]}|� kr|�qS rF rF )rS r� )� axis_labelsrF rG rV � s ) rY rB r� r� rW rA r r r r� )r� r} r{ r� r� rB rF )rl rk rG �validate_data_columns� s$ zTable.validate_data_columnsr& )rb r� c / s� t �t�s,| jj}td|� dt��� d���� dkr:dg� �fdd�� D �� | �� rzd}d d� | jD �� t| j �}| j }nd }| j } | j d ks�t �t� �| j d kr�td ��g } |dkr�d}� fdd�dD �d } �j| } t| �} |�r<t| �}| j| d }tt�| �t�|���s<tt�t| ��t�t|����r<|} | �| i �}t| j�|d< t| �j|d< | �| | f� � d } �j| } ��| �}t|| | j | j �}| |_ |� d� |� | � |�!|� |g}t|�}|d k�s�t �t| �d k�s�t �x"| D ]} t"�| d | d ���q�W |j d k}| �#||| �}| �$�|��%� }| �&||| | j'|�\}}g }�x�t(t)||��D �]�\}\}} t*} d} |�r�t| �d k�r�| d |k�r�t+} | d } | dk�s�t | t,��s�td��|�r.|�r.y| j'| } W nB t-t.fk �r* } z td|� d| j'� d��| �W dd} ~ X Y nX nd} | �p@d|� �}!t/|!|| ||| j | j | d�}"t0|!| j1�}#| �2|"�}$t3|"j4j5�}%d}&t6|"dd�dk �r�t7|"j8�}&d }' }(})t9|"j4��r�|"j:})d}'tj|"j;d d��<� }(t=|"�\}*}+| |#|!t| �|$||%|&|)|'|(|+|*d� },|,� | � |�|,� |d 7 }�qlW d d� |D �}-t| �| j>| j| j | j || ||-| |d � }.t?| d ��rr| j@|._@|.�A|� |�r�|�r�|.�B| � |.S ) a0 Create and return the axes. Parameters ---------- axes: list or None The names or numbers of the axes to create. obj : DataFrame The object to create axes on. validate: bool, default True Whether to validate the obj against an existing object already written. nan_rep : A value to use for string column nan_rep. data_columns : List[str], True, or None, default None Specify the columns that we want to create to allow indexing on. * True : Use all available columns. * None : Use no columns. * List[str] : Use the specified columns. min_itemsize: Dict[str, int] or None, default None The min itemsize for a column in bytes. z/cannot properly create the storer for: [group->z,value->r� Nr c s g | ]}� � |��qS rF )�_get_axis_number)rS rq )rb rF rG rV � s z&Table._create_axes.<locals>.<listcomp>Tc S s g | ] }|j �qS rF )r� )rS rq rF rF rG rV � s Fr� rQ z<currently only support ndim-1 indexers in an AppendableTable�nanc s g | ]}|� kr|�qS rF rF )rS r� )r rF rG rV s )r rQ r r� z-cannot have non-object label DataIndexableColz Incompatible appended table [z]with existing table [Z values_block_)� existing_colr{ r rJ r~ r� rv r� )r= ) rM rx r ry rz r) rv r# r� r{ r� r< c S s g | ]}|j r|j�qS rF )r� rM )rS r� rF rF rG rV � s ) r� r� rJ r~ rB r� rC r} rB r r )CrA r& r� r� r� r� r� rB rW r} r rB r r� rY r� r r� r% rB �arrayr r� r r] rx Z_get_axis_namer rJ r~ r� r| r� r� � _reindex_axisrm rj r� �_get_blocks_and_itemsrC r r� r� r� rL � IndexErrorr� � _maybe_convert_for_string_atomr[ r� r� r� r� rM r/ r rv r r# r� r� r� r� r� r rZ r� )/r� r rb r� r r} r{ r� � table_existsZnew_info�new_non_index_axesr� rq Z append_axisZindexerZ exist_axisrB � axis_nameZ new_indexZnew_index_axes�jr rK r? r: Zvaxesr! rA �b_itemsr\ rM rp r �new_name�data_convertedr] ry r) rv r� r{ r# r< r� r� ZdcsZ new_tablerF )r rb rG � _create_axes� s " zTable._create_axes)rK rv c C s� t | jt�r| �d�} dd� }| j}tt|�}t|j�}||�}t|�r�|d \} } t | �� t |��} | j | | d�j}t|j�}||�}x8|D ]0} | j | g| d�j}|� |j� |� ||�� q�W |�rzdd� t ||�D �} g }g }x�|D ]�}t|j�}y&| �|�\}}|�|� |�|� W q� ttfk �rl } z*d�d d � |D ��}td |� d ��|�W d d }~X Y q�X q�W |}|}||fS ) Nr6 c s � fdd�� j D �S )Nc s g | ]}� j �|j��qS rF )r� r r@ )rS rA )�mgrrF rG rV � s zFTable._get_blocks_and_items.<locals>.get_blk_items.<locals>.<listcomp>)r? )r~ rF )r~ rG � get_blk_items� s z2Table._get_blocks_and_items.<locals>.get_blk_itemsr )r� c S s" i | ]\}}||ft |�� ��qS rF )rX �tolist)rS �brz rF rF rG r � s z/Table._get_blocks_and_items.<locals>.<dictcomp>r� c s s | ]}t |�V qd S )N)r; )rS �itemrF rF rG r � s z.Table._get_blocks_and_items.<locals>.<genexpr>z+cannot match existing table structure for [z] on appending data)rA r= r7 r> r r8 rW r? rY r( r r" r r� rX r r! rx rt r� r� r� )rK rv rw rC r} r r~ r? r: r� rl Z new_labelsr� Zby_itemsZ new_blocksZ new_blk_itemsZear� r� rz r ZjitemsrF rF rG rs � sH zTable._get_blocks_and_itemsrf )rg c s� |dk rt |�}|dk rR�jrRt�jt �s.t�x"�jD ]}||kr6|�d|� q6W x �jD ]\}}t� |||�� qZW |jdk r�x2|j� � D ]$\}�}� ��fdd�} | ||�� q�W � S )zprocess axes filtersNr c s� x�� j D ]�}� �|�}� �|�}|d k s,t�| |krd�jrJ|�t�j��}�||�}� j|d�| S | |krt t � | �j �}t |�}t � t �r�d| }�||�}� j|d�| S qW td| � d���d S )N)r� rQ zcannot find the field [z] for filtering!)Z _AXIS_ORDERSrn � _get_axisr� rK �unionr( r r r6 r/ r rA r& r� )�field�filtrx Z axis_numberZ axis_valuesZtakersr )rb �opr� rF rG �process_filter s" z*Table.process_axes.<locals>.process_filter) rW rK rA r r� �insertr� rr �filterry ) r� rb rg r� r� r� �labelsr� r� r� rF )rb r� r� rG � process_axes� s !zTable.process_axes)rv r� r r� c C s� |dkrt | jd�}d|d�}dd� | jD �|d< |rj|dkrH| jpFd}t� j|||pZ| jd �}||d < n| jdk r~| j|d < |S ) z:create the description of the table from the axes & valuesNi' re )rM r c S s i | ]}|j |j�qS rF )ry rx )rS rq rF rF rG r 8 s z,Table.create_description.<locals>.<dictcomp>r� � )rv rw r� r� )�maxrN r r� rp r� r� r� )r� rw rv r� r r r� rF rF rG �create_description) s zTable.create_description)r� r� c C s� | � |� | �� sdS t| |||d�}|�� }|jdk r�xR|j�� D ]D\}}}| j||�� |�� d d�} ||| j ||�� |�j }qDW t |�S )zf select coordinates (row numbers) from a table; return the coordinates object F)rZ r� r� NrQ )r� r� ) r� r� rf � select_coordsr� ry r� rr r� �ilocr r( ) r� rZ r� r� rg Zcoordsr� r� r� r< rF rF rG r� H s "zTable.read_coordinates)r� r� r� c C s� | � � | �� sdS |dk r$td��x�| jD ]v}||jkr,|jsPtd|� d���t| jj |�}|� | j � |j |||� | j | j| jd�}tt|d |j�|d�S q,W td|� d ���dS ) zj return a single column from the table, generally only indexables are interesting FNz4read_column does not currently accept a where clausezcolumn [z=] can not be extracted individually; it is not data indexable)r rJ r~ rQ )rM z] not found in the table)r� r� r� r rM r� r� r/ re r r� rB r� r rJ r~ r, r� rv r� )r� r� rZ r� r� rq r� Z col_valuesrF rF rG r� b s& zTable.read_column)Nrj NNNNNN)N)NNN)NN)TNNN)N)NNN)NNN)4r] r^ r_ rn r� r� ro r r� r� rp rE r� r� r� rK rM rN rd r� re r� r� r rF rP rQ rS rT rU rV r� r� r� r� r� rZ r rW r* ri r� rj rm r} � staticmethodrs r� r� r� r� r� rF rF )r� rG r� � s~ JU!* d?: r� c @ s. e Zd ZdZdZd ddd�dd�Zdd � ZdS ) r] z� a write-once read-many table: this format DOES NOT ALLOW appending to a table. writing is a one-time operation the data are stored in a format that allows for searching the data on disk rS Nz int | None)r� r� c C s t d��dS )z[ read the indices and the indexing array, calculate offset rows and return z!WORMTable needs to implement readN)r� )r� rZ r� r� r� rF rF rG r� � s zWORMTable.readc K s t d��dS )z� write in a format that we can search later on (but cannot append to): write out the indices and the values using _write_array (e.g. a CArray) create an indexing table so that we can search z"WORMTable needs to implement writeN)r� )r� r� rF rF rG rf � s zWORMTable.write)NNNN)r] r^ r_ rn rG r� rf rF rF rF rG r] � s r] c @ sV e Zd ZdZdZddd�Zddd d �d d �Zd dddd�dd�Zdddd�dd�ZdS )r� z(support the new appendable table formatsZ appendableNFTc C s� |s| j r| j�| jd� | j||||| | d�}x|jD ] }|�� q8W |j s�|j|||| d�}|�� | |d< |jj |jf|� |j |j _ x|jD ]}|� ||� q�W |j || d� d S )Nre )r rb r� r{ r r} )rw rv r� r r )r| )rd r� rg r� r} r r� r� r� Z create_tablerB r� r� � write_data)r� rb r rx rw rv r� r{ r� r r| r r} r re rq �optionsrF rF rG rf � s0 zAppendableTable.writez int | Nonerr )r� r| c s� | j j}| j}g }|rXx@| jD ]6}t|j�jdd�}t|tj �r |� |j ddd�� q W t |�r�|d }x |dd� D ] }||@ }qvW |� � }nd}dd � | jD �} t | �} | dks�t| ��d d � | jD �} d d � | D �} g } xBt| �D ]6\} }|f| j || | j }| � | | �|�� q�W |dk�r0d }tjt||�| j d �}|| d }x�t|�D ]t} | | �t| d | |�� �� k�r�P | j|� �fdd �| D �|dk �r�|�� � nd� �fdd �| D �d� �q\W dS )z` we form the data into a 2-d including indexes,values,mask write chunk-by-chunk r )r� �u1F)r= rQ Nc S s g | ] }|j �qS rF )r� )rS rq rF rF rG rV s z.AppendableTable.write_data.<locals>.<listcomp>c S s g | ] }|� � �qS rF )r� )rS rq rF rF rG rV s c S s, g | ]$}|� t�t�|j�|jd ���qS )rQ )Z transposerB Zrollr� r )rS r rF rF rG rV s i�� )r� c s g | ]}|�� � �qS rF rF )rS rq )�end_i�start_irF rG rV # s c s g | ]}|�� � �qS rF rF )rS r )r� r� rF rG rV % s )�indexesr� r )r� r rN rC r/ r< r rA rB r� rx r� rY r� rB r� r r� �reshapera rr r �write_data_chunk)r� r� r| r r� �masksrq r� �mr� �nindexesr �bvaluesr! r Z new_shape�rows�chunksrF )r� r� rG r� � sJ zAppendableTable.write_dataz np.ndarrayzlist[np.ndarray]znp.ndarray | None)r� r� r� r c C s� x |D ]}t �|j�sdS qW |d jd }|t|�krHt j|| jd�}| jj}t|�}x t|�D ]\} } | ||| < qbW x$t|�D ]\} }|||| | < q�W |dk r�|�� j t dd� } | � � s�|| }t|�r�| j � |� | j �� dS )z� Parameters ---------- rows : an empty memory space where we are putting the chunk indexes : an array of the indexes mask : an array of the masks values : an array of the values Nr )r� F)r= )rB r& r� rY ra r� r r r� r� rr r re rx r� ) r� r� r� r� r r r� r r� r! r� r� rF rF rG r� ( s& z AppendableTable.write_data_chunk)r� r� c C sf |d kst |�sf|d kr:|d kr:| j}| jj| jdd� n(|d krH| j}| jj||d�}| j�� |S | �� srd S | j}t | |||d�}|� � }t |�� � }t |�} | �rb|� � } t| | dk j�} t | �s�dg} | d | kr�| �| � | d dk�r| �dd� | �� } xLt| �D ]@} |�t| | ��}|j||jd ||jd d d� | } �qW | j�� | S )NT)r )r� r� rQ r r� )rY r� r� rg r� re Z remove_rowsr� r� rf r� r, Z sort_values�diffrW rz rx r� r! �reversedr r )r� rZ r� r� r� re rg r Z sorted_series�lnr� r� Zpgr� r� rF rF rG r T sB " zAppendableTable.delete) NFNNNNNNFNNT)F)NNN) r] r^ r_ rn rG rf r� r� r rF rF rF rG r� � s" .;,r� c @ s` e Zd ZU dZdZdZdZeZde d< e dd�d d ��Z e dd �d d ��Z dddd�dd�ZdS )r[ z(support the new appendable table formatsrH rQ r� ztype[FrameOrSeriesUnion]r� rr )r� c C s | j d jdkS )Nr rQ )rB r� )r� rF rF rG rP � s z"AppendableFrameTable.is_transposed)r c C s |r |j }|S )z these are written transposed)r )r` rb r rF rF rG rj � s z AppendableFrameTable.get_objectNz int | None)r� r� c s6 � � |� � �� sd S � j|||d�}t� j�rH� j�� jd d i �ni }� fdd�t� j�D �}t|�dkstt �|d }|| d } g } �x^t� j�D �]N\} } | � j kr�q�|| \} }|�d�dkr�t � | �}nt | �}|�d�}|d k r�|j|d d � � j�r$|}|}t | t| d d �d �}n |j}t | t| d d �d �}|}|jdk�rpt|tj��rp|�d|jd f�}t|tj��r�t|j||d �}n.t|t ��r�t|||d �}ntj|g||d �}|j|jk� � �s�t |j|jf��| � |� q�W t| �dk�r| d }n t | dd�}t � |||d�}� j |||d�}|S )N)rZ r� r� r c s" g | ]\}}|� j d kr|�qS )r )rB )rS r! r8 )r� rF rG rV � s z-AppendableFrameTable.read.<locals>.<listcomp>rQ r� r* r T)�inplacerM )rM )r� rz )r� )rg r� )!r� r� ri rY r� rB r� r r r� rC r* � from_tuplesr( � set_namesrP r/ r r rA rB r� r� r� r& Z _from_arraysZdtypesr� r rx r. rf r� )r� rZ r� r� r� r rB Zinds�indrz �framesr! rq Z index_valsr� r r r Zindex_Zcols_r; rg rF )r� rG r� � sV " " zAppendableFrameTable.read)NNNN)r] r^ r_ rn r� rG r r& r� ro rp rP r� rj r� rF rF rF rG r[ � s r[ c sn e Zd ZdZdZdZdZeZe dd�dd��Z e dd �d d ��Z d� fd d� Z ddddd�� fdd� Z� ZS )rY z(support the new appendable table formatsrN rO r� rr )r� c C s dS )NFrF )r� rF rF rG rP � s z#AppendableSeriesTable.is_transposed)r c C s |S )NrF )r` rb r rF rF rG rj s z AppendableSeriesTable.get_objectNc s< t |t�s |jpd}|�|�}t� jf ||j�� d�|��S )z+we are going to write this as a frame tabler )rb r} )rA r& rM Zto_framer� rf r� r� )r� rb r} r� rM )r� rF rG rf s zAppendableSeriesTable.writez int | Noner, )r� r� r� c s� | j }|d k rF|rFt| jt�s"t�x"| jD ]}||kr*|�d|� q*W t� j||||d�}|rn|j| jdd� |j d d �df }|j dkr�d |_ |S )Nr )rZ r� r� r� T)r� r ) rK rA r rW r� r� r� r� � set_indexr� rM )r� rZ r� r� r� rK r� rE )r� rF rG r� s zAppendableSeriesTable.read)N)NNNN)r] r^ r_ rn r� rG r r, r� rp rP r� rj rf r� r� rF rF )r� rG rY � s rY c s( e Zd ZdZdZdZ� fdd�Z� ZS )rZ z(support the new appendable table formatsrN rP c s^ |j pd}| �|�\}| _t| jt�s*t�t| j�}|�|� t|�|_t � j f d|i|��S )z+we are going to write this as a frame tabler rb ) rM rM r rA rW r� rx r( r� r� rf )r� rb r� rM Znewobjr )r� rF rG rf , s z AppendableMultiSeriesTable.write)r] r^ r_ rn r� rG rf r� rF rF )r� rG rZ & s rZ c @ sd e Zd ZU dZdZdZdZeZde d< e dd�d d ��Z e d d � �Z d d� Z edd� �Zdd� ZdS )rX z:a table that read/writes the generic pytables table formatrH rI r� zlist[Hashable]r rL )r� c C s | j S )N)r� )r� rF rF rG r, @ s zGenericTable.pandas_typec C s t | jdd �p| jS )Nre )r/ r� )r� rF rF rG r� D s zGenericTable.storablec C sL g | _ d| _g | _dd� | jD �| _dd� | jD �| _dd� | jD �| _dS )zretrieve our attributesNc S s g | ]}|j r|�qS rF )r� )rS rq rF rF rG rV N s z*GenericTable.get_attrs.<locals>.<listcomp>c S s g | ]}|j s|�qS rF )r� )rS rq rF rF rG rV O s c S s g | ] }|j �qS rF )rM )rS rq rF rF rG rV P s )r� r r rW rB rC r} )r� rF rF rG r� H s zGenericTable.get_attrsc C s� | j }| �d�}|dk r dnd}tdd| j||d�}|g}xlt|j�D ]^\}}t|t�s\t�t ||�}| �|�}|dk r|dnd}t |||g|| j||d�} |� | � qFW |S )z0create the indexables from the table descriptionrz Nr� r )rM r� re r� r{ )rM rz r ry re r� r{ ) r� r� r� re r Z_v_namesrA rL r� r/ r� rx ) r� r r^ r� rc rb r! r� r� r$ rF rF rG rW R s* zGenericTable.indexablesc K s t d��d S )Nz cannot write on an generic table)r� )r� r� rF rF rG rf u s zGenericTable.writeN)r] r^ r_ rn r� rG r r& r� ro rp r, r� r� r rW rf rF rF rF rG rX 7 s #rX c s` e Zd ZdZdZeZdZe� d�Z e dd�dd��Z d� fd d � Z dd d d �� fdd� Z� ZS )r\ za frame with a multi-indexrR r� z ^level_\d+$rL )r� c C s dS )NZappendable_multirF )r� rF rF rG rE � s z*AppendableMultiFrameTable.table_type_shortNc s| |d krg }n|dkr |j �� }| �|�\}| _t| jt�s@t�x"| jD ]}||krH|�d|� qHW t� j f ||d�|��S )NTr )rb r} ) r� r� rM r rA rW r� r� r� rf )r� rb r} r� r� )r� rF rG rf � s z AppendableMultiFrameTable.writez int | None)r� r� c sD t � j||||d�}|�� j�}|j�� fdd�|jjD ��|_|S )N)rZ r� r� r� c s g | ]}� j �|�rd n|�qS )N)� _re_levels�search)rS rM )r� rF rG rV � s z2AppendableMultiFrameTable.read.<locals>.<listcomp>)r� r� r� r rz r� r )r� rZ r� r� r� r; )r� )r� rG r� � s z AppendableMultiFrameTable.read)N)NNNN)r] r^ r_ rn rG r& r� r �re�compiler� rp rE rf r� r� rF rF )r� rG r\ y s r\ r& r( )rb r� r� r� c C s� | � |�}t|�}|d k r"t|�}|d ks4|�|�rB|�|�rB| S t|�� �}|d k rlt|�� �j|dd�}|�|�s�td d �g| j }|||< | jt|� } | S )NF)�sort) r� r6 �equals�uniquer �slicer r rX )rb r� r� r� r8 ZslicerrF rF rG rr � s rr r z str | tzinfo)rv r� c C s t �| �}|S )z+for a tz-aware type, return an encoded zone)r Z get_timezone)rv �zonerF rF rG r � s r znp.ndarray | Indexzstr | tzinfo | Noneznp.ndarray | DatetimeIndex)r rv r� r� c C s� t | t�r"| jdks"| j|ks"t�|dk rtt | t�rB| j}| j} n d}| �� } t|�}t| |d�} | �d�� |�} n|r�t j | dd�} | S )a coerce the values to a DatetimeIndex if tz is set preserve the input shape if possible Parameters ---------- values : ndarray or Index tz : str or tzinfo coerce : if we do not have a passed timezone, coerce to M8[ns] ndarray N)rM r zM8[ns])r� ) rA r' rv r� rM r1 r� rH r r rB r� )r rv r� rM rF rF rG r� � s r� )rM rz rJ r~ r� c C st t | t�st�|j}t|�\}}t|�}t�|�}t |t�sFt |j �rlt | |||t |dd �t |dd �|d�S t |t �r~td��tj|dd�} t�|�} | dkr�tjdd � | D �tjd �}t | |dt� �� |d �S | d k�rt| ||�}|j j} t | |d t� �| �|d �S | d k�r$t | ||||d�S t |tj��r>|j tk�sBt�|dk�sTt|��t� �� }t | ||||d �S d S )Nru rv )r r) ry ru rv rw z MultiIndex not supported here!F)r* r c S s g | ] }|� � �qS rF )� toordinal)rS r rF rF rG rV s z"_convert_index.<locals>.<listcomp>)r� )rw r� )�integerZfloating)r r) ry rw r� ) rA rL r� rM r� r� r� r� r) r$ r� rt r/ r* r� r r- rB r� Zint32rp Z Time32Col�_convert_string_arrayr} r� r� r� r/ ) rM rz rJ r~ rw r� r� r) r� r3 r r} rF rF rG r � sL r )r) rJ r~ r� c C s� |dkrt | �}n�|dkr$t| �}n�|dkrxy tjdd� | D �td�}W q� tk rt tjdd� | D �td�}Y q�X nT|dkr�t�| �}n@|d kr�t| d ||d �}n&|d kr�t�| d �}ntd |� ���|S )Nr� r� r c S s g | ]}t �|��qS rF )r r� )rS r rF rF rG rV 0 s z$_unconvert_index.<locals>.<listcomp>)r� c S s g | ]}t �|��qS rF )r r� )rS r rF rF rG rV 2 s )r� �floatr� )r rJ r~ r� r zunrecognized index type )r' r- rB r� r� r� r� )r< r) rJ r~ rz rF rF rG r' ' s$ r' r? z list[str])rM r6 r� c C s� |j }|jtkr|S |jj} tj|dd�} | dkr<td��n&| dkrNtd��n| dksb| dksb|S |j|dd �} t| �d ks�t �| d }|j } tj| dd�} | dk�rxjt | j d �D ]X} |� | �}tj|dd�} | dkr�t|�| kr�|| nd | � �}td |� d| � d���q�W t | ||��| j �}|j}t|t��rXt|�| ��pT|�d��pTd �}t|�pbd |�}|d k �r�|�|�}|d k �r�||k�r�|}|jd|� �dd�}|S )NF)r* r z+[date] is not implemented as a table columnr� z>too many timezones in this block, create separate data columnsr� r� )ZdowncastrQ r zNo.z Cannot serialize the column [z2] because its data contents are not [string] but [z] object dtyper z|S)r= )r r� r� rM r r- r� ZfillnarY r� r r� �igetr� r� r} rA r rO r� r� r� r� )rM r6 rp r{ r rJ r~ r� r� r� r3 r? r< r! r� Zerror_column_labelr| r} ZecirF rF rG ru @ sL ru z np.ndarray)r< rJ r~ r� c C s\ t | �r(t| �� �j�||�j�| j�} t| �� �}t dt � |��}t j | d|� �d�} | S )a Take a string-like that is object dtype and coerce to a fixed size string type. Parameters ---------- data : np.ndarray[object] encoding : str errors : str Handler for encoding errors. Returns ------- np.ndarray[fixed-length-string] rQ �S)r� )rY r, r� rL �encoder� r� r� r r� � libwriters�max_len_string_arrayrB r� )r< rJ r~ �ensuredr} rF rF rG r� � s r� c C s� | j }tj| �� td�} t| �rvt�t| ��}d|� �}t | d t �r^t | �j j ||d�j} n| j|dd�jtdd�} |dkr�d}t�| |� | �|�S ) a* Inverse of _convert_string_array. Parameters ---------- data : np.ndarray[fixed-length-string] nan_rep : the storage repr of NaN encoding : str errors : str Handler for encoding errors. Returns ------- np.ndarray[object] Decoded data. )r� �Ur )r~ F)r= Nro )r� rB r� r� r� rY r� r� r rA r r, rL rD r� r� Z!string_array_replace_from_nan_repr� )r< r rJ r~ r� r} r� rF rF rG r� � s r� )r r� rJ r~ c C s6 t |t�stt|���t|�r2t|||�}|| �} | S )N)rA rL r� r� � _need_convert�_get_converter)r r� rJ r~ �convrF rF rG r� � s r� )r) rJ r~ c s8 | dkrdd� S | dkr&� �fdd�S t d| � ���d S )Nr� c S s t j| dd�S )NzM8[ns])r� )rB r� )r� rF rF rG r� � � z _get_converter.<locals>.<lambda>r� c s t | d � �d�S )N)r rJ r~ )r� )r� )rJ r~ rF rG r� � s z invalid kind )r� )r) rJ r~ rF )rJ r~ rG r� � s r� )r) r� c C s | dkr dS dS )N)r� r� TFrF )r) rF rF rG r� � s r� z Sequence[int])rM r� r� c C sl t |t�st|�dk r td��|d dkrh|d dkrh|d dkrht�d| �}|rh|�� d }d|� �} | S ) z� Prior to 0.10.1, we named values blocks like: values_block_0 an the name values_0, adjust the given name if necessary. Parameters ---------- name : str version : Tuple[int, int, int] Returns ------- str � z6Version is incorrect, expected sequence of 3 integers.r rQ r� r� zvalues_block_(\d+)Zvalues_)rA rL rY r� r� r� r� )rM r� r� �grprF rF rG r[ � s $ r[ )� dtype_strr� c C s� t | �} | �d�s | �d�r"d}n�| �d�r2d}n�| �d�rBd}n�| �d�sV| �d�r\d}nn| �d�rld}n^| �d �r|d }nN| �d �r�d }n>| �d �r�d }n.| �d �r�d}n | dkr�d}ntd| � d���|S )zA Find the "kind" string describing the given dtype name. r� r r� rd rO r� r� r� � timedeltar� rr r� r� r� zcannot interpret dtype of [r� )rH r7 r� )r� r) rF rF rG r� s. r� r )r< c C sb t | t�r| j} | jj�d�d }| jjdkr@t�| � d��} nt | t �rP| j } t�| �} | |fS )zJ Convert the passed data into a storable form and a dtype string. r� r )r� �Mr, ) rA r0 r� r� rM ri r) rB r� r0 r+ r1 )r< r� rF rF rG r� + s r� c @ s< e Zd ZdZddddd�dd�Zdd � Zd d � Zd d � ZdS )rf z� Carries out a selection operation on a tables.Table object. Parameters ---------- table : a Table object where : list of Terms (or convertible to) start, stop: indices to start and/or stop selection Nr� z int | None)re r� r� c C sR || _ || _|| _|| _d | _d | _d | _d | _t|��rt t ��� t j |dd�}|dksd|dk�rt �|�}|jt jkr�| j| j }}|d kr�d}|d kr�| j j}t �||�| | _nVt|jjt j��r| jd k r�|| jk �� �s | jd k �r|| jk�� �rt d��|| _W d Q R X | jd k�rN| �|�| _| jd k �rN| j�� \| _| _d S )NF)r* r� �booleanr z3where must have index locations >= start and < stop)re rZ r� r� � conditionr� Ztermsr� r! r r� r r- rB r� r� Zbool_r� r� � issubclassr� r� r� �generate�evaluate)r� re rZ r� r� �inferredrF rF rG r� M s< zSelection.__init__c C s~ |dkr dS | j �� }yt||| j jd�S tk rx } z2d�|�� �}td|� d|� d��}t|�|�W dd}~X Y nX dS )z'where can be a : dict,list,tuple,stringN)rS rJ r� z- The passed where expression: a* contains an invalid variable reference all of the variable references must be a reference to an axis (e.g. 'index' or 'columns'), or a data_column The currently defined references are: z ) re rS r3 rJ � NameErrorr� r� r r� )r� rZ rY r Zqkeysr� rF rF rG r� | s zSelection.generatec C sX | j dk r(| jjj| j �� | j| jd�S | jdk rB| jj�| j�S | jjj| j| jd�S )z( generate the selection N)r� r� ) r� re Z read_wherery r� r� r� r� r� )r� rF rF rG r� � s zSelection.selectc C s� | j | j }}| jj}|dkr$d}n|dk r4||7 }|dkrB|}n|dk rR||7 }| jdk rx| jjj| j�� ||dd�S | jdk r�| jS t� ||�S )z( generate the selection Nr T)r� r� r� ) r� r� re r� r� Zget_where_listry r� rB r� )r� r� r� r� rF rF rG r� � s zSelection.select_coords)NNN)r] r^ r_ rn r� r� r� r� rF rF rF rG rf A s * rf ) rq NNFNTNNNNrj r@ ) Nr� rj NNNNFN)N)F)�rn � __future__r � contextlibr r= r� r r r� r� r� �textwrapr �typingr r r r r r r� �numpyrB Zpandas._configr r Z pandas._libsr r r� Zpandas._libs.tslibsr Zpandas._typingr r r r r Zpandas.compat._optionalr Zpandas.compat.pickle_compatr Z pandas.errorsr Zpandas.util._decoratorsr Zpandas.core.dtypes.commonr r r r r r r! r" r# r$ Zpandas.core.dtypes.missingr% r� r& r' r( r) r* r+ r, r- r. r/ Zpandas.core.arraysr0 r1 r2 Zpandas.core.common�core�commonr Z pandas.core.computation.pytablesr3 r4 Zpandas.core.constructionr5 Zpandas.core.indexes.apir6 Zpandas.core.internalsr7 r8 Zpandas.io.commonr9 Zpandas.io.formats.printingr: r; rl r< r= r> r? r� rI rH rK rN rR r[ r r\ r` �Warningra rX rb r� rc Z duplicate_docr. rC r Z dropna_docZ format_docZ config_prefixZregister_optionZis_boolZis_one_of_factoryrk ro rp r� r� r� r� r� rt r� r� r� r� r� r� rU r4 rV r� r] r� r[ rY rZ rX r\ rr r r� r r' ru r� r� r� r� r� r[ r� r� rf rF rF rF rG �<module> s$ 0 0 (+ Np 1 b _ l fd1B+ %=R ' !
| ver. 1.4 |
Github
|
.
| PHP 8.3.23 | Генерация страницы: 0.05 |
proxy
|
phpinfo
|
Настройка