Файловый менеджер - Редактировать - /home/digitalm/venv/lib/python3.7/site-packages/pandas/tests/io/parser/__pycache__/test_multi_thread.cpython-37.pyc
Назад
B �5�g) � @ s� d Z ddlmZ ddlmZ ddlmZ ddlZddl Z ddl Z ddl m Z ddl mZ dd� Ze jjd d � �Zd d � Ze jjd d� �ZdS )zd Tests multithreading behaviour for reading and parsing files for each parser defined in parsers.py � )� ExitStack)�BytesIO)� ThreadPoolN)� DataFramec C s\ t tj�| d�td�d�}d|d< d|d< d|d< tjd| dd �|d < tj| d d �|d < |S )z� Construct a DataFrame for testing. Parameters ---------- num_rows : int The number of rows for our DataFrame. Returns ------- df : DataFrame � Zabcde)�columnsZfoo�barZbazz20000101 09:00:00�s)Zperiods�freq�dateZint64)Zdtype�int)r �np�randomZrand�list�pdZ date_rangeZarange)�num_rows�df� r �^/home/digitalm-up/venv/lib/python3.7/site-packages/pandas/tests/io/parser/test_multi_thread.py�_construct_dataframe s r c s� | }d� d}� fdd�t |�D �}t� �V��fdd�|D �}��td��}|�|j|�}|d }x|D ]}t�||� qfW W d Q R X d S )Ni' �d c s( g | ] }d � dd� t� �D ���� �qS )� c s s( | ] }|d �d|d �d|d ��V qdS )�d�,Nr )�.0�ir r r � <genexpr>/ s zBtest_multi_thread_string_io_read_csv.<locals>.<listcomp>.<genexpr>)�join�range�encode)r �_)� max_row_ranger r � <listcomp>/ s z8test_multi_thread_string_io_read_csv.<locals>.<listcomp>c s g | ]}� � t|���qS r )� enter_contextr )r �b)�stackr r r" 5 s � r )r r r# r �map�read_csv�tm�assert_frame_equal) � all_parsers�parserZ num_filesZ bytes_to_df�files�pool�resultsZ first_result�resultr )r! r% r �$test_multi_thread_string_io_read_csv' s r1 c s| ��fdd�}� �fdd�t ��D �}t�d��}|�||�}W dQ R X |d j}x|dd� D ] } || _q`W t�|�} | S ) a� Generate a DataFrame via multi-thread. Parameters ---------- parser : BaseParser The parser object to use for reading the data. path : str The location of the CSV file to read. num_rows : int The number of rows to read per task. num_tasks : int The number of tasks to use for reading this DataFrame. Returns ------- df : DataFrame c sB | \}}|s"� j �dd|dgd�S � j �ddt|�d |dgd�S )aj Create a reader for part of the CSV. Parameters ---------- arg : tuple A tuple of the following: * start : int The starting row to start for parsing CSV * nrows : int The number of rows to read. Returns ------- df : DataFrame r r )� index_col�header�nrows� parse_datesN� � )r2 r3 Zskiprowsr4 r5 )r( r )�arg�startr4 )r, �pathr r �readerT s z0_generate_multi_thread_dataframe.<locals>.readerc s g | ]}� | � � � f�qS r r )r r )r � num_tasksr r r" w s z4_generate_multi_thread_dataframe.<locals>.<listcomp>)Z processesNr r6 )r r r' r r �concat) r, r: r r< r; �tasksr. r/ r3 �r�final_dataframer )r r< r, r: r � _generate_multi_thread_dataframe@ s # rA c C sV d}d}| }d}t |�}t�|��*}|�|� t||||�}t�||� W d Q R X d S )N� i�� z__thread_pool_reader__.csv)r r) Z ensure_cleanZto_csvrA r* )r+ r< r r, � file_namer r: r@ r r r �)test_multi_thread_path_multipart_read_csv� s rD )�__doc__� contextlibr �ior Zmultiprocessing.poolr �numpyr ZpytestZpandasr r Zpandas._testingZ_testingr) r �markZslowr1 rA rD r r r r �<module> s F
| ver. 1.4 |
Github
|
.
| PHP 8.3.23 | Генерация страницы: 0.03 |
proxy
|
phpinfo
|
Настройка