Файловый менеджер - Редактировать - /home/digitalm/venv/lib/python3.7/site-packages/pip/_vendor/chardet/__pycache__/universaldetector.cpython-37.pyc
Назад
B �5�g : � @ s� d Z ddlZddlZddlZddlmZmZmZ ddlm Z ddl m Z ddl m Z mZmZ ddlmZ dd lmZ dd lmZ dd lmZ dd lmZ dd lmZ ddl m Z G dd� d�Z dS )a Module containing the UniversalDetector detector class, which is the primary class a user of ``chardet`` should use. :author: Mark Pilgrim (initial port to Python) :author: Shy Shalom (original C code) :author: Dan Blanchard (major refactoring for 3.0) :author: Ian Cordasco � N)�List�Optional�Union� )�CharSetGroupProber)� CharSetProber)� InputState�LanguageFilter� ProbingState)�EscCharSetProber)� Latin1Prober)�MacRomanProber)�MBCSGroupProber)� ResultDict)�SBCSGroupProber)� UTF1632Proberc @ s� e Zd ZdZdZe�d�Ze�d�Ze�d�Z dddd d d d d d�Z dddd dddd�Z e j dfe edd�dd�Zeed�dd��Zeed�d d ��Zeee d�d d ��Zdd�d d!�Zeeef dd"�d#d$�Ze d�d%d&�Z dS )'�UniversalDetectoraq The ``UniversalDetector`` class underlies the ``chardet.detect`` function and coordinates all of the different charset probers. To get a ``dict`` containing an encoding and its confidence, you can simply run: .. code:: u = UniversalDetector() u.feed(some_bytes) u.close() detected = u.result g�������?s [�-�]s (|~{)s [�-�]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)z iso-8859-1z iso-8859-2z iso-8859-5z iso-8859-6z iso-8859-7z iso-8859-8z iso-8859-9z iso-8859-13z ISO-8859-11ZGB18030�CP949zUTF-16)�asciiz iso-8859-1ztis-620z iso-8859-9�gb2312zeuc-krzutf-16leFN)� lang_filter�should_rename_legacy�returnc C sd d | _ d | _g | _d dd d�| _d| _d| _tj| _d| _ || _ t � t �| _d| _|| _| �� d S )Ng )�encoding� confidence�languageF� )�_esc_charset_prober�_utf1632_prober�_charset_probers�result�done� _got_datar � PURE_ASCII� _input_state� _last_charr �logging� getLogger�__name__�logger�_has_win_bytesr �reset)�selfr r � r- �F/tmp/pip-install-251nq386/pip/pip/_vendor/chardet/universaldetector.py�__init__d s zUniversalDetector.__init__)r c C s | j S )N)r$ )r, r- r- r. � input_state{ s z UniversalDetector.input_statec C s | j S )N)r* )r, r- r- r. � has_win_bytes s z UniversalDetector.has_win_bytesc C s | j S )N)r )r, r- r- r. �charset_probers� s z!UniversalDetector.charset_probersc C sj dddd�| _ d| _d| _d| _tj| _d| _| jr>| j� � | j rN| j � � x| j D ] }|� � qVW dS )z� Reset the UniversalDetector and all of its probers back to their initial states. This is called by ``__init__``, so you only need to call this directly in between analyses of different documents. Ng )r r r Fr ) r r! r" r* r r# r$ r% r r+ r r )r, �proberr- r- r. r+ � s zUniversalDetector.reset)�byte_strr c C s� | j r dS |sdS t|t�s$t|�}| js�|�tj�rFdddd�| _nv|�tjtj f�rhdddd�| _nT|�d�r�dddd�| _n:|�d �r�d ddd�| _n |�tj tj f�r�d ddd�| _d | _| jd dk r�d | _ dS | j t jk�r(| j�|�r�t j| _ n*| j t jk�r(| j�| j| ��r(t j| _ |dd� | _| j�sFt� | _| jjtjk�r�| j�|�tjk�r�| jj | j� � dd�| _d | _ dS | j t jk�r�| j �s�t | j �| _ | j �|�tjk�r�| j j | j � � | j j!d�| _d | _ n�| j t jk�r�| j"�sBt#| j �g| _"| j t$j%@ �r&| j"�&t'� � | j"�&t(� � | j"�&t)� � x@| j"D ]6}|�|�tjk�rJ|j |� � |j!d�| _d | _ P �qJW | j*�|��r�d | _+dS )a� Takes a chunk of a document and feeds it through all of the relevant charset probers. After calling ``feed``, you can check the value of the ``done`` attribute to see if you need to continue feeding the ``UniversalDetector`` more data, or if it has made a prediction (in the ``result`` attribute). .. note:: You should always call ``close`` when you're done feeding in your document if ``done`` is not already ``True``. Nz UTF-8-SIGg �?� )r r r zUTF-32s �� zX-ISO-10646-UCS-4-3412s ��zX-ISO-10646-UCS-4-2143zUTF-16Tr ���),r! � isinstance� bytearrayr" � startswith�codecs�BOM_UTF8r � BOM_UTF32_LE� BOM_UTF32_BE�BOM_LE�BOM_BEr$ r r# �HIGH_BYTE_DETECTOR�search� HIGH_BYTE� ESC_DETECTORr% � ESC_ASCIIr r �stater � DETECTING�feed�FOUND_IT� charset_name�get_confidencer r r r r r r �NON_CJK�appendr r r �WIN_BYTE_DETECTORr* )r, r4 r3 r- r- r. rG � s� zUniversalDetector.feedc C s� | j r | jS d| _ | js&| j�d� n�| jtjkrBdddd�| _n�| jtjk�rd}d}d}x,| j D ]"}|snqd|� � }||krd|}|}qdW |�r|| j k�r|j }|dk s�t �|�� }|� � }|�d �r�| jr�| j�||�}| jr�| j�|p�d�� |�}|||jd�| _| j�� tjk�r�| jd dk�r�| j�d � xn| j D ]d}|�sL�q>t|t��r�xF|jD ] }| j�d |j |j|� � � �q`W n| j�d |j |j|� � � �q>W | jS ) z� Stop analyzing the current document and come up with a final prediction. :returns: The ``result`` attribute, a ``dict`` with the keys `encoding`, `confidence`, and `language`. Tzno data received!r g �?r5 )r r r Ng ziso-8859r z no probers hit minimum thresholdz%s %s confidence = %s) r! r r" r) �debugr$ r r# rB r rJ �MINIMUM_THRESHOLDrI �AssertionError�lowerr9 r* � ISO_WIN_MAP�getr � LEGACY_MAPr �getEffectiveLevelr& �DEBUGr7 r �probers) r, �prober_confidence�max_prober_confidence� max_proberr3 rI �lower_charset_namer � group_proberr- r- r. �close sh zUniversalDetector.close) r( � __module__� __qualname__�__doc__rO �re�compiler@ rC rM rR rT r �ALL�boolr/ �property�intr0 r1 r r r2 r+ r �bytesr8 rG r r] r- r- r- r. r 8 sB r ) r` r: r& ra �typingr r r �charsetgroupproberr � charsetproberr �enumsr r r � escproberr � latin1proberr Zmacromanproberr �mbcsgroupproberr Z resultdictr �sbcsgroupproberr Z utf1632proberr r r- r- r- r. �<module>$ s
| ver. 1.4 |
Github
|
.
| PHP 8.3.23 | Генерация страницы: 0.03 |
proxy
|
phpinfo
|
Настройка