$64 GRAYBYTE WORDPRESS FILE MANAGER $67

B

<_9Y�0�@s�dZd
dl
Z
d
dlZd
dlZddlmZ
ddlmZmZm	Z	
ddl
mZ
ddlm
Z

ddlmZ
dd	lmZ
Gd
d�de�ZdS)a

Module containing the UniversalDetector detector class, which is the primary
class a user of ``chardet`` should use.

:author: Mark Pilgrim (initial port to Python)
:author: Shy Shalom (original C code)
:author: Dan Blanchard (major refactoring for 3.0)
:author: Ian Cordasco
�N�
)
�CharSetGroupProber)�
InputState�LanguageFilter�ProbingState)
�EscCharSetProber)
�Latin1Prober)
�MBCSGroupProber)
�SBCSGroupProberc	@sneZ
dZd
ZdZe�d�
Ze�d�
Ze�d�
Z	dddd	d
ddd
d�Z
ejf
dd�
Z
dd�Zdd�Zdd�ZdS)�UniversalDetectoraq

    The ``UniversalDetector`` class underlies the ``chardet.detect`` function
    and coordinates all of the different charset probers.

    To get a ``dict`` containing an encoding and its confidence, you can simply
    run:

    .. code::

            u = UniversalDetector()
            u.feed(some_bytes)
            u.close()
            detected = u.result

    g�������?s[�-�]s(|~{)s[�-�]zWindows-1252zWindows-1250zWindows-1251zWindows-1256zWindows-1253zWindows-1255zWindows-1254zWindows-1257)z
iso-8859-1z
iso-8859-2z
iso-8859-5z
iso-8859-6z
iso-8859-7z
iso-8859-8z
iso-8859-9ziso-8859-13cCsNd|_g|_
d|_d|_d|_d|_d|_|
|_t�	t
�
|_d|_|�
�
dS)
N)�_esc_charset_prober�_charset_probers�result�done�	_got_data�_input_state�
_last_char�lang_filter�logging�	getLogger�__name__�logger�_has_win_bytes�reset)�selfr�r�J/opt/alt/python37/lib/python3.7/site-packages/chardet/universaldetector.py�__init__Qs










zUniversalDetector.__init__c
CsZd
dd
d�|_d|_
d|_d|_tj|_d|_|jr>|j�	�
x|j
D]}
|
�	�
qFWd
S)z�
        Reset the UniversalDetector and all of its probers back to their
        initial states.  This is called by ``__init__``, so you only need to
        call this directly in between analyses of different documents.
        Ng)�encoding�
confidence�languageF�)rrrrr�
PURE_ASCIIrrrrr
)r�proberrrrr^s









zUniversalDetector.resetcCs>|jr
d
St
|
�
sd
St|
t�s(t|
�
}
|js�|
�tj�
rJdddd�|_nv|
�tj	tj
f�
rldddd�|_nT|
�d�
r�dddd�|_n:|
�d	�
r�d
ddd�|_n |
�tjtjf�
r�dddd�|_d|_|jd
d
k	r�d|_d
S|j
tjk�
r.|j�|
�
�
rtj|_
n*|j
tjk�
r.|j�|j|
�
�
r.tj|_
|
dd
�|_|j
tjk�
r�|j�
s^t|j�
|_|j�|
�
tjk�r:|jj|j��|jjd�|_d|_n�|j
tjk�r:|j�
s�t |j�
g
|_|jt!j"@�
r�|j�#t$��

|j�#t%��

x@|jD]6}|�|
�
tjk�
r�|j|��|jd�|_d|_P�
q�W|j&�|
�
�r:d|_'d
S)a�

        Takes a chunk of a document and feeds it through all of the relevant
        charset probers.

        After calling ``feed``, you can check the value of the ``done``
        attribute to see if you need to continue feeding the
        ``UniversalDetector`` more data, or if it has made a prediction
        (in the ``result`` attribute).

        .. note::
           You should always call ``close`` when you're done feeding in your
           document if ``done`` is not already ``True``.
        Nz	UTF-8-SIGg�?�)rrr zUTF-32s��zX-ISO-10646-UCS-4-3412s��zX-ISO-10646-UCS-4-2143zUTF-16Tr���)(r�len�
isinstance�	bytearrayr�
startswith�codecs�BOM_UTF8r�BOM_UTF32_LE�BOM_UTF32_BE�BOM_LE�BOM_BErrr"�HIGH_BYTE_DETECTOR�search�	HIGH_BYTE�ESC_DETECTORrZ	ESC_ASCIIrrr�feedrZFOUND_IT�charset_name�get_confidencer r
r	rZNON_CJK�appendr
r�WIN_BYTE_DETECTORr)rZbyte_strr#rrrr4os|
















































zUniversalDetector.feedc
		Cs�
|jr|j
Sd
|_|js&|j�d�

n�|jtjkrBdddd�|_
n�|jtjkr�d}
d}d}x,|j	D]"}|slqb|�
�}
|
|krb|
}|}qbW|r�||jkr�|j}|j�
�}|�
�}|�d	�
r�|jr�|j�||�}|||jd�|_
|j��tjk�
rz|j
d
dk�
rz|j�d�

xn|j	D]d}|�
s �
qt|t��
rZxF|jD] }|j�d|j|j|�
��
�
q4Wn|j�d|j|j|�
��
�
qW|j
S)
z�
        Stop analyzing the current document and come up with a final
        prediction.

        :returns:  The ``result`` attribute, a ``dict`` with the keys
                   `encoding`, `confidence`, and `language`.
        Tzno data received!�asciig�?r$)rrr Ngziso-8859rz no probers hit minimum thresholdz%s %s confidence = %s)rrrr�debugrrr"r2r
r6�MINIMUM_THRESHOLDr5�lowerr)r�ISO_WIN_MAP�getr �getEffectiveLevelr�DEBUGr'rZprobers)	rZprober_confidenceZmax_prober_confidenceZ
max_proberr#r5Zlower_charset_namerZgroup_proberrrr�close�s`	










































zUniversalDetector.closeN)r�
__module__�__qualname__�__doc__r;�re�compiler0r3r8r=rZALLrrr4rArrrrr3s"














mr)rDr*rrEZcharsetgroupproberrZenumsrrrZ	escproberrZlatin1proberrZmbcsgroupproberr	Zsbcsgroupproberr
�objectrrrrr�<module>$s