xi d Z ddlmZmZ ddlmZmZmZmZm Z m
Z
m Z ddl m
Z
ddlmZ ddlmZ ddlZddlZddlmZmZmZmZmZmZmZm Z m Z m Z d Z G d
d e Z G d d
e Z!y)aK
Constrained Logits Processor for ACE-Step Language Model
This module implements a finite state machine (FSM) based logits processor that constrains
the language model's output to follow specific formats and value ranges during music generation.
Key Features:
- Enforces structured metadata generation (BPM, duration, keyscale, etc.)
- Validates numeric ranges (BPM: 30-300, Duration: 10-600s)
- Ensures proper formatting for musical metadata
- Prevents generation of invalid tokens or formats
- Supports constrained audio code generation (0-63999)
The FSM guides the model through different states to ensure outputs conform to expected
schema requirements without post-processing corrections.
Usage:
processor = ConstrainedLogitsProcessor(tokenizer, mode="metadata")
outputs = model.generate(inputs, logits_processor=[processor])
)Enumauto)OptionalDictAnyTupleListCallableSet)logger)
AutoTokenizer)LogitsProcessorN)
VALID_LANGUAGESKEYSCALE_NOTESKEYSCALE_ACCIDENTALSKEYSCALE_MODESVALID_KEYSCALESBPM_MINBPM_MAX DURATION_MIN DURATION_MAXVALID_TIME_SIGNATURESi c ` e Zd ZdZ e Z e Z e Z e Z e Z e Z
e Z e Z e Z
e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z y)FSMStatez3Finite State Machine states for metadata generationN) __name__
__module__ __qualname____doc__r THINK_TAGNEWLINE_AFTER_THINKBPM_NAME BPM_VALUENEWLINE_AFTER_BPM CAPTION_NAME
CAPTION_VALUE
DURATION_NAMEDURATION_VALUENEWLINE_AFTER_DURATION GENRES_NAME GENRES_VALUENEWLINE_AFTER_GENRES
KEYSCALE_NAMEKEYSCALE_VALUENEWLINE_AFTER_KEYSCALE
LANGUAGE_NAMELANGUAGE_VALUE TIMESIG_NAME
TIMESIG_VALUENEWLINE_AFTER_TIMESIG
THINK_END_TAGCODES_GENERATION COMPLETED C/mnt/workspace/ACE-Step-1.5/acestep/constrained_logits_processor.pyr r 5 s =I &vHI 6LFMFMVN !V&K6L 6FMVN !VFMVN6LFM FFMvIr8 r c e Zd ZdZ dVdedededee dedee f d Z d
ed ee
fd Z d
Z defdZ
defdZdefdZeded efd ZdefdZdefdZdWdeeeee f fdZd Zd Zd ed ee fd Zd Zd ej6 d e e d dfd!Z d ee ed"f e e f fd#Z dXd$e e d%ed&ed ee ed"f e e f fd'Z!d ee ed"f e e f fd(Z"d) Z#d* Z$d+ Z%defd,Z&dYd-ed.ed/e'd0efd1Z(d2 Z)d3 Z*d.ed ee fd4Z+d5ed efd6Z,d7ed.ed ee fd8Z-d e e fd9Z.d: Z/d;ee0 fd<Z1defd=Z2d>ed e e fd?Z3d@edAed e e fdBZ4dCee ed"f e e f d e e fdDZ5dEej6 d@edAed efdFZ6dEej6 d efdGZ7d e e fdHZ8d efdIZ9d e e fdJZ:d e e fdKZ;dLejx d ejz d ejz fdMZ>dLejx d efdNZ?d ejz d ejz fdOZ@dPed ee e fdQZAdLejx d ejz d ejz fdRZBdS ZCdTefdUZDy)Z"MetadataConstrainedLogitsProcessoru8
FSM-driven LogitsProcessor that constrains generation to produce valid metadata.
This processor enforces the following format:
bpm: [30-300]
caption: [text without code blocks, ends with period + newline]
duration: [10-600]
keyscale: [A-G][#/♭]? [major/minor]
language: [en/zh/ja/ko/es/fr/de/uk/ru/...]
timesignature: [2/3/4/6]
It uses token masking (setting invalid token logits to -inf) to enforce constraints.
For numeric fields, it uses early-blocking to prevent out-of-range values.
For field transitions (e.g., end of numeric value), it compares P(newline) vs P(digit).
For caption field, it blocks code blocks and newlines, and only transitions when
the previous token was a period and newline has the highest probability.
N tokenizerenableddebuggenres_vocab_path skip_genres max_durationc || _ || _ || _ || _ ||nt | _ d| _ d| _ d| _ dddddddd| _ d| _
d| _ d| _ d| _
d| _ d| _ d| _ t" j$ | _ d| _ d| _ g | _ d| _ d| _ d| _ d| _ g | _ d| _ | j; |xs^ t< j> jA t< j> jC t< j> jE tF d | _$ g | _% d| _& i | _' i | _( g | _) i | _* | jW tX tZ d t\ | j
d d
t^ id | _0 tc | j` d d
| j` d d dz D cg c]
}te | c}| _3 tc | j` d d
| j` d d dz D cg c]
}te | c}| _4 | j` d d
D cg c]
}te | c}| _5 | jm | _7 | jq | jf dd | _9 | jq | jh dd | _: | jq | jj dd | _; | jy | _= | j} t" j$ dt" j~ dt" j dt" j dt" j dt" j d t" j d t" j d t" j dt" j d i
| _H | j yc c}w c c}w c c}w ) aT
Initialize the constrained logits processor.
This processor should be initialized once when loading the LLM and reused
for all generations.
Args:
tokenizer: The tokenizer to use for encoding/decoding
enabled: Whether to enable constrained decoding
debug: Whether to print debug information
genres_vocab_path: Path to genres vocabulary file
skip_genres: Whether to skip genres field generation
max_duration: Maximum duration in seconds (default: DURATION_MAX from constants)
NFbpmcaptiondurationkeyscalelanguage
timesignaturegenresr cot zgenres_vocab.txtg )minmax valid_values)rD rF rI rD rM rN rF rI zbpm:bpm: context_prefix_for_matching context_prefix_for_tokenization duration:
duration: ztimesignature:timesignature: z
zcaption:zgenres: keyscale: language:)Jr< r= r> r@ r rA skip_caption
skip_languagerE user_provided_metadatametadata_temperaturecodes_temperaturetarget_duration target_codes codes_countstop_at_reasoninggeneration_phaser r stateposition_in_stateaccumulated_valueaccumulated_token_idscaption_after_newlinecaption_token_countcaption_endingpending_field_nameuser_field_token_queuecurrent_user_field_precompute_tokensospathjoindirnameabspath__file__r? genres_vocabgenres_vocab_mtime genres_triecaption_genres_triecaption_matched_genres_char_to_tokens _precompute_char_token_mappingr r r r field_specsrangestrvalid_bpm_valuesvalid_duration_valuesvalid_timesig_values_build_keyscale_prefix_treekeyscale_prefix_tree_build_numeric_prefix_treebpm_prefix_treeduration_prefix_treetimesig_prefix_tree_build_language_prefix_treelanguage_prefix_tree_load_genres_vocabr r! r$ r& r) r, r/ r1 r4
fixed_strings_build_state_transitions)selfr< r= r> r? r@ rA vs r9 __init__z+MetadataConstrainedLogitsProcessor.__init__f s , #
& -9,DL, ! "&* !A
# 6:!26 15 +/ ! (- &+ ''
!" !# 02" &+"#$ #"$ 24#15
! "3 "
bggll GGOOBGGOOH 5 68J7
(*), !#)+ 13#/1
++-
#73 ,T5F5FG ,.C D
27t7G7G7Nu7UW[WgWghmWnotWuxyWy1z {1zAQ1z { 6;D > ! !(.,3 ?
%)$C$C & &(3,8 %D %
!
$(#B#B % %(8,= $C $
%)$D$D$F! !
( ($ v ! !: " "K ) " "K " "K ! !#3 " "J
%%'c !| &K$gs $O/O O
current_fieldreturnc g d}t j t j t j t j t j
t j t j d} |j | }t |dz t | D ]C }|| }|dk( r
| j r|dk( r
| j r,|dk( r
| j r>|| c S t j S # t $ r t j cY S w xY w)a
Get the next field state. Always returns the next field's NAME state,
even if the field is user-provided (we still need to generate the field name).
Args:
current_field: Current field name ("bpm", "caption", "duration", "genres", "keyscale", "language", "timesignature")
Returns:
Next FSMState (NAME state of next field), or THINK_END_TAG if no more fields
)rD rE rF rJ rG rH rI rP rJ rE rH )r r! r$ r& r) r, r/ r1 index
ValueErrorr4 r lenr@ r\ r] )r r field_orderfield_to_state current_idxifields r9 _get_next_field_statez8MetadataConstrainedLogitsProcessor._get_next_field_state s h $$ ,, .. ** .. .. % 2 2
*%++M:K
{QK(89A NE T%5%5 !d&7&7
"t'9'9 "%( ( : %%%' *)) ) *s 2C1 1D
D
c
t j t j t j t j t j t j
t j
t j i| _ t j | j t j < | j d | j t j < | j sWt j | j t j < | j d | j t j < t j | j t j < | j d | j t j < | j sWt j | j t j" < | j d | j t j < t j$ | j t j& < | j d | j t j$ < | j( sWt j* | j t j, < | j d | j t j* < t j. | j t j0 < t j | j t j. < y)z;Build state transition map based on user-provided metadata.rD rE rF rJ rG rH N)r r r r! r4 r5 r6
next_stater" r r\ r% r$ r' r& r@ r* r) r- r, r] r0 r/ r2 r1 r s r9 r z;MetadataConstrainedLogitsProcessor._build_state_transitions5 s
< < ( ((*;*; " "H$=$= % %x'9'9
.6-?-?))*.2.H.H.O**+ 5=5K5KDOOH 1 1 26:6P6PQZ6[DOOH 2 2 3 3;2I2I../373M3Mj3Y//0 4<4I4IDOOH 0 0 1595O5OPX5YDOOH 1 1 2 3;2I2I../373M3Mj3Y//0 !!6>6M6MDOOH 2 2 37;7Q7QR\7]DOOH 3 3 4 2:1G1G--.2:2H2H../r8 skipc 2 || _ | j y)zDSet whether to skip genres generation and rebuild state transitions.N)r@ r r r s r9 set_skip_genresz2MetadataConstrainedLogitsProcessor.set_skip_genres` s %%'r8 c 2 || _ | j y)zESet whether to skip caption generation and rebuild state transitions.N)r\ r r s r9 set_skip_captionz3MetadataConstrainedLogitsProcessor.set_skip_captione s %%'r8 c 2 || _ | j y)zFSet whether to skip language generation and rebuild state transitions.N)r] r r s r9 set_skip_languagez4MetadataConstrainedLogitsProcessor.set_skip_languagej s ! %%'r8 rE c | s| S | j d }g }|D ]& }|j }|s|j | ( dj | S )a
Post-process caption to remove YAML multi-line formatting.
Converts YAML-style multi-line text (with newlines and leading spaces)
to a single-line string.
Example:
Input: "An emotional ballad.\n The track opens with piano.\n More text."
Output: "An emotional ballad. The track opens with piano. More text."
Args:
caption: Raw caption text with possible YAML formatting
Returns:
Clean single-line caption
rX )splitstripappendrs )rE lines
cleaned_lineslinestrippeds r9 postprocess_captionz6MetadataConstrainedLogitsProcessor.postprocess_captiono sZ " N
d#
Dzz|H $$X. xx
&&r8 stopc || _ y)a
Set whether to stop generation after tag.
Args:
stop: If True, generation will stop immediately after tag is generated.
If False, generation continues to codes generation phase.
N)rd )r r s r9 set_stop_at_reasoningz8MetadataConstrainedLogitsProcessor.set_stop_at_reasoning s "& r8 phasec 8 |dvrt d|d || _ y)u
Set the generation phase.
Args:
phase: "cot" for CoT metadata generation, "codes" for audio codes generation,
or "understand" for audio understanding (codes → metadata + lyrics).
When phase is "codes" and the input prompt already contains ,
the FSM will skip metadata generation and go directly to codes generation.
When phase is "understand", generate CoT metadata then free-form lyrics.
)rK codes
understandzInvalid generation phase: z). Must be 'cot', 'codes', or 'understand'N)r re )r r s r9 set_generation_phasez7MetadataConstrainedLogitsProcessor.set_generation_phase s+ 6 6 9%Bk lm m % r8 metadatac j |i }dD ]( }||v r|| | j |< d| j |< * | j | j r`| j j D cg c]
\ }}| | }}}|rt j d| yt j d yyc c}}w )a
Set user-provided metadata fields. Fields that are provided will be used directly
instead of generating. Fields that are None will be generated.
Args:
metadata: Dictionary with optional fields:
- "bpm": Optional[str] - e.g., "120"
- "caption": Optional[str] - e.g., "A melodic piano piece..."
- "duration": Optional[str] - e.g., "234"
- "keyscale": Optional[str] - e.g., "G major"
- "language": Optional[str] - e.g., "en"
- "timesignature": Optional[str] - e.g., "4"
- "genres": Optional[str] - e.g., "Pop Rock"
If None, clears all user-provided metadata.
NrC z User provided metadata fields: z7No user-provided metadata, all fields will be generated)r^ r r> itemsr )r r r kr provided_fieldss r9 set_user_metadataz4MetadataConstrainedLogitsProcessor.set_user_metadata s H gE 5=e_++E259++E2 g
%%' ::-1-H-H-N-N-P b-PTQTUTaq-PO b >>O PQ VW bs +
B/6B/c D i | _ t d D ]= }| j j t | d }|s,|d | j |< ? | j j dd }|r|d nd| _ i | _ t D ]4 }| j j |d }|s#|d | j |< 6 g | _ dD ]@ }| j j |d }|s#| j j |d B g | _
dD ]@ }| j j |d }|s#| j j |d B | j j d d }|r|d nd| _ g | _ g | _
d
D ]r }| j j |d }|s#|j d k( s7| j j |d | j j |d t t | j | _ | j j d d } | r| d nd| _ | j j$ | _ | j j d
d }
|
r|
d nd| _ | j j dd } | r| d nd| _ t* | _ t/ | _ | j3 d| _ d| _ | j9 t; j< | _ y)z3Pre-compute commonly used token IDs for efficiency.
Fadd_special_tokensrX N)#u ♯)bu ♭r )mMr ,.`) digit_tokensr r< encoder
newline_token note_tokensr sharp_tokensr flat_tokens space_tokenmajor_start_tokensminor_start_tokenslowerr
vocab_size comma_token eos_token_id period_tokenbacktick_tokenr valid_languagessetaudio_code_token_ids _precompute_audio_code_tokensaudio_code_masknon_audio_code_mask_build_audio_code_maskr copyvalid_keyscales) r dtokensnewline_tokensnotesf space_tokensprefix comma_tokens
period_tokensbacktick_tokenss r9 rp z5MetadataConstrainedLogitsProcessor._precompute_tokens s rA^^**3q6e*LF'-bz!!!$ ..t.N3A^B /t "D^^**4E*JF)/ & # A^^**1*GF!!((4
A^^**1*GF ''r
3 ~~,,SU,K /;<+ #% "$ F^^**6e*LF<<>S(++226":>++226":> ! dnn- ~~,,SU,K /;<+ !NN 7 7 --ce-L
1>M" -D ..///N5Dob 1$ / /2e! **, 8< ;? ##% / 3 3 5 r8 c
ddl }|j d }d}t | j D ] } | j j |g }|j
| }|r|t |j d }d|cxk r t k r n n | j j | n4|dz
}| j r#t j d| d| dt d |dkD r t j d | d
t d t | j dk( r t j d t d
y| j r4t j d t | j dt d yy# t $ r Y Yw xY w)z
Precompute audio code token IDs (tokens matching <|audio_code_\d+|>).
These tokens should be blocked during caption generation.
Only tokens with code values in range [0, MAX_AUDIO_CODE] are included.
r N^<\|audio_code_(\d+)\|>$rP zSkipping audio code token z with invalid code value z (max: )zFound z7 audio code tokens with values outside valid range [0, ]z:No valid audio code tokens found in vocabulary (range [0, z ]). Code generation may fail.z$ valid audio code tokens (range [0, z]))recompiler r r< decodematchintgroupMAX_AUDIO_CODEr addr> r Exceptionr warning)r r audio_code_patterninvalid_tokens_counttoken_id
token_textr
code_values r9 r z@MetadataConstrainedLogitsProcessor._precompute_audio_code_tokens$ s ZZ(C D doo.H
!^^ 2 2H: >
*00<!$U[[^!4JJ8.8 1155h?,1, :: "LL+EhZOhishtt{ }K |L LM *N O /" ! # LL6"6!77no}n~~ A
B t(( )Q . NN WXfWg hE F
G
ZZ LL6#d&?&?"@!AAefteuuwx y
s B+E:: FFr c ddl }|j d } | j j |g }|j | }|rt |j
d S y# t $ r Y yw xY w)z
Extract audio code value from a token ID.
Args:
token_id: Token ID to extract code value from
Returns:
Code value if token is a valid audio code token, None otherwise
r Nr rP )r r r< r r r r r )r r r r r r s r9 _extract_code_from_tokenz;MetadataConstrainedLogitsProcessor._extract_code_from_tokenI sy ZZ(C D ..z:J&,,Z8E5;;q>**
s AA" " A.-A.c 0 | j sd| _ d| _ yt j d| j
t j }t | j }t d |d|f< || _ t j d| j
ft d t j }d|d|f< | j d|d| j f< || _ | j r-t j dt | j d yy)a
Build a precomputed mask tensor for blocking audio code tokens.
This mask can be added to scores in O(1) time instead of O(n) loop.
The mask is [1, vocab_size] tensor with -inf at audio code token positions.
Also builds the inverse mask (non_audio_code_mask) for CODES_GENERATION state,
which blocks all non-audio-code tokens.
NrP )dtype-infr zBuilt audio code masks for z tokens)r r r torchzerosr float32listfloatfullr r> r r )r maskaudio_code_indices inverse_masks r9 r z9MetadataConstrainedLogitsProcessor._build_audio_code_mask` s ((#'D '+D $ {{1dooU]]C "$";"; < ',FmQ"
"# # zz1doo"6f
U]][ ./ Q**+ (12LD - -- .#/ :: LL6s4;T;T7U6VV]^ _ r8 scoresallowed_tokensc
|s|j t d yt j ||j t j
}|d|f j
}|j t d ||d|f< y)a
Apply whitelist constraint inplace: only allow specified tokens, block all others.
This is more efficient than creating a mask tensor because:
1. No memory allocation for mask
2. No tensor addition operation
Args:
scores: [1, vocab_size] scores tensor to modify inplace
allowed_tokens: List of token IDs to allow (all others will be set to -inf)
r Ndevicer r )fill_r r tensorr longclone)r r r allowed_indices saved_valuess r9 _apply_whitelist_inplacez;MetadataConstrainedLogitsProcessor._apply_whitelist_inplace st LLv ' ,,~fmmSXS]S]^ a 01779 U6]# &2q/!"r8 .c
i }d}d}| j j |d }| j rD|D cg c] }| j j |g }}t j d| d| | j
D ] }||z }| j j |d } d}
t
| t
| k\ r | dt
| |k( r t
| }
|
&| j rt j d| d | |
d } | s&| j rt j d
| d | d } | j j | g }
|
j r!|
j d j nd }|d
vr-| j r t j d| d|
d| d 5t t
| dz D ]t }t | d| }||vr
t ||< |t
| k r| | }|| j | J| j sW|| j | j v | j rt j dt
| d t }||v rZ|| }t | D cg c]) }|t! | j j |g f+ }}t j d| |S c c}w c c}w )a
Build keyscale prefix to allowed tokens mapping based on ACTUAL tokenization.
IMPORTANT: Uses token ID sequences as keys, NOT strings, to avoid tokenization mismatches.
CRITICAL FIX: The tokenizer may merge the context's trailing space into the next token.
For example:
- "keyscale: " tokenizes to [10563, 2246, 25, 220] -> ['keys', 'cale', ':', ' ']
- "keyscale: G major" tokenizes to [10563, 2246, 25, 479, 3598] -> ['keys', 'cale', ':', ' G', ' major']
The space ' ' (220) is merged into ' G' (479), so we can't use simple slicing.
Strategy:
1. For each keyscale (e.g., "G major"), encode the FULL string "keyscale: G major"
2. Tokenize to get: [10563, 2246, 25, 479, 3598] -> ['keys', 'cale', ':', ' G', ' major']
3. Find where context prefix ends by matching token sequences (handling space merging)
4. Extract keyscale value tokens: [479, 3598] (for "G major")
5. Build prefix tree using token ID sequences as keys
This ensures we get the exact tokenization that occurs during generation.
rY
keyscale: Fr z.Context for matching 'keyscale:' tokenizes to -> N7Could not find context prefix in full tokenization of ' ', skippingz"No tokens extracted for keyscale 'r rL ABCDEFGzSkipping keyscale 'z': first token is 'z' (id=z
), not a noterP z Built keyscale prefix tree with token sequence prefixesz%First tokens allowed (empty prefix): )r< r r> r r r r r lstripupperr tupler r r sortedrepr)r prefix_to_tokensrS rT context_token_idstcontext_tokens_strrG full_textfull_token_idscontext_end_idxkeyscale_token_idsfirst_token_idfirst_token_str
first_charr token_prefix
next_token_id empty_prefix first_tokens
decoded_firsts r9 r z>MetadataConstrainedLogitsProcessor._build_keyscale_prefix_tree s * =? '2#*6' !NN 1 12Mbg 1 h ::FW!XFW$.."7"7""c*;&<<!"93'8#9:>OO&)*;& set of allowed token IDs
Fr Nr r! rP ) r< r r r> r r r r& r r r )r rO rS rT r) r* value_strr- token_idsr/ value_token_idsr r4 r5 s r9 r z=MetadataConstrainedLogitsProcessor._build_numeric_prefix_tree sn & =? mHDNN 1 12Mbg 1 h NP &I7)CI --iE-RI #O9~%6!77 4c"3 459JJ&)*;& r r r r r r r& r r r r' r( )r r) rS rT r* r+ r, langr- r. r/ lang_token_idsr r4 r5 r6 r7 r8 s r9 r z>MetadataConstrainedLogitsProcessor._build_language_prefix_treeS sv =?&1#*6' NN 1 12Mbg 1 h ::FW!XFW$.."7"7"I !^^ 2 29QV 2 WN "O>"c*;&<<!"93'8#9:>OO&)*;&**$21$5M$\266}E))( 6::4;M;MN 4+ )D :: LL;C@P , 3 $ +C0D0D,E+FGHT 1 123CR8B BtBxj/ " hr8 c < t j j | j s/| j r"t j d| j y t j j
| j }|| j k ryt | j dd 5 }g }|D ]F }|j }|s|j d r(|j |j H || _
|| _ | j | j r8t j dt | j d| j ddd y# 1 sw Y yxY w# t $ r"}t j" d | Y d}~yd}~ww xY w)
z
Load genres vocabulary from file. Supports hot reload by checking file mtime.
File format: one genre per line, lines starting with # are comments.
z Genres vocab file not found: Nrzutf-8)encodingr zLoaded z
genres from z Failed to load genres vocab: )rq rr existsr? r> r getmtimerx openr
startswithr r rw _build_genres_trier r r )r mtimer rJ r es r9 r z5MetadataConstrainedLogitsProcessor._load_genres_vocab sG
ww~~d 4 45zz ? ? @sI 8E0 E0 +E$E$BE$E0 $E-)E0 -E0 0 F9FFc i | _ | j D ]( }| j }|D ] }||vri ||< || } d|d< * | j r-t j dt | j d yy)z
Build a trie (prefix tree) from genres vocabulary for efficient prefix matching.
Each node is a dict with:
- '_end': True if this node represents a complete genre
- other keys: next characters in the trie
T_endzBuilt genres trie with z entriesN)ry rw r> r r )r genrenodechars r9 rW z5MetadataConstrainedLogitsProcessor._build_genres_trie s &&E##D t#!#DJDz DL
' :: LL23t7H7H3I2J(S T r8 c . |r | j sy|j }t }ddl}|j d| }|D cg c]@ }|j st
|j dk\ s1|j B }}|D ]) }| j | }|| j ||| + t | j } |D ] }|| v s|j | |s"| j rt j d yt | | _
i | _ |D ]( }
| j }|
D ] } | |vri || < || } d|d< * | j r2t j dt
| d t | dd
d yyc c}w ) a
Extract genres from the user's caption that match entries in the vocabulary.
This creates a smaller trie for faster and more relevant genre generation.
Strategy (optimized - O(words * max_genre_len) instead of O(vocab_size)):
1. Extract words/phrases from caption
2. For each word, use trie to find all vocab entries that START with this word
3. Build a separate trie from matched genres
Nr z
[,\s\-_/\\|]+ z.No genres matched in caption, using full vocabTr[ zMatched z genres from caption: z...)rw r r r r r r _get_genres_trie_node_collect_complete_genresr r> r r r{ rz ) r rE
caption_lowermatched_genresr wordswwordr]
genres_setr\ r^ s r9 _extract_caption_genresz:MetadataConstrainedLogitsProcessor._extract_caption_genres s d / /
)=9$)OEqQWWY3qwwy>Q;NEO D--d3D --dD.I **+
Dz! ""4( zz MO '+>&:##% #E++D t#!#DJDz DL
$ :: LL8C$7#88NtTbOcdfefOgNhhkl m E Ps F F7Fr] r result max_depthc |dk ry|j dd r|j | t | dk\ ry|j D ]$ \ }}|dvs | j |||z ||dz
& y)z}
Recursively collect all complete genres under a trie node.
Limited depth to avoid too many matches.
r Nr[ Fd r[ _tokensrP )getr r r rc )r r] r rk rl r^
child_nodes r9 rc z;MetadataConstrainedLogitsProcessor._collect_complete_genres$ sv
> 88FE " JJv v;# $
D*..--j&4-QZ]^Q^_ !-r8 c 4 i | _ i | _ t | j D ]( } | j j |g }|s$|j
}|j r|j }nd}|| j |< |d j
}|| j vrt | j |< | j | j | |j }|r[||k7 rV|d j
}|| j vrt | j |< | j | j | + | j r-t j dt | j d yy# t $ r Y rw xY w)a
Precompute mapping from characters to token IDs and token decoded texts.
This allows O(1) lookup instead of calling tokenizer.encode()/decode() at runtime.
Time complexity: O(vocab_size) - runs once during initialization
Note: Many subword tokenizers (like Qwen) add space prefixes to tokens.
We need to handle both the raw first char and the first non-space char.
r r z$Precomputed char->token mapping for z unique charactersN)r| _token_to_textr r r< r r r rstripr r r$ r r> r r )r r text
text_lowernormalized_textr3
stripped_textfirst_nonspace_chars r9 r} zAMetadataConstrainedLogitsProcessor._precompute_char_token_mapping7 s} 02 .0 doo.H
~~,,hZ8
"ZZ\
##%&0&7&7&9O&)O0?##H- "!W]]_
T%9%997:uD((4$$Z044X> !%
]d%:*7*:*@*@*B'*$2F2FFDGE,,-@A(()<=AA(K= /F :: LL?DDXDX@Y?ZZlm n
s F
DF
FFc
t j j | j sy t j j | j }|| j
kD r| j
yy# t $ r Y yw xY w)zDCheck if genres vocab file has been updated and reload if necessary.N)rq rr rS r? rT rx r r )r rX s r9 _try_reload_genres_vocabz;MetadataConstrainedLogitsProcessor._try_reload_genres_vocabk sk ww~~d 4 45 GG$$T%;%;Check if the given text is a complete genre in the vocabulary.Nr[ F)rb r rq )r rv r] s r9 _is_complete_genrez5MetadataConstrainedLogitsProcessor._is_complete_genre s3 ))$**,74 ;DHHVU$;;r8 triec J |}|j D ]
}||vr y|| } |S )zGGet a trie node from a specific trie (helper for caption vs full trie).N)r )r r r r] r^ s r9 _get_trie_node_from_triez;MetadataConstrainedLogitsProcessor._get_trie_node_from_trie s2 LLND4 :D # r8 c | j sg S | j j }|j }d}d}| j r4|dk( r| j }d}n | j | j | }|d}|#|dk( r
| j }n| j | }|| j r
| j gS g S t d |j D }|j dd }|s>t }|r'| j r|j | j t | S t }|D ]/ } | | j v s|j | j | 1 |r | j n | j }
t }|D ] } | j j | d } | r| j sd|v sd|v r|j | K| j# d s| j# d r|| z }
n|| z }
| j |
|
}||j | |r'| j r|j | j t | S ) a
Get allowed tokens for genres field based on trie matching.
The entire genres string (including commas) must match a complete entry in the vocab.
For example, if vocab contains "pop, rock, jazz", the generated string must exactly
match that entry - we don't treat commas as separators for individual genres.
Strategy:
1. If caption-matched genres exist, use that smaller trie first (faster + more relevant)
2. If no caption matches or prefix not in caption trie, fallback to full vocab trie
3. Get valid next characters from current trie node
4. For each candidate token, verify the full decoded text forms a valid trie prefix
FNrL Tc 3 * K | ] }|d vs|
yw)ro Nr7 ).0r s r9 zPMetadataConstrainedLogitsProcessor._get_allowed_genres_tokens.. s ^*=QJ]A]q*=s r[ r r )rw rh r r rz r ry rb r r keysrq r r r| updatert rV )r accumulatedcurrent_genre_prefixuse_caption_trie current_nodevalid_next_chars is_completeallowedcandidate_tokensr^ active_trier decoded_normalized
new_prefixnew_nodes r9 _get_allowed_genres_tokensz=MetadataConstrainedLogitsProcessor._get_allowed_genres_tokens so I ,,224 * 0 0 2 ! # ##r) # 7 7 #' # < z codesz/Target duration cleared, no duration constraint)ra r rb r> r )r rF s r9 set_target_durationz6MetadataConstrainedLogitsProcessor.set_target_duration s| ( HqL #HqL 1D zz 4XJeDDUDUCVV\ ]^ !%D zz NO r8 c
|| j k( ry| j }|| _ || j d d< t | j d d | j d d dz D cg c]
}t | c}| _ | j | j dd | _ | j r3t j d | d
| d t | j d yyc c}w )
a
Dynamically update the maximum allowed duration for constrained decoding.
This method should be called when GPU configuration changes (e.g., LM initialization state changes).
It rebuilds the duration prefix tree to constrain duration values to the new maximum.
Args:
max_duration: Maximum duration in seconds (e.g., 120 for 2 minutes, 360 for 6 minutes)
NrF rN rM rP rU rV rR zUpdated max duration: r z s, rebuilt prefix tree with z values)
rA r~ r r r r r r> r r )r rA old_maxr s r9 set_max_durationz3MetadataConstrainedLogitsProcessor.set_max_duration s8 4,, , ## ( /;$U+ 7zYMetadataConstrainedLogitsProcessor._get_allowed_tokens_for_fixed_string..u s QqTr8 T)keyreversezFallback: returning z tokens: ra zFixed string: z , position: z
, remaining: )rg r> r r( r r r< r r rM r$ r rV r' r )r r remaining
best_tokenbest_prefix_lenendr r r first_token
decoded_tokennormalized_prefixnormalized_decoded
sorted_tokenstoken_rk r+ s r9 $_get_allowed_tokens_for_fixed_stringzGMetadataConstrainedLogitsProcessor._get_allowed_tokens_for_fixed_string7 sZ d 4 4 56 I :: LLKDQZOK\\pqu rH rH qI IU VZ [d Ve Uf g
h
Y B/C t_F^^**6e*LF#f+* $AY
"%::LL#DT&\NR]^h]iivw{ }A }K }K }R }R T^ S_ }` xa wb "c d 0 ! < CI 2B 78C t_F^^**6e*LF $Qi $ 5 5{m D
$*MMO$9$9$;!%2%9%9%;%A%A%C" &001BCGXGcGcdvGw".8C.Q\B]<]69{3 9& ~ 3 3 5>SWX
:G
6
HE1%
6R :: LL/F }Iv|}~ wA GB wAqr4PTP^P^PePeghfiPjKkGl wA GB FC D
E ~d9o->l4KaKaJbboptu~p pA B C
7 GBs K
.Kmin_valmax_valc d | j spt }t ||dz D ]( }|j t t | d * |D cg c] }|| j v s| j | " c}S t | j }g }t d D ]Z }t | j t | z }||kD r*||k\ r|j | A|dz |k sJ|j | \ |D cg c] }|| j v s| j | " c}S c c}w c c}w )z
Get allowed digit tokens based on accumulated value and range constraints.
Uses early-blocking to prevent out-of-range values.
rP r r )rh r r r r r r r ) r r r allowed_digitsr r currentr new_values r9 _get_allowed_digit_tokensz2 12@[.QAIZIZDZD%%a(.[ [d,,-rAD 2 2SV ;O>O9O!!!$gPP3 \2 Qs D()D( D-D- prefix_treec T t | j }||v rt || S g S )a
Get allowed tokens for numeric field using the precomputed prefix tree.
IMPORTANT: Uses token ID sequence as key (not string) to avoid tokenization mismatches.
Args:
prefix_tree: Precomputed prefix tree mapping token ID sequence -> set of allowed token IDs
Returns:
List of allowed token IDs for current accumulated_token_ids
)r& ri r )r r r4 s r9 _get_allowed_numeric_tokensz>MetadataConstrainedLogitsProcessor._get_allowed_numeric_tokens s3 T 7 78 ; & L12 2 r8 logitsc | j syt | j }||k s||kD ryt j |d | j r d| j f j nd}| j
|| }|syt fd|D }| j r t j d|dd |d ||kD S )
z
Determine if we should end the current numeric field.
Returns True if P(newline) > P(any valid digit) AND current value is valid.
Fr dimr Tc 3 H K | ] }d |f j yw)r N)item)r r+ probss r9 r zOMetadataConstrainedLogitsProcessor._should_end_numeric_field.. s# H AU1a4[ - - / s "z%Numeric field decision: newline_prob=z.4fz, max_digit_prob=)
rh r r softmaxr r r rN r> r ) r r r r r newline_probr max_digit_probr s @r9 _should_end_numeric_fieldzB>P>PuQ 2 2 2388:VW 77I H HH :: LL@ c@RRcdrsvcwx yn,,r8 c f | j j syt j |d }| j r |d| j f j nd}|j
}| j rd|d| j f< |d j j }||kD S )z
Determine if we should end a text field (genres).
Returns True if P(newline) > P(any other token) AND we have some content.
Fr r r )rh r r r r r r rN )r r r r masked_probsmax_other_probs r9 _should_end_text_fieldz9MetadataConstrainedLogitsProcessor._should_end_text_field s
%%++-
f"->B>P>PuQ 2 2 2388:VW {{} 23LD . .. /%a ,,.335n,,r8 c | t | j }|| j v rt | j | S g S )z
Get allowed tokens for keyscale field using the precomputed prefix tree.
Uses token ID sequence as key (not string) to avoid tokenization mismatches.
)r& ri r r r r4 s r9 _get_allowed_keyscale_tokensz?MetadataConstrainedLogitsProcessor._get_allowed_keyscale_tokens s? T 7 78 444 411,?@ @ r8 c t | j }|| j v r| j | j | v S y)z
Check if keyscale value is complete and valid.
Uses token ID sequence to check if current prefix allows newline.
F)r& ri r r r s r9 _is_keyscale_completez8MetadataConstrainedLogitsProcessor._is_keyscale_complete sA
T 7 78 444 4%%)B)B<)PP Pr8 c | t | j }|| j v rt | j | S g S )z
Get allowed tokens for language field using the precomputed prefix tree.
Uses token ID sequence as key (not string) to avoid tokenization mismatches.
Similar to keyscale.
)r& ri r r r s r9 _get_allowed_language_tokensz?MetadataConstrainedLogitsProcessor._get_allowed_language_tokens s? T 7 78 444 411,?@ @ r8 c | t | j }|| j v rt | j | S g S )z
Get allowed tokens for timesignature field using the precomputed prefix tree.
Uses token ID sequence as key (not string) to avoid tokenization mismatches.
)r& ri r r r s r9 _get_allowed_timesig_tokensz>MetadataConstrainedLogitsProcessor._get_allowed_timesig_tokens s?
T 7 78 433 300 >? ? r8 input_idsc 4 | j s| j | S | j t j k( r| j
dk( r| j | j j |j k7 s#| j j |j k7 r6| j j |j |j | _ || j z }| j | S | j
dk( rk| j t j k( rN| j | r=t j | _ d| _
| j rt j d | j t j k( r| j | j j |j k7 s#| j j |j k7 r6| j j |j |j | _ || j z }| j" | j$ | j | j" k rYt' d |dd| j$ f< | j rt j d| j d | j" d
n|dd| j$ f j) }|j+ t' d ||dd| j$ f< | j r0t j d| j d | j" d | j | S |j, d }t/ | D ]% }| j1 || |||d z }|d ||< ' | j | S )
aU
Apply constrained decoding by modifying logits.
Args:
input_ids: [batch_size, seq_len] input token IDs
scores: [batch_size, vocab_size] logits for next token
Returns:
Modified scores with invalid tokens masked to -inf and temperature scaling applied
r Nr r r zECodes phase: detected in input, skipping to CODES_GENERATIONr zCodes generation: /z, blocking EOSz
, forcing EOSrP )r= _apply_temperature_scalingrf r r6 re r r r tor _input_contains_think_end_tagr5 rc r> r r rb r r r r shaper _process_single_sequence)r r r
eos_scores
batch_sizer rk s r9 __call__z+MetadataConstrainedLogitsProcessor.__call__ s ||226: : ::++ +$$ 49M9M9Y''..&--?4CWCWC]C]agamamCm+/+?+?+B+B&--_e_k_k+B+lD( $"6"66226: : G +
h>P>P0P11)< % 6 6
#$ ::LL!hi ::22 2 ''3++22fmmCtG_G_GeGeioiuiuGu/3/G/G/J/JRXR_R_gmgsgs/J/tD, $":":: ,1B1B1N##d&7&7738=F1d / //0zz '9$:J:J9K1TM^M^L__m%no "(4+<+<( closing tag.
Args:
input_ids: [batch_size, seq_len] input token IDs
Returns:
True if is found in the input (any sequence in batch)
r[ Fr r rP T)r< r r r tolistr )r r think_end_tokensr seqr s r9 r z@MetadataConstrainedLogitsProcessor._input_contains_think_end_tagi s >>00PU0V yq)*AA,%%'C3s8c*:&;;a?@q3 /0015EE A + r8 c | j t j k( s | j t j k( r
| j }n | j
}||S |dk rd}||z S )a
Apply temperature scaling based on current generation phase.
Temperature scaling: logits = logits / temperature
- Lower temperature (< 1.0) makes distribution sharper (more deterministic)
- Higher temperature (> 1.0) makes distribution flatter (more diverse)
Args:
scores: [batch_size, vocab_size] logits
Returns:
Temperature-scaled logits
r gư>)rf r r5 r6 r` r_ )r r temperatures r9 r z=MetadataConstrainedLogitsProcessor._apply_temperature_scaling sf ::22 2djjHDVDV6V 00K 33K M ! K ##r8
field_namec | j j | }|ydddddddd }|| }| | d
}| j j |d }|d
z }| j j |d }t | t | k\ r |dt | |k( r|t | d S | j
rt
j d| d |S )a
Get token sequence for a user-provided field (field_name + value + newline).
Uses the same tokenization logic as prefix tree building.
Args:
field_name: Field name ("bpm", "caption", "duration", "keyscale", "language", "timesignature")
Returns:
List of token IDs for the complete field, or None if field is not provided
NrQ z caption: rV r r> rW zgenres: rC rX Fr :z(Could not match prefix tokens for field z, using all tokens)r^ rq r< r r r> r r ) r r valuefield_to_prefixr r- r prefix_for_matching
prefix_tokenss r9 _get_user_provided_field_tokenszBMetadataConstrainedLogitsProcessor._get_user_provided_field_tokens s ++//
; = "$$$ .
!, hugR( &&yU&K )3 . --.AV[-\
v;#m, ,8K]9K1LP]1]#m,-. . zz!I* Ug hiMr8 c
| j r$| j d }| j ||g |S | j | j v r| j | j }| j | }|r| j t
j k( ry| j rmt | | j z
}|dk rP| j D| j || j g | j rt j d| d |S | j || |S | j t
j k( rX| j rL| j @| j || j g | j rt j d |S | j }| j | j | j v rH| j r:t j d|j d| j j d |S |j! | j# || S | j t
j$ k( r| j& d r| j sf| j( sZ| j& d }d
| d } | j* j- | d
}
|
r&|
| _ d | _ | j ||
d g |S | j1 | j2 }t5 | j( } | | j2 v r+| j6 | j2 | v r|| j6 gz }| j || |S | j t
j8 k( r | j& d r| j sf| j: sZ| j& d }d
| d } | j* j- | d
}
|
r&|
| _ d| _ | j ||
d g |S | j< rut? j@ |d jC } | j* jE | g }
t |
dkD r |
d dvrd | _ d| _# d| _$ |S d | _ | jF r|S | jJ tM d |d| jJ f< | jN | jN jP |jP k7 s#| jN jR |jR k7 r6| jN jU |jP |jR | _' || jN z }| jV dk\ r+| j6 | j || j6 g |S |S | j t
jX k( r| j& d r| j sf| j( sZ| j& d }d
| d } | j* j- | d
}
|
r&|
| _ d| _ | j ||
d g |S | jZ t] t_ | jZ }t | j: }|t | k r>t_ || }|| j` v r| j || j` | g |S | j6 r| j || j6 g |S | j1 | jb }t5 | j( } | | jb v r+| j6 | jb | v r|| j6 gz }| j || |S | j t
jd k( r| j& d r| j sf| j: sZ| j& d }d
| d } | j* j- | d
}
|
r&|
| _ d| _ | j ||
d g |S | jg | ji }|r| j || |S | jj rZ| j6 r| j r#t j d| j: d | j || j6 g |S | jm | r;| j6 ro| j || j6 g | j |S | j: jo s&| j6 rtM d |d| j6 f< |S | j t
jp k( r;| j& d r| j sf| j( sZ| j& d }d
| d } | j* j- | d
}
|
r&|
| _ d| _ | j ||
d g |S t5 | j( } | | jr v rF| j6 | jr | v r+| j6 rn| j || j6 g |S | ju }|r| j || |S | j6 r | j || j6 g |S | j t
jv k( r| j& d r| j sf| j( sZ| j& d }d
| d } | j* j- | d
}
|
r&|
| _ d| _ | j ||
d g |S | j( sXt5 }|| jx v rt{ | jx | }|rt? j| ||jP t> j~ }|d|f }t? j@ | jC }|| } | j || g | j r^| j* jE | g }
t j d| d t |
d t | d |S | j6 r| j || j6 g |S | j6 r| j || j6 g |S t5 | j( } | | jx v rF| j6 | jx | v r+| j6 rn| j || j6 g |S | j }|r| j || |S | j6 r | j || j6 g |S | j t
j k( r | j& d r| j sf| j( sZ| j& d }d
| d } | j* j- | d
}
|
r&|
| _ d | _ | j ||
d g |S t5 | j( } | | j v rF| j6 | j | v r+| j6 r | j || j6 g |S | j }| j || |S ) zMProcess a single sequence and return modified scores (inplace when possible).r r zIstop_at_reasoning=True: forcing EOS near end of tag (remaining: z chars)zAstop_at_reasoning=True: forcing EOS after completing tagzState transition from z to z+ still in fixed_strings, avoiding recursionrD r rX Fr rE z TrL r r i rF rJ z!No valid genre continuation for 'z', forcing newlinerG rH z%Language field: selected top-1 token z (z) from z candidatesrI )Ern r rf r r r r4 rd r rg r r> r _transition_to_next_stater namezero_r r" r^ ri r< r ro r r r& r r% rh rj r argmaxr r rl rm r r r r r r rk r' ra r r r r r* r| r rw r r r- r r r0 r r r r r( r r2 r r )r r r
next_tokenr r remaining_chars old_stater
value_text value_tokensr4 top_token_idtop_token_text
target_str current_pos
next_digitr6 r candidate_indicescandidate_scoresbest_idxs r9 r z;MetadataConstrainedLogitsProcessor._process_single_sequence s/ & & 44Q7J ) )&:, ?M ::++ + **4::6I?? JG ::!7!77DR
I
::!7!77DQR :: "LL+l n %
JJ ..0::!3!33zz )? ?OtTXT^T^TcTcSd eP (Q R!M 44YGG
ZZ8--
-**51=dFaFajn kE kE 33E: r]
#~~ 4 4ZTY 4 Z 2>D/.3D+11&<?:KL!M 66t7K7KLG !!;!; r]
#~~ 4 4ZTY 4 Z 2>D/.7D+11&<?:KL!M )) $||F1I 6 ; ; = !% !6!6 ~!F ~&*~a/@/M 27D.*.D'.0D+!M 27D. ""
"".16v q$---. ##/''..&--?4CWCWC]C]agamamCm+/+?+?+B+B&--_e_k_k+B+lD( $"6"66 ''3.%%111&4;M;M:NO!M M
ZZ822
2**:6B4KfKfos pJ pJ 33J? r]
#~~ 4 4ZTY 4 Z 2>D/.8D+11&<?:KL!M ##/ T%9%9!: ;
!$"8"8 9 Z 0!$Z %]^r
m )) 55ft?Q?Q>RSj
c ::4;T;TU %T%?%? @ 4#<#<<ASASW[WpWpq}W~A~ %););(< T
Q ZZ800
0**84@IdIdmq nD nD 33H= r]
#~~ 4 4ZTY 4 Z 2>D/.6D+11&<?:KL!M
) ) + 557G--fg>d
c "" %%zz 'HI_I_H``r%st11&4;M;M:NOV
Q ..v6)) 55ft?Q?Q>RS 668J
E 11779 --D/.8D+11&<?:KL!M !!;!;RS@
} ZZ822
2 **:6B4KfKfos pJ pJ 33J? r]
#~~ 4 4ZTY 4 Z 2>D/.8D+11&<?:KL!M -- $w 4#<#<<'+D,E,El,S'T$' -2LL9IRXR_R_glgqgq,r)+1!5F2F+G( $)<<0@#A#F#F#H'7'A 55f|nM ::-1^^-B-BL>-RN "LL+PQ]P^^`aeftau`vv} B CS T ~U U` *a bh
c -- 9 9&4CUCUBV W`
[ )) 55ft?Q?Q>RSX
Q %T%?%? @ 4#<#<<ASASW[WpWpq}W~A~)) 55ft?Q?Q>RSH
C # ? ? AG 55fgF>
9 -- 9 9&4CUCUBV W6
3 ZZ811
1**?;GPTPkPktx uO uO 33OD r]
#~~ 4 4ZTY 4 Z 2>D/.=D+11&<?:KL!M !!;!;
r8 c " | j | j v r| j }| j | j }|t j k( r@| j dk( r1t j
}| j rt j d || _ d| _ d| _ g | _
d| _ d| _ d| _
d| _ | j r:t j d|j d| j j yyy) z!Transition to the next FSM state.r zGgeneration_phase='understand': allowing free-form lyrics after r rL FzFSM transition: r N)rf r r r4 re r6 r> r rg rh ri rj rk rl rm r )r r r s r9 r z rb r rn r ro popr rg rh ri r r r< r r( r r r4 rd r" r' r2 r r r isdigitr* r% rk rj rl rm ru r r- r0 )
r r expected_tokenr r r token_strr field_name_fullfield_name_to_value_states
r9 update_statez/MetadataConstrainedLogitsProcessor.update_state[ s || ::++ + ::22 2 ! zzd / / ; }T-=-=,>a@Q@Q?R ST & & ! 8 8 ;N!^3::NN_^4DIN`Maaz{ |S |S {T $U V
' ' + +A . ..::LL#OPTPgPgOh!ij ! 4 4
*.' " 7 7
C
$
I!+DJ-.D*-/D*13D.zz 'TU^UcUcTddhimisisixixhy%z{
224 NN))+=*>? :: LL,T)_,=UCUBVV_`d`j`j`o`o_pq r ::++ + **4::6I " "c)n 4 " %%Y 7::!7!77D>$(JJ %>z%J
12.13.572.3+24/ :: "LL+LY^^L\\`aeakakapap`q)r s :: "NN-OPTU_P`Oaau+v w.3+24/ 668 &9 (@ #T ZZ822
2!T%7%77..0
::!3!33 **112DE&&)3&
4 ZZ822
2!T%7%77..0::!3!33 4 **112DE&&)3& 3r8 )TFNTN)N)rL rL )2 )Er r r r r
boolr r r r r r r r r r staticmethodr r r r r rp r r r r Tensorr r r r r r r rO r rW rj r rc r} r| rb r r r r r r r r r r r r r r r r
LongTensor FloatTensorr r r r r r r r7 r8 r9 r; r; Q s . +/ &*^( ^( ^( ^(
$C= ^(
^( sm^(@-&3 -&8H;M -&^)IV(D (
(T (
(d (
'S 'S ' '@&$ &
&#
& "X(4Xc]8J3K*L "XHR6n#zJ # .'`R2u|| 2TRUY 2[_ 26n T%S/3s82K-L n f ,./1 = 3i= &)= *- =
eCHos3x' ( = ~< T%S/3s82K-L < |%P @>U(6ns 6np`T `3 ` `X[ `&2oh
C
HTN
d DI d L % PHUO P$ TS T@Fc Fd3i FP#Q #Qs #QtCy #QJtE#s(OSQTX,# ,(4PS9BU ,\l##l !!l
l\ Wr sW * B B B & B " t 8Q#4 Q#4r8