Ë
|
WñúhsH ã ó d Z ddlZddlmZmZmZmZmZmZ ddl Z
|
ddl m Z ddl mZ ddlmZ ddlmZ ddlmZ dd lmZmZmZmZmZmZm Z m Z m Z m Z dd
|
l m!Z! d Z"d Z#e d Z$e dZ%e dZ&e dZ'e dZ(dZ) G d de« Z*y)zÌMySQL-backed vector store for embeddings and semantic document retrieval.
|
|
Provides a VectorStore implementation persisting documents, metadata, and
|
embeddings in MySQL, plus similarity search utilities.
|
é N)ÚAnyÚIterableÚListÚOptionalÚSequenceÚUnion)ÚDocument)Ú
|
Embeddings)Ú VectorStore)Ú PrivateAttr)Ú MyEmbeddings)
|
ÚVAR_NAME_SPACEÚatomic_transactionÚdelete_sql_tableÚ execute_sqlÚextend_sql_tableÚformat_value_sqlÚget_random_nameÚis_table_emptyÚ source_schemaÚ table_exists)ÚMySQLConnectionAbstractz Hello world!Úexternal_sourcez
|
.embeddingz.contextz .context_mapz.retrieval_infoz.optionsÚinternal_ai_id_c ó( e Zd ZU dZ e« Zeed< e« Ze ed< e« Z
|
e ed< e« Z e e ed< e« Zeed< e« Zeed< d#d ed
|
e e d df fd Zd ed ee fdZd$dZd#de ee ded dfdZd$dZ d%dee de ee de ee ded ee f
|
dZ e d%dee d
|
e de ee d ed e f
|
d« Z d#dee dee d ee fdZ! d&de ded ed ee fd Z"d'd Z#d e$e%df d e$e&df d!e$e'df d dfd"Z( xZ)S )(Ú MyVectorStorea
|
MySQL-backed vector store for handling embeddings and semantic document retrieval.
|
|
Supports adding, deleting, and searching high-dimensional vector representations
|
of documents using efficient storage and HeatWave ML similarity search procedures.
|
|
Supports use as a context manager: when used in a `with` statement, all backing
|
tables/data are deleted automatically when the block exits (even on exception).
|
|
Attributes:
|
db_connection (MySQLConnectionAbstract): Active MySQL database connection.
|
embedder (Embeddings): Embeddings generator for computing vector representations.
|
schema_name (str): SQL schema for table storage.
|
table_name (Optional[str]): Name of the active table backing the store
|
(or None until created).
|
embedding_dimension (int): Size of embedding vectors stored.
|
next_id (int): Internal counter for unique document ID generation.
|
Ú_db_connectionÚ _embedderÚ _schema_nameÚ _table_nameÚ_embedding_dimensionÚ_next_idNÚ db_connectionÚembedderÚreturnc óî t | « d| _ t |« | _ |xs t |« | _ || _ d| _ t | j j t « « | _ y)añ
|
Initialize a MyVectorStore with a database connection and embedding generator.
|
|
Args:
|
db_connection: MySQL database connection for all vector operations.
|
embedder: Embeddings generator used for creating and querying embeddings.
|
|
Raises:
|
ValueError: If the schema name is not valid
|
DatabaseError:
|
If a database connection issue occurs.
|
If an operational error occurs during execution.
|
r N) ÚsuperÚ__init__r" r r r r r r ÚlenÚ embed_queryÚBASIC_EMBEDDING_QUERYr! )Úselfr# r$ Ú __class__s úNH:\Change_password\venv_build\Lib\site-packages\mysql/ai/genai/vector_store.pyr( zMyVectorStore.__init__d sg ø ô$ ÑÔØ ä )¨-Ó 8ÔØ!Ò@¤\°-Ó%@Ø +ÔØ*.Ôô %(Ø NN× &Ñ &Ô'<Ó =ó%
|
Õ!ó Únum_idsc ó t | j | j |z « D cg c] }d| }}| xj |z c_ |S c c}w )zä
|
Generate a batch of unique internal document IDs for vector storage.
|
|
Args:
|
num_ids: Number of IDs to create.
|
|
Returns:
|
List of sequentially numbered internal string IDs.
|
r )Úranger" )r, r0 ÚiÚidss r. Ú_get_idszMyVectorStore._get_ids sT ô ,1°· ± ÀÇ Á ÐPWÑ@WÓ+Xö
|
Ø&'oaSÒ !ð
|
ð
|
ð Ñ Ø
|
ùò
|
s ¦ A c óú j at j « 5 t fd« }d j d| d}t | j f¬« | _ ddd« yy# 1 sw Y yxY w)a²
|
Create a backing SQL table for storing vectors if not already created.
|
|
Returns:
|
None
|
|
Raises:
|
DatabaseError:
|
If a database connection issue occurs.
|
If an operational error occurs during execution.
|
|
Notes:
|
The table name is randomized to avoid collisions.
|
Schema includes content, metadata, and embedding vector.
|
Nc ó4 t j | « S ©N)r r )Ú
|
table_nameÚcursorr, s r. ú<lambda>z2MyVectorStore._make_vector_store.<locals>.<lambda>§ s ø ¬<Ø × 1Ñ 1°:ó,ð ( r/ z
|
CREATE TABLE ú.a (
|
`id` VARCHAR(128) NOT NULL,
|
`content` TEXT,
|
`metadata` JSON DEFAULT NULL,
|
`embed` vector(%s),
|
PRIMARY KEY (`id`)
|
) ENGINE=InnoDB;
|
©Úparams)r r r r r r r! )r, r9 Úcreate_table_stmtr: s ` @r. Ú_make_vector_storez MyVectorStore._make_vector_store s ù ð × Ñ Ð #Ü# D×$7Ñ$7Ó8ð .¸FÜ ,ôó
|
ð ) Ø "× /Ñ /Ð 0°°*° ð >ð%Ð!ô ØÐ -°t×7PÑ7PÐ6Rõð $.Ô ÷) .ð .ð $÷ .ð .ús ¤AA1Á1A:r4 Ú_c
|
ó t | j « 5 }|r1|D ], }t |d| j d| j d|f¬« . t || j | j « r| j « ddd« y# 1 sw Y yxY w)a#
|
Delete documents by ID. Optionally deletes the vector table if empty after deletions.
|
|
Args:
|
ids: Optional sequence of document IDs to delete. If None, no action is taken.
|
|
Returns:
|
None
|
|
Raises:
|
DatabaseError:
|
If a database connection issue occurs.
|
If an operational error occurs during execution.
|
|
Notes:
|
If the backing table is empty after deletions, the table is dropped and
|
table_name is set to None.
|
z DELETE FROM r< ú WHERE id = %sr= N)r r r r r r Ú
|
delete_all)r, r4 rA r: Ú_ids r. ÚdeletezMyVectorStore.delete» s ô& × 3Ñ 3Ó 4ð
|
"¸ÙØ ò CÜ Ø Ø& t×'8Ñ'8Ð&9¸¸4×;KÑ;KÐ:LÈNÐ[Ø #vöðô f d×&7Ñ&7¸×9IÑ9IÔJØÔ!÷
|
"÷
|
"ñ
|
"ús A%BÂB c óÂ | j Gt | j « 5 }t || j | j « d| _ ddd« yy# 1 sw Y yxY w)zc
|
Delete and drop the entire vector store table.
|
|
Returns:
|
None
|
N)r r r r r )r, r: s r. rD zMyVectorStore.delete_allÚ s_ ð × Ñ Ð 'Ü# D×$7Ñ$7Ó8ð (¸FÜ ¨×):Ñ):¸D×<LÑ<LÔMØ#'Ô ÷ (ð (ð (÷ (ð (ús ¢)AÁA ÚtextsÚ metadatasc ó¾ t |« }t ||xs i gt |« z « D cg c] \ }}t ||¬« }}}| j ||¬« S c c}}w )ap
|
Add a batch of text strings and corresponding metadata to the vector store.
|
|
Args:
|
texts: List of strings to embed and store.
|
metadatas: Optional list of metadata dicts (one per text).
|
ids: Optional custom document IDs.
|
|
Returns:
|
List of document IDs corresponding to the added texts.
|
|
Raises:
|
DatabaseError:
|
If a database connection issue occurs.
|
If an operational error occurs during execution.
|
|
Notes:
|
If metadatas is None, an empty dict is assigned to each document.
|
)Ú page_contentÚmetadata)r4 )ÚlistÚzipr) r Ú add_documents)r, rH rI r4 rA ÚtextÚmetaÚ documentss r. Ú add_textszMyVectorStore.add_textsæ sk ô4 U ô " %¨Ò)G°r°d¼SÀ»ZÑ6GÓ H÷
|
ádô $°Ö 6ð
|
ñ
|
ð ×!Ñ! )°Ð!Ó5Ð5ùó
|
s ¬Ac óp | t d« t |« } | ||¬« }|j ||¬« |S )a
|
Construct and populate a MyVectorStore instance from raw texts and metadata.
|
|
Args:
|
texts: List of strings to vectorize and store.
|
embedder: Embeddings generator to use.
|
metadatas: Optional list of metadata dicts per text.
|
db_connection: Active MySQL connection.
|
|
Returns:
|
Instance of MyVectorStore containing the added texts.
|
|
Raises:
|
ValueError: If db_connection is not provided.
|
DatabaseError:
|
If a database connection issue occurs.
|
If an operational error occurs during execution.
|
z@db_connection must be specified to create a MyVectorStore object)r# r$ )rI )Ú
|
ValueErrorrM rS )ÚclsrH r$ rI r# Úinstances r. Ú
|
from_textszMyVectorStore.from_texts sI ð4 Ð Ü ØRóð ô U á ]¸XÔFØ×Ñ5¨IÐÔ6àr/ rR c ó |r=t |« t |« k7 r&dt |« dt |« d}t |« t |« dkD r| j « ng S || j t |« « }|D cg c] }|j }}| j
|
j |« }t j « }||d< ||d< ||d< |D cg c] }|j c}|d < t | j « 5 }t || j | j |dd
|
i¬ « ddd« |S c c}w c c}w # 1 sw Y |S xY w) a«
|
Embed and store Document objects as high-dimensional vectors with metadata.
|
|
Args:
|
documents: List of Document objects (each with 'page_content' and 'metadata').
|
ids: Optional list of explicit document IDs. Must match the length of documents.
|
|
Returns:
|
List of document IDs stored.
|
|
Raises:
|
ValueError: If provided IDs do not match the number of documents.
|
DatabaseError:
|
If a database connection issue occurs.
|
If an operational error occurs during execution.
|
|
Notes:
|
Automatically creates the backing table if it does not exist.
|
z.ids must be the same length as documents. Got z ids and z documents.r NÚidÚcontentÚembedrL zstring_to_vector(%s))Ú col_name_to_placeholder_string)r) rU r@ r5 rK r Úembed_documentsÚpdÚ DataFramerL r r r r r ) r, rR r4 ÚmsgÚdocr[ ÚvectorsÚdfr: s r. rO zMyVectorStore.add_documents. sH ñ, 3s8s 9~Ò-ðÜ3xj ¬#¨i«.Ð)9¸ðFð ô S/Ð !ä y>AÒ Ø × #Ñ #Õ %àIà ;Ø--¤ I£Ó/Cà/8Ö9¨3×#Ó#Ð9Ð9Ø..×0Ñ0°Ó9ä \\^Ø4Ø 9 Ø 7 Ø2;Ö<¨3#,,Ò<:ä × 3Ñ 3Ó 4ð ¸Ü ØØ×!Ñ!Ø× Ñ ØØ07Ð9OÐ/Põ ÷ ð
|
ùò% :ùò =÷ ð
|
ús ÂD7ÃD<Ä'EÅE ÚqueryÚkÚkwargsc ó¶ | j g S | j j |« }t | j « 5 }t |dt dt |« g¬« |j dd« }|j dd« |j d d
|
« |j d d « d }t |« \ }} dt d| j d| j d| d| dt dt dt d}
|
t ||
|
|g| ¢¬« t |dt « g } t j |j! « d « } | D ]~ } t |d| j d| j d| d f¬« |j! « \ }}}||d}|t j |« |d< t# di |¤}| j% |« | cddd« S # 1 sw Y yxY w) aä
|
Search for and return the most similar documents in the store to the given query.
|
|
Args:
|
query: String query to embed and use for similarity search.
|
k: Number of top documents to return.
|
kwargs: options to pass to ML_SIMILARITY_SEARCH. Currently supports
|
distance_metric, max_distance, percentage_distance, and segment_overlap
|
|
Returns:
|
List of Document objects, ordered from most to least similar.
|
|
Raises:
|
DatabaseError:
|
If provided kwargs are invalid or unsupported.
|
If a database connection issue occurs.
|
If an operational error occurs during execution.
|
|
Implementation Notes:
|
- Calls ML similarity search within MySQL using stored procedures.
|
- Retrieves IDs, content, and metadata for search matches.
|
- Parsing and retrieval for context results are handled via intermediate JSONs.
|
NzSET @z = string_to_vector(%s)r= Údistance_metricÚCOSINEÚ max_distanceg333333ã?Úpercentage_distanceg 4@Úsegment_overlapr )rk rl rm z=
|
CALL sys.ML_SIMILARITY_SEARCH(
|
@z3,
|
JSON_ARRAY(
|
'r< zÛ'
|
),
|
JSON_OBJECT(
|
"segment", "content",
|
"segment_embedding", "embed",
|
"document_name", "id"
|
),
|
zR,
|
%s,
|
NULL,
|
NULL,
|
z,
|
@z
|
)
|
zSELECT @z"SELECT id, content, metadata FROM rC Ú document_name)rZ rK rL © )r r r* r r r Ú VAR_EMBEDDINGÚstrÚgetr r Ú VAR_CONTEXTÚVAR_CONTEXT_MAPÚVAR_RETRIEVAL_INFOÚjsonÚloadsÚfetchoner Úappend)r, re rf rg Ú embeddingr: ri Úretrieval_optionsÚ retrieval_options_placeholderÚretrieval_options_paramsÚsimilarity_search_queryÚresultsÚ context_mapsÚcontextÚdoc_idr[ rL Údoc_argsrb s r. Úsimilarity_searchz MyVectorStore.similarity_searchg s? ð: × Ñ Ð #ØIàNN×.Ñ.¨uÓ5 ä × 3Ñ 3Ó 4ð H ¸ä ØØ Ð&=Ð>ÜIÐ'õ ð %jjÐ):¸HÓ EOà &§
|
¡
|
¨>¸3Ó ?Ø'-§z¡zÐ2GÈÓ'NØ#)§:¡:Ð.?ÀÓ#Cñ!Ð ô GWØ!óGÑ CÐ )Ð+Cð+ä ð !à×'Ñ'Ð(¨¨$×*:Ñ*:Ð);ð <ð ð ð /Ð/ð 0Ü ð Ü!Ð"ð #Ü$Ð%ð & ð%'Ð #ô, ØØ'Ø'ÐCÐ*BÐCõ ô
|
(¬?Ð*;Ð <Ô =àGä :: f§o¡oÓ&7¸Ñ&:Ó;LØ'ò $ÜØð Ø $× 1Ñ 1Ð 2°!°D×4DÑ4DÐ3Eð F(ð(ð $ OÑ 4Ð6õð -3¯O©OÓ,=Ñ) ð !Ø$+ñ ð Ð'Ü+/¯:©:°hÓ+?HZÑ(ä Ñ* Ñ*ØsÕ#ð) $ð, ÷QH ÷ H ò H ús ¿FGÇGc ó | S )aÃ
|
Enter the runtime context related to this vector store instance.
|
|
Returns:
|
The current MyVectorStore object, allowing use within a `with` statement block.
|
|
Usage Notes:
|
- Intended for use in a `with` statement to ensure automatic
|
cleanup of resources.
|
- No special initialization occurs during context entry, but enables
|
proper context-managed lifecycle.
|
|
Example:
|
with MyVectorStore(db_connection, embedder) as vectorstore:
|
vectorstore.add_texts([...])
|
# Vector store is active within this block.
|
# All storage and resources are now cleaned up.
|
ro )r, s r. Ú __enter__zMyVectorStore.__enter__Ó s ð& r/ Úexc_typeÚexc_valÚexc_tbc ó$ | j « y)aÄ
|
Exit the runtime context for the vector store, ensuring all storage
|
resources are cleaned up.
|
|
Args:
|
exc_type: The exception type, if any exception occurred in the context block.
|
exc_val: The exception value, if any exception occurred in the context block.
|
exc_tb: The traceback object, if any exception occurred in the context block.
|
|
Returns:
|
None: Indicates that exceptions are never suppressed; they will propagate as normal.
|
|
Implementation Notes:
|
- Automatically deletes all vector store data and backing tables via `delete_all()`
|
upon exiting the context.
|
- This cleanup occurs whether the block exits normally or due to an exception.
|
- Does not suppress exceptions; errors in the context block will continue to propagate.
|
- Use when the vector store lifecycle is intended to be temporary or scoped.
|
|
Example:
|
with MyVectorStore(db_connection, embedder) as vectorstore:
|
vectorstore.add_texts([...])
|
# Vector store is active within this block.
|
# All storage and resources are now cleaned up.
|
N)rD )r, r r r s r. Ú__exit__zMyVectorStore.__exit__è s ð> Õr/ r8 )r% N)NN)é )r% r )*Ú__name__Ú
|
__module__Ú __qualname__Ú__doc__r r r Ú__annotations__r r
|
r rq r r r! Úintr" r( rM r5 r@ r r rF rD r Údictr rS Ú classmethodr rX r rO r r r ÚtypeÚ BaseExceptionÚobjectr Ú __classcell__)r- s @r. r r I s0 ø
ññ& /:«mNÐ+Ó;Ù 'MIzÓ)Ù# L#Ó%Ù!,£ K# Ó.Ù +£ Ð#Ó-Ù MHcÓ!ð
|
*.ñ
|
à.ð
|
ð :Ñ&ð
|
ð
|
õ
|
ð@ 𠨨S© ó ó %.ñN "( 8¨C¡=Ñ1ð "¸sð "Àtó "ó>
|
(ð +/Ø#'ñ 6à}ð 6ð D JÑ'ð 6ð d3iÑ ð 6ð
|
ð 6ð
|
có 6ðD ð
|
+/Ø15ñ #à}ð#ð ð#ð D JÑ'ð #ð
|
/ð #ð
|
ò #ó ð#ðL ;?ñ7Ø h ð7Ø.2°3©ið7à có7ðx ñjàðjð ðjð ð jð
|
|
h ó jóXð* àd
|
Ñ#ð ð } dÐ*Ñ+ð ð fdlÑ#ð ð
|
|
÷ r/ r )+r rv Útypingr r r r r r Úpandasr_ Úlangchain_core.documentsr Úlangchain_core.embeddingsr
|
Úlangchain_core.vectorstoresr Úpydanticr Úmysql.ai.genai.embeddingr Úmysql.ai.utilsr r r r r r r r r r Úmysql.connector.abstractsr r+ ÚEMBEDDING_SOURCErp rs rt ru Ú VAR_OPTIONSÚID_SPACEr ro r/ r. ú<module>r¥ s ðñ:ó ç A× Aã å -Ý 0Ý 3Ý å 1÷ ÷ ÷ õ >à&Ð Ø$Ð à!Ð" *Ð- Ø Ð Ð) Ø#Ð$ LÐ1Ø&Ð' Ð7Ð Ø Ð Ð) à ô~Kõ ~r/
|
