Demos
Redesign of embedding models
Recent developments in NLP rely on vector representations of text, commonly known as embeddings. To support the utilization, training, and fine-tuning of models for the legal domain, Legal NLP is introducing new models: word embedding models, which generate vector representations of words (or tokens), and sentence embedding models, which create vector representations for longer pieces of text such as sentences, paragraphs, and documents.
Word Embedding model
For word embedding models, we aimed to create fast and lightweight models that can be used on token classification tasks such as Named Entity Recognition (NER). The newly released model has a dimension of 200 and was trained from scratch on a mix of legal and general texts, allowing the model to learn usual English words, but with a special focus on domain-specific terms.
The architecture is based on Word2Vec, which provides the best tradeoff between representation capabilities and speed, allowing for fast and accurate models built on top of it (e.g., NER, Relation Extraction).
To use the model in Legal NLP, use the pretrained method of the corresponding annotator:
word_embedding_model = (
nlp.WordEmbeddings.pretrained(
"legal_word_embeddings", "en", "legal/models"
)
.setInputColumns(["sentence", "token"])
.setOutputColumn("word_embedding")
)
With the new word embedding model, we trained new models and improved existing models:
- De-identification model:
legner_deid_le - NER models:
legner_subpoenas_sm,legner_sec_edgar_le,legner_contract_doc_parties_le
These models achieved on average an improvement of 12% over the previous models by using the new word embedding model, with a small reduction in performance metrics for the contract doc parties model.
Sentence Embedding model
Now for the sentence embedding model, we based our model on the BGE architecture and trained from scratch on our own augmented Legal datasets together with public general English datasets. The model maps sentences/documents/paragraphs to a vector of 768 dimension and is designed to improve RAG applications.
To use the model, use the corresponding annotator:
sentence_embeddings = (
nlp.BGEEmbeddings.pretrained(
"legal_bge_base_embeddings", "en", "legal/models"
)
.setInputCols("document")
.setOutputCol("sentence_embeddings")
)
With better sentence embedding models, Legal practitioners, researchers, and developers can design better text classification, entity linking, and RAG systems.
New multilabel models
Using the previously release E5 embedding fine-tuned with Legal datasets, we trained improved versions of the following models:
- Contracts clause classification:
legmulticlf_edgar_le - Agreement clause classification:
legmulticlf_sec_mnda_le
These models improved the previous versions up to 16%, demonstrating the better capacity of the E5 model for classification tasks.
Conclusion
Legal NLP 2.0 arrives with better capability to train specialized models based on performant embedding models. Practitioners and data scientists in this area can benefit from them. All models can be found in the Spark NLP Models Hub.
We are motivated to start releasing new specialized models in future releases, stay tuned.
Fancy trying?
We’ve got 30-day free licenses for you with technical support from our legal team of technical and SMEs. This trial includes complete access to more than 926 models, including Classification, NER, Relation Extraction, Similarity Search, Summarization, Sentiment Analysis, Question Answering, etc., and 120+ legal language models.
Just go to https://www.johnsnowlabs.com/install/ and follow the instructions!
