textproc/py-sentencepiece: import py-sentencepiece-0.1.97

SentencePiece is an unsupervised text tokenizer and detokenizer
mainly for Neural Network-based text generation systems where the
vocabulary size is predetermined prior to the neural model training.
SentencePiece implements subword units (e.g., byte-pair-encoding
(BPE)) and unigram language model with the extension of direct
training from raw sentences. SentencePiece allows us to make a
purely end-to-end system that does not depend on language-specific
pre/postprocessing.

This package contains the Python module.


(wiz)

SentencePiece is an unsupervised text tokenizer and detokenizer
mainly for Neural Network-based text generation systems where the
vocabulary size is predetermined prior to the neural model training.
SentencePiece implements subword units (e.g., byte-pair-encoding
(BPE)) and unigram language model with the extension of direct
training from raw sentences. SentencePiece allows us to make a
purely end-to-end system that does not depend on language-specific
pre/postprocessing.

This package contains the Python module.

# $NetBSD: Makefile,v 1.1 2023/03/13 14:18:27 wiz Exp $

PKGNAME=	${PYPKGPREFIX}-${DISTNAME}

WRKSRC=		${WRKDIR}/${DISTNAME}/python

USE_TOOLS+=	pkg-config

.include "../../textproc/sentencepiece/Makefile.common"
.include "../../textproc/sentencepiece/buildlink3.mk"
.include "../../lang/python/egg.mk"
.include "../../mk/bsd.pkg.mk"

@comment $NetBSD: PLIST,v 1.1 2023/03/13 14:18:27 wiz Exp $
${PYSITELIB}/${EGG_INFODIR}/PKG-INFO
${PYSITELIB}/${EGG_INFODIR}/SOURCES.txt
${PYSITELIB}/${EGG_INFODIR}/dependency_links.txt
${PYSITELIB}/${EGG_INFODIR}/top_level.txt
${PYSITELIB}/sentencepiece/__init__.py
${PYSITELIB}/sentencepiece/__init__.pyc
${PYSITELIB}/sentencepiece/__init__.pyo
${PYSITELIB}/sentencepiece/_sentencepiece.so
${PYSITELIB}/sentencepiece/_version.py
${PYSITELIB}/sentencepiece/_version.pyc
${PYSITELIB}/sentencepiece/_version.pyo
${PYSITELIB}/sentencepiece/sentencepiece_model_pb2.py
${PYSITELIB}/sentencepiece/sentencepiece_model_pb2.pyc
${PYSITELIB}/sentencepiece/sentencepiece_model_pb2.pyo
${PYSITELIB}/sentencepiece/sentencepiece_pb2.py
${PYSITELIB}/sentencepiece/sentencepiece_pb2.pyc
${PYSITELIB}/sentencepiece/sentencepiece_pb2.pyo

$NetBSD: distinfo,v 1.1 2023/03/13 14:18:27 wiz Exp $

BLAKE2s (sentencepiece-0.1.97.tar.gz) = 969788b6d87e8c992f6df4349f984fb2d6e80f978d4007127174222ec7fcb3ab
SHA512 (sentencepiece-0.1.97.tar.gz) = 4c35488e3661e45be677b04299c0d0b1f0d46421098f0b1625a1bb5e7725d175dfd55328a5a7bbf88badeb03c2ba087aef942b0d7520a29f6bf34eae211a99eb
Size (sentencepiece-0.1.97.tar.gz) = 11945436 bytes