New release of python module
Today, we are happy to release a new version of the fastText python library. The main goal of this release is to merge two existing python modules: the official
fastText module which was available on our github repository and the unofficial
fasttext module which was available on pypi.org. We hope that this new version will address the confusion due to the previous existence of two similar, but different, python modules.
The new version of our library is now available on pypi.org as well as on our github repository, and you can find an overview of its API here.
fastText vs fasttext: what happened?
There was an ongoing confusion among our user community about the existence of both
When fastText was first released in 2016, it was a command line only utility. Very soon, people wanted to use fastText's capabilities from python without having to call a binary for each action. In August 2016, Bayu Aldi Yansyah, a developer outside of Facebook, published a python wrapper of fastText. His work was very helpful to a lot of people in our community and he published his unofficial python library on pypi with the pretty straighforward module name
fasttext (note the lowercase
Later, our team began to work on an official python binding of fastText, that was published under the same github repository as the C++ source code. However, the module name for this official library was
fastText (note the uppercase
Last year, Bayu Aldi Yansyah gave us admin access to the pypi project so that we could merge the two libraries.
To sum up, we ended up with two libraries that had:
- almost the same name
- different APIs
- different versions
- different ways to install
That was a very confusing situation for the community.
What actions did we take?
Today we are merging the two python libraries. We decided to keep the official API and top level functions such as
train_supervised as well as returning numpy objects. We remove
supervised functions from the unofficial API. However, we bring nice ideas from the unofficial API to the official one. In particular, we liked the pythonic approach of
WordVectorModel. This new python module is named
fasttext, and is available on both pypi and our github repository.
From now, we will refer to the tool as "fastText", however the name of the python module is
What is the right way to do now?
Before, you would either use
import fastText # and call: fastText.train_supervised fastText.train_unsupervised
import fasttext # and call: fasttext.cbow fasttext.skipgram fasttext.supervised
Now, the right way to do is to
import fasttext (lowercase
import fasttext # and call: fasttext.train_supervised fasttext.train_unsupervised
We are keeping the lowercase
fasttext module name, while we keep the
This is because:
- the standard way to name python modules is all lowercases
- the API from
fastTextis exposing numpy arrays, which is widely used by the machine learning community.
You can find a more comprehensive overview of our python API here.
Should I modify my existing code?
Depending on the version of the python module you were using, you might need to do some little modifications on your existing code.
1) You were using the official
You don't have to do much. Just replace your
import fastText lines by
import fasttext and everything should work as usual.
2) You were using the unofficial
If you were using the functions
SupervisedModel objects, you were using the unofficial
Updating your code should be pretty straightforward, but it still implies some little changes.
cbow function: use
For example, replace:
fasttext.cbow("train.txt", "model_file", lr=0.05, dim=100, ws=5, epoch=5)
model = fasttext.train_unsupervised("train.txt", model='cbow', lr=0.05, dim=100, ws=5, epoch=5) model.save_model("model_file.bin")
skipgram function: use
For example, replace:
fasttext.skipgram("train.txt", "model_file", lr=0.05, dim=100, ws=5, epoch=5)
model = fasttext.train_unsupervised("train.txt", model='skipgram', lr=0.05, dim=100, ws=5, epoch=5) model.save_model("model_file.bin")
supervised function: use
For example, replace:
fasttext.supervised("train.txt", "model_file", lr=0.1, dim=100, epoch=5, word_ngrams=2, loss='softmax')
model = fasttext.train_supervised("train.txt", lr=0.1, dim=100, epoch=5, , word_ngrams=2, loss='softmax') model.save_model("model_file.bin")
- As you can see, you can use either
wordNgramsas parameter name. Because the parameter names from the unofficial API are mapped to the official ones:
silentparameter is not supported. Use
encodingparameter is not supported, every input should be encoded in
SupervisedModel objects, we return a model object that mimics some nice ideas from the unofficial API.
model = fasttext.train_unsupervised("train.txt", model='skipgram') print(model.words) # list of words in dictionary print(model['king']) # get the vector of the word 'king' print('king' in model) # check if a word is in dictionary
model = fasttext.train_supervised("train.txt") print(model.words) # list of words in dictionary print(model.labels) # list of labels
The model object also contains the arguments of the training:
print(model.epoch) print(model.loss) print(model.wordNgrams)
We want to thank our incredible community. We truly appreciate your feedback, a big thank you to everyone reporting issues and contributing to the project. In particular we want to express how grateful we are to Bayu Aldi Yansyah who did a great job with his python library and for giving us the ownership of the pypi