Glossary term
Glossary term
Foundations
A special hidden layer that trains on a high-dimensional categorical feature to gradually learn a lower dimension embedding vector. An embedding layer enables a neural network to train far more efficiently than training just on the high-dimensional categorical feature.
For example, Earth currently supports about 73,000 tree species. Suppose tree species is a feature in your model, so your model's input layer includes a one-hot vector 73,000 elements long. For example, perhaps baobab would be represented something like this:
A 73,000-element array is very long. If you don't add an embedding layer to the model, training is going to be very time consuming due to multiplying 72,999 zeros. Perhaps you pick the embedding layer to consist of 12 dimensions. Consequently, the embedding layer will gradually learn a new embedding vector for each tree species.
In certain situations, hashing is a reasonable alternative to an embedding layer.
See Embeddings in Machine Learning Crash Course for more information.
For example, Earth currently supports about 73,000 tree species. Suppose tree species is a feature in your model, so your model's input layer includes a one-hot vector 73,000 elements long. For example, perhaps baobab would be represented something like this:
Created for this library
A retail recommendation team places an embedding layer for product IDs at the front of its ranker so the model learns a dense representation of each SKU.
An NLP team uses an embedding layer for tokens at the input of its sequence model, sharing weights with the output projection.
A search team uses an embedding layer for user IDs and query terms in its candidate generator and ships the embeddings to the retrieval index.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License