What Is TF-IDF, and How Do You Calculate and Interpret It?

TF-IDF is a statistical method for measuring how important a word is to a particular document. It is calculated by multiplying term frequency (TF) by inverse document frequency (IDF). It considers both how often a word appears in a document and how widespread it is across the entire document collection. Words that appear frequently in one document but in relatively few other documents generally receive higher weights. However, a high score indicates only relative importance under this calculation; it does not mean TF-IDF actually understands the document’s content.

Why not just count how often words appear?

Frequency alone cannot tell you whether a word is characteristic of a particular document or common across many documents. TF-IDF combines two types of information to help identify words that better represent a document:

• TF looks at an individual document: the number of times a word appears as a proportion of the document’s total word count.
• IDF looks at the document collection: the fewer documents that contain a word, the higher its IDF under this formula; the more widespread the word, the lower its IDF.

So a word that appears frequently in one document does not necessarily receive a high final score. How widespread it is across the collection also matters. This is how TF-IDF distinguishes common words from rare ones.

How to calculate TF-IDF in three steps

The formulas below use the source’s definition of relative term frequency and base-10 logarithms to match the values in its example.

1. Calculate TF: the number of times the word appears ÷ the document’s total word count. If a document contains 10,000 words and a particular word appears 25 times, TF＝25 ÷ 10,000＝0.0025.
2. Calculate IDF: log₁₀(total number of documents ÷ number of documents containing the word). If the collection contains 10,000 documents and 500 contain the word, IDF＝log₁₀(20)≈1.30.
3. Multiply the two: TF-IDF≈0.0025 × 1.30＝0.00325.

Note that the two denominators are different: TF divides by the total word count of a single document; IDF compares document counts, not the sum of the word’s occurrences across all documents.

How should you interpret a high score, and what are the limitations?

Under this method, a higher score indicates that a word is more relevant to the document, while a lower score indicates less relevance. However, you must consider the chosen document collection and calculation method when interpreting scores. A number alone cannot tell you whether a word is important.

The source also cautions that extremely rare words may be overvalued because of their high IDF. TF-IDF does not understand word meanings or context, nor does it consider word order, so its scores alone cannot tell you what phrases or documents mean. It is easy to calculate and serves as a useful starting point for identifying important words and conducting further analysis, but it cannot replace an understanding of the content.