Sentiment Analysis: Comparing Azure, AWS, and Custom Fine-Tuned Models

I ran 22,719 student evaluations through five sentiment models side by side — Azure’s cloud API, AWS Comprehend, a pre-trained RoBERTa model, Microsoft’s Phi-2, and fine-tuned versions of the last two. The cloud APIs weren’t just less accurate than the custom models. They were classifying the same feedback in ways that would’ve pointed a business in the wrong direction.

Azure called over 40% of reviews “mixed” sentiment, a label the human reviewers barely used. AWS did the same thing. The dataset was mostly positive and negative with a thin neutral middle; both APIs turned it into a distribution the ground truth — the human-labeled answers I treated as correct — did not resemble.

I expected the fine-tuned models to win. I did not expect the gap to be this wide. My advice to clients is now direct: if sentiment classification matters to your business, build a custom model. Fine-tuning a Transformer — the neural network architecture behind modern language AI — is doable with as few as 20,000 labeled examples. Skip the API.

The dataset

I used a publicly available dataset for aspect-based sentiment analysis — sentiment detection applied to a specific dimension, here teacher performance. It contains student feedback from American International University-Bangladesh, labeled by undergraduates as positive, negative, or neutral. After cleaning, the full dataset contains over 2 million records.

I used a filtered subset of 22,719 entries where students wrote more than 200 characters. Short one-liners are too easy to classify and would make the scores look better than they should.

The dataset includes several columns I didn’t use: a numerical rating, a pre-trained model’s sentiment guess, subjectivity classifications and scores, and a boolean flag for whether the manual and pre-trained labels matched. I only needed StudentComments (the free-form text) and Sentiment (the human label). Everything else sat untouched.

Figure-1: Dataset preview

Figure-2: Dataset summary

Before training, I checked whether the labels made sense. The box plot showed positive sentiment clustered around high ratings (4–5), negative sentiment clustered low with a wider spread, and neutral in the middle. The scatter plot showed the same relationship across comment length: sentiment tracked ratings regardless of how many words the student wrote.

The labels weren’t noise. That matters, because if the ground truth is sloppy, every comparison downstream is meaningless.

Figure-3: Relationship between Rating and Sentiment

Figure-4: Rating vs. Total Words with Sentiment

What I compared

I compared five approaches in three groups.

Cloud APIs. Azure Cognitive Text Analytics and AWS Comprehend take text at an endpoint and return a sentiment label. There is no training and no configuration beyond an API key. Azure fits naturally in Microsoft’s ecosystem. AWS scales well at volume. They are easy to try, which is why they are easy to choose too quickly.

Off-the-shelf models. RoBERTa and Phi-2, straight from Hugging Face with no additional training. RoBERTa — cardiffnlp/twitter-roberta-base-sentiment-latest — had an advantage because it was already fine-tuned on roughly 124 million tweets from January 2018 through December 2021 with the TweetEval benchmark. It had seen sentiment classification before, just not this dataset. Phi-2 — microsoft/phi-2 — is a Transformer with 2.7 billion parameters (2.7 billion adjustable weights). It was trained on the same data sources as Phi-1.5, plus synthetic NLP texts and filtered web content. It showed near state-of-the-art performance among models under 13 billion parameters on benchmarks for common sense, language understanding, and logical reasoning. This Phi-2 had not been fine-tuned through reinforcement learning from human feedback. It shipped raw so researchers could study safety problems such as toxicity reduction and bias control in an unrestricted small model. I included it untuned to establish a baseline.

The original Twitter-based RoBERTa model lives here and the reference paper is TweetEval.

Fine-tuned models. Same RoBERTa and Phi-2, but trained further on my 22,719 labeled evaluations. This is the custom path: you take a general-purpose language model and teach it your specific classification task. The model keeps everything it learned during its original training — word meanings, sentence structure, world knowledge — but adjusts its weights to get better at your particular job.

What I measured

Accuracy alone hid too much, so I used three measures.

Sentiment distribution. I compared how each model spread its predictions across positive, negative, neutral, and mixed against the ground truth. If a model’s distribution is a different shape from the truth, it’s misclassifying systematically — not just making random errors. That’s bias a single accuracy number won’t surface.

F1 score. The standard measure that balances precision and recall. Precision answers: when the model says “positive,” how often is it right? Recall answers: of all the actual positives, how many did the model catch? F1 is the harmonic mean of the two — it punishes models that are good at one and terrible at the other. Range is 0 to 1. Models scoring above 0.85 are generally considered production-ready.

Agreement analysis. How often do two models give the same answer to the same input? High agreement between independent models suggests the classification is unambiguous. Low agreement means at least one of them is guessing differently — and you don’t know which.

What I found

The distribution charts showed the first problem.

The ground truth was mostly positive, with a solid block of negative and a thin band of neutral. Azure and AWS overproduced “mixed” sentiment — a label the reviewers barely used — and underproduced neutral. Their output did not match the data.

Pre-trained RoBERTa was closer to the ground truth on positive sentiment but still underestimated neutral. Its Twitter fine-tuning gave it a head start. Raw Phi-2 was nearly useless. It struggled across most of the dataset, and its bars in the distribution chart were barely visible. That is what I would expect from a general-purpose model doing sentiment classification without task-specific training.

Fine-tuning fixed both. Fine-tuned RoBERTa moved closer to the ground truth, especially on positives, although it still undercounted neutral. Fine-tuned Phi-2 was best and matched the ground truth distribution almost exactly. Its improvement over fine-tuned RoBERTa was incremental but real.

Figure-6: Sentiment Distribution

Then the F1 scores.

AWS was my benchmark. Azure scored lower. Pre-trained RoBERTa beat both cloud APIs, which fits its Twitter sentiment background. Pre-trained Phi-2 was last.

Fine-tuned RoBERTa jumped ahead of the cloud APIs by a wide margin. Fine-tuned Phi-2 was the clear winner — it significantly outperformed AWS, Azure, and even fine-tuned RoBERTa in accuracy, precision, recall, and F1 score. The best cloud API and the best fine-tuned model were not close: one was usable, the other was reliable.

Figure-7: F1 Scores

Fine-tuned RoBERTa and fine-tuned Phi-2 agreed with each other 90.84% of the time — the highest agreement in the entire matrix. Fine-tuned models consistently agreed more with each other than with any cloud API. Azure and raw Phi-2 had the lowest agreement, which is what you’d expect when two models are systematically wrong in different directions.

Figure-8: Pairwise agreement analysis

A fine-tuned model learns the shape of your data rather than the product-review and social-media patterns that calibrate a general-purpose API.

The next post covers how to fine-tune Phi-2 for a sentiment task.