LLM Assignment 1

LLM Assignment 1
Name: Drishti Singh

Roll No: MT23117
I. Hallucinations:
As mentioned in the assignment, LLaMa 3.1 and OpenHathi Models have been
tested for mainly 2 types of Hallucinations
1. Self consistency - Self-consistency hallucination in Large Language
Models (LLMs) occurs when the model generates internally contradictory
responses or fails to maintain coherence across multiple parts of its output.
This type of hallucination happens when the model contradicts itself in
answers that should logically be consistent or when it changes its stance
on factual information within the same conversation or across repeated
queries.
2. Factual - Factual hallucinations in Large Language Models (LLMs) occur

when the model generates information that appears plausible but is
factually incorrect or not grounded in any real-world data. These
hallucinations are problematic because the model confidently presents
incorrect or fabricated information as if it were true.
After analyzing the hallucinations tried on LLaMa. We can infer the following’
1. LLaMa hallucinates a lot for mathematical problems. It doesn’t have the
capability to break a large problem step by step and then solve it. It gives
different answers for the same input asked many times.
2. In mathematics, it tends to not apply required concepts at correct places.
For eg: when asked if 9.9 is greater than 9.11, it simply says that since 9 <
11, 9.9 < 9.11, which is not how decimals are compared.
3. It also tends to lose control over the answer when a question is asked in a
different way, and tends to get confused between the two and gives
different answers.
4. It is also not good enough in remembering some dates and particular
events.’
5. LLaMa doesn’t work well on Indian History and politics. It tends to give
wrong facts.
But, LLaMa has the capacity to work well with RAG, where when particular
information is given , it learns it correctly and applies it as well, for eg, when
given how to solve a math problem step by step, it tends to catch the method and
give the correct answer.
II. Probing:
For probing, IMDB dataset for top 1000 movies and tv shows is used, both for
classification and regression, where in classification, the model is predicting the
certificate of the movie and in regression, it is predicting the rating of the movie,
based on audience.
1. Regression:
Prompt: How well would you recommend the movie {title} with perspective
of the audience.
Analysis:
● The highest MSE for the first layer embeddings suggests that this
layer’s representations are less accurate or less informative for the
regression task. This might indicate that the raw, initial embeddings
do not capture enough relevant information for the task.
● The mid-layer embeddings have a significantly lower MSE compared
to the first layer. This suggests that the model has started to refine
and transform the initial embeddings into more meaningful
representations that are more suited to the task.
● The MSE for the final layer embeddings is slightly higher than the
mid-layer but still lower than the first layer. This could indicate that
the final layer embeddings are very close to the optimal
representation but might be overfitting due to less data available and
more parameters to train.
● There might be additional noise or complexity introduced in the final
layer that slightly increases the MSE.
Therefore, it is evident that as the layers progress, the model tends to learn
more about the context and the meaning behind the text. Little overfitting
happened from the middle layer to the last layer due to less data availability to
the model.
2. Classification:
Prompt: How good is the movie {title} to watch with family including
children
● The lowest accuracy for the first layer embeddings suggests that
these embeddings are less effective for classification. This indicates
that the raw, initial embeddings do not capture sufficient
discriminative features needed for accurate classification. The
model's performance is relatively poor with the first layer
embeddings
● The mid-layer embeddings show a significant improvement in
accuracy compared to the first layer. This suggests that the model’s
intermediate layers are effectively transforming and refining the initial
embeddings into more informative features.
● The accuracy for the final layer embeddings remains the same as for
the mid-layer embeddings. This stability in accuracy might indicate
that the final layer's embeddings are as effective as the mid-layer
embeddings for the classification task.
Therefore, the increase in accuracy from the first layer to the mid-layer shows
that the model is learning and refining features effectively through intermediate
layers. The accuracy does not change between the mid-layer and final layer
embeddings. This suggests that the final layer’s embeddings are as effective as
the mid-layer embeddings for classification purposes. This happens due to less
data in the probing to make complex and less intuitive adjustments in the last
layers.

LLM Assignment 1

Uploaded by

LLM Assignment 1

Uploaded by

LLM Assignment 1

Name: Drishti Singh

2. Factual - Factual hallucinations in Large Language Models (LLMs) occur

You might also like