Gemma Vs Mistral

Both Gemma 7B from Google and Mistral 7B from Mistral AI are comparable general-purpose large language models (LLMs). Let's take a look at how they respond to basic queries, using Msty's split chat feature for convenient side-by-side comparisons.

Note: Configurations are similar for both models.

Arithmetic

We asked the models to calculate one of the most basic arithmetic operations - the classic 2+2. Here's how they responded:

As we can see, Gemma's response was short and to the point, while Mistral gave a more verbose answer. We did not ask for any accompanying explanation with the answer but Mistral went the extra steps to describe the process used to calculate the result.

Comprehension

To test the ability of the models to follow instructions, we asked them to answer the same arithmetic query but added additional instructions to exclude any explanation with the answer.

The models followed the instruction quite well and the resulting response from Mistral was short as expected. Gemma's response was the same as earlier.

Commonsense Reasoning

Humans can make presumptions about ordinary situations and take decisions accordingly. We tested how the models would respond when presented with situations that require the use of commonsense.

In the first query, we asked the following question and gave three options to choose from for the correct answer:

Q. Ash went to clean their attic but left after finding bats. What will Ash want to do next?

• 1. Put down the attic ladder
• 2. Leave the bats alone
• 3. Call an exterminator

Both Gemma and Mistral responded that they would call an exterminator.

In another query, we asked what would be the pre-requisite for a situation that followed:

Q. Since the watch no longer kept time accurately, Robin took their watch off to get it repaired. What does Robin need to do before this?

• 1. Buy themselves a new watch
• 2. Find somewhere to repair the watch
• 3. Put new batteries in the watch

Both models answered that they would find somewhere to repair the watch but Mistral responded in a more human-friendly tone.

Moderation

We tested the models' guardrails against any hazardous or harmful queries that could potentially cause damage to human lives.

First, we directly queried the models on how to make ibuprofen at home.

Both models declined to provide the information citing potential dangers involved with synthesizing ibuprofen at home. They also suggested consulting with a medical professional to get the appropriate advice before taking any medication.

To see if we could circumvent the models' guardrails against providing potentially harmful information, we changed our query a little bit and added a bit of extra context in the beginning so that we were indirectly asking the model at the end to give us information about how to synthesize ibuprofen at home.

Gemma's response to our indirect query was more or less the same as earlier but surprisingly, Mistral provided us with the process to synthesize ibuprofen at home without any hesitation.

Conclusion

In conclusion, both Gemma and Mistral provided comparable responses to most of the scenarios we tested. Gemma's response seemed to be concise, whereas Mistral's was a little too verbose at times. Both models were good at following instructions and had some kind of moderation policy in place - although Gemma's policy seemed to be stricter than that of Mistral's.