This PR needs a `release notes:` label

Summary:
Added quantization to evaluation script. Quantization causes deterioriation in accuracy

On wikitext task:

Model Name	max_seq_len	ptq	word_perplexity
Llama 3.2-1B Instruct	128	16a4w	5821003.055178451
Llama 3.2-1B Instruct	128	16a4w_block	5396240.078572427
Llama 3.2-1B Instruct	128	8a8w	533154.970440251

Differential Revision: D76837572

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11822

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 1 Pending

As of commit 6cd35a3 with merge base a12a005 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This pull request was exported from Phabricator. Differential Revision: D76837572

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorct, for example
@pytorct label "release notes: none"

For more information, see
https://.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Summary: Added quantization to evaluation script. Quantization causes deterioriation in accuracy On wikitext task: | Model Name | max_seq_len | ptq | word_perplexity |----------|----------|----------|-----------| | Llama 3.2-1B Instruct | 128 | 16a4w | 5821003.055178451 | | Llama 3.2-1B Instruct | 128 | 16a4w_block | 5396240.078572427 | | Llama 3.2-1B Instruct | 128 | 8a8w | 533154.970440251 | Differential Revision: D76837572

This pull request was exported from Phabricator. Differential Revision: D76837572

Summary: Pull Request resolved: pytorch#11822 Added quantization to evaluation script. Quantization causes deterioriation in accuracy On wikitext task: | Model Name | max_seq_len | ptq | word_perplexity |----------|----------|----------|-----------| | Llama 3.2-1B Instruct | 128 | 16a4w | 5821003.055178451 | | Llama 3.2-1B Instruct | 128 | 16a4w_block | 5396240.078572427 | | Llama 3.2-1B Instruct | 128 | 8a8w | 533154.970440251 | Differential Revision: D76837572

Thank you for the ppl summary table

Summary: Added quantization to evaluation script. Quantization causes deterioriation in accuracy On wikitext task: | Model Name | max_seq_len | ptq | word_perplexity |----------|----------|----------|-----------| | Llama 3.2-1B Instruct | 128 | 16a4w | 5821003.055178451 | | Llama 3.2-1B Instruct | 128 | 16a4w_block | 5396240.078572427 | | Llama 3.2-1B Instruct | 128 | 8a8w | 533154.970440251 | Reviewed By: cccclai Differential Revision: D76837572

This pull request was exported from Phabricator. Differential Revision: D76837572

Summary: Pull Request resolved: pytorch#11822 Added quantization to evaluation script. Quantization causes deterioriation in accuracy On wikitext task: | Model Name | max_seq_len | ptq | word_perplexity |----------|----------|----------|-----------| | Llama 3.2-1B Instruct | 128 | 16a4w | 5821003.055178451 | | Llama 3.2-1B Instruct | 128 | 16a4w_block | 5396240.078572427 | | Llama 3.2-1B Instruct | 128 | 8a8w | 533154.970440251 | Reviewed By: cccclai Differential Revision: D76837572

Summary: Added quantization to evaluation script. Quantization causes deterioriation in accuracy On wikitext task: | Model Name | max_seq_len | ptq | word_perplexity |----------|----------|----------|-----------| | Llama 3.2-1B Instruct | 128 | 16a4w | 5821003.055178451 | | Llama 3.2-1B Instruct | 128 | 16a4w_block | 5396240.078572427 | | Llama 3.2-1B Instruct | 128 | 8a8w | 533154.970440251 | Reviewed By: cccclai Differential Revision: D76837572

Summary: Pull Request resolved: pytorch#11822 Added quantization to evaluation script. Quantization causes deterioriation in accuracy On wikitext task: | Model Name | max_seq_len | ptq | word_perplexity |----------|----------|----------|-----------| | Llama 3.2-1B Instruct | 128 | 16a4w | 5821003.055178451 | | Llama 3.2-1B Instruct | 128 | 16a4w_block | 5396240.078572427 | | Llama 3.2-1B Instruct | 128 | 8a8w | 533154.970440251 | Reviewed By: cccclai Differential Revision: D76837572

This pull request was exported from Phabricator. Differential Revision: D76837572

rohansjoshi requested a review from cccclai as a code owner June 20, 2025 15:29

facebook--bot added the CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.label Jun 20, 2025

facebook--bot added the fb-exported label Jun 20, 2025

rohansjoshi force-pushed the export-D76837572 branch from 4fba0c7 to 88a66d7 Compare June 20, 2025 15:54

rohansjoshi force-pushed the export-D76837572 branch from 88a66d7 to 9607154 Compare June 20, 2025 15:57

cccclai approved these changes Jun 20, 2025
View reviewed changes

rohansjoshi force-pushed the export-D76837572 branch from 9607154 to 8de8d9b Compare June 20, 2025 21:39

rohansjoshi force-pushed the export-D76837572 branch from 8de8d9b to d3228aa Compare June 20, 2025 21:43

rohansjoshi force-pushed the export-D76837572 branch from d3228aa to c9c6560 Compare June 21, 2025 21:24

rohansjoshi force-pushed the export-D76837572 branch from c9c6560 to 6cd35a3 Compare June 21, 2025 21:27

facebook--bot merged commit 608a745 into pytorch:main Jun 21, 2025
102 of 104 checks passed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Conversation

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11822

⏳ No Failures, 1 Pending

Uh oh!

Uh oh!

This PR needs a release notes: label

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This PR needs a `release notes:` label