Suppose n and p are unsigned int variables in a C program. We wish to set p to $^n\text{C}_3$. If n is large, which one of the following statements is most likely to set p correctly?

We need to compute:

When $n$ is large, computing the product $n(n-1)(n-2)$ directly will easily cause integer overflow before the division by 6 is performed.

Let's evaluate each option:

• (A)p = n * (n-1) * (n-2) / 6;: Computes the full 3-term product first, which is most vulnerable to integer overflow.

• (B)p = n * (n-1) / 2 * (n-2) / 3;:

  • One of $n$ or $n-1$ is always even, so $n(n-1)$ is always divisible by 2. Thus, integer division n * (n-1) / 2 loses no precision and yields an exact integer.

  • The product of any three consecutive integers $n(n-1)(n-2)$ is divisible by 6. Since division by 2 was performed first without remainder, the remaining factor must be divisible by 3. Thus, ... * (n-2) / 3 also divides evenly with no fractional loss.

  • Performing the division early keeps intermediate values smaller, preventing overflow.

• (C)p = n * (n-1) / 3 * (n-2) / 2;: $n(n-1)$ is not necessarily divisible by 3 (for instance, if $n=5$, $5 \times 4 = 20$, and $20/3 = 6$ truncates the fractional part), resulting in an incorrect answer.

• (D)p = n * (n-1) * (n-2) / 6.0;: The integer expression n * (n-1) * (n-2) is still evaluated first in unsigned int arithmetic before promotion to double, so overflow still happens.

Therefore, (B) is the correct statement.