By EnginStack Editorial Team | Reviewed against NIST, BIPM and primary-source documentation About →

1. One Wire, Four Answers

Ask four converters for 11 AWG in millimeters and you can get four different numbers. Not four values differing in the eighth decimal place — four values differing in the second decimal place: 2.286, 2.304, 2.305, 2.311. Each one is defensible. Each one comes from the same formula, applied by someone who stopped rounding at a different point.

American Wire Gauge is defined by an expression rather than a table. The diameter in inches is 0.005 times 92 raised to the power (36 − n) divided by 39. For 11 AWG that returns 0.0907420 inch, and the exact factor of 25.4 millimeters per inch turns it into 2.305 mm. Everything else on a published chart is that same number seen through a different amount of rounding.

Where the chain was cut Inch value used Exact result (mm) Printed as
The formula, all digits0.09074202.3052.305 mm
Rounded to four places first0.09072.3042.304 mm
Rounded to three places first0.0912.3112.311 mm
Rounded to two places first0.092.2862.29 mm

The spread is 0.025 mm, which is 1.1 per cent of the answer. For scale, the step from one gauge to the next at this size is 0.283 mm, so the disagreement between two published charts is about 9 per cent of a single gauge step. That is small enough to be invisible when you are estimating a cable run and large enough to settle an argument about which chart is right. Neither chart is right. They answered slightly different questions.

The useful thing to take from that table is not which row is correct. It is that the answer moved by more than one per cent while everyone involved did competent arithmetic.

2. The One Step That Cannot Be Wrong

Three of those four chains differ because of rounding. None of them differs because of the multiplication, and there is a structural reason: the conversion factors are exact by definition rather than measured.

  • One inch is exactly 25.4 millimeters, fixed by the 1959 International Yard and Pound Agreement and given force in law on both sides of the Atlantic.
  • One pound is exactly 0.45359237 kilograms, from the same agreement.
  • One US gallon is exactly 231 cubic inches, which is exactly 3.785411784 liters.
  • The scale factor between Celsius and Fahrenheit is the fraction 9/5 one way and 5/9 the other, not a decimal that approximates either.

A factor that is exact cannot introduce error. Multiply 0.0907420 by 25.4 and you get 2.305 mm carrying as many digits as you care to write down. The multiplication is the one link in the chain that is safe to trust, which narrows the whole problem to two places: how precise the input was, and where the rounding happened.

There is a subtler consequence, and it explains a great deal of the confusion on datasheets. Because the definitions are exact, a short decimal is a display choice rather than a loss of definition. Writing one pound as 0.4536 kg does not make the pound approximate; it makes that sentence approximate. The distinction sounds academic until you meet a drawing that carries 0.4536 into a chain of three more calculations and treats it as the definition it is not.

3. Where the Rounding Actually Happens

Take 3/32 inch, a size that appears on every drill chart. As a fraction it is exactly 0.09375 inch, and in millimeters that is 2.38125 before rounding, which prints as 2.381.

Now do what a person with a chart and a calculator actually does. The chart gives the inch value to four places: 0.0938. Multiply that by 25.4 and the result is 2.38252, which prints as 2.383.

Two conversions, both arithmetically correct, 0.002 mm apart. Both values appear on drill charts published today. Neither chart made a mistake: one rounded at the end and the other rounded first. The rule that separates them is short enough to memorise.

Round once, at the end. Every intermediate value stays at full precision until the last operation, and then it is rounded once. The second half of that sentence is the part that gets skipped.

Half the fractional drill sizes are exposed to this, and the reason is arithmetic rather than carelessness. A sixty-fourth is exact, but its decimal expansion is not: n/64 runs to six decimal places at most, and it ends in a digit of 5 whenever the numerator is odd, which is half of them. When it ends in 5, a chart that rounds half up and a chart that truncates produce a different last digit, and both are describing the same drill.

Fraction Inch, four places mm from that inch mm from the fraction Outcome
1/320.03130.7950.794differs
3/320.09382.3832.381differs
19/640.29697.5417.541same
21/640.32818.3348.334same

2 of these four differ, and the difference is one to two thousandths of a millimeter. That is not the point. The point is that copying 0.0938 off a chart and converting it is a different act from converting 3/32, because the chart's 0.0938 is already a rounded output. A conversion applied to a rounded number inherits the rounding.

4. A Factor Rounded Before It Was Used

Rounding an answer is a small sin. Rounding the factor is a larger one, because a factor is applied to every input that touches it, while a rounded answer is applied to nothing.

98.6 °F is exactly 37.00 °C: subtract 32 to get 66.6, multiply by 5/9. The factor 5/9 is 0.5555 recurring. Type 0.5556, which is what four places buys you, and the answer comes back as 37.00 °C. Type 0.56, which is what “about 0.56” buys you, and the same body temperature reads 37.30 °C. Three tenths of a degree is the difference between a normal temperature and a fever.

The lesson is not that 0.56 is sloppy. It is that the position of the rounding decides how far the error travels. Two rules of thumb make the same point with more familiar numbers.

Shortcut Exact factor At this input Error
2.2 lb per kg2.2046230 metric tons → 66,139 lb exactly, 66,000 lb by the shortcut139 lb short · 0.21%
1.6 km per mile1.609344500 miles → 804.7 km exactly, 800.0 km by the shortcut4.7 km short · 0.58%

Neither shortcut is wrong for the job it was made for, which is deciding roughly whether a number is the right order of magnitude. Both are wrong if the result is going into another calculation, because the 0.21 per cent does not stay 0.21 per cent — it gets multiplied by whatever comes next.

5. Measured Numbers and Defined Numbers

This is the distinction that decides how many digits to write, and it is not the one most people reach for. The question is not how precise a number looks. It is where the number came from.

A measurement carries the resolution of the instrument that produced it. Read 2.5 inches off a tape measure and the last digit is a statement about the tape, not about the board: the true length lies somewhere between 2.45 and 2.55, and converting 2.5 to 63.5 millimeters claims a precision the tape never had. The honest conversion is 64 mm, or 63 mm, or a range.

A specification carries no uncertainty at all. 3/32 inch is the diameter a standard assigns to a drill, and it is exactly 0.09375 inch because a committee wrote it down that way. Converting it to 2.381 mm is not false precision, because there was never a measurement to overstate. The digits are limited only by what the job needs.

Two numbers that look equally precise, two opposite answers. A conversion cannot create precision the input did not have, and it should not throw away precision the input did have. Both halves follow from the same requirement: the last digit of the output has to correspond to something real about the input.

Applied to a printed chart, the question has an uncomfortable answer. The number in the chart is neither measured nor specified — it is a display, and its precision is whatever the chart's author chose. That is precisely why converting 0.0938 off a chart is a different act from converting 3/32 inch, even though the two numbers are meant to represent the same drill.

6. How Many Digits to Keep

Five rules cover every case on this site, and the first two are the ones that get confused with each other.

  1. Multiplication and division keep significant figures. The answer carries no more significant figures than the least precise input. A two-figure measurement times an exact factor gives a two-figure answer.
  2. Addition and subtraction keep decimal places. Offsets, altitude corrections and gauge-to-absolute conversions follow this rule instead, and the output keeps no more decimal places than the least precise term. This is a different count from rule one, and applying rule one here is a common error.
  3. Exact factors cost nothing. 25.4, 0.45359237, 5/9 and 231 are exact, so they never limit the answer's digit count. Only measured and specified inputs do.
  4. Round once, at the end. Intermediate values stay at full precision, however ugly they look in a spreadsheet cell.
  5. An affine conversion applies both rules in order. Celsius to Fahrenheit is an addition followed by a multiplication, so the offset decides the decimal places and the scale decides nothing about them.
Input What it is Digits out Result
2.5 inmeasured, two figurestwo64 mm
3/32 inspecified exact fractionas many as the job needs2.381 mm
0.0907 inspecified, already rounded to four placesfour2.304 mm
98.6 °Fmeasured, three figuresthree37.00 °C
150 psirated, two to three figurestwo to three1.0 MPa

Read the second and third rows together, because they are the same wire of the same problem in two different states. An exact fraction can support four digits. The same fraction already rounded to four places in inches can support four digits in millimeters, but only if you accept that the fourth is inherited rather than computed.

7. When Rounding Is Not Allowed

Three places where the rules above stop being enough, and where rounding in the wrong direction stops being an accuracy question: pressure test limits, pharmaceutical doses, and fuel loads. In all three the practice is identical. Carry full precision, round once at the end, and round toward the conservative side rather than the nearest one, because the two are not the same when the number is a limit.

Wall thickness is a fourth case, and the quietest, because nobody looks there.

NPS 24 Schedule 40 has a wall of 0.688 inch, and the mass works out at 171.48 lb/ft. Round that wall to 0.69 inch first — a change of two thousandths of an inch, which looks like nothing — and the identical formula returns 171.96 lb/ft. A difference of 0.48 lb per foot, or 0.28 per cent.

On one joint of pipe that is invisible. On a thousand twelve-meter joints, which is 39,370 feet of pipe, it is 19,044 pounds — about 9.5 short tons of steel that exists in one calculation and not in the other. Nothing exotic happened: the same rounding as the drill chart, applied to a quantity that then gets multiplied by 39,370.

That is the general shape of the problem. A rounding error is a percentage, but a percentage of a shipment is a number with units, and the units are the part that gets signed for.

8. Why the Charts on This Site Round Differently

Five reference charts here use five different rounding policies, and a careful reader will notice. The differences are not stylistic. Each chart's inch column holds a different kind of number, and the millimeter column has to follow from what that number actually is.

Chart What the inch column holds Millimeters computed from
Sheet metal gaugea thickness back-calculated from weight per unit area, so its fourth digit is approximatethat rounded inch value — there is nothing behind it
Drill bit sizesan exact fraction, n/64the fraction, never the printed inch
AWG wire gaugethe defining formula, 0.005 × 92(36−n)/39the formula, at full precision
Nominal pipe sizea defined table value from ASME B36.10Mthe defined inch × 25.4, exactly
Unit conversion charta legal definitionthe definition, exactly

The sheet metal gauge chart is the one that looks inconsistent and is not. Its inch figure originally came out of a calculation — a nominal weight per square foot divided by a density — and that calculation does not support more than four places. So the millimeters are computed from the rounded inch, because the rounding is part of the number. Computing them from the unrounded internal value would dress the chart in a precision it does not have.

The drill and AWG charts go the other way, and for the same reason. Their inch values are exact — one an exact fraction, the other a formula — so full precision is genuinely available, and the millimeters are computed from it. Using the printed inch there would import a display error into an input that never had one.

The pipe chart sits closest to the second group. Its outside diameters and wall thicknesses are not measured and not calculated: they are values a standard assigns, in inches, and the millimeter column is those values multiplied by an exact 25.4. That is why its inch column carries three places and its millimeter column carries two, and why neither of them is a rounding of the other.

Underneath all five is a single sentence. The millimeter column is computed from whatever the inch column really is, not from what it looks like.

9. Twenty Seconds of Self-Check

Three questions, asked in this order, catch nearly every precision error before it reaches a drawing.

  1. Was this number measured or specified? If you cannot answer, the last digit means nothing and should not be quoted. This is the question that decides everything downstream.
  2. How many times did I round? The answer should be once. If it is twice, the second one is the one that did damage, because it was applied to an already-lossy number.
  3. Did my output gain digits? A conversion can preserve an input's precision and it can shorten it. It cannot add to it. If the output carries more significant figures than the input, one of them was invented somewhere in the middle.

The third question is the one that catches unit converters, and it is worth asking of any tool that prints eight decimal places. Such a tool is not eight times more accurate than one that prints three. It has declined to make a judgement that belongs to whoever is holding the number.

Which brings the argument back to where it started. The four converters that disagree about 11 AWG are all telling the truth about the same wire. The value of the number depends entirely on knowing where its last digit came from — and that is a question no converter can answer for you.

Frequently Asked Questions

Why do two unit converters give different answers for the same conversion?

Because they stop rounding at different points, not because one of them is broken. The factors themselves are usually exact definitions — 25.4 millimeters to the inch, 0.45359237 kilograms to the pound — so the multiplication is identical everywhere. What differs is whether a converter rounds an intermediate value before the next step, and how many digits it prints. For 11 AWG, four published values between 2.286 and 2.311 millimeters all derive from the same defining formula, and each is arithmetically correct.

How many significant figures should a conversion have?

No more than the least precise input supports, and no fewer than the input justifies. A measurement with two significant figures produces an answer with two. A specified value such as the fraction 3/32 inch has no uncertainty of its own, so its conversion can carry as many digits as the job requires. Exact factors such as 25.4 and 0.45359237 never limit the count, because they are exact rather than measured.

Is it ever wrong to round a conversion?

Rounding once, at the end, to the precision the input supports, is correct and necessary. Rounding twice is where errors appear, and rounding an intermediate value is the commonest version of that mistake: 3/32 inch is 2.381 millimeters from the exact fraction but 2.383 millimeters from the four-place inch value. Rounding the factor before using it is worse still, because the error reaches every conversion the factor is applied to rather than one answer.

Why does 3/32 inch appear as both 2.381 and 2.383 millimeters?

Because 3/32 inch is exactly 2.38125 millimeters. Converting the exact fraction and rounding once gives 2.381. Rounding the inch value to four places first gives 0.0938, and multiplying that by 25.4 gives 2.38252, which prints as 2.383. Both charts are arithmetically correct; they applied the rounding at different stages. Half of the fractional drill sizes are exposed to this, because n/64 runs to six decimal places at most and ends in a digit of 5 whenever the numerator is odd.

Does a more precise conversion factor make the answer more accurate?

Only up to the precision of the input, and never beyond it. Multiplying a two-figure measurement by a factor with ten decimal places produces a number with ten decimal places and two figures of real information. The extra digits are not arithmetically wrong, but they describe a measurement that was never made. The factor needs to be more precise than the input, not more precise than the output.

Sources and Further Reading

  • NIST Special Publication 811, Guide for the Use of the International System of Units, Section 7, which sets out the rules for rounding and for significant figures in converted values, including the instruction to round only once and only at the end.
  • NIST Handbook 44, Appendix C, General Tables of Units of Measurement, which marks the exact conversion factors. Every factor quoted as exact in this article is underlined as exact in that appendix, including the inch at 2.54 centimeters and the pound at 0.45359237 kilograms.
  • The International Yard and Pound Agreement of 1 July 1959, which fixed the yard at 0.9144 meter and the pound at 0.45359237 kilogram in the United States, the United Kingdom, Canada, Australia, New Zealand and South Africa. Everything above the pound — the stone, the hundredweight, the ton — diverged between the two countries before that date and never converged.
  • ASTM E29, Standard Practice for Using Significant Digits in Test Data to Determine Conformance with Specifications, for the industrial convention on rounding when a measured value is compared against a limit rather than reported.
  • The wire gauge figures come from the defining expression for American Wire Gauge, d = 0.005 × 92(36−n)/39 inch, evaluated at fifty digits and rounded once. The drill figures come from the exact fractions in ANSI/ASME B94.11M. The pipe figures come from ASME B36.10M, with mass from the geometry at a density of 7850 kilograms per cubic meter.
  • The charts whose rounding policies are compared in the eighth section are all published on this site and generated from the models described above, so each policy can be checked against the page it belongs to rather than taken on trust.

Related Guides

Reprint & Attribution. This article was written and fact-checked by the EnginStack Editorial Team. It was first published on November 10, 2026 at enginstack.com/guides/why-converters-disagree. You are welcome to quote it, share it, or translate it — we ask only that you link back to the original page and credit EnginStack as the source. If you'd like to republish the full text on your own site, please contact us first: [email protected]. We license syndication at no cost for educational and non-commercial use with proper attribution.