Specimen · Process Calibration Audit
Stated tolerances against measured parts: a radial forge
A worked example of the deliverable, formatted as a client receives it. It is run on a public research data set rather than on client records, and every number in it regenerates from the linked scripts.
Verdict
No part in the record meets every stated tolerance. 1 dimension is outside its limits on all 81 parts by a constant offset, part length runs at its upper limit, and 8 of the 18 tolerances are so wide that these records never tested them. Of 9 findings, 3 are Material, 2 are Significant and 4 are Observations. 9 of 81 parts are over the stated length limit. A 90% range around a prediction of length covered 73% when built the usual way, and 94% when built from days the model had not seen.
| ID | Finding | Severity |
|---|---|---|
| F-01 | No part meets the stated tolerance on the inner diameter at 55 mm: all 81 are outside | Material |
| F-02 | Part length runs at the upper limit: 9 of 81 parts exceed it on at least one of the 4 measurements | Material |
| F-03 | A 90% range built from the model's own training errors covered 73% of parts on days it never saw | Material |
| F-04 | The two files that state the length tolerance disagree, and one sensor channel is constant | Significant |
| F-05 | 2 length measurements are inside the limit on every part and too close to it | Significant |
| F-06 | 8 of 18 stated tolerances were never tested by these records | Observation |
| F-07 | A 90% range built from errors on held-out days covered 94% of parts on days it never saw | Observation |
| F-08 | A 90% range built from errors on held-out rows covered 88% of parts on days it never saw | Observation |
| F-09 | The process features improve the prediction of part length by 0.13 mm on an error of 0.99 mm | Observation |
Data-structure check, delivered on day 2
Before anything is scored, the records are checked for whether they can support the analysis at all. This section is what a client receives inside 48 hours of sending data. If the answer were no, the engagement would stop here.
| Check | Result |
|---|---|
| Every measured part has a sensor trace | 81 of 81 |
| Missing values in dimensions and features | 0 |
| The unit that repeats | The production day: 10-31 10, 11-01 25, 11-02 27, 11-03 13, 11-06 6 parts |
| Days stated in the description against days in the records | 1 stated, 5 recorded |
| Lower length tolerance in the two files that state it | -5 mm in the measurement file; "plus 15, minus 8" in the structure summary |
| Sensor channels with no variation across parts | 3 of 14 features: temp_at_forge_start, l_pos_min, r_pos_min |
| Trace length | median 17,880 rows; 12 traces more than 20% longer |
| Can the records support the analysis | Yes, with the 3 questions in the register |
2 of the constant channels are hammer end positions, which are meant to be constant. The third is the pyrometer at the start of forging, which reads the bottom of its range on every part: billet temperature at the hammer is not in these records.
Scope and exclusions
- Whether the tolerances are the right ones. They are taken as stated in the data set.
- The measuring machine. There is one reading per part and no repeatability study, so measurement error is inside every number here.
- Anything about a production line. This is a research testbed running a trial over 5 days.
- Change intervals or tool life. The records carry no tool changes; the tool-wear paper covers that question on other data.
- What a rejected or reworked part costs. The data set has no costs.
Findings register
F-01 · Stated limit Material
No part meets the stated tolerance on the inner diameter at 55 mm: all 81 are outside
- Why this severity
- 100% of parts lie outside the stated tolerance (lower bound 95.5%, above the 1% floor)
- Measurement
- Stated 30 mm, limits 29.7 to 30.3. Measured mean 29.568 mm, standard deviation 0.008 mm, range 29.544 to 29.591. 81 of 81 parts outside. The same diameter at 100 mm averages 29.857 mm and is inside on every part.
- Consequence
- Read as stated, every part in the record is a reject. The scatter is 8 thousandths of a millimetre, so this is not a process out of control: it is a process holding a value the specification does not describe.
- Recommended action
- Resolve the specification before anything else. Either the nominal at this position is not 30 mm, or the tolerance was copied from the 100 mm position. Only the owner of the drawing can say which.
- Limit of this finding
- The audit cannot tell a wrong specification from a wrong process. It can tell that one of them is wrong, and that the offset is constant across 5 days.
F-02 · Stated limit Material
Part length runs at the upper limit: 9 of 81 parts exceed it on at least one of the 4 measurements
- Why this severity
- 7% of parts on Top4 lie outside the stated tolerance (lower bound 3.4%, above the 1% floor)
- Measurement
- Stated 222 mm, plus 15 and minus 5, so the upper limit is 237 mm. Top1: mean 234.5 mm, 0 of 81 over [0.0%, 4.5%], capability index 0.76; Top2: mean 235.0 mm, 3 of 81 over [1.3%, 10.3%], capability index 0.60; Top3: mean 233.8 mm, 0 of 81 over [0.0%, 4.5%], capability index 0.98; Top4: mean 235.5 mm, 6 of 81 over [3.4%, 15.2%], capability index 0.48. Intervals are 95% Wilson on 81 parts. By day, parts over the limit on any measurement: 2017-10-31 1 of 10, 2017-11-01 7 of 25, 2017-11-02 1 of 27, 2017-11-03 0 of 13, 2017-11-06 0 of 6.
- Consequence
- 11.1% of parts are over the stated length limit, and the mean sits 1.4 standard deviations below it on the worst measurement. The decision table prices that at your own rates.
- Recommended action
- Either re-centre the process about 2 mm shorter, or confirm that the upper limit is the one the customer holds you to. If the length is trimmed later, the limit that matters is the trim allowance, and this finding should be re-read against it.
- Limit of this finding
- 81 parts over 5 days from a testbed, not a production line. The Wilson intervals treat parts as independent, which parts forged on one day are not; the per-day counts are given so the reader can see how uneven they are.
F-03 · Predicted range Material
A 90% range built from the model's own training errors covered 73% of parts on days it never saw
- Why this severity
- the stated 90% lies outside the interval of delivered coverage [62%, 81%], and on the longest parts coverage is 67% with an upper bound of 83%
- Measurement
- Predicting the longest of the 4 lengths from 14 process features, each day held out whole. Covered 73% [62%, 81%] of 81 parts against a stated 90%; on the 21 longest parts 67% [45%, 83%]. Median width 2.62 mm. By day: 2017-10-31 70%, 2017-11-01 64%, 2017-11-02 93%, 2017-11-03 38%, 2017-11-06 100%.
- Consequence
- A range that covers less than it states will pass parts it should have flagged.
- Recommended action
- Do not quote this range at 90%.
- Limit of this finding
- 81 parts, 5 days. With 5 days, one unusual day moves every figure here, and the intervals treat parts as independent.
F-04 · Data structure Significant
The two files that state the length tolerance disagree, and one sensor channel is constant
- Why this severity
- the analysis runs, but a stated limit or a channel is ambiguous and only the owner of the specification can resolve it
- Measurement
- The measurement file gives the lower tolerance on Top4 as -5 mm; the structure summary gives 'plus 15, minus 8'. The data set's description says the parts were forged over one day; the measurement dates span 5 days (2017-10-31 to 2017-11-06). The pyrometer reading at the start of forging is the same on all 81 parts (243.8), which is the bottom of its range, so billet temperature at the hammer is not recorded. 12 traces are more than 20% longer than the median (parts 1, 22, 43, 53, 56, 60, 61, 65, 70, 71, 75, 77).
- Consequence
- None of these stops the analysis. Each changes how a result should be read, and the first is a question about which document is the specification.
- Recommended action
- Confirm the lower length tolerance, the number of production days, and whether a billet temperature at transfer is logged anywhere else.
- Limit of this finding
- A structure check finds what is inconsistent inside the records. It cannot find what the records leave out without being told.
F-05 · Stated limit Significant
2 length measurements are inside the limit on every part and too close to it
- Why this severity
- no share outside the tolerance is established, but the capability index is 0.76, below the conventional 1.33
- Measurement
- Top1: maximum 236.3 mm against 237, capability index 0.76; Top3: maximum 236.2 mm against 237, capability index 0.98.
- Consequence
- No reject yet on these measurements. At this capability a run several times longer would be expected to produce some.
- Recommended action
- Treat with the finding above: it is the same length, measured at another angle.
- Limit of this finding
- A capability index assumes a stable, roughly normal process, and 81 parts cannot establish either.
F-06 · Stated limit Observation
8 of 18 stated tolerances were never tested by these records
- Why this severity
- every part is inside, and the nearest limit is more than 6 standard deviations from the mean on each
- Measurement
- ID @100 (47 SD); 38 dia @200 (13 SD); 42 dia @140 (11 SD); 42 dia @80 (9 SD); Base angle BL (7 SD); 40.5mm taper F (8 SD); 40.5mm taper BR (11 SD); 40.5mm taper BL (10 SD).
- Consequence
- These limits are met, and by so wide a margin that the record says nothing about what would happen near them. A coverage figure over all 18 dimensions would be dominated by limits that cannot fail.
- Recommended action
- None. Do not average them into a headline.
- Limit of this finding
- Wide margins on a testbed part do not show the tolerances are right for a customer.
F-07 · Predicted range Observation
A 90% range built from errors on held-out days covered 94% of parts on days it never saw
- Why this severity
- the stated 90% lies inside the interval of delivered coverage [86%, 97%]
- Measurement
- Predicting the longest of the 4 lengths from 14 process features, each day held out whole. Covered 94% [86%, 97%] of 81 parts against a stated 90%; on the 21 longest parts 100% [85%, 100%]. Median width 6.38 mm. By day: 2017-10-31 100%, 2017-11-01 100%, 2017-11-02 100%, 2017-11-03 62%, 2017-11-06 100%.
- Consequence
- On this evidence the range does what it states, at the width shown.
- Recommended action
- Usable as stated. Watch the per-day figures: a single day far below the rest is how a regime change first shows.
- Limit of this finding
- 81 parts, 5 days. With 5 days, one unusual day moves every figure here, and the intervals treat parts as independent.
F-08 · Predicted range Observation
A 90% range built from errors on held-out rows covered 88% of parts on days it never saw
- Why this severity
- the stated 90% lies inside the interval of delivered coverage [79%, 93%]
- Measurement
- Predicting the longest of the 4 lengths from 14 process features, each day held out whole. Covered 88% [79%, 93%] of 81 parts against a stated 90%; on the 21 longest parts 81% [60%, 92%]. Median width 3.37 mm. By day: 2017-10-31 100%, 2017-11-01 80%, 2017-11-02 100%, 2017-11-03 62%, 2017-11-06 100%.
- Consequence
- On this evidence the range does what it states, at the width shown.
- Recommended action
- Usable as stated. Watch the per-day figures: a single day far below the rest is how a regime change first shows.
- Limit of this finding
- 81 parts, 5 days. With 5 days, one unusual day moves every figure here, and the intervals treat parts as independent.
F-09 · Predicted range Observation
The process features improve the prediction of part length by 0.13 mm on an error of 0.99 mm
- Why this severity
- recorded as a limit on what the ranges above can be used for, not as a failure of a stated claim
- Measurement
- Mean absolute error 0.86 mm with 14 features, 0.99 mm predicting the mean of the training days, against a standard deviation of 1.11 mm. Predictions span 233.6 to 237.3 mm; measured lengths span 232.2 to 237.9 mm.
- Consequence
- A flag-or-ship rule built on this prediction separates long parts only weakly. The decision curve shows the price: catching every over-limit part means checking about 9 parts in 10.
- Recommended action
- Before paying for in-process prediction of length, find the variable that moves it. Billet length or volume before forging is the obvious candidate and is not in these records.
- Limit of this finding
- A fixed, short feature list on 81 parts. A richer model might do better and would need far more parts to show it honestly.
Evidence
Each stated tolerance against the measured dimension
Evidence. 81 parts per row. The interval on the share outside is a 95% Wilson interval. The last column is the distance from the mean to the nearest limit in standard deviations: above 6, the limit was never tested.
| Dimension | Limits | Mean | SD | Outside | 95% interval | Nearest limit, SD | Severity |
|---|---|---|---|---|---|---|---|
| ID @100 | 29.7 to 30.3 | 29.857 | 0.003 | 0 of 81 | [0.0%, 4.5%] | 46.8 | Observation |
| ID @55 | 29.7 to 30.3 | 29.568 | 0.008 | 81 of 81 | [95.5%, 100.0%] | -16.3 | Material |
| 38 dia @200 | 37.6 to 38.4 | 38.012 | 0.029 | 0 of 81 | [0.0%, 4.5%] | 13.2 | Observation |
| 42 dia @140 | 41.6 to 42.4 | 41.945 | 0.031 | 0 of 81 | [0.0%, 4.5%] | 11.2 | Observation |
| 42 dia @80 | 41.6 to 42.4 | 42.136 | 0.029 | 0 of 81 | [0.0%, 4.5%] | 9.0 | Observation |
| Base angle F | 69 to 71 | 69.818 | 0.150 | 0 of 81 | [0.0%, 4.5%] | 5.5 | Observation |
| Base angle BR | 69 to 71 | 69.798 | 0.152 | 0 of 81 | [0.0%, 4.5%] | 5.3 | Observation |
| Base angle BL | 69 to 71 | 69.896 | 0.131 | 0 of 81 | [0.0%, 4.5%] | 6.8 | Observation |
| 162mm taper F | 161 to 163 | 161.595 | 0.105 | 0 of 81 | [0.0%, 4.5%] | 5.7 | Observation |
| 162mm taper BR | 161 to 163 | 161.538 | 0.108 | 0 of 81 | [0.0%, 4.5%] | 5.0 | Observation |
| 162mm taper BL | 161 to 163 | 161.522 | 0.101 | 0 of 81 | [0.0%, 4.5%] | 5.2 | Observation |
| 40.5mm taper F | 39.5 to 41.5 | 40.624 | 0.105 | 0 of 81 | [0.0%, 4.5%] | 8.3 | Observation |
| 40.5mm taper BR | 39.5 to 41.5 | 40.565 | 0.088 | 0 of 81 | [0.0%, 4.5%] | 10.7 | Observation |
| 40.5mm taper BL | 39.5 to 41.5 | 40.492 | 0.104 | 0 of 81 | [0.0%, 4.5%] | 9.6 | Observation |
| Top1 | 217 to 237 | 234.525 | 1.083 | 0 of 81 | [0.0%, 4.5%] | 2.3 | Significant |
| Top2 | 217 to 237 | 235.038 | 1.091 | 3 of 81 | [1.3%, 10.3%] | 1.8 | Material |
| Top3 | 217 to 237 | 233.763 | 1.099 | 0 of 81 | [0.0%, 4.5%] | 2.9 | Significant |
| Top4 | 217 to 237 | 235.460 | 1.071 | 6 of 81 | [3.4%, 15.2%] | 1.4 | Material |
3 ways to state a 90% range for part length
Evidence. The quantity predicted is the longest of the 4 measured lengths, from 14 process features fixed before any was compared with a dimension. Every prediction is for a day the model never saw. The range is then built 3 ways, and the only difference between them is which errors set its width.
| How the range was built | States | Covered | 95% interval | Longest quarter | Median width | 10-31 | 11-01 | 11-02 | 11-03 | 11-06 |
|---|---|---|---|---|---|---|---|---|---|---|
| From errors on its own training data | 90% | 73% | [62%, 81%] | 67% | 2.62 mm | 70% | 64% | 93% | 38% | 100% |
| From errors on held-out days | 90% | 94% | [86%, 97%] | 100% | 6.38 mm | 100% | 100% | 100% | 62% | 100% |
| From errors on held-out rows | 90% | 88% | [79%, 93%] | 81% | 3.37 mm | 100% | 80% | 100% | 62% | 100% |
The range built from training errors is the narrowest and the only one that fails. It is narrow because a model's errors on the data it was fitted to are smaller than its errors on anything new. The range built from held-out rows is about half the width of the day-held-out one and covered 88% [79%, 93%]: a shortfall these 81 parts cannot establish. Both held-out ranges covered 62% on the same single day. The range built from held-out days holds, and pays for it: at 6.4 mm it is wider than the 5.8 mm spread of the lengths themselves.
Prediction error: 0.86 mm mean absolute error with the features, 0.99 mm using only the average of the other days. Predictions span 233.6 to 237.3 mm; measured lengths span 232.2 to 237.9 mm.
Flag or ship, in counts and then in money
9 of 81 parts are over the 237 mm limit, which is 11.1 in every 100. A policy decides which parts get checked before they ship. The counts below are measured on held-out days. What a check costs and what an over-limit part costs once it has shipped are yours to enter.
Illustration. Both boxes start at placeholder values that are not estimates of anything in this data set. Enter your own and the ranking below changes with them. The counts in the table are evidence; the euro column is arithmetic on your two numbers.
| Policy | Checked | Checked, in limit | Over limit, shipped | Cost per 100 parts |
|---|---|---|---|---|
| Ship every part unchecked | 0.0 | 0.0 | 11.1 | EUR 4,444 |
| Check every part | 100.0 | 88.9 | 0.0 | EUR 1,500 |
| Check when the top of the range passes the limit | 90.1 | 79.0 | 0.0 | EUR 1,352 |
| Check when the prediction itself passes the limit | 3.7 | 2.5 | 9.9 | EUR 4,006 |
| Check when the prediction passes 234.75 mm, the cheapest threshold at these rates | 84.0 | 72.8 | 0.0 | EUR 1,259 |
Assumptions register
| Assumption | If it is wrong |
|---|---|
| The tolerance rows in the measurement file are the specification | The largest lever here. The structure summary states a different lower length tolerance, and the inner diameter at 55 mm fails on every part. If the specification is elsewhere, F-01 and F-02 are re-read against it. |
| Parts are independent trials for the Wilson intervals | Understates every interval. Parts forged on one day share a setup. With 5 days there are too few blocks to resample, so the per-day figures are printed instead. |
| A part is over the limit when any of its 4 length readings is | A customer who inspects one position would see fewer rejects. |
| The measured dimension is the true dimension | Either direction. One reading per part and no gauge study. |
| The feature list and the penalty were fixed before scoring | They were. A tuned model would report a smaller error on these 81 parts and deserve less trust. |
| A check finds every over-limit part it is applied to, and costs the same each time | Flatters every policy that checks. A real inspection misses some. |
| The 2 cost rates in the control | Illustrative until the reader replaces them. No rate comes from the data set. |
Method note
How severity is assigned
Severity is derived by rule. Each kind of finding has its own, fixed in the script.
- Stated limit
- Material when the lower bound of the share of parts outside the tolerance exceeds 1%. Significant when no such share is established and the capability index is below 1.33, a conventional minimum. Observation otherwise.
- Predicted range
- Observation when the stated level lies inside the interval of delivered coverage. Material when it lies outside and the upper bound of coverage on the longest quarter of parts is also below the stated level. Significant otherwise.
- Data structure
- Material when part of the analysis cannot run on the records as supplied. Significant when it runs but a stated limit or a channel is ambiguous. Observation otherwise.
The 1% floor and the capability threshold are commercial judgements, stated so they can be argued with. They were fixed after the share of parts inside each tolerance was known and before any range or policy was scored.
Why whole days are held out
Parts forged on one day share a setup and a warm-up. A model tested on parts from a day it also trained on is being tested on something it has partly seen. Here the row split gave a range 0.53 times the width of the day split and covered 88% against 94%. With 5 days that difference is suggestive and not established. The background note on clustered data explains the mechanism, and the tool-wear paper shows a case where it is established.
Lineage
| Data | Sensor data set radial forging at AFRC testbed v2. Tachtatzis, C.; Gourlay, G.; Andonovic, I.; Panni, O. University of Strathclyde, Advanced Forming Research Centre, via Zenodo, 2019-09-11. doi:10.5281/zenodo.3405265. Creative Commons Attribution 4.0 International. Archive md5 549e0e2f365c51748373f265dff7b268. |
|---|---|
| Scripts | src/features.py, then src/audit_forging.py |
| Code | src/audit_forging.py, sha256 f67b6e65065f9a0fc761391a7f87da0edf3436eb3edbc0ac3957f45d3218ed81; src/features.py, sha256 ca53eaf08e8ae8f29abc742b6d476e44a5eb0bfd5137f638e00665e16c03f932 |
| Artifact digest | sha256 47541f0fb311e5abd7dc81753442dc9e214f791dfc1d6cf4861026bde8e2e187 |
| Digest rule | sha256 over every file in artifacts/specimen except meta.json, in filename order, feeding each file's name as UTF-8 followed by its bytes. meta.json is excluded because it carries the digest and cannot hash itself. |
| Repository | kwantil-research, analyses/forging-tolerance-audit |
What this report does not support
4 conclusions do not follow from it.
- That this forge makes bad parts. It is a research testbed, the tolerances may not be a customer's, and one failing dimension looks like a specification question.
- That in-process prediction of dimensions does not work. It shows that 14 features on 81 parts barely move the error. A variable that is not in these records may do far more.
- Any yield, scrap rate or cost for a production line. 81 parts over 5 days, and no cost in the data.
- That the day-held-out range would hold next month. It held on 4 of 5 days and covered 62% on the other.
A client engagement replaces the public records with your own and changes nothing else: the day-2 check, the limits, the ranges and the trade-off at your own rates. Scope, timing and price are on the services page.