Eval report
40 questions, 3 personas, 39 documents. Retrieval metrics are computed; answers are recorded claude-sonnet-5-5 runs graded by claude-opus-5-5. Full write-up in evals/REPORT.md.
Retrieval, computed
29 questions with a readable source| Retriever | Doc hit@5 | Section hit@5 | Section MRR@10 | Multi-doc recall@5 |
|---|---|---|---|---|
| BM25 | 1.000 | 0.952 | 0.754 | 0.854 |
| Embeddings (bge-small, local) | 1.000 | 1.000 | 0.905 | 0.792 |
| Hybrid, RRF k=60 | 1.000 | 0.952 | 0.911 | 0.833 |
Iteration: run-1 → run-2
| Metric | run-1 | run-2 |
|---|---|---|
| Context passages (k) | 6 | 8 |
| Strictly correct | 90% | 95% |
| Multi-doc score | 75% | 87.5% |
| Grounded in excerpts (judge) | 100% | 100% |
| Expected section in context | 0.952 | 1.000 |
Every question (run-2)
A01How often do torque wrenches need to be calibrated?correct
Answer
Torque wrenches are calibrated every 6 months, performed in-house with the torque analyzer QP-7.6 §5.2. The interval was changed to 6 months in Rev E (2022-03-07) QP-7.6 §7.
The Quality Engineer may shorten the interval after an OOT result. It can be lengthened only after two consecutive in-tolerance calibrations and with Quality Manager approval QP-7.6 §5.2.
Judge (claude-opus-5-5)
Sources
A02What do we do when a gage is found out of tolerance at calibration?correct
Answer
Follow QP-7.6 §5.6:
- Red-label the gage and remove it from use. QP-7.6 §5.6
- The Quality Engineer opens an OOT Investigation (F-760-02) within 5 working days. QP-7.6 §5.6
- The investigation assesses product impact back to the last good calibration, using the calibration database and the ERP to identify every job inspected with that gage in that period. QP-7.6 §5.6
- Affected product still at Tallgrass is placed on hold per WI-610 and re-inspected. If affected product has shipped, it is handled as an escape under QP-7.5, including customer notification. QP-7.6 §5.6
- The result and actions are recorded on F-760-02 and signed by the Quality Manager. QP-7.6 §5.6
Judge (claude-opus-5-5)
Sources
A03How long does an aluminum part have to soak in the metrology lab before I measure it on the CMM?correct
Answer
An aluminum part needs a minimum soak of 2 hours in the Bay 4 lab (20 +/- 1 C) before CMM measurement WI-210 §5.2. Record the soak start time and lab temperature on the traveler. If the lab is outside 20 +/- 1 C, do not measure the part and notify the Quality Engineer WI-210 §5.2.
Judge (claude-opus-5-5)
Sources
A04What is the probe qualification limit on the CMM and when do I have to requalify?correct
Answer
The qualification standard deviation on reference sphere RS-0412 must be 0.002 mm or less WI-210 §5.3.
You must requalify every stylus you will use at the start of each shift, after any crash, and after any stylus change WI-210 §5.3. If the result is above the limit, clean the stylus and sphere and requalify. If it still fails, tag the CMM out of service and notify the Quality Engineer WI-210 §5.3.
Judge (claude-opus-5-5)
Sources
A05How should I name a new CMM program?correct
Answer
Name the program <PartNumber>_<DrawingRev>_OP<op>_v<n> WI-215 §5.1. For example: 48210-3_C_OP20_v2 WI-215 §3.
- DrawingRev: the customer drawing revision letter WI-215 §5.1.
- OP: the operation number from the traveler at which the inspection is performed WI-215 §5.1.
- v<n>: starts at v1 and increases by one for each change WI-215 §5.1.
A new drawing revision always starts a new program name at v1, even if you copy the content from the previous revision WI-215 §5.1.
Judge (claude-opus-5-5)
Sources
A06The drawing just says break sharp edges with no value. What edge break do I use?correct
Answer
Use an edge break of 0.005-0.015 in, as a chamfer or radius WI-305 §5.1. This default applies when the drawing says "break sharp edges" or "break all edges" without a value WI-305 §5.1.
Measure it with a calibrated radius gage or the optical comparator, and the gage must carry a green calibration label (QP-7.6) WI-305 §5.1. If the edge break exceeds the maximum, stop and red-tag the part per WI-610 WI-305 §5.6.
Judge (claude-opus-5-5)
Sources
A07How much PTO does an employee with 4 years of service earn in a year?correct
Answer
An employee with 4 years of service falls in the 3-6 years tier and earns 160 hours of PTO per year HR-100 §6.1. That works out to 0.0769 hours per hour paid, or approximately 6.15 hours per biweekly pay period HR-100 §6.1. Accrual is based on hours paid, up to 80 hours per pay period, so overtime does not earn additional PTO HR-100 §6.1.
Judge (claude-opus-5-5)
Sources
A08What is the second shift differential?correct
Answer
The 2nd shift differential is $1.75 per hour for all hours worked on 2nd shift (14:30-23:00) HR-110 §7.2. It is included in the regular rate used to calculate overtime, but it is not paid on PTO or holiday pay HR-110 §7.2. It applies to non-exempt (hourly) employees HR-110 §2.
Judge (claude-opus-5-5)
Sources
A09I lost my company laptop. How fast do I have to report it, and who do I call after hours?correct
Answer
Report it to IT within 1 hour of noticing it is missing. IT-06 §5.2
After hours (outside help desk hours of Monday to Friday, 6:00 to 16:30), call the IT after-hours on-call number, 316-555-0142. Lost or stolen devices are covered by that line. IT-06 §5.4
If the laptop was stolen, it is also reported to the police, and the report number is added to the ticket. IT-06 §5.2
Judge (claude-opus-5-5)
Sources
A10Which AI tools am I allowed to use for work?correct
Answer
The only AI tool approved for company work is Microsoft 365 Copilot signed in with your Tallgrass account IT-01 §5.5. Do not paste Confidential or Restricted data into any other AI tool, chatbot, translation site or browser extension IT-01 §5.5.
Even in Copilot, ITAR and other export-controlled data must never be entered into any AI tool IT-01 §5.5IT-05 §5.4EXP-01 §5.2. Other Restricted data may only go into Copilot signed in with a Tallgrass account, and never into any other AI tool IT-05 §5.4.
Judge (claude-opus-5-5)
Sources
A11What do I need before doing welding or cutting work in the plant?correct
Answer
You need a Hot Work Permit, EHS-F-07, issued by the EHS Coordinator before work begins. This covers welding, cutting, brazing, grinding outside Bay 2, or any other spark-producing work EHS-04 §5.5.
Conditions that go with the permit:
- The permit is valid for one shift in one location EHS-04 §5.5.
- Combustibles within 35 feet must be removed or covered EHS-04 §5.5.
- A fire extinguisher must be at the work area EHS-04 §5.5.
- A fire watch must remain for 30 minutes after work ends EHS-04 §5.5.
- The completed permit is returned to EHS EHS-04 §5.5.
If you are a contractor, you must also complete the EHS-04 orientation before working unescorted anywhere at the plant EHS-04 §5.1.
Judge (claude-opus-5-5)
Sources
A12How long do we keep a personnel file after an employee leaves?correct
Answer
Personnel files are kept for 7 years after separation HR-230 §6. HR-200 confirms the same period for discipline records filed in the personnel file HR-200 §6.
If a record is under a legal hold, audit or open claim, it is kept until the HR Manager releases the hold, even after the 7 years have passed HR-230 §6.
Other record types, such as I-9 forms, medical files and training records, have their own retention periods HR-230 §6.
Judge (claude-opus-5-5)
Sources
A13When is a full FAI required instead of a partial one?correct
Answer
A full FAI covering every characteristic is required for:
- a new part number QP-8.6 §5.1;
- a part number made for a new customer QP-8.6 §5.1;
- a change of manufacturing location, including moving the work to a different Tallgrass building or to an outside supplier QP-8.6 §5.1;
- a lapse in production of the part number of 2 years or more, measured from the last shipment QP-8.6 §5.1.
Also, if there is doubt about whether a partial FAI is enough, a full FAI is performed QP-8.6 §5.2.
Judge (claude-opus-5-5)
Sources
A14When are merit increases effective, and what is the 2026 merit budget?correct
Answer
Merit increases are effective in March, on the first full pay period of March HR-220 §6.1. The 2026 merit budget is 3.5% of eligible base payroll HR-220 §6.2.
Judge (claude-opus-5-5)
Sources
A15How far ahead does a visit by a foreign national need to be approved?correct
Answer
A visit by a foreign national needs Empowered Official (EO) approval at least 5 business days before the visit EXP-01 §5.3. To request it, send the visitor's name, citizenship, employer and purpose to exportcompliance@tallgrasspc.com EXP-01 §5.3.
The same 5-business-day lead time applies to foreign-national contractors, whether or not they are escorted EHS-04 §5.3.
Judge (claude-opus-5-5)
Sources
M01I found a bad part at my machine. Walk me through what happens from tagging it to the MRB decision.correct
Answer
Here is the sequence from the documents:
- Stop, tag, segregate, notify. Stop, tag the part with the red HOLD tag (F-750-04), segregate it, and notify your lead and Quality WI-610 §4WI-610 §3.
- Hold, then move to the cage. Put the part on the red-lined hold shelf in your area until it can go to the cage WI-610 §5.2. It must be in the MRB cage in Bay 3 within 4 hours or by end of shift, whichever comes first QP-7.5 §5.2. Quality unlocks the cage and logs the material (date, job, quantity, NCR number) WI-610 §5.2. If the part is too large to move, tag it in place and rope it off with red barrier tape. Quality records the location in the cage log and on the NCR WI-610 §5.2.
- NCR. An NCR (F-750-01) is entered in the ERP during the same shift. It states the requirement, the actual condition, the quantity affected and the source QP-7.5 §5.2. Quality enters or confirms it WI-610 §4. At least one photo of the nonconforming condition must be attached QP-7.5 §5.2.
- MRB decision. The MRB dispositions the NCR within 5 working days of the NCR date. The options are use-as-is, rework, repair, return to supplier, or scrap QP-7.5 §5.4. Use-as-is and repair require the Engineering Manager on the MRB and customer approval on F-750-03 before the part ships QP-7.5 §5.4. The decision is recorded on the MRB Disposition Record (F-750-02) and signed by each member QP-7.5 §5.4. If the MRB cannot agree, the Quality Manager's decision stands QP-7.5 §5.4. An NCR not dispositioned within 5 working days is escalated to the Production Manager and reviewed at the weekly production meeting QP-7.5 §5.4.
Only Quality may remove the red tag. Material leaves the cage only after the disposition is recorded on F-750-02 WI-610 §5.4.
Judge (claude-opus-5-5)
Sources
M02We just learned we shipped nonconforming parts to a customer. What are Quality's obligations and deadlines?partial
Answer
- The Quality Manager must notify the customer within 24 hours of discovery, by phone or email. The notice must identify the part number, lot or serial numbers, quantity shipped and the nature of the nonconformance QP-7.5 §5.5.
- If the contract specifies a notification method, follow it QP-7.5 §5.5.
- Every escape requires a CAR per QP-8.5 QP-7.5 §5.5.
- Stock at Tallgrass from the same lot is placed on hold QP-7.5 §5.5.
- Anyone who learns that nonconforming product may have shipped must tell the Quality Manager immediately WI-610 §5.5.
The excerpts give no deadline for the CAR and don't cover the QP-8.5 requirements. Quality Assurance would need to confirm those.
Judge (claude-opus-5-5)
Sources
M03As a contract CMM programmer, can I release my own CMM program and sign off the FAI?correct
Answer
No to both.
- CMM program release: Contractors may write and prove out programs but may not approve release WI-215 §4. The Quality Engineer reviews the prove-out, signs F-215-01, sets the status to Released and sets the file to read-only WI-215 §5.4. A program is not Released until both prove-out and Quality Engineer approval are complete WI-215 §5.4.
- FAI sign-off: Contract CMM programmers may program and run the CMM but do not approve FAIs QP-8.6 §4. The Quality Engineer approves the FAI, or the Quality Manager may approve it in place of the Quality Engineer QP-8.6 §4.
Only Released programs may be used for FAI WI-215 §3, so your program needs Quality Engineer release before it can be used for an FAI.
Judge (claude-opus-5-5)
Sources
M04What are the retention periods for training records and for I-9 forms?correct
Answer
- Training records (F-620-01, F-620-02): Retained for the duration of employment plus 5 years HR-230 §6QP-6.2 §6QP-4.2 §6.2.
- I-9 forms: Retained 3 years after hire or 1 year after termination, whichever is later HR-230 §6.
Judge (claude-opus-5-5)
Sources
M05A supplier shipped bar stock with a mill cert that doesn't match the heat number. What happens at receiving and what does the supplier owe us?partial
Answer
At receiving: The Quality Technician checks that the heat or lot number on the cert matches the material marking SHP-01 §5.2. Material stays on receiving hold, and nothing on hold may be issued to production SHP-01 §5.2. For a mismatched cert, apply a red HOLD tag, quarantine per WI-610, and notify the Supplier Quality Engineer SHP-01 §5.2. The Supplier Quality Engineer handles supplier cert problems and SCARs (F-740-03) SHP-01 §4.
What the supplier owes us: Every shipment of raw material must include a mill certification showing the heat or lot number, specification and results SQM-01 §6.1. Material must be traceable to the producing mill and heat or lot number SQM-01 §6.2. Distributors must supply the original mill certification, not a re-typed copy SQM-01 §6.2. Shipments without the required documentation are held at Tallgrass receiving SQM-01 §6.1.
Not covered: The excerpts do not say what remedy the supplier must provide for a mismatched cert, such as a corrected cert, replacement or return. They also do not say whether a SCAR is required in this case. The Supplier Quality Engineer would be the person to ask.
Judge (claude-opus-5-5)
Sources
M06What are the rules for handling ITAR drawings, and can I paste one into Copilot to summarize it?correct
Answer
No. ITAR data must not be entered into any AI tool, including Microsoft 365 Copilot, so you can't paste a drawing in to summarize it EXP-01 §5.2IT-01 §5.5IT-05 §5.4.
Rules for handling ITAR drawings:
- Keep drawings and models only in the restricted "ITAR Programs" SharePoint site, which is limited to U.S. persons verified by HR EXP-01 §5.2.
- ITAR data is classified Restricted under IT-05 and carries the Restricted sensitivity label EXP-01 §5.2.
- Never save it to local drives, USB media, personal cloud storage or other SharePoint sites, and never send it to personal email EXP-01 §5.2.
- Printed drawings stay at your workstation during the shift and go back to the controlled drawing cabinet at end of shift EXP-01 §5.2.
- If a drawing or PO is marked export-controlled but the job isn't flagged "ITAR", stop and contact the Empowered Official (EO) EXP-01 §5.1.
- Annual ITAR awareness training is required, and must be completed before you work on any ITAR-flagged job EXP-01 §6.
If in doubt, contact the EO, Gail Thornton, at exportcompliance@tallgrasspc.com EXP-01 §1.
Judge (claude-opus-5-5)
Sources
M07What PPE do I need for deburring, and can I wear my cut gloves while deburring at the machine?correct
Answer
No, you can't wear your cut gloves while deburring at the machine. Gloves are not allowed when working at a machine with a rotating spindle, and WI-305 names deburring at the machine specifically WI-305 §5.5. EHS-02 also prohibits gloves of any kind near rotating spindles or drill presses EHS-02 §5.1. In Bay 5, cut gloves are allowed only with the spindle stopped EHS-02 §5.2.
PPE for deburring:
- Cut-resistant gloves (ANSI A4) for deburring and handling sharp parts, except at a machine with a rotating spindle as described above WI-305 §5.5EHS-02 §5.1.
- Safety glasses with side shields at all deburr benches WI-305 §5.5. EHS-02 requires safety glasses (ANSI Z87.1 with side shields) everywhere past the yellow line on the production floor EHS-02 §5.1.
- Safety-toe shoes in all production and shipping areas EHS-02 §5.1.
Judge (claude-opus-5-5)
Sources
M08We are terminating an employee today. When does IT disable the account, and what happens to their mailbox and OneDrive?correct
Answer
Account disable: For involuntary terminations, IT disables the account immediately, coordinated with HR so access ends before or during the separation meeting IT-04 §6.1. (For other departures the window is within 4 hours of HR's offboarding ticket IT-04 §6.1.)
Mailbox: IT converts it to a shared mailbox and retains it per IT-07. The manager may request access on IT-F-04 IT-04 §6.2.
OneDrive: The manager gets access to the departed user's OneDrive for 30 days to move business files to SharePoint IT-04 §6.2.
The excerpts don't state the retention period for the mailbox, since that is set in IT-07, which isn't included here.
Judge (claude-opus-5-5)
Sources
U01What's on the cafeteria menu on Fridays?correct
Answer (refused)
I can't find that in the documents available to you. For cafeteria questions, you could ask HR or Facilities.
Judge (claude-opus-5-5)
Sources
U02What is the torque spec for the M6 fasteners on the 48210-3 assembly?correct
Answer (refused)
I can't find that in the documents available to you. The excerpts don't include a torque spec for the M6 fasteners on the 48210-3 assembly, so check with Engineering or Quality for the drawing or work instruction that covers it.
The only torque-related content is that torque wrenches are calibrated every 6 months using the in-house torque analyzer QP-7.6 §5.2.
Judge (claude-opus-5-5)
Sources
U03Does Tallgrass offer tuition reimbursement?correct
Answer (refused)
I can't find that in the documents available to you. HR may be able to help with tuition reimbursement.
Judge (claude-opus-5-5)
Sources
U04What is the guest Wi-Fi password?correct
Answer (refused)
I can't find that in the documents available to you. The excerpts only say that personal phones and laptops connect to the guest Wi-Fi network, not the corporate one IT-01 §5.6. For the password itself, ask IT.
Judge (claude-opus-5-5)
Sources
U05What is the company 401(k) match?correct
Answer (refused)
I can't find that in the documents available to you. HR would be the best department to ask about the 401(k) match.
Judge (claude-opus-5-5)
Sources
U06Which customer was involved in our most recent escape?correct
Answer (refused)
I can't find that in the documents available to you. The excerpts describe how escapes are handled (the Quality Manager notifies the customer within 24 hours QP-7.5 §5.5), but they don't name any specific escape or customer. The Quality department would be the place to ask.
Judge (claude-opus-5-5)
Sources
P01What is the pay range for a CMM Programmer?correct
Answer (refused)
I can't find that in the documents available to you. The excerpts only describe the CMM programmer's role and responsibilities, not pay, so HR would be the department to ask.
Judge (claude-opus-5-5)
Sources
P02How many PTO hours do employees get per year?correct
Answer (refused)
I can't find that in the documents available to you. HR would be the best department to ask.
Judge (claude-opus-5-5)
Sources
P03What are the steps of progressive discipline, and how long does a warning stay active?correct
Answer (refused)
I can't find that in the documents available to you. The only mention of progressive discipline is that HR-100 refers a single no-call/no-show to HR-200 Progressive Discipline HR-100 §5.3, but HR-200 itself isn't among the excerpts. You could ask HR for the steps and how long a warning stays active.
Judge (claude-opus-5-5)
Sources
P04What are the RPO and RTO for the ERP system?correct
Answer (refused)
I can't find that in the documents available to you. The excerpts mention the ERP only in passing (job flags, CAR/NCR entry, access roles) and say nothing about RPO or RTO. IT would be the best department to ask.
Judge (claude-opus-5-5)
Sources
P05When does an export shipment need an EEI filing in AESDirect?correct
Answer (refused)
I can't find that in the documents available to you. The excerpts only mention that EEI filings are kept as export records for 5 years from the date of export EXP-01 §7, and that every export shipment needs EO review and an SHP-02 EXP-01 §5.5. They don't say when an EEI filing in AESDirect is required. Export Control (the Empowered Official) or Shipping would be the place to ask.
Judge (claude-opus-5-5)
Sources
S01How many working days does the MRB have to disposition an NCR?correct
Answer
The MRB must disposition every NCR within 5 working days of the NCR date QP-7.5 §5.4. A working day is Monday through Friday, excluding company holidays QP-7.5 §3. An NCR not dispositioned within that time is escalated to the Production Manager and reviewed at the weekly production meeting QP-7.5 §5.4.
Judge (claude-opus-5-5)
Sources
S02How often do I have to change my password, and how long does it have to be?correct
Answer
You don't have to change your password on a schedule, since there is no scheduled password expiry. Change it only if you suspect it's been compromised, for example after entering it on a phishing page, seeing an unexpected MFA prompt, or being asked to by IT IT-02 §5.2.
Your password must be at least 14 characters, and a passphrase of several words is recommended IT-02 §5.1.
Judge (claude-opus-5-5)
Sources
S03Can I use text message codes for MFA?correct
Answer
No. SMS text messages and voice calls are not allowed MFA methods IT-02 §5.3. The approved method is Microsoft Authenticator with number matching IT-02 §5.3. SMS was removed as an MFA method in revision 4, effective 2026-01-12 IT-02 §7.
Judge (claude-opus-5-5)
Sources
S04Where do I save the CMM report for an FAI, and who signs Form 3?correct
Answer
Where to save: Save the CMM report and the completed F-860-01 as PDF to the SharePoint Quality site > "FAI Records" library > Customer / Part number / Drawing rev WI-210 §5.5. Do not save reports to local drives or network shares WI-210 §5.5.
Who signs Form 3: Only Tallgrass Quality may sign AS9102 Form 3, meaning the Quality Engineer or the Quality Manager QP-8.6 §5.4WI-210 §4. The Quality Engineer signs after a second-person review WI-210 §5.6. Contractors, operators and production personnel may not sign it QP-8.6 §5.4.
Judge (claude-opus-5-5)
Sources
S05How quickly does Quality have to notify the customer about an escape?correct
Answer
The Quality Manager must notify the customer within 24 hours of discovery, by phone or email. The notice identifies the part number, lot or serial numbers, quantity shipped and the nature of the nonconformance QP-7.5 §5.5. If the contract requires a customer-specified notification method, that method is followed QP-7.5 §5.5.
Judge (claude-opus-5-5)
Sources
S06What changed in the latest revision of QP-7.5?correct
Answer
Revision D of QP-7.5 (effective 2026-03-02) made these changes: