Candidate-instrument pilot protocol

Status

Draft. Waiting for the cognitive interviews.

Estimate item performance and measurement structure; do not produce individual decisions or population prevalence claims.

Set after feasibility review and before data collection; justify separately for each instrument and planned model.

Document channels, eligibility, duplicates, completion, exclusions, incentives, and achieved sample gaps.

Cognitive-interview participants are excluded from the primary pilot analysis unless preregistered otherwise.

• career stage
• work arrangement
• business model
• jurisdiction
• disability/care context
• resource access
• platform dependence

Before collection, the public record must freeze:

• frozen candidate version
• construct-item map
• primary and secondary analyses
• exclusions
• missing/not-applicable handling
• fairness analyses
• decision thresholds
• deviations process

Store public registration URL, timestamp, protocol version, and analysis-plan hash.

• Item response distributions, floor/ceiling effects, missingness, and not-applicable patterns.
• Inter-item associations and redundancy, interpreted against construct definitions.
• Dimensionality using preregistered models appropriate to ordinal items and sample size.
• Reliability with uncertainty; no alpha-only validation claim.
• Differential item functioning or structured fairness review where sample size permits.
• Sensitivity analyses for missing data, exclusions, and plausible alternative specifications.

A versioned R analysis script accompanies the protocol and preserves missing, do-not-know, and structurally not-applicable states.

Every pilot report must address each limitation even when the result is “none observed.”

• Recruitment channels and resulting coverage
• Eligibility and exclusion counts
• Achieved sampling dimensions and unfilled cells
• Interview completion, withdrawal, and missingness
• Items not administered to the required number of participants
• Unresolved response-process failures
• Accessibility and structural-fairness limitations
• Researcher interpretation and coding limitations
• Claims the evidence does not support

The default decision is blocked. The record must contain:

• instrument and frozen candidate version
• dated gate snapshot
• preregistration record and analysis-plan hash
• sample and exclusions
• item-performance findings
• dimensionality and reliability findings with uncertainty
• fairness review and unresolved gaps
• approved claim language
• migration and crosswalk approval
• named approver and decision date

Automated activation is prohibited.

Every requirement must be recorded as met. Waivers do not unlock scoring.

1. First round of cognitive interviews complete
2. Second round of cognitive interviews complete
3. Pilot preregistered
4. Pilot data collection complete
5. Item performance reviewed
6. Dimensionality reviewed
7. Reliability reviewed
8. Fairness reviewed
9. Claim language approved
10. Crosswalk approved
11. Methodology record published
12. Dated activation decision approved

Pilot findings inform a dated decision; passing automated checks never activates scoring by itself.