# 选择效应与多重检验 保留全部尝试，证明 Bonferroni，区分 FWER/FDR 并按严格规则执行 BH。 Entry: zh-qt22 | Node: QT22 | Language: zh | Editorial revision: 2026-09-21 ## Teaching instructions 请以具有微积分、线性代数与基本概率背景的高年级本科至研究生为对象，围绕“选择效应与多重检验”完成一次学习。先实际取得并读完 required_readings 的完整指定单元，记录实际版本/定位/假设/支持范围；目录、摘要或入口不抵扣阅读。可先读取本篇所有必读以备课，但不要把它们全变成读者额外作业。来源打不开时先尝试同版作者可读入口；仍缺承重单元应说明具体缺口，不伪称已读，也不要求无关新批准。先给全部候选与开发/保留独立关系；让读者独立证明Bonferroni并计算严格BH。要求解释M10模拟超过理论界、混合FDR不等FWER、人工等号例、研究者已看数据不可通过折叠界面恢复独立。 先让读者尝试，再根据本文完整解析反馈；引用范围内解释，不把模拟当市场事实、不把最优目标当收益保证。只在读者选择分支时启用 optional_readings。完成后用一项新输入迁移检验，明确其能独立完成什么。 Before substantive teaching, actually retrieve every required reading unit for the selected scope. Read its complete designated section, including necessary assumptions, tables and footnotes. A working URL or an editorial access date is not a runtime reading receipt. Record the actual version, location, scope and what it supports. If unavailable, use a previously verified equivalent source; if the required unit remains unavailable, identify that gap rather than teach it from memory. Start runtime_reading_log empty. Once reading is complete, use a substantive diagnostic or follow the reader's request for direct explanation. Advance one complete reasoning task at a time; skip mastered basics. Distinguish original facts, supplied teaching assumptions and inference. ## Required readings and runtime protocol ```json { "export_mode": "public", "required_readings": [ { "source_id": "QTDE-ISLP", "access": { "kind": "pdf_full_text", "uri": "https://drive.google.com/uc?export=download&id=1ajFkHO6zjrdGNqhqW1jKBZdiNGh_8YQ1", "author_gateway": "https://hastie.su.domains/ISLP/ISLP_website.pdf.download.html", "verified_access_at": "2026-09-21" }, "required_unit": { "locator": "§13.1.1–13.1.2与§13.2 pp.558–564；§13.3.1及Bonferroni子节pp.565–568；§13.4.1–13.4.2 pp.573–577", "scope": "p值、FWER、Bonferroni、FDP/FDR、Algorithm13.2", "purpose": "定义与证明边界；采用严格<；一般BH控制证明不在本篇" }, "version": "First Printing July 5, 2023", "supports": "p值、FWER、Bonferroni、FDP/FDR、Algorithm13.2", "title": "An Introduction to Statistical Learning with Applications in Python", "authors": [ "James", "Witten", "Hastie", "Tibshirani", "Taylor" ] } ], "optional_readings": [], "runtime_reading_log": [], "supplied_inputs": { "input_identity": "QT-DE-adopted-inputs-20260921-v1", "config": { "content_version": "2026-09-21-QT-DE-review-v2", "input_version": "QT-DE-adopted-inputs-20260921-v1", "dataset": { "file": "https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/data/BusEq-value-weighted-monthly-199001-202512.csv", "origin_input_version": "QT-C-inputs-20260921-v1", "transport_sha256": "72dd0f35d9437d268971de8553827a5c7a4f85ebcf9712c0c1b85ea2ec7cfa74", "source_url": "https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/ftp/30_Industry_Portfolios_CSV.zip", "source_details_url": "https://mba.tuck.dartmouth.edu/pages/Faculty/ken.french/Data_Library/det_30_ind_port.html", "regime_url": "https://mba.tuck.dartmouth.edu/pages/faculty/ken.french/data_library.html", "vintage": "202607 CRSP database; current CIZ reconstruction", "download_date": "2026-09-21", "block": "Average Value Weighted Returns -- Monthly", "column": "BusEq", "sample": [ "199001", "202512" ], "n": 432, "source_lines": [ 775, 1206 ], "missing_removed": 0, "missing_codes": [ -99.99, -999 ], "raw_unit": "percent", "internal_unit": "monthly decimal return", "display_errors": "百分点/月", "annualized": false, "identity": "单一当前重建快照；不是FIZ/CIZ时间拼接，不是逐月当时可见版本。" }, "forecast": { "experiment_id": "EXP-QT-D-FORECAST-01", "p_grid": [ 1, 3, 12 ], "lambda_grid": [ 0.0, 0.1, 1.0, 10.0 ], "default_candidate": "p1-l10", "target": "y_t=r_t", "features": "(r_(t-1),...,r_(t-p))", "warmup": [ "199001", "199012" ], "initial_train": [ "199101", "200912", 228 ], "validation": [ "201001", "201912", 120 ], "historical_evaluation": [ "202001", "202512", 72 ], "train_start_index": 12, "objective": "RSS/n + lambda*||beta||^2", "matrix_penalty": "n*lambda*I", "standardization_ddof": 0, "intercept_penalized": false, "selection": "minimum validation SSE including zero and expanding-mean baselines", "tie_rule": "regression grid: smaller p then smaller lambda", "frozen_hyperparameters_after_validation": true, "refit": "每原点用新成熟标签扩展重拟合；不再次选择p或lambda", "availability": "教学月末可知收益假设；French历史真实发布和接收时点未重建", "future_scaling_control": { "p": 12, "lambdas": [ 0.0, 0.1 ], "scope": "validation only", "identity": "故意信息越界对照" }, "default_clock_month": "201001", "default_clock_lambda": 0.1 }, "selection": { "experiment_id": "EXP-QT-D-SELECT-01", "B": 5000, "m_grid": [ 1, 10, 100, 500 ], "default_m": 100, "seed_all_null": 2201, "seed_mixed": 2202, "rng": "numpy.random.Generator(PCG64)", "n_development": 120, "n_holdout": 120, "sigma": 0.05, "alpha": 0.05, "hypothesis": "事前单侧H0:mu=0 versus H1:mu>0; sigma known", "pvalue": "0.5*erfc(Z/sqrt(2))", "draw_order": "先取(5000,500)开发Z矩阵，再取独立保留Z矩阵；M为共同500列前缀。mixed另重置2202。", "bh_adopted_comparison": "strict <", "source_pilot_comparison": "<=", "empty_BH": "k=0, no rejections, FDP=0", "mixed": { "m": 100, "signals": 10, "signal_mu": 0.01, "null_mu": 0.0 }, "identity": "直接模拟正态样本均值的Z分布；不含逐月路径，不是真实策略面板", "equal_example": { "p": [ 0.025, 0.06 ], "q": 0.05 }, "manual_example": { "p": [ 0.006, 0.012, 0.601, 0.756, 0.918 ], "q": 0.05 } }, "cost": { "experiment_id": "EXP-QT23-COST-01", "rf": 0, "continuous_default": { "mu": 0.015, "rho": 0, "sigma": 0.1, "gamma": 2, "w0": 0.5, "kappa": 0.002, "cap": 0.8, "cash_min": 0.2 }, "integer_default": { "mu": 0.002, "rho": 0, "sigma": 0.02, "gamma": 10, "wealth": 10000, "initial_shares": 0, "mid": 100, "ask": 100.05, "cap": 80, "cash_min": 2000, "mode": "spread_and_commission" }, "fee": { "id": "QTDE-IBKR", "url": "https://www.interactivebrokers.com/en/pricing/commissions-stocks.php?re=amer", "accessed_at": "2026-09-21", "scope": "IBKR Pro US Fixed ordinary whole-share single-order commission component only", "formula": "q=0:0; otherwise min(0.01*execution_price*abs(q),max(1,0.005*abs(q)))" }, "identity": "收益、波动、风险系数、报价、价差、容量为教学设定；佣金是有日期的实际规则组件", "end_value": "期末中价标记；没有退出委托，非往返净利", "excluded_costs": [ "监管/交易所/清算等其他收费", "税", "市场冲击", "持仓融资费", "退出交易费", "零股及改单最低费规则" ] }, "clocks": { "view_id": "EXP-QT-D-CLOCK-01", "shared_experiment": "EXP-QT-D-FORECAST-01", "BEA": [ { "period": "2020Q2", "vintage": "advance", "release_at": "2020-07-30T08:30:00-04:00", "received_at": null, "value_percent_SAAR": -32.9 }, { "period": "2020Q2", "vintage": "second", "release_at": "2020-08-27T08:30:00-04:00", "received_at": null, "value_percent_SAAR": -31.7 } ], "h3_task": { "formula": "prod(1+r_(s+j),j=1..3)-1", "cutoff": "202003", "latest_mature_origin": "201912", "performance_run": false } }, "scale_examples": { "raw_x": [ -1, 1 ], "y": [ -1, 1 ], "lambda": 1.0, "default_feature_scale": 10.0, "response_scale": 100.0 }, "source_identities": { "d_pilot": { "file": "qt-d-experiment-pilot.py", "sha256": "b8a23b01294d543b3b74c6a153168529db74ab7c84837ea3724a55f5071fd3fb", "read_scope": "1–193" }, "ef_pilot": { "file": "qt-ef-calculations.py", "sha256": "28eff60cf87400805af145fabd22d4fe17e2c25bac3708cd9e71506d939acb20", "adopted_scope": "commission, continuous_case, integer_case, decision_calculation" }, "approval": { "file": "qt-de-draft-approved.md", "read_scope": "1–7", "sha256": "80e92333d057772de9463d786a9be435c6dc1208744bf828d5eb2a4d2a5e9a48" }, "plan_review": { "file": "qt-def-plan-review.md", "read_scope": "1–20", "sha256": "56fddfa847ca0f83ec4822652c506f847e9da4e45f7223c039b31428f3aedfd4" } } }, "static_equivalent_html": "

已知σ=.05、n=120、单侧检验；直接生成Z。PCG64先2201开发5000×500再独立保留，2202真假混合；M是共同前缀。BH正文/实现严格<，无合格序号则R=0。

M未校正模拟Bonf模拟Bonf精确Bonf MCSE开发冠军pp/月独立保留pp/月
10.04740.04740.050000000.003005-0.000216-0.001749
100.40800.05700.048889870.0032790.710968-0.006699
1000.99340.04980.048782470.0030771.1461830.009726
5001.00000.04580.048772950.0029571.384628-0.007626
FWER=P(V≥1)；Bonferroni ≤Σ_{j∈I0}P(pj≤α/M)≤α。FDR=E[V/max(R,1)]，不是E[V]/E[R]。
混合真值：100项中10项μ=.01实际输出
mean_fdp_fdr_estimate0.04527148
mc_se_fdr_estimate0.00178395
probability_any_false_rejection0.14880000
mean_rejections2.44380000
mean_true_discoveries2.26920000

q=.05；混合FDR约.04527而FWER=.1488。它们是不同对象。人工p=(.025,.06)、q=.05：严格规则R=0，非严格R=1。模拟使用完整精度；人工等号例单独比较两种规则。约束条件由理论给出，不由这轮频率证明。

", "files": { "frozen_csv": "https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/data/BusEq-value-weighted-monthly-199001-202512.csv", "complete_outputs": "https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/data/results.json", "reproduction": "https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/compute/reproduce.py", "interactive_engine": "https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/engine.js", "selection_arrays": "https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/data/selection-arrays.npz", "shared_inputs": "https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/shared_inputs.json", "sources": "https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/sources.json" }, "algorithm_identity": "forecast/selection使用真实计算的冻结网格及PCG64结果；成本和人工边界可重算。完整reader在body_markdown中仅一份。", "actual_selection_summary": { "config": { "experiment_id": "EXP-QT-D-SELECT-01", "B": 5000, "m_grid": [ 1, 10, 100, 500 ], "default_m": 100, "seed_all_null": 2201, "seed_mixed": 2202, "rng": "numpy.random.Generator(PCG64)", "n_development": 120, "n_holdout": 120, "sigma": 0.05, "alpha": 0.05, "hypothesis": "事前单侧H0:mu=0 versus H1:mu>0; sigma known", "pvalue": "0.5*erfc(Z/sqrt(2))", "draw_order": "先取(5000,500)开发Z矩阵，再取独立保留Z矩阵；M为共同500列前缀。mixed另重置2202。", "bh_adopted_comparison": "strict <", "source_pilot_comparison": "<=", "empty_BH": "k=0, no rejections, FDP=0", "mixed": { "m": 100, "signals": 10, "signal_mu": 0.01, "null_mu": 0.0 }, "identity": "直接模拟正态样本均值的Z分布；不含逐月路径，不是真实策略面板", "equal_example": { "p": [ 0.025, 0.06 ], "q": 0.05 }, "manual_example": { "p": [ 0.006, 0.012, 0.601, 0.756, 0.918 ], "q": 0.05 } }, "all_null": [ { "m": 1, "naive_fwer_estimate": 0.0474, "naive_fwer_exact": 0.050000000000000044, "naive_fwer_mcse": 0.0030054042142277936, "bonferroni_fwer_estimate": 0.0474, "bonferroni_fwer_exact": 0.050000000000000044, "bonferroni_fwer_mcse": 0.0030054042142277936, "selected_development_mean": -2.1623449612290013e-06, "selected_holdout_mean": -1.7490274376658475e-05, "selected_holdout_mcse": 6.481222509088888e-05, "bh_strict": { "mean_fdp_fdr_estimate": 0.0474, "mc_se_fdr_estimate": 0.0030054042142277936, "probability_any_false_rejection": 0.0474, "mean_rejections": 0.0474, "mean_true_discoveries": 0.0 }, "bh_inclusive_original": { "mean_fdp_fdr_estimate": 0.0474, "mc_se_fdr_estimate": 0.0030054042142277936, "probability_any_false_rejection": 0.0474, "mean_rejections": 0.0474, "mean_true_discoveries": 0.0 }, "first_replica": { "p_values": [ 0.8534229641244158 ], "development_means": [ -0.0047981755611324756 ], "holdout_means": [ 0.007658912384705444 ], "winner_zero_based": 0 } }, { "m": 10, "naive_fwer_estimate": 0.408, "naive_fwer_exact": 0.4012630607616213, "naive_fwer_mcse": 0.006951033259356089, "bonferroni_fwer_estimate": 0.057, "bonferroni_fwer_exact": 0.048889869534228136, "bonferroni_fwer_mcse": 0.003279077685888825, "selected_development_mean": 0.007109678475910141, "selected_holdout_mean": -6.6989271930215e-05, "selected_holdout_mcse": 6.365529307790786e-05, "bh_strict": { "mean_fdp_fdr_estimate": 0.0578, "mc_se_fdr_estimate": 0.0033006076413533963, "probability_any_false_rejection": 0.0578, "mean_rejections": 0.0634, "mean_true_discoveries": 0.0 }, "bh_inclusive_original": { "mean_fdp_fdr_estimate": 0.0578, "mc_se_fdr_estimate": 0.0033006076413533963, "probability_any_false_rejection": 0.0578, "mean_rejections": 0.0634, "mean_true_discoveries": 0.0 }, "first_replica": { "p_values": [ 0.8534229641244158, 0.5778490160856251, 0.9854505456797473, 0.035868607290115956, 0.14613659518590374, 0.602942573910611, 0.3346558994306192, 0.7023767914227861, 0.5991395721062277, 0.8628232179394502 ], "development_means": [ -0.0047981755611324756, -0.0008964108416191423, -0.00996007898675733, 0.008219408627936102, 0.0048069407294246734, -0.0011911643085693565, 0.0019494026179431084, -0.002424807684851433, -0.0011462035588896365, -0.00498925792670572 ], "holdout_means": [ 0.007658912384705444, -0.0007174443004839976, 0.0022785446916859953, 0.0017711203965957904, 0.005561338668448594, 0.0036157631884963745, -0.0015705522935678213, -0.005820472142382774, 0.006766608177975697, 0.003749632355527969 ], "winner_zero_based": 3 } }, { "m": 100, "naive_fwer_estimate": 0.9934, "naive_fwer_exact": 0.994079470779666, "naive_fwer_mcse": 0.0011452293700661073, "bonferroni_fwer_estimate": 0.0498, "bonferroni_fwer_exact": 0.04878246975766576, "bonferroni_fwer_mcse": 0.0030766678691460034, "selected_development_mean": 0.01146182624537545, "selected_holdout_mean": 9.726230135979109e-05, "selected_holdout_mcse": 6.432114162014367e-05, "bh_strict": { "mean_fdp_fdr_estimate": 0.0518, "mc_se_fdr_estimate": 0.0031345361799495442, "probability_any_false_rejection": 0.0518, "mean_rejections": 0.0598, "mean_true_discoveries": 0.0 }, "bh_inclusive_original": { "mean_fdp_fdr_estimate": 0.0518, "mc_se_fdr_estimate": 0.0031345361799495442, "probability_any_false_rejection": 0.0518, "mean_rejections": 0.0598, "mean_true_discoveries": 0.0 }, "first_replica": { "p_values": [ 0.8534229641244158, 0.5778490160856251, 0.9854505456797473, 0.035868607290115956, 0.14613659518590374, 0.602942573910611, 0.3346558994306192, 0.7023767914227861, 0.5991395721062277, 0.8628232179394502, 0.3072786163486795, 0.6446638134127374, 0.8905408742223062, 0.7814423709540815, 0.15233734280723954, 0.9414434782647944, 0.1515293405276159, 0.8559904247744021, 0.549195403866199, 0.6300978737951761, 0.876368111076663, 0.5099685373915479, 0.13607060349302227, 0.3570967955420285, 0.010184575122785604, 0.45336974765186416, 0.33157330585329603, 0.38519442076067906, 0.36259461367265133, 0.6468485291852406, 0.9724755760050876, 0.3190162761612558, 0.3688018604797179, 0.16252414150420214, 0.881957800853761, 0.58567847219524, 0.9238707410840241, 0.4508122773293958, 0.9347832868741658, 0.4226742109294276, 0.7690322790591826, 0.25516183889560085, 0.3900618061798127, 0.42723134491087766, 0.12695444444396306, 0.010792527059349575, 0.8343375255226564, 0.48368347251122373, 0.4012711110504669, 0.9646153292882157, 0.40847452798791956, 0.4604913584312707, 0.03868018507366879, 0.12183876004984262, 0.22933730707065417, 0.8695174584502872, 0.522240132088451, 0.9907317132322845, 0.42177850685294715, 0.10299501922305858, 0.6344321965027212, 0.2601203611372047, 0.597491790532756, 0.8976900747933818, 0.16724088427720096, 0.43859100866607736, 0.6312658952249004, 0.4826617822238235, 0.839574953842144, 0.8017363719982666, 0.4721948352697341, 0.6676501515363849, 0.9168177212012609, 0.893489262957827, 0.7140993815817195, 0.4490479749631837, 0.8526064517392637, 0.6202444221868904, 0.3083930137954006, 0.2793049040758189, 0.4300927040491767, 0.6808734356949653, 0.46432971765003084, 0.07471568061580637, 0.4220200257206621, 0.6098580999882854, 0.4371363615922268, 0.9185855305368961, 0.6980193037475122, 0.46930304811276885, 0.9535543405951088, 0.5645291753143131, 0.7802015059086396, 0.7557856716117084, 0.8698136111359869, 0.7919605716283196, 0.27473036656775607, 0.3510909356917084, 0.4801554673996442, 0.2941946525866236 ], "development_means": [ -0.0047981755611324756, -0.0008964108416191423, -0.00996007898675733, 0.008219408627936102, 0.0048069407294246734, -0.0011911643085693565, 0.0019494026179431084, -0.002424807684851433, -0.0011462035588896365, -0.00498925792670572, 0.0022985132782164415, -0.0016931620625910693, -0.005611461677177663, -0.0035468401964051, 0.004685128663308346, -0.007152375914058517, 0.004700812027357641, -0.0048495222692586945, -0.0005642856648236809, -0.0015158795907090443, -0.005281054208458644, -0.00011406330698361012, 0.005012322931284437, 0.0016716027885953484, 0.010586918703139535, 0.0005347238978518669, 0.0019881093454438476, 0.0013321813580639112, 0.0016045169761117995, -0.001719967355290149, -0.008756670939355676, 0.002147307010489967, 0.001529188082656663, 0.004491949115944484, -0.005407987434385508, -0.0009879190741889014, -0.006534327111463503, 0.0005641971091248882, -0.006903106783646581, 0.0008903087591911849, -0.0033578296080499096, 0.003004868837199822, 0.0012741758858595298, 0.0008372276393875034, 0.005207501284307683, 0.010487022098548206, -0.0044340358218156895, 0.000186731757764035, 0.0011413550074582375, -0.008247582115450165, 0.0010565148904382261, 0.000452765343292763, 0.00806161572761358, 0.005321325519831879, 0.0033823285147733377, -0.005130850478963029, -0.00025458441589719014, -0.010747785757690417, 0.0009007557936905378, 0.005772397426017361, -0.0015683821508893942, 0.0029347631119845227, -0.0011267575009895306, -0.005789878756487811, 0.004405176769052742, 0.0007053866980756999, -0.0015300080597557173, 0.00019843147232562937, -0.004531091810655898, -0.0038698410261924087, 0.00031838079023234256, -0.001978345282170842, -0.00631697585896543, -0.005683991108723206, -0.0025806893904984445, 0.0005845432134588905, -0.004781972861396629, -0.0013972529835981607, 0.0022840512102963663, 0.0026697259924966543, 0.0008039569947192937, -0.0021458976353289586, 0.0004086540085749986, 0.006579713120351349, 0.0008979383786702229, -0.001273223138311876, 0.0007222342968647964, -0.0063701048969554835, -0.002367586834289378, 0.0003515554209254143, -0.0076696644901966884, -0.0007415365084817654, -0.0035276707640320604, -0.0031622315530179844, -0.005137228999981466, -0.003711928617211329, 0.002732078461526574, 0.0017453034765488496, 0.0002271377962338248, 0.0024700991426852043 ], "holdout_means": [ 0.007658912384705444, -0.0007174443004839976, 0.0022785446916859953, 0.0017711203965957904, 0.005561338668448594, 0.0036157631884963745, -0.0015705522935678213, -0.005820472142382774, 0.006766608177975697, 0.003749632355527969, -0.001672794024594719, -0.004599427590402427, -0.003992286481063795, -0.001056893338078012, -0.002944929268282168, -0.0005602826915198105, -0.003509401986775069, 0.0016284786117830193, 0.0024925254680253843, 0.010222345991360812, -0.004851755596328836, 0.0005352026587780661, 0.004959742429063509, -0.005291006717233991, 0.0016349888234482514, 0.009285597602908363, -0.010707768637042767, -0.0022688749604447274, -0.004736682223363454, -0.0030572348717810348, -0.002438936642894018, -0.00160762467651023, -0.003407700021871669, 0.0057013588939509125, -0.0017420882349641735, 0.0027822173073378965, -0.006234732726281721, -0.006828187936153975, -0.0029366314394212484, -0.005375359675385418, -0.001074404676391136, -0.0024355159170225986, -0.002710899110562593, -0.00036958322009809653, 0.007155454728852111, 0.00500079608721685, 0.0006653906263716094, -0.004529246055835025, 0.004111502767292098, -0.000830810549292612, 0.003039832457664698, -0.004789208086900036, 0.0032141738744052117, 6.098473965308105e-05, -0.005093483848699583, -0.0037788068634633525, 0.0024385585543757454, -0.000980078096339036, -0.0013275150390482913, -0.0004971434219030999, 0.0020436665629693326, -0.006163119621906277, -0.006255331712484443, 0.01578227615776235, -0.002089250926599542, -0.006652935944108395, 0.010945773949246602, 0.013263956558886512, -0.005436427447832911, 0.0015590119978978552, 0.00462193152418235, -0.0050539163113293385, 0.0030714799550927707, -0.005636878881185781, -0.0036415524189248745, -0.0031680016338319857, -0.0036608432295858165, -0.0052072072036564295, -0.00012246782385121272, 0.006191927315243657, 0.00558575732054903, 0.005972754792654958, -0.0006953859196873227, -0.00124355868066654, 0.002366672649197871, -0.003200766361824028, 0.0003822828976154978, -0.0007704415868099267, -0.00022073437465164854, -0.0028956177872436245, -0.0003762342831913942, 0.004689022901160395, 0.005606899266753244, 0.006393367508378689, -0.00599284233780417, 0.002532865519888206, 0.0008458603691643703, -0.0018775580338739248, 0.005407625706551953, -0.0006184602782087109 ], "winner_zero_based": 24 } }, { "m": 500, "naive_fwer_estimate": 1.0, "naive_fwer_exact": 0.9999999999927255, "naive_fwer_mcse": 0.0, "bonferroni_fwer_estimate": 0.0458, "bonferroni_fwer_exact": 0.048772953728419166, "bonferroni_fwer_mcse": 0.0029567246141784694, "selected_development_mean": 0.013846280153571394, "selected_holdout_mean": -7.626474858791914e-05, "selected_holdout_mcse": 6.479093999990008e-05, "bh_strict": { "mean_fdp_fdr_estimate": 0.0466, "mc_se_fdr_estimate": 0.002981185249711701, "probability_any_false_rejection": 0.0466, "mean_rejections": 0.0514, "mean_true_discoveries": 0.0 }, "bh_inclusive_original": { "mean_fdp_fdr_estimate": 0.0466, "mc_se_fdr_estimate": 0.002981185249711701, "probability_any_false_rejection": 0.0466, "mean_rejections": 0.0514, "mean_true_discoveries": 0.0 }, "first_replica": { "p_values": [ 0.8534229641244158, 0.5778490160856251, 0.9854505456797473, 0.035868607290115956, 0.14613659518590374, 0.602942573910611, 0.3346558994306192, 0.7023767914227861, 0.5991395721062277, 0.8628232179394502, 0.3072786163486795, 0.6446638134127374, 0.8905408742223062, 0.7814423709540815, 0.15233734280723954, 0.9414434782647944, 0.1515293405276159, 0.8559904247744021, 0.549195403866199, 0.6300978737951761, 0.876368111076663, 0.5099685373915479, 0.13607060349302227, 0.3570967955420285, 0.010184575122785604, 0.45336974765186416, 0.33157330585329603, 0.38519442076067906, 0.36259461367265133, 0.6468485291852406, 0.9724755760050876, 0.3190162761612558, 0.3688018604797179, 0.16252414150420214, 0.881957800853761, 0.58567847219524, 0.9238707410840241, 0.4508122773293958, 0.9347832868741658, 0.4226742109294276, 0.7690322790591826, 0.25516183889560085, 0.3900618061798127, 0.42723134491087766, 0.12695444444396306, 0.010792527059349575, 0.8343375255226564, 0.48368347251122373, 0.4012711110504669, 0.9646153292882157, 0.40847452798791956, 0.4604913584312707, 0.03868018507366879, 0.12183876004984262, 0.22933730707065417, 0.8695174584502872, 0.522240132088451, 0.9907317132322845, 0.42177850685294715, 0.10299501922305858, 0.6344321965027212, 0.2601203611372047, 0.597491790532756, 0.8976900747933818, 0.16724088427720096, 0.43859100866607736, 0.6312658952249004, 0.4826617822238235, 0.839574953842144, 0.8017363719982666, 0.4721948352697341, 0.6676501515363849, 0.9168177212012609, 0.893489262957827, 0.7140993815817195, 0.4490479749631837, 0.8526064517392637, 0.6202444221868904, 0.3083930137954006, 0.2793049040758189, 0.4300927040491767, 0.6808734356949653, 0.46432971765003084, 0.07471568061580637, 0.4220200257206621, 0.6098580999882854, 0.4371363615922268, 0.9185855305368961, 0.6980193037475122, 0.46930304811276885, 0.9535543405951088, 0.5645291753143131, 0.7802015059086396, 0.7557856716117084, 0.8698136111359869, 0.7919605716283196, 0.27473036656775607, 0.3510909356917084, 0.4801554673996442, 0.2941946525866236, 0.6977415309210189, 0.5024140403429673, 0.19420428627456748, 0.7725037804513217, 0.20883745132079287, 0.7085055826873295, 0.9844485661010738, 0.7633900778405074, 0.32530289216080827, 0.4181302502659828, 0.9621891578383085, 0.7256384871040971, 0.30321815984161216, 0.9532040595687885, 0.46959865216492386, 0.5984679169100675, 0.2670394822307416, 0.21984726803023555, 0.9131701358401505, 0.39747796876560304, 0.08899855197796458, 0.8459472114436662, 0.1952277116084028, 0.9912411712614922, 0.2699942314803315, 0.14031585529462254, 0.01326364648923651, 0.6294687866453097, 0.6005935693267885, 0.4055317563840517, 0.7905702575000998, 0.6846126404107178, 0.06316254881831615, 0.14646523097463168, 0.5227401068570818, 0.1701051301958056, 0.7902349218232594, 0.501098652946115, 0.16268656963609202, 0.9864974145383364, 0.3490416245336693, 0.8067035110249288, 0.044579746306345806, 0.6904628265817359, 0.5188256631634425, 0.4780306158525292, 0.5115621572995785, 0.9719120244152251, 0.9396931617357918, 0.81983523010092, 0.5192465991687436, 0.8811418655893876, 0.4221924923390805, 0.6298609624988686, 0.7445735041282298, 0.04420395880967749, 0.5125139015900569, 0.6658084568167971, 0.517518197681329, 0.9531392368963412, 0.2595565567335288, 0.6729705679621412, 0.49242924254191295, 0.4004910547949401, 0.0934583202795866, 0.6339791849075991, 0.7763295731183172, 0.8001048396991061, 0.3834297215713862, 0.6994917269376573, 0.3844229691011764, 0.7482474093765166, 0.3885308005650732, 0.23914786670399477, 0.36319214576944014, 0.5264324555032195, 0.7552380275360913, 0.7938013140418965, 0.7736560935060021, 0.891682312494806, 0.47642339233660036, 0.14642110266848624, 0.31303096496765354, 0.9037095201140877, 0.07948745279811323, 0.5562976618747998, 0.9390152818119806, 0.31181434617084414, 0.6531393510904538, 0.039167541291285315, 0.365968260823681, 0.36168143916157003, 0.03855544955304563, 0.5124736334724934, 0.5386351420959107, 0.8158898820553788, 0.9654687857523268, 0.1214456714661371, 0.03107654877402274, 0.4758497996755591, 0.2523561120597068, 0.8230127950981354, 0.4096436790256799, 0.8160744824112673, 0.49563945983333807, 0.36487661773454205, 0.10022416754004958, 0.8922917448602201, 0.5946537995423363, 0.7866866420262011, 0.010210125297596512, 0.762482320518053, 0.6294015451592937, 0.08862556753728729, 0.17664939976257157, 0.8046348367583662, 0.6709499172299381, 0.774110066921858, 0.36087859461783167, 0.09768310637014777, 0.7718027701805144, 0.9569688039304004, 0.2570887307261538, 0.5311837727306734, 0.5116681502987066, 0.8690345570431239, 0.6139705263800441, 0.578688230941549, 0.4060279373493232, 0.3814327767396905, 0.443660011900307, 0.584075781805963, 0.47295670303888276, 0.5327746375230162, 0.13043033775021307, 0.9261660059424508, 0.3488347935980832, 0.3431569995020796, 0.39247583210211295, 0.45408392779109213, 0.5763232856294063, 0.5713979732097869, 0.2327283246733468, 0.49523020586287364, 0.7743707516034681, 0.007984848598931303, 0.571324162244996, 0.934598148316171, 0.7880867051405536, 0.6803394109417076, 0.8092195002926952, 0.1827684830074947, 0.6956460137316305, 0.7368248029053233, 0.4054162534536427, 0.4835969436418468, 0.7005436946936663, 0.6668203655629275, 0.6747742077405124, 0.15869512760198215, 0.008829458292342142, 0.6126115395192061, 0.2551576012035198, 0.5676015416037454, 0.19817506269389004, 0.4625145993971254, 0.8697533028512614, 0.03666789251778998, 0.680475481530404, 0.42737968807228555, 0.12009520362050777, 0.8299129119149837, 0.39613624608363984, 0.6991178979568512, 0.7014258053166849, 0.21090266641166328, 0.4954426026734301, 0.5696680122544033, 0.8499082419613309, 0.015455133675933626, 0.6918649371730082, 0.01672948738843818, 0.044819175376217955, 0.06578557718293884, 0.39196397088194235, 0.9608974826550178, 0.8298968007638532, 0.6708704871718725, 0.18479137994641776, 0.37385516198442337, 0.08648143392077827, 0.6388859409839606, 0.876377099368909, 0.08705032060559212, 0.9799790120493399, 0.6039353163992977, 0.3824384663164095, 0.20659156102904755, 0.5270369471706327, 0.19479210973757627, 0.8589902579027415, 0.3640747455101905, 0.7657775438261656, 0.2732466038826931, 0.520017561707202, 0.6370733085271851, 0.6830632743660273, 0.4641677832089236, 0.04844950039590895, 0.043052613762762065, 0.9184248861965092, 0.6711965987638174, 0.4658999046200608, 0.02580185475054453, 0.840873528348936, 0.15085670346374297, 0.5727391024002164, 0.49823049682812437, 0.26157400535654013, 0.33811502951789607, 0.3489334900227565, 0.9450094241116105, 0.864529536002978, 0.9216827471870335, 0.6922502894175918, 0.3333049404850443, 0.6239127641493674, 0.4119415644164437, 0.3481628934055626, 0.7500564893370429, 0.8155082061286623, 0.7323412949895709, 0.786021376615162, 0.1333487814029005, 0.5653728152337605, 0.9429348702214195, 0.3254615393926969, 0.9231973162708946, 0.779070597610837, 0.20466477126527388, 0.2772216829458845, 0.3213916802699418, 0.6919191632713613, 0.9591478751300635, 0.47435306637836777, 0.2812185604709479, 0.1296825007287567, 0.09609454735912518, 0.02581353139860128, 0.6577344928314396, 0.04495472734996527, 0.34489667896764187, 0.6900516588753214, 0.1921616668038256, 0.3093475335274711, 0.3211015800566085, 0.8393340710752721, 0.1261916025558781, 0.23584735577151, 0.5754120802873427, 0.759906198203566, 0.7581845638391882, 0.29453332247722713, 0.5498980706378941, 0.9785083037672149, 0.5407205580664983, 0.8267179404264413, 0.4125461377184654, 0.648302928590822, 0.12470363131121948, 0.5018709283683465, 0.19263551609862414, 0.3677239122521036, 0.9482811209391552, 0.9710046630448066, 0.7246068975455704, 0.14410232548871788, 0.6338635194626108, 0.865113377704483, 0.8387375683629137, 0.0201514766304572, 0.15406364219027985, 0.5422272441365151, 0.03888481706409419, 0.5639797520428309, 0.050567627807332066, 0.538734590998457, 0.5692668975724224, 0.7263179872332034, 0.9744343741900373, 0.20065021514723508, 0.7530692107862712, 0.7256722844398289, 0.41953983784222326, 0.3450936587419866, 0.3744889587987322, 0.976633799381478, 0.969747366061723, 0.7300024200786188, 0.5146179437923363, 0.061459117716680875, 0.6324924163585088, 0.5272039059031345, 0.3111331824896798, 0.4200668202459331, 0.45554610047979094, 0.18893999024104946, 0.8008440392637604, 0.3585841135155762, 0.7213066844862184, 0.8647082548436087, 0.9304391358857109, 0.2964785795577449, 0.07464774483959913, 0.5068144369885509, 0.2039640141308817, 0.8720531532317909, 0.0739385262280136, 0.4233354902116333, 0.7721350389108503, 0.6924162798452524, 0.8562979747322916, 0.2496562413220742, 0.004683256935137429, 0.7138756546684925, 0.5177299733741771, 0.5153075779389618, 0.7531142897698552, 0.6238453120514262, 0.4144180709610268, 0.24663366743243834, 0.8543798107193378, 0.921738033258253, 0.8997607557795191, 0.21830597056800624, 0.005491310796416403, 0.9654500626623105, 0.31778188537166074, 0.27395514456226283, 0.05508205982845443, 0.9592054746289422, 0.5081964733854482, 0.3748613323289083, 0.7481983978863813, 0.7660777100060117, 0.7122906935216258, 0.006775397388540632, 0.7884075394378364, 0.5836710619891738, 0.23669984772141017, 0.2770508501877966, 0.13914110112832434, 0.08621180287884705, 0.5078601947376415, 0.3458875090893699, 0.412359650386924, 0.2426010612911438, 0.6591067719243702, 0.7934419265156993, 0.7178491840550183, 0.5533449724894552, 0.6316951346012472, 0.9244008326222798, 0.22294773268071943, 0.10517779104204482, 0.17141271273632674, 0.7060373833683954, 0.555832891303625, 0.6311515647152197, 0.3502372262628532, 0.49370464297980926, 0.8910182117568825, 0.800951571179628, 0.05532012212940471, 0.12426471956852553, 0.13166988492414772, 0.10260478902588002, 0.8045233238356827, 0.3129816527346089, 0.5409755109224552, 0.29594035578266153, 0.9015836624311755, 0.6229864343805418, 0.41467861660815675, 0.352557073020474, 0.500256116590172, 0.0811914137414858, 0.10219683454612849, 0.15865305225697174, 0.61663141447383, 0.5390584956189902, 0.12670184642279347, 0.5952054046783304, 0.2153867789886469, 0.10149339676719825, 0.2817866663724713, 0.6438407297897654, 0.41127580988380835, 0.5848458781661179, 0.4701966393849881 ], "development_means": [ -0.0047981755611324756, -0.0008964108416191423, -0.00996007898675733, 0.008219408627936102, 0.0048069407294246734, -0.0011911643085693565, 0.0019494026179431084, -0.002424807684851433, -0.0011462035588896365, -0.00498925792670572, 0.0022985132782164415, -0.0016931620625910693, -0.005611461677177663, -0.0035468401964051, 0.004685128663308346, -0.007152375914058517, 0.004700812027357641, -0.0048495222692586945, -0.0005642856648236809, -0.0015158795907090443, -0.005281054208458644, -0.00011406330698361012, 0.005012322931284437, 0.0016716027885953484, 0.010586918703139535, 0.0005347238978518669, 0.0019881093454438476, 0.0013321813580639112, 0.0016045169761117995, -0.001719967355290149, -0.008756670939355676, 0.002147307010489967, 0.001529188082656663, 0.004491949115944484, -0.005407987434385508, -0.0009879190741889014, -0.006534327111463503, 0.0005641971091248882, -0.006903106783646581, 0.0008903087591911849, -0.0033578296080499096, 0.003004868837199822, 0.0012741758858595298, 0.0008372276393875034, 0.005207501284307683, 0.010487022098548206, -0.0044340358218156895, 0.000186731757764035, 0.0011413550074582375, -0.008247582115450165, 0.0010565148904382261, 0.000452765343292763, 0.00806161572761358, 0.005321325519831879, 0.0033823285147733377, -0.005130850478963029, -0.00025458441589719014, -0.010747785757690417, 0.0009007557936905378, 0.005772397426017361, -0.0015683821508893942, 0.0029347631119845227, -0.0011267575009895306, -0.005789878756487811, 0.004405176769052742, 0.0007053866980756999, -0.0015300080597557173, 0.00019843147232562937, -0.004531091810655898, -0.0038698410261924087, 0.00031838079023234256, -0.001978345282170842, -0.00631697585896543, -0.005683991108723206, -0.0025806893904984445, 0.0005845432134588905, -0.004781972861396629, -0.0013972529835981607, 0.0022840512102963663, 0.0026697259924966543, 0.0008039569947192937, -0.0021458976353289586, 0.0004086540085749986, 0.006579713120351349, 0.0008979383786702229, -0.001273223138311876, 0.0007222342968647964, -0.0063701048969554835, -0.002367586834289378, 0.0003515554209254143, -0.0076696644901966884, -0.0007415365084817654, -0.0035276707640320604, -0.0031622315530179844, -0.005137228999981466, -0.003711928617211329, 0.002732078461526574, 0.0017453034765488496, 0.0002271377962338248, 0.0024700991426852043, -0.0023639519082297275, -2.7619543074444164e-05, 0.0039367879171438944, -0.0034101107329875223, 0.0036992343578395215, -0.002505941209215972, -0.00983962644680144, -0.003273783431842525, 0.002067291084119779, 0.0009433565947732077, -0.008109375997863086, -0.002737128186399772, 0.002351404937372287, -0.007653269938060243, 0.0003481634235399957, -0.0011382746244372874, 0.002838077003462128, 0.003526918554302193, -0.006209978056336826, 0.0011861868611689364, 0.0061479466155860574, -0.004652013810658454, 0.003919830087266598, -0.010843348833807168, 0.002797175453832722, 0.004924488352328562, 0.01012558298318219, -0.0015082761307739603, -0.0011633799828552386, 0.0010911283987899648, -0.0036898311719504697, -0.0021937964055882483, 0.006977781721354253, 0.00480039890336584, -0.0002603138044153397, 0.004353252736854689, -0.0036845143364877854, -1.2569858507655199e-05, 0.004488934014131974, -0.01009381039574231, 0.0017705550390787856, -0.003951875433576119, 0.007758709783659634, -0.0022692267216641773, -0.00021546705511358484, 0.0002514819870380752, -0.00013230278943548264, -0.008716402804093118, -0.0070848046305991555, -0.004175185600986711, -0.00022028853324809788, -0.00538919845557639, 0.0008959266954214937, -0.0015130156703653213, -0.0030011091882280083, 0.007777004888266289, -0.0001431967920808689, -0.001955224299481122, -0.00020049261483636536, -0.007650246781437926, 0.002942699317283771, -0.002045427498655683, 8.66232987029637e-05, 0.0011505655060491764, 0.0060238324596435, -0.0015628851251486393, -0.003468250821328608, -0.0038431673093154194, 0.0013532643455697886, -0.002386880086652071, 0.0013413944232011166, -0.0030534838288380785, 0.0012923985278855722, 0.0032363388614020075, 0.0015972468311529291, -0.0003026390392738357, -0.0031542711846655894, -0.0037413196811445144, -0.0034275637051597724, -0.005639371857259754, 0.00026990048617883574, 0.0048012767484692555, 0.0022241057778621363, -0.005947271237884011, 0.0064290194121870495, -0.0006462622936382162, -0.007059047281711369, 0.002239793053761213, -0.0017974886456435459, 0.008035222643994647, 0.0015635227235051062, 0.0016156354039286963, 0.008068414370921309, -0.0001427358528376225, -0.00044272331099746585, -0.004107061694232701, -0.008298047642268106, 0.0053302092939929374, 0.008513443341954544, 0.00027647480600879613, 0.0030448526570394183, -0.004230735899108805, 0.0010427800440353035, -0.00411022873238324, 4.9890545743006814e-05, 0.0015767736666125155, 0.00584363046541672, -0.00565436028530246, -0.001093313030870089, -0.003628558260966786, 0.010582617048070949, -0.0032603652822566277, -0.0015074636646693132, 0.00615853432404009, 0.004236678294015706, -0.003917556678701482, -0.002019898944643125, -0.0034344533720754556, 0.0016254184239879786, 0.0059102309032208165, -0.0033995175651282405, -0.00783491882210065, 0.0029775422470080955, -0.00035714201683026897, -0.00013351598445902072, -0.005120471210318353, -0.0013222148476626974, -0.000906201451964862, 0.0010852878667893132, 0.0013771568869935737, 0.0006467514361306147, -0.0009691561855410221, 0.0003096435005497914, -0.00037540203403642347, 0.005131974114090351, -0.0066083543122295585, 0.0017731066052353141, 0.0018433706512842772, 0.0012454842636012097, 0.0005264974616733517, -0.000878621597703367, -0.0008212845021064499, 0.0033314826064409163, 5.4573184678630286e-05, -0.0034384131570654375, 0.01099830249911333, -0.0008204262531221658, -0.0068964666363845375, -0.003650571580973763, -0.0021390762085290323, -0.003993918582768407, 0.00413012429371647, -0.00233657808901709, -0.0028919157167606856, 0.0010924882365282607, 0.00018772257993459305, -0.0024006902305151916, -0.001967921744256246, -0.0020682685220476844, 0.004563602560760677, 0.010829814045384927, -0.0013060085329286634, 0.0030049290533439884, -0.0007771778106857027, 0.0038712927497687646, 0.00042950876919985414, -0.005135929269971195, 0.008173550904816917, -0.002140813867751309, 0.0008355016908391157, 0.005360884754544153, -0.0043535780465362075, 0.0012020721459583192, -0.00238197775530947, -0.002412287464026592, 0.0036665146903344862, 5.2142956865732296e-05, -0.0008011766664868413, -0.004728853652162755, 0.009850908724448661, -0.0022873967189436335, 0.00970624692316407, 0.007747117945346298, 0.00688275029021324, 0.0012515637820129512, -0.008038728544417546, -0.004353287552116274, -0.0020188967337279097, 0.004095390908467785, 0.0014681700743140339, 0.006220051796993636, -0.0016225484129929553, -0.005281255049307417, 0.006203619858684596, -0.009372060737678255, -0.0012029198152232123, 0.0013651195905743207, 0.0037350336666371383, -0.0003095706843800546, 0.003927041244479217, -0.004910304444671697, 0.001586515684059571, -0.003309210077518258, 0.002752412072511618, -0.00022911992073672283, -0.0016004760997895603, -0.0021739201902075895, 0.00041051419773825216, 0.007577179314957219, 0.007833849805020849, -0.00636524125539948, -0.0020230120768168877, 0.00039062027574025804, 0.008884173971361353, -0.0045554745856720355, 0.004713910403452201, -0.0008368837891189593, 2.0245200629487365e-05, 0.0029143421481704576, 0.001906134032667205, 0.0017718889694787448, -0.007295106989516703, -0.005024890207745685, -0.006465325027336622, -0.0022923968559723853, 0.0019663485778324036, -0.001441302012999197, 0.0010158125242037876, 0.0017813993199624085, -0.003079421852963674, -0.004100519852980023, -0.00282948653518911, -0.003618127770476964, 0.005069628063161103, -0.0007513186462066163, -0.007211214134241119, 0.0020652801238917376, -0.006512930738680909, -0.003510254032279414, 0.003765930644155329, 0.0026980585248446293, 0.002116996625559589, -0.0022881001634634025, -0.00794600322857619, 0.0002936325792250074, 0.002643789965060994, 0.005148104832923584, 0.005952514060600229, 0.008883286032911256, -0.0018544424404978973, 0.0077405773538875185, 0.0018217960229601426, -0.002263905210493324, 0.003970797035713831, 0.0022716821629145748, 0.0021206933048658863, -0.004526583062229925, 0.005224268775463988, 0.0032850773233953487, -0.0008680037844993279, -0.0032224382983076067, -0.0031972151523219504, 0.0024656145221161963, -0.0005723875392357604, -0.009237666510685576, -0.0004667015681809091, -0.004296311419551677, 0.0010087233002440718, -0.0017378449465913495, 0.00525717932016885, -2.140563262521331e-05, 0.003962887968159176, 0.0015422392477559268, -0.0074326454989811245, -0.008652959380274177, -0.0027230137843434685, 0.0048476566694736715, -0.0015614819574271107, -0.005037152862870663, -0.004515436925743685, 0.009359804407744804, 0.004651805072237879, -0.00048403352874044464, 0.008050500957248637, -0.000735167692808031, 0.007482687347525898, -0.00044386649872204197, -0.0007965166287322354, -0.002746439545030744, -0.008902210386945133, 0.0038308673529386895, -0.0031228396658025777, -0.002737591050248084, 0.0009268872588684868, 0.001819355751295929, 0.0014605357623314328, -0.009077211631684881, -0.008567720845461126, -0.0027971292303470945, -0.00016728338960992194, 0.00704115439079778, -0.001544859944161187, -0.00031148530851645426, 0.0022485876415172124, 0.0009207332124002222, 0.0005096604180213405, 0.00402489004705178, -0.003855236076510151, 0.0016534190930373792, -0.002678033099508666, -0.005028640054804901, -0.006750997958577866, 0.002439901558663374, 0.0065819107805128625, -7.796872209133444e-05, 0.0037772104688777964, -0.005185792607133846, 0.006604945026129365, 0.0008825989304249946, -0.0034045362874003944, -0.002294551511987765, -0.0048557142040618045, 0.003083549774284512, 0.011859921766579289, -0.002577686607187533, -0.00020291793732943073, -0.00017517914585516277, -0.00312349146186481, -0.001440490855908721, 0.0009867883624057328, 0.003127136917784477, -0.004817240304322569, -0.006467050428172164, -0.005843239013619643, 0.0035507355591731997, 0.011608300792174852, -0.008296929601853282, 0.0021630953399568195, 0.0027426953806396885, 0.0072913552745073025, -0.007949004037084602, -9.378360134560973e-05, 0.0014560522945579286, -0.003052782491243428, -0.0033136782190506995, -0.0025564456329932633, 0.011269189963045654, -0.003655628050445802, -0.0009644206834206511, 0.0032724529224235147, 0.002700386513608533, 0.004948610801230016, 0.006227868157701, -8.99354107635386e-05, 0.0018095264462824816, 0.0010109097895778523, 0.003185737604272746, -0.0018715066842047681, -0.0037355691922298776, -0.0026311858958787794, -0.0006121575636631636, -0.0015352038479280721, -0.006551270992270784, 0.0034792966975340303, 0.005717257137783656, 0.0043297348457227635, -0.0024731731080658843, -0.0006408916652219476, -0.0015286244646638672, 0.0017558163285076947, 7.202905309255495e-05, -0.005623107847793084, -0.003856993974971467, 0.007281615466945185, 0.005266939405334913, 0.005105377560338519, 0.005782344410485672, -0.003915713002422315, 0.002224741105788123, -0.0004696339042525766, 0.002447008174749559, -0.005890884261169833, -0.00143016623815865, 0.0009837371582870301, 0.0017272705359257052, -2.930266070977048e-06, 0.006376870112654464, 0.005792772633296889, 0.004564396176804991, -0.0013539952573917396, -0.0004475900493232689, 0.005213045673938339, -0.0010998088036580538, 0.003596111795789256, 0.005810825425074809, 0.0026361067954381763, -0.0016830784421742564, 0.0010236219957365238, -0.000978169732495677, 0.0003413022256793034 ], "holdout_means": [ 0.007658912384705444, -0.0007174443004839976, 0.0022785446916859953, 0.0017711203965957904, 0.005561338668448594, 0.0036157631884963745, -0.0015705522935678213, -0.005820472142382774, 0.006766608177975697, 0.003749632355527969, -0.001672794024594719, -0.004599427590402427, -0.003992286481063795, -0.001056893338078012, -0.002944929268282168, -0.0005602826915198105, -0.003509401986775069, 0.0016284786117830193, 0.0024925254680253843, 0.010222345991360812, -0.004851755596328836, 0.0005352026587780661, 0.004959742429063509, -0.005291006717233991, 0.0016349888234482514, 0.009285597602908363, -0.010707768637042767, -0.0022688749604447274, -0.004736682223363454, -0.0030572348717810348, -0.002438936642894018, -0.00160762467651023, -0.003407700021871669, 0.0057013588939509125, -0.0017420882349641735, 0.0027822173073378965, -0.006234732726281721, -0.006828187936153975, -0.0029366314394212484, -0.005375359675385418, -0.001074404676391136, -0.0024355159170225986, -0.002710899110562593, -0.00036958322009809653, 0.007155454728852111, 0.00500079608721685, 0.0006653906263716094, -0.004529246055835025, 0.004111502767292098, -0.000830810549292612, 0.003039832457664698, -0.004789208086900036, 0.0032141738744052117, 6.098473965308105e-05, -0.005093483848699583, -0.0037788068634633525, 0.0024385585543757454, -0.000980078096339036, -0.0013275150390482913, -0.0004971434219030999, 0.0020436665629693326, -0.006163119621906277, -0.006255331712484443, 0.01578227615776235, -0.002089250926599542, -0.006652935944108395, 0.010945773949246602, 0.013263956558886512, -0.005436427447832911, 0.0015590119978978552, 0.00462193152418235, -0.0050539163113293385, 0.0030714799550927707, -0.005636878881185781, -0.0036415524189248745, -0.0031680016338319857, -0.0036608432295858165, -0.0052072072036564295, -0.00012246782385121272, 0.006191927315243657, 0.00558575732054903, 0.005972754792654958, -0.0006953859196873227, -0.00124355868066654, 0.002366672649197871, -0.003200766361824028, 0.0003822828976154978, -0.0007704415868099267, -0.00022073437465164854, -0.0028956177872436245, -0.0003762342831913942, 0.004689022901160395, 0.005606899266753244, 0.006393367508378689, -0.00599284233780417, 0.002532865519888206, 0.0008458603691643703, -0.0018775580338739248, 0.005407625706551953, -0.0006184602782087109, 0.01307606971249608, -0.010416946099521405, -0.0013589784657142318, -0.0005395851582790393, -0.0012497717898587212, 0.00046921762585769956, 0.0017516201443399323, 0.0008642336046057999, 0.0034646997197286724, 0.0018151051644114467, -0.0001931870892070097, 0.004036229398498335, -0.0047927130324297434, -0.004331392406910495, 0.0014586819751383063, 0.00023718047769741426, 0.0017700481120449063, 0.0018139967152815016, -0.003919301639016274, 0.0018550982640112093, 0.0025199674404627124, 0.002882136075484432, -0.01410739026772266, -0.0032862375788431995, -0.0031333542023565426, -0.003915851840584958, -0.0037585965771355107, -0.011039086500627921, 0.003916404909691047, 0.0016461795514684048, 0.0005943630439550033, 0.0009535972972305773, -0.0035272371650496024, 0.004594426396384581, -0.0033780057862144645, -0.0007752380313113755, 0.00035963332619644143, 0.006921166833551329, -0.0022279744804130372, -0.005511286889878139, -0.0037809659504403096, -0.007276525366666369, 0.001523837327202965, 0.009652202830647782, 0.003574310494142085, 0.0014864294975561714, 0.0013950819197998151, -0.002162510108374455, -0.007177204640482086, 0.006871350348830373, -0.00014062370074406007, 7.458969421800773e-05, 0.001445843353013186, -0.00027264694495655, 0.0013559508935882085, 0.0026377857032892486, -0.001269356415578432, -0.0021104904977538165, 0.0029964291592867423, 0.002793182353474894, 0.010219551433002797, 0.0018414327408377308, -0.005739854561278292, 0.0006158000368383814, -0.002479225854422263, 0.0036305916449200283, -0.0016271841111403168, -0.0007963392493796791, -0.009876542357820987, -0.003155207795055957, -0.008002026055839618, 0.0004804894030311122, -0.010650528064253803, 0.0004653888398777627, -0.0011054378251926624, -0.0018800492894123105, 0.0011171059271976658, -0.0019687270335402656, 0.008656389443143972, 0.0008995634878518116, 0.006123223869064781, -0.00302431147914376, -0.004176775700276873, 0.005916159563554937, -0.0033590127577971464, -0.0008456571243950335, -0.0035048569215542246, 0.0020323641230799235, -0.008459680231659751, 0.006286197732281372, 0.003210808316206338, -0.0011944090659410927, -0.0007064149925171395, 0.0010102911874078975, 0.007333784748629898, -0.008480224216557706, 0.004903239215125869, 0.00012334080743322366, -0.0025948335251572184, -0.006445823675443204, -0.0029807418401388984, 0.005495007192280218, 0.00153567224102422, -0.006623703136692725, -0.004320786233338355, 0.002140686016304405, 0.0037903262920116134, -0.0035506485947404687, 0.0016484551074046265, 0.009639599621401212, -0.003091791909473042, 0.00529310356000881, -0.0015845666676906546, 0.004224396763526028, -0.003606249261453526, 0.009442572037690217, -0.0022882841044167862, -0.006711192567572825, -0.001832535107572656, 0.0005925880212215259, 0.00412510241285651, 0.0015985755037502968, 0.0008201170424239661, -0.002324805473124611, 0.011080322005103954, -0.0022808010356342697, 0.008239058544786602, -0.0006714798930967876, 0.002605346814895096, -0.0015667875313956049, 0.0012861307952439344, -0.005187307166330497, -4.960940412424048e-05, -0.0005826414706974848, -0.0031265262271386405, 0.0049068267619956285, -0.0004320259690746444, -0.006103039769282945, -0.00489742369275031, -0.002990199231833599, -0.0027787317948553135, 0.0030463614176842323, 0.0018613326742693255, -0.008025178828696458, 0.0030197360282577454, 0.0014544353834260325, 0.004660439756472443, -0.0015976670790784676, -0.0005363176572394173, 0.0013003821635785688, -0.004392428999814146, -0.0022525842903724537, -0.00013628408374442654, 0.0029208898478881046, -0.00018058531412347417, 0.0011184205810162524, -0.002812457949993003, 0.001081722351659579, -0.005267003939382538, -0.00026212862254894846, 0.003318323312155441, -0.0017684649487985896, -0.005290406033235768, -0.001924593731594651, 0.0003171088443566651, -0.003018432664925843, -0.00304065237250408, 0.0009114557621524744, 0.005216732823403003, -0.0019170139545777445, -0.0046970130307686766, -0.008075521986925066, -0.005119388341874328, 0.004781910139806068, -0.006533881281559878, -0.0007903746384706136, -0.0005117254628926568, -0.0009171686873009579, 0.0002366298748779959, 0.007213010068171241, 0.005169912370527408, 0.0057028539107911586, -0.0054015849322978865, 0.0003584636263438479, 0.002304598736970687, 0.003562015072173523, 0.0071799711495011955, 0.000156611967590321, -0.0018114925442044353, -0.004215168830841639, -0.006674757298508806, -0.0019120234456670051, 0.008460776593323994, 0.008504227444975157, -1.89718973520183e-05, 0.0037175128466586872, 0.003290692639241827, -0.0054205854876691155, -0.002128864010324102, -0.0043959469019508755, 0.0021556587026729224, -0.0016999168024171745, 0.0005569808587167829, -0.0005803736290725467, -0.006138567145378035, 0.007111272941626229, -0.0028684657017468106, 0.0023214555219873275, -0.004047142382220693, 0.0013208648994669564, 0.010481169279351086, 0.0009290331031472408, 0.001833774654159743, 0.012028532488217588, 0.0069115769959795246, -0.0004856200096454379, 0.003526940148772515, 0.004290157700966018, 0.0024616281543922234, 0.0016427597465667719, 0.0033328364120469483, -0.0026688210340773963, -0.0008311069426741818, -0.001157025011760869, 0.00038106841817354975, 0.0020228782163379476, -0.004441261828104462, -0.004515918585221834, 0.000491511025683889, 0.006552324204027022, 0.004745976750934364, -0.0030066051681806624, 0.008380915378755807, -0.006976933260631139, -0.00987558095554786, -0.00752233144406355, -0.005150263410346343, -0.002314567390092911, -0.00711635978360604, 0.003573047242690416, 0.0013218549962317422, -0.006853324389622499, 0.001681220097309655, 0.005660577275596607, -0.005430804017320564, 0.007350848316811632, -0.0009568547744149179, -0.0012486614609265412, -0.007624210118055466, -0.0028656007066040273, 0.0037212104530594202, -9.283785197809424e-05, 0.007978128384003564, 0.0018195424851390545, 0.0020113360798998487, -0.0005044458216061854, -0.001414244613672532, -0.0008647405821274888, -0.0012844991146668343, -0.0015890943433153597, 0.0026907885391320623, -0.0003850949271666315, -0.0016609819663905862, 0.001237898107393143, -0.0028594239003388758, 0.009335559083016401, -0.0004751105679725704, -0.006706030613038241, 0.0033293845704678317, -0.00787852697431093, 0.0016166242575224835, 0.008933027287909604, 0.0019040330824089365, -0.001496547798226257, -0.003507516951822364, -0.0038896369878091397, -0.0005064077996063575, 0.0012313834738223528, -0.007703152054514268, -0.0052430272481305325, 0.002801415704900334, -0.005910047400696269, 0.006373792356708663, -0.0013837797985012334, -0.0007178698015530866, 0.00012186396680884295, 0.00046977976936163904, -0.00033575119990371453, 0.0014709007970490344, -0.0025987051516454165, 0.006154406912766153, -0.004016282502358063, 0.0016805196114437887, -5.9703706759118414e-05, 0.0011866415460709673, -0.0030810740265549727, 0.004147547665886879, 0.0023486643441993295, -0.002865241777030534, 0.002561099652638543, 0.00350657536946471, 0.008872012476914316, -0.002397391825549041, -0.003660456357081933, 0.002070176708743287, -0.00034863758655725096, -0.0014588583673498373, -0.0028063569501345868, -0.0017636581769051524, -0.003980993132997927, 0.0014807142562605548, 0.00430417649601649, 0.004870650873763, 0.00473988446301119, 0.0009955402603081082, 0.001997697598441392, 0.004742900389642199, -0.00472120268506272, -0.004260405754930051, -0.0007657585497247356, -0.002153578767098434, -0.002589828350065598, -0.008313072693994951, 0.009383412895362169, 0.003443999570172722, 0.0015333985062490195, -0.00370781330961663, -0.0015257988539856496, 0.0018708555485697524, -0.005717867568250162, -0.006968266868640191, 0.0042175794519150964, 0.0033553331082624154, -0.0010387999858284306, 0.004834205572985626, -0.002667502167171221, 0.0023863267685437757, -0.0027729377933987357, 0.005585204003632436, 0.015844031269870145, 0.0067896708223069546, 0.0007387827743622568, 0.003663802249436887, -0.0020804160670506214, 0.004347120706289768, -0.009624767236464648, 0.0012757479783175652, 0.00472071040212146, -0.001961957136151467, -0.0012341172262825388, -0.0037569332941299236, 0.004708443316555055, 0.0020018506288405533, -0.012861919612488353, -0.0031536133708531394, 0.0028508186247168644, 0.0027718516629135956, -0.003166442630649686, -0.0032247403143210083, 0.008396763696162662, 0.005519116535334079, 0.004954966040023979, -0.0022272564841066085, 0.00547790674578865, -0.006224474384198015, -0.0037138819548734867, -0.011273406975126932, 0.0006268207335624748, 0.0030147582096108744, 0.00289503650350741, -0.004921031631531033, 0.008646911346154394, 0.0021293565843676695, -0.0014970352966755082, 0.005266671086686753, -0.0016503335778668131, 0.00433049525360365, -0.003683465667227376, -0.00019483058165896123, -0.003601970533350547, 0.00577158857362483, -0.00966683402360808, -0.0017486954956029196, -0.0012070155914004788, 0.004049319568769893, 0.002645287924821145, -0.002947885825089047, 0.005627900044510663, 0.0031155725186436216, 0.004912640914562181, 0.008563574579955139, 0.005136105856562553, -0.0012389427780464051, -0.01266231320634511, -0.0018261753686895572, -0.006459869346895336, 0.0018599603870309755, -0.00458420317927716, 0.005618648576941394, -0.0006867843472203861 ], "winner_zero_based": 423 } } ], "bh_equality_audit": [ { "design": "all-null-M1", "rank_threshold_comparisons": 5000, "equal_count": 0, "same_rejections": true }, { "design": "all-null-M10", "rank_threshold_comparisons": 50000, "equal_count": 0, "same_rejections": true }, { "design": "all-null-M100", "rank_threshold_comparisons": 500000, "equal_count": 0, "same_rejections": true }, { "design": "all-null-M500", "rank_threshold_comparisons": 2500000, "equal_count": 0, "same_rejections": true }, { "design": "mixed-M100", "rank_threshold_comparisons": 500000, "equal_count": 0, "same_rejections": true } ], "equality_example": { "p_values": [ 0.025, 0.06 ], "q": 0.05, "strict_R": 0, "inclusive_R": 1 } }, "mixed_summary": { "mean_fdp_fdr_estimate": 0.04527148074148074, "mc_se_fdr_estimate": 0.001783952983602667, "probability_any_false_rejection": 0.1488, "mean_rejections": 2.4438, "mean_true_discoveries": 2.2692 }, "evaluation_candidate_ledger": [ { "key": "p1-l0", "p": 1, "lambda": 0.0, "n": 120, "rmse_decimal": 0.05045915640801843, "mae_decimal": 0.041517385484932286, "sse_decimal_squared": 0.3055351758490641, "benchmark_sse_decimal_squared": 0.30355158324232817, "r2_os_vs_expanding_mean": -0.006534614596796162, "mean_train_rmse_decimal": 0.07836753483816335, "mean_beta_norm": 0.0014615416134109065 }, { "key": "p1-l0.1", "p": 1, "lambda": 0.1, "n": 120, "rmse_decimal": 0.0504434493225498, "mae_decimal": 0.04150503553732705, "sse_decimal_squared": 0.305344989546798, "benchmark_sse_decimal_squared": 0.30355158324232817, "r2_os_vs_expanding_mean": -0.005908077583763216, "mean_train_rmse_decimal": 0.07836765669505215, "mean_beta_norm": 0.001328674194009915 }, { "key": "p1-l1", "p": 1, "lambda": 1.0, "n": 120, "rmse_decimal": 0.050374725305139666, "mae_decimal": 0.04144946077310352, "sse_decimal_squared": 0.30451355394819346, "benchmark_sse_decimal_squared": 0.30355158324232817, "r2_os_vs_expanding_mean": -0.0031690518480917973, "mean_train_rmse_decimal": 0.07837122090042192, "mean_beta_norm": 0.0007307708067054532 }, { "key": "p1-l10", "p": 1, "lambda": 10.0, "n": 120, "rmse_decimal": 0.05030921470775232, "mae_decimal": 0.04139388600887999, "sse_decimal_squared": 0.3037220501412867, "benchmark_sse_decimal_squared": 0.30355158324232817, "r2_os_vs_expanding_mean": -0.0005615747318388209, "mean_train_rmse_decimal": 0.07837971931183126, "mean_beta_norm": 0.00013286741940099153 }, { "key": "p3-l0", "p": 3, "lambda": 0.0, "n": 120, "rmse_decimal": 0.05061215322545209, "mae_decimal": 0.04106420261487309, "sse_decimal_squared": 0.30739080649399686, "benchmark_sse_decimal_squared": 0.30355158324232817, "r2_os_vs_expanding_mean": -0.012647679879184803, "mean_train_rmse_decimal": 0.07807872252516207, "mean_beta_norm": 0.006829627564629242 }, { "key": "p3-l0.1", "p": 3, "lambda": 0.1, "n": 120, "rmse_decimal": 0.05057073274925227, "mae_decimal": 0.04107964782144354, "sse_decimal_squared": 0.30688788129555555, "benchmark_sse_decimal_squared": 0.30355158324232817, "r2_os_vs_expanding_mean": -0.010990876797911442, "mean_train_rmse_decimal": 0.07808116766480033, "mean_beta_norm": 0.006216922808392738 }, { "key": "p3-l1", "p": 3, "lambda": 1.0, "n": 120, "rmse_decimal": 0.050415189132425355, "mae_decimal": 0.04119319794088291, "sse_decimal_squared": 0.3050029554309864, "benchmark_sse_decimal_squared": 0.30355158324232817, "r2_os_vs_expanding_mean": -0.004781303306527596, "mean_train_rmse_decimal": 0.07815353738113272, "mean_beta_norm": 0.003440490044876775 }, { "key": "p3-l10", "p": 3, "lambda": 10.0, "n": 120, "rmse_decimal": 0.05031109715989309, "mae_decimal": 0.04134451295699682, "sse_decimal_squared": 0.30374477969186436, "benchmark_sse_decimal_squared": 0.30355158324232817, "r2_os_vs_expanding_mean": -0.0006364534405407518, "mean_train_rmse_decimal": 0.07832893924850517, "mean_beta_norm": 0.0006296889364991363 }, { "key": "p12-l0", "p": 12, "lambda": 0.0, "n": 120, "rmse_decimal": 0.05135414958326497, "mae_decimal": 0.04166338926915809, "sse_decimal_squared": 0.31646984153044244, "benchmark_sse_decimal_squared": 0.30355158324232817, "r2_os_vs_expanding_mean": -0.04255704467138788, "mean_train_rmse_decimal": 0.07757181431487904, "mean_beta_norm": 0.011459832684340695 }, { "key": "p12-l0.1", "p": 12, "lambda": 0.1, "n": 120, "rmse_decimal": 0.051207657288217544, "mae_decimal": 0.04161090779009132, "sse_decimal_squared": 0.3146668997937047, "benchmark_sse_decimal_squared": 0.30355158324232817, "r2_os_vs_expanding_mean": -0.03661755419836865, "mean_train_rmse_decimal": 0.07757928723331557, "mean_beta_norm": 0.010343850489743778 }, { "key": "p12-l1", "p": 12, "lambda": 1.0, "n": 120, "rmse_decimal": 0.050688303663885714, "mae_decimal": 0.041466238857951566, "sse_decimal_squared": 0.3083164953986748, "benchmark_sse_decimal_squared": 0.30355158324232817, "r2_os_vs_expanding_mean": -0.015697207392072032, "mean_train_rmse_decimal": 0.07778195597693044, "mean_beta_norm": 0.005581722194977279 }, { "key": "p12-l10", "p": 12, "lambda": 10.0, "n": 120, "rmse_decimal": 0.050349393777743084, "mae_decimal": 0.0413836039246465, "sse_decimal_squared": 0.30420737445434803, "benchmark_sse_decimal_squared": 0.30355158324232817, "r2_os_vs_expanding_mean": -0.0021603946354524783, "mean_train_rmse_decimal": 0.07824331776883953, "mean_beta_norm": 0.001015524644240653 } ], "reading_base": "公开同源包；按完整 URL 取得冻结输入与复算文件。来源网站后续更新不覆盖本份 202607 快照。" }, "entry_id": "zh-qt22", "content_version": "2026-09-21-QT-DE-review-v2", "audience": "高年级本科至研究生；默认微积分、线性代数与基本概率", "learning_task": "保留全部尝试，证明 Bonferroni，区分 FWER/FDR 并按严格规则执行 BH。" } ``` ## Supplied entry 前两篇已把预测目标与信息时钟写清，但研究者还可以尝试许多变量、窗口和参数，最后只展示最好的结果。即使每一次拟合都没偷看未来，这个挑选过程也会改变结果的分布。本篇保留完整候选集合，先在真值已知的模拟里观察选择效应，再区分“至少犯一次错”与“发现中错误的比例”。 ## 1. 被评价的不只是冠军，还有挑冠军的方法 BusEq 实验有 12 个回归候选和两条基线，采用同一 202607 CIZ 快照、同一训练和验证月份，最终按预定 SSE 规则选择扩展历史均值。那 12 次回归不能在报告中消失；若还尝试过后来放弃的变量或起止日，也应进入研究记录。只对最终展示的一项作检验，会把“它为何留下来”排除在问题之外。[^pilot] 为了把选择机制看清，我们暂时离开真实数据。构造 $M$ 个候选，每个开发样本有 $n=120$ 个独立正态月收益，已知标准差 $\sigma=0.05$。先设所有均值都为零。对每个候选事前固定单侧检验 $H_0:\mu=0$、$H_1:\mu>0$，则 $$ Z_j=\sqrt n\,\bar R_j/\sigma\sim N(0,1),\qquad p_j=1-\Phi(Z_j). $$ 模拟直接生成这个 $Z$ 的精确分布，再换算样本均值，不需要制造 120 行月收益。它不是实际策略面板，也没有逐月净值路径。开发与独立保留样本各生成一组 $5000\times500$ 的独立正态矩阵；种子为 PCG64(2201)，先开发、再保留。$M=1,10,100,500$ 使用相同 500 列的前缀，所以这些规模的结果彼此相关。[^simulation] 每轮只按开发 $Z$ 最大值选出编号 $J$，冻结编号后才看独立保留样本的同一列。$p$ 值问的是“在原假设及所设抽样模型下，至少这样极端的统计量有多常见”，不是“看完数据后原假设为真的概率”。单侧方向也不能看完符号再改。[^testing] ## 2. 没有真信号，冠军仍会越来越漂亮 | 候选数 $M$ | 开发冠军平均样本收益 | 同编号独立保留样本平均收益 | |---:|---:|---:| | 1 | −0.000216 | −0.001749 | | 10 | 0.710968 | −0.006699 | | 100 | 1.146183 | 0.009726 | | 500 | 1.384628 | −0.007626 | 数值为 5000 次实验的平均，单位百分点/月。开发冠军变好不是均值从零变正，而是取最大值这项操作改变了统计量。保留样本的小幅正负差仍是有限模拟波动。[^simulation] 还可以直接解释为什么保留样本不继承这种提升。$J$ 完全由开发数据决定，保留向量与全部开发数据独立，且每列均值为零。因此给定开发数据后， $$ E[\bar R^{hold}_J\mid\text{开发数据}] =\sum_{j=1}^M1_{\{J=j\}}E[\bar R^{hold}_j]=0. $$ 再取期望仍是零。若看过保留结果再换编号，这个证明的独立性步骤便不再适用。 ## 3. FWER：整组中至少错一次 令 $V$ 表示错误拒绝的真实原假设个数。族错误率是 $FWER=P(V\ge1)$。若全部 $M$ 个原假设为真，每个独立检验以 $\alpha$ 的概率误报，那么 $$ FWER=1-P(\text{一个也不错})=1-(1-\alpha)^M. $$ 这里的乘法需要独立性，等号也用了每项精确误报率为 $\alpha$。对一般仅“有效”的 $p$ 值，我们只知道原假设下 $P(p_j\le u)\le u$，不应自动把所有界写成等号。[^fwer] 用 $\alpha=0.05$ 比较理论和本次模拟： | $M$ | 未校正 FWER 精确值 | 未校正模拟 | Bonferroni 模拟 | Bonferroni 精确值（本独立全零模型） | |---:|---:|---:|---:|---:| | 1 | 0.050000 | 0.0474 | 0.0474 | 0.050000 | | 10 | 0.401263 | 0.4080 | 0.0570 | 0.048890 | | 100 | 0.994079 | 0.9934 | 0.0498 | 0.048782 | | 500 | 约 1 | 1.0000 | 0.0458 | 0.048773 | Bonferroni 把每一项门槛改成 $\alpha/M$。其一般控制可以在这里完整证明：设 $I_0$ 是真实原假设集合，只需各项 $p$ 值有效，便有 $$ \begin{aligned} P(V\ge1) &=P\left(\bigcup_{j\in I_0}\{p_j\le\alpha/M\}\right)\\ &\le\sum_{j\in I_0}P(p_j\le\alpha/M)\\ &\le |I_0|\alpha/M\le\alpha. \end{aligned} $$ 这次使用的是并集上界，不需要检验相互独立。但它也不能挽救本来无效的单项 $p$ 值，或只把公布的候选数当作全部尝试数。[^bonf] 为什么表中 $M=10$ 的模拟 Bonferroni 值 0.0570 反而超过 0.05？它是 5000 次试验的频率，不是理论概率。该频率的 Monte Carlo 标准误约 0.003279。我们保留真实输出，不把它改成符合定理的数；定理靠证明成立，不靠这轮随机数恰好落在界内。$M=500$ 未校正模拟全为误报，其代入式 MCSE 为零，也不能解释成风险估计毫无不确定性。[^simulation] ## 4. FDR：控制的是比例的期望 记 $R$ 为拒绝总数，$V$ 为其中的误报数。定义 $$ FDP=\frac{V}{\max(R,1)},\qquad FDR=E[FDP]. $$ 没有拒绝时 FDP 为零。FDR 不是一般意义下的 $E[V]/E[R]$，也不是观测到 $R=100$ 后保证恰有至多 $100q$ 个误报。它是对重复抽样中每轮比例取期望。全零假设时 $V=R$，故 FDP 是 $R>0$ 的示性函数，FDR 与 FWER 恰好相同；为了区分两者，需要有真假混合的实验。[^fdr] 本篇的 BH 规则采用 ISLP Algorithm 13.2 的严格比较。对 $M$ 个 $p$ 值排序，取 $$ k=\max\{j:p_{(j)} 先在全零模型下改变候选数，比较开发冠军与固定编号的独立保留结果；再切入混合模型，看 FWER 和平均 FDP 的不同。图中是候选或重复试验的分布，不是未生成的月度收益路径。 ## 5. 复核与迁移 解释题。 同一轮 BH 发现很多候选，后来知道其中 20% 是误报。能否据此断言 $q=5\%$ 的 FDR 控制失效？ 解析。 不能仅凭这一轮断言。控制对象是重复抽样中 FDP 的期望，不是每轮确定上限。不过仍须检查原假设 $p$ 值是否有效、独立性等采用条件是否满足，以及研究者是否按预定规则执行。满足某个观测比例也反过来不能证明这些条件成立。 迁移题一。 $M=10$ 个独立全零检验，每项门槛 0.05，至少一次误报的概率是多少？Bonferroni 的证明哪一步允许任意依赖？ 解析。 未校正值为 $1-0.95^{10}\approx0.401263$。Bonferroni 证明用 $P(\cup A_j)\le\sum P(A_j)$，不使用乘法分解，所以无需独立；只要每个真实原假设的误报概率都不超过 $0.05/10$，总 FWER 就不超过 0.05。 迁移题二。 开发期尝试 100 项，挑出编号 7。先看它的保留成绩，不喜欢，再改成编号 12。为什么不能继续引用本篇“保留均值为零”的条件证明？ 解析。 新编号不再只由开发数据决定；在给定开发数据后，它仍依赖保留结果，因此不能把 $E[\bar R_J^{hold}\mid开发]$ 拆成固定编号的零均值。新尝试应写回研究账本，之后的确认需要未用于这次调整的数据。网页把成绩折叠起来只是一种学习顺序，不会让已被研究者查看的历史重新独立。 迁移题三。 对上面的人工等号例分别执行 `<` 和 `≤`，再说明只保存三位小数的 $p$ 值有何风险。 解析。 拒绝数分别为 0 和 1。若原值略大于 0.025 却显示成 0.025，用显示值重算会改变决定；应保留原始精度、比较规则和全部候选数量，显示精度与运算精度分开。 最后，把这种记录意识带回真实研究：目标、期限、数据版本、全部特征和参数、所有尝试、选择准则，以及何时查看评价结果，都属于方法本身。即使得到可信预测，下一步仍须把它转成有预算和成本的可行决策。[成本、约束与稳健决策](https://ou-liu-red-sugar.github.io/zh/notebook/costs-constraints-robust-decisions/) [^pilot]: [共同输入](https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/shared_inputs.json)与[实际结果](https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/data/results.json)，forecast 组；BusEq 版本与数据来源见 [预测目标与正则化](https://ou-liu-red-sugar.github.io/zh/notebook/prediction-regularization-complexity/)。 [^simulation]: [复算程序](https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/compute/reproduce.py)、[完整结果](https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/data/results.json) selection 组及[重复试验数组](https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/data/selection-arrays.npz)。模拟按严格 `<` 比较完整精度的 p 值；人工等号例单独展示两种比较规则的差别。 [^testing]: James 等，ISLP 2023，[作者全文入口](https://hastie.su.domains/ISLP/ISLP_website.pdf.download.html)，§13.1.1–13.1.2、§13.2，印刷 pp.558–564。 [^fwer]: 同上，§13.3.1，pp.565–566，FWER 定义及独立全零模型。 [^bonf]: 同上，§13.3.2 的 Bonferroni 子单元，pp.567–568；本文以真实原假设集合写出完整并集上界证明。 [^fdr]: 同上，§13.4.1，pp.573–575，包括 $R=0$ 时比例取零的约定。 [^bh]: 同上，§13.4.2，pp.575–577，Algorithm 13.2 的严格比较；该书不提供一般控制定理证明，本篇明确限定到全部 $p$ 值相互独立、真实原假设下有效。 ## Experiment inputs and static equivalents ```json [ { "id": "EXP-QT-D-SELECT-01", "title": "选择效应与多重检验：可复算实验", "anchor": "qt22-fdr", "description": "保留全部尝试，证明 Bonferroni，区分 FWER/FDR 并按严格规则执行 BH。", "inputs": { "identity": "QT-DE-adopted-inputs-20260921-v1", "manifest": "https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/shared_inputs.json" }, "outputs": { "equalities": [ { "design": "all-null-M1", "rank_threshold_comparisons": 5000, "equal_count": 0, "same_rejections": true }, { "design": "all-null-M10", "rank_threshold_comparisons": 50000, "equal_count": 0, "same_rejections": true }, { "design": "all-null-M100", "rank_threshold_comparisons": 500000, "equal_count": 0, "same_rejections": true }, { "design": "all-null-M500", "rank_threshold_comparisons": 2500000, "equal_count": 0, "same_rejections": true }, { "design": "mixed-M100", "rank_threshold_comparisons": 500000, "equal_count": 0, "same_rejections": true } ], "mixed": { "mean_fdp_fdr_estimate": 0.04527148074148074, "mc_se_fdr_estimate": 0.001783952983602667, "probability_any_false_rejection": 0.1488, "mean_rejections": 2.4438, "mean_true_discoveries": 2.2692 } } } ] ``` ## Sources - [QT-D/E adopted teaching experiments](https://ou-liu-red-sugar.github.io/notebook/labs/qt-de/compute/reproduce.py): 本轮实际沙盒复算，预测遵守冻结算法及数据；选择沿PCG64调用顺序并核strict/inclusive，成本用Fraction。数学演算与假设模型不等于真实投资有效性。 - [An Introduction to Statistical Learning with Applications in Python](https://drive.google.com/uc?export=download&id=1ajFkHO6zjrdGNqhqW1jKBZdiNGh_8YQ1): 采用单元的完整镜像原文已在规划轮读取。Ridge 的截距、尺度、参数选择；有效p值、FWER、Bonferroni及BH。BH一般控制定理是引用，原书不提供证明。正文采用严格<；不采用任意相关p值也有效的扩展。 ## Content relations ```json [ { "from": "zh-qt22", "relation": "part_of", "to": "quant-estimation", "reason": "主要 topic 归属" }, { "from": "zh-qt22", "relation": "requires", "to": "zh-qt19", "required_competence": "抽样不确定性", "reason": "本篇使用该能力，不要求整门随机过程课程。" }, { "from": "zh-qt22", "relation": "requires", "to": "zh-qt21", "required_competence": "完整时间验证与保留期", "reason": "本篇使用该能力，不要求整门随机过程课程。" }, { "from": "qt22-fdr", "relation": "illustrated_by", "to": "EXP-QT-D-SELECT-01", "reason": "完整输入、默认及静态等价支持本篇独立任务。" }, { "from": "zh-qt22", "relation": "supported_by", "to": "QTDE-ISLP", "locator": "§13.1.1–13.1.2与§13.2 pp.558–564；§13.3.1及Bonferroni子节pp.565–568；§13.4.1–13.4.2 pp.573–577", "scope": "p值、FWER、Bonferroni、FDP/FDR、Algorithm13.2", "reason": "定义与证明边界；采用严格<；一般BH控制证明不在本篇" } ] ``` ## Related entries ## Optional reading path 理解模型并亲手算: step 8/9 证明 Bonferroni，计算严格 BH，并区分 FWER 与 FDR。 把预测作为决策输入时，进一步明确资金、风险与交易成本。 Next: [成本、约束与稳健决策](https://ou-liu-red-sugar.github.io/zh/notebook/costs-constraints-robust-decisions/)