Grossman, Maura R., Special Master
Plaintiffs, v.
WALGREEN CO. and WALGREENS BOOTS ALLIANCE, INC.,
Defendants.
SPECIAL MASTER’S ORDER ON REMEDIATION PROTOCOL PURSUANT TO HER ORDER ON PRELIMINARY DIAGNOSTIC PROTOCOL TO ASSESS THE ADEQUACY OF INITIAL PLAINTIFFS’ ORIGINAL AND SUPPLEMENTAL PRODUCTIONS [ECF 465], BOTH AS RELATED TO DEFENDANTS’ MOTION TO COMPEL DOCUMENT DISCOVERY AND RELATED METRICS [ECF 374]
WHEREAS, on May 2, 2023, the Court appointed Dr. Maura R. Grossman as Special Master “to address and resolve all issues raised in Defendants’ [referring to Walgreen Co. and Walgreens Boot Alliance, Inc.’s] Motion to Compel Document Discovery and Related Metrics [ECF 374] (the “MTC”), including asserted issues related to (i) the Initial Plaintiffs’ use of technology-assisted review (“TAR”), and (ii) the Initial Plaintiffs’ collection and production of documents from additional custodians.” Order Appointing Special Master [ECF 450], at 2;
WHEREAS, on July 21, 2023, the Special Master issued an Order on Preliminary Diagnostic Protocol to Assess the Adequacy of Initial Plaintiffs’ Original and Supplemental Productions ([ECF 465] (the “PDP”);
WHEREAS the Initial Plaintiffs complied with the PDP as ordered, and the Special Master performed the calculations set forth in the Appendix to the PDP based on information supplied by Initial Plaintiffs, as well as the responsiveness coding provided by Initial Plaintiffs with respect to certain samples (collectively, the “PDP Sample”), as provided for in the PDP;
WHEREAS, on September 16, 2023, the Special Master provided the parties with several worksheets containing the information contemplated in the Appendix of the PDP (the “Validation Worksheets”) (attached hereto as Exhibits 1 through 7);
WHEREAS the Validation Worksheets indicated an overall recall of 11.3% for all Initial Plaintiffs combined for the Original Collection as of August 22, 2022; an overall recall of 23.2% for all Initial Plaintiffs combined for the Original Collection after August 22; an overall recall of 21.5% for all Initial Plaintiffs combined for the Supplemental Collection; and an overall recall of 22.8% for all Initial Plaintiffs combined for the Combined Collection;
WHEREAS the results of the PDP suggest that a substantial number of additional responsive documents likely remain within the documents collected for the previous (agreed-upon) custodians;
WHEREAS, while the Initial Plaintiffs do not entirely agree with the methods and/or results of the PDP and/or the Special Master’s conclusions therefrom, and do not waive their rights to challenge them at a later date that may be beyond the deadline established in Fed. R. Civ. P. 53(f), the Order Appointing Special Master [ECF 450], or any other applicable rules, the Parties have nonetheless agreed that the Initial Plaintiffs will voluntarily engage in this Remediation Protocol designed to identify and lead to the production of additional responsive documents remaining within the documents collected for the previous (agreed-upon) custodians;
WHEREAS the Parties have also agreed that Initial Plaintiffs will produce responsive documents from thirteen (13)[1] additional custodians identified in Appendix A to Walgreens’ October 13, 2023 “Proposed Stipulation Regarding Remediation of Deficiencies in Initial Plaintiffs’ Production,” set forth at Footnote 2 below (the “Additional Custodians);[2]
WHEREAS no Party waives any arguments regarding compliance with its obligations under applicable rules, court orders or the appropriateness of cost-shifting or any other potential “collateral relief” as may later be sought by any Party;
WHEREAS nothing in this Order overrides the Parties’ prior agreements or the Court’s orders regarding non-waiver and claw-back of privileged information, confidentiality protections, privilege log requirements, or any other aspect of the collection, handling, or production of discovery materials;
WHEREAS all Parties reserve all rights to seek modification of the time limitations or other provisions of this Order from the Special Master for good cause;
WHEREAS it is contemplated that successful completion of this Remediation Protocol by the Initial Plaintiffs, as jointly determined by the Parties and if not, as ultimately determined by the Special Master (and if appealed, by the Court), shall resolve, in full, the MTC [ECF 374] and the issues raised in Defendants’ Memorandum in Support of the Motion to Compel [ECF 376], provided that within fourteen (14) days of when the Remediation Protocol is determined to have been successfully completed, as set forth in Paragraph 21 below, the Parties shall jointly file with the Court a proposed revised Scheduling Order (or competing proposals if they cannot agree) that will (i) take into account the time needed by Defendants to complete their review of any documents newly produced as a result of this Remediation Protocol, and (ii) include a schedule for the Parties to brief any motion for collateral relief that any Party may choose to make.
IT IS HEREBY ORDERED AS FOLLOWS:
1. Initial Plaintiffs shall take the steps set forth below to supplement their Original Production, as of August 22, 2022, their subsequent Supplemental Production, and the new production from the Additional Custodians.
THE STEP-ONE TRAINING SET
2. Defendants shall review the 6,000 documents in the PDP Sample for responsiveness under the same “blind review” and other parameters set forth in the PDP and this Order; in other words, Defendants shall have no knowledge of how the 6,000 documents were previously coded by the Initial Plaintiffs and shall code them de novo. Responsiveness shall be defined by Initial Plaintiffs’ Responses and Objections to Defendants’ Requests for Production and the December 24, 2021 letter to Defendants from counsel for the Initial Plaintiffs. Initial Plaintiffs may withhold documents they have previously identified as potentially privileged from the Step-One Training set, subject to the conditions set forth in Paragraph 4 below. For the purpose of this review, Initial Plaintiffs will arrange for their vendor (Consilio) to provide a secure workspace for Defendants to make their coding determinations, subject to the terms outlined in one or more separate agreement(s) to be negotiated between the Parties and Consilo.
3. Defendants shall also be provided with and review a random sample of up to 300 documents for each of the Additional Custodians3 (together with the PDP Sample, the “Step-One Training Set”). Prior to the drawing of these random samples, Initial Plaintiffs shall apply the same steps previously agreed to between the Parties regarding domain, date range (August 1, 2006 to Present, unless otherwise specified), and search-string parameters to the Additional Custodian collection as Initial Plaintiffs applied to the prior collections (as referenced in Plaintiffs’ Proposed Search Strings, [ECF 377-4], at 7-12). If any Additional Custodian’s document set after processing, date restriction, and search-term culling consists of less than 300 documents (including family members), the full post-processing document set for that custodian shall be deemed sufficient and added to the Step-One Training Set.
4. Initial Plaintiffs are permitted, but not required, to withhold from the Step-One Training Set any documents that hit on their Special Master-approved privilege search-term filter, plus any of their attachments or family members, for further privilege review. Prior to the Initial Plaintiffs’ application of any privilege search-term filter, the Special Master shall review in camera the Initial Plaintiffs’ potential privilege filter to determine whether the filter contains only such terms that are appropriate for its intended purpose and, if not, the Special Master will instruct the Initial Plaintiffs as to the appropriate revisions to that filter. Initial Plaintiffs shall implement the Special Master’s instructions in full.
5. Defendants shall complete their assessment of the Step-One Training Set no later than five (5) weeks after receiving access to it, which shall occur no later than two (2) business days following notification by the Initial Plaintiffs that the Step-One Training Set is ready for Defendants to review. Should Defendants in good faith determine that they cannot complete their review of the Step-One Training Set in the specified time frame, they will notify the Initial Plaintiffs and the Special Master of their revised time estimate no later than two weeks after receiving access to the Step-One Training set. The Special Master may for good cause approve the extension.
6. During this same five-week period, Initial Plaintiffs shall review any documents they withheld as potentially privileged from the Step-One Training Set, make final privilege determinations, and make available to Defendants any non-privileged or privilege-redacted documents (the “Step-One Training Set Privilege Review”). Initial Plaintiffs shall produce a privilege log in accordance with Section 6 of the ESI Protocol [ECF 140], but also including the identity of the Initial Plaintiff that is asserting the privilege claim as to each document, for all redacted and withheld documents on or before the close of this five-week period.
7. If the PDP Sample contains less than 5,500 documents at the close of the Step-One Training Set Privilege Review, Initial Plaintiffs shall patch the deficit with a random selection of documents that are not potentially privileged, so that the PDP Sample size is restored to no less than 5,500 documents. If the sample size for any Additional Custodian contains less than 250 documents at the close of the Step-One Training Set Privilege Review, Initial Plaintiffs shall patch the deficit with additional documents randomly selected from the given Additional Custodian’s files that are not potentially privileged, to the extent available, so that the sample size for each Additional Custodian is restored to no less than 250 documents per custodian.
8. Defendants shall notify Initial Plaintiffs of any challenges to the privilege log no later than seven (7) business days after receipt of the privilege log, or any portion thereof, if it is produced on a rolling basis. The Parties agree to use their best efforts to resolve all challenges within ten (10) business days after receiving notice of such challenge(s) and Defendants may promptly raise any privilege disputes that the Parties are unable to resolve within the ten-business-day period as follows. It is presumed that the Court shall resolve such privilege disputes, but if the volume or timing of the disputes will result in substantial delay to the completion of this Remediation Protocol, the Special Master may resolve such privilege disputes. If the resolution of those challenges results in additional documents made available to Defendants, those documents shall be evaluated for responsiveness in the same manner as all other documents in the Step-One Training Set.
9. After the review of the Step-One Training Set is completed, Defendants shall allow Initial Plaintiffs to have access to Defendants’ responsiveness coding determinations for the Step-One Training Set (whether responsive, non-responsive or technical issue), but not to any work product (e.g., issue tagging, reviewer notes, etc.). At the same time, Initial Plaintiffs shall make available to Defendants the current TAR ranking scores of all documents in the Step-One Training set.
10. Initial Plaintiffs’ vendor (Consilio) shall propagate Defendants’ responsiveness coding of the Step-One Training Set to any duplicates outside of that set by matching MD5 hash values (or by using a similarly accepted methodology to identify exact duplicates).
11. Initial Plaintiffs’ vendor (Consilio) shall update the training of the existing TAR model based solely on Defendants’ responsiveness determinations on the Step-One Training Set; none of the Initial Plaintiffs’ prior coding shall be used to update the TAR model. Rankings derived from the new model shall be applied to the combined Original and Supplemental Collections (approximately 4,779,000 documents), as well as the Additional Custodian Collection (total to be determined by the Initial Plaintiffs and provided to Defendants and the Special Master once known) (together, the “Updated Review Set”).
12. Initial Plaintiffs shall produce to Defendants all documents in the Step-One Training Set that were coded as responsive by Defendants and not previously produced, plus their family members, following reconciliation of all privilege and duplication issues.
THE STEP-TWO TRAINING SET
13. The top-ranked 100,000, unproduced documents from the Updated Review Set (together with any attachments or family members), that do not hit the Initial Plaintiffs’ Special Master approved privilege search term filter (the “Step-Two Training Set”), will promptly be made available to Defendants. Defendants will then code the Step-Two Training Set for responsiveness in good faith using the same protocol described above in Paragraph 2. Defendants shall complete their assessment of the Step-Two Training Set no later than nine (9) weeks after receiving access to it, which shall occur no later than two (2) business days following notification by the Initial Plaintiffs that the Step-Two Training Set is ready for Defendants to review. Should Defendants in good faith determine that they cannot complete the review in the specified time frame, they will notify the Initial Plaintiffs and the Special Master of their revised time estimate no later than two weeks after receiving access to the Step-Two Training Set. The Special Master may for good cause approve the extension.
14. Initial Plaintiffs shall submit a proposal to the Defendants and the Special Master indicating how documents from the Additional Custodian collection that were deemed unsuitable for TAR (if any) (“Non-TARable Documents”) will be assessed by Initial Plaintiffs for responsiveness and when any responsive, non-privileged documents from that set will be produced. Any disputes involving the Non-TARable Documents that cannot be resolved within ten (10) business days shall be raised with the Special Master for a ruling. In addition, Initial Plaintiffs shall timely produce a privilege log for the Non-TARable Documents in accordance with Section 6 of the ESI Protocol [ECF 140], but also including the identity of the Initial Plaintiff that is asserting the privilege claim as to each document. The Parties shall meet and confer to resolve any questions or disagreements over the treatment or non-treatment of these documents. Defendants shall notify Initial Plaintiffs of any challenges to the privilege log no later than seven (7) business days after receipt of the privilege log, or any portion thereof, if it is produced on a rolling basis. The Parties agree to use their best efforts to resolve all challenges within ten (10) business days after receiving notice of such challenge(s) and Defendants may promptly raise any privilege disputes that the Parties are unable to resolve within the ten-business-day period as follows. It is presumed that the Court shall resolve such privilege disputes, but if the volume or timing of the disputes will result in substantial delay to the completion of this Remediation Protocol, the Special Master may resolve such privilege disputes.
15. Concurrent with Defendants’ review of the Step-Two Training Set, Initial Plaintiffs shall review the documents withheld as potentially privileged from the Step-Two Training Set, make final privilege determinations, and make available any non-privileged or privilege-redacted documents a rolling basis to Defendants. Initial Plaintiffs shall produce a privilege log in accordance with Section 6 of the ESI Protocol [ECF 140], but also including the identity of the Initial Plaintiff that is asserting the privilege claim, for all redacted and withheld documents before the close of Defendants’ date for completing the review of the Step-Two Training Set.
16. When they have completed their assessment of the Step-Two Training Set, Defendants will make available their responsiveness determinations to Initial Plaintiffs, in the same manner as set forth in Paragraph 9, above. Likewise, at the same time, the Initial Plaintiffs will make available to Defendants the current TAR ranking scores of all documents in the Step-Two Training Set.
17. Defendants shall notify Initial Plaintiffs of any challenges to the privilege log no later than seven (7) business days after receipt of the privilege log, or any portion thereof if produced on a rolling basis. The Parties agree to use their best efforts to resolve all challenges within ten (10) business days after receiving notice of such challenge(s) and Defendants may promptly raise any privilege disputes that the Parties are unable to resolve within the ten-business-day period as follows. It is presumed that the Court shall resolve such privilege disputes, but if the volume or timing of the disputes will result in substantial delay, the Special Master may resolve the privilege disputes. If the resolution of those challenges results in making additional documents available to Defendants, those documents will be evaluated for responsiveness in the same manner as the other documents in the Step-Two Training Set.
18. At the close of the privilege challenge period and/or resolution of privilege challenges, Initial Plaintiffs’ vendor will update the TAR model based on Defendants’ responsiveness coding, adjusted to include any additional documents made available as a result of the privilege challenge process. Ranking derived from that updated model will be applied to recalibrate the previous ranking of the documents in the Updated Review Set. After the model is updated , Initial Plaintiffs will generate a ranking report like that attached hereto at Appendix A and provide that report to Defendants and the Special Master.
19. Initial Plaintiffs will produce to Defendants documents in the Step-Two Training Set coded as responsive by Defendants and not previously produced, plus any family members, following reconciliation of all privilege issues.
POTENTIAL STEPS “THREE PLUS” TRAINING SETS
20. Using the Appendix A report described in Paragraph 18 as a guide, the Parties and the Special Master will meet to discuss whether additional training and review is warranted based on the information contained in the report, the marginal precision of the lowest-scoring documents contained in the Step-Two Review Set, and whether a substantial number of non-marginal, non-duplicative responsive documents are estimated to remain in documents not yet produced to Defendants. If that appears to be the case, a Step-Three Training Set will be generated and reviewed in the same manner as set forth above for the Step-Two Training Set. If it is determined that there is no need for further training and review, the Parties and the Special Master shall jointly determine what final validation testing should be performed at this point (the “Final Validation Protocol”). If the Parties cannot agree on a reasonable and proportionate Final Validation Protocol, the Special Master shall devise one.
21. Once the Final Validation Protocol has been successfully completed to the satisfaction of the Parties, or if they cannot agree, the Special Master (and if appealed, the Court), the Remediation Protocol shall be deemed completed.
22. This Remediation Protocol shall not be considered complete while any unresolved privilege disputes relating to any Training Set or Non-TARable Documents review described in this Remediation Protocol are still pending, or while any privilege logs from Initial Plaintiffs relating to any Training Set or Non-TARable Documents review described in this Remediation Protocol are outstanding.
23. It is impossible to anticipate and address in advance every potential issue that may arise during a TAR review process. Should any unanticipated circumstances arise in the course of implementing this Remediation Protocol, any Party may raise the issue with the other Party. The Parties shall meet and confer in good faith to resolve the issue. If they are unable to do so within ten (10) business days, the Parties shall raise the issue with the Special Master for resolution.
SO ORDERED.
While the Special Master was drafting this Order, the Parties agreed to dismiss the claims, counterclaims, and third-party claims involving the Cambia Plaintiffs (i.e., Cambia Health Solutions, Inc.; Regence BlueShield of Idaho, Inc.; Regence BlueCross BlueShield of Oregon; Regence BlueCross BlueShield of Utah; Regence BlueShield; and Asuris Northwest Health). An Order of Dismissal was entered by the Court on Jan. 2, 2024 [Dkt. 506]. The dismissal of the Cambia Plaintiffs from this action implicates six (6) of the thirteen (13) Additional Custodians. In addition, it is my understanding that there are no documents for one custodian whose employment with Blue Cross and Blue Shield of Massachusetts was terminated in 2015. Accordingly, the number of Additional Custodians agreed to by the Parties and/or ordered by the Special Master is now six (6), rather than thirteen (13).