Plaintiffs and Defendants 3M Company, Aearo Technologies LLC, Aearo Holdings, LLC, Aearo Intermediate, LLC, Aearo, LLC, and any of their related or affiliated entities or individuals named as defendants herein (collectively, “3M Defendants”) submitted the following Protocol to conduct Technology Assisted Review (“TAR”) of certain electronically stored information (“ESI”). The Court hereby adopts the parties’ Protocol as follows.

By agreeing to use TAR in this MDL and to follow this TAR Protocol, the 3M Defendants do not acknowledge or concede that TAR is appropriate, or that they are obligated to use TAR in any other matter, including, without limitation, all earplug-related matters pending in any state courts. Moreover, by agreeing to use TAR in this MDL and to follow this TAR Protocol, the 3M Defendants do not intend to waive any rights or protections pursuant to privacy, confidentiality, attorney-client privilege, attorney work product, and any other privileges, protections, or objections to discovery (individually, “Privilege”; collectively, “Privileges”). The 3M Defendants preserve all such Privileges.

The 3M Defendants reserve the right to withhold from production and/or to redact any document that contains information subject to any appropriate objections or protections, including, without limitation, any Privilege. The 3M Defendants intend for all information and documents produced pursuant to this TAR Protocol to be subject to the Order Governing Production of Documents and Electronically Stored Information and the Joint Stipulated Order for Preservation of Documents and Electronically Stored Information entered in this MDL.

Except as explicitly set forth herein, nothing in this Order supersedes any rights or obligations of the Parties pursuant to Pre-Trial Order No. 10 (Order Governing Production of Documents and Electronically Stored Information), including Section IV thereof.

1. TAR Tool

The Parties will use Xact Data Discovery’s licensed TAR tool Relativity Active Learning (version 9.5.370.136), herein referred to as Text Classification or TAR Tool. Text Classification leverages machine learning and natural language processing techniques to automatically assign a Classification Score to each document in the review population. The predictive models used to generate these scores are trained through supervised learning, meaning they are built based on the coding of human-reviewed documents. The TAR tool automatically selects documents likely to be relevant or improve the predictive model and presents those documents to reviewers on an ongoing basis, continually updating the predictive model based on the results of that human review. The updated model is used to recalculate the Classification Scores of the document population, and the process is continually repeated until all likely relevant documents have been reviewed, as determined by a score threshold determined by the Parties, or by an agreed-upon elusion test to validate the accuracy of the model.

2. Corpus

(a) The Corpus shall include populations of (i) documents associated with or related to the custodians reasonably believed to possess relevant information identified by the Plaintiffs or the 3M Defendants, agreed to by the Parties, or ordered by the Court, and (ii) documents from non-custodial sources reasonably believed to possess relevant information identified by the Plaintiffs or the 3M Defendants, agreed to by the Parties, or ordered by the Court.

(b) The Corpus shall not include any documents lacking adequate text to subject them to TAR-based text classification, including Excel documents and nontext based documents, e.g., image files, audio files, video files, CAD files, etc. 

(c) The 3M Defendants will disclose to Plaintiffs the total number of documents in the Corpus when created, and a provisional estimate (subject to updates as may be required to facilitate the parties’ conferrals hereunder) of the number of documents being reviewed outside of the TAR process. The 3M Defendants will disclose to Plaintiffs if documents are added to or deleted from the Corpus at a later date, and will disclose to Plaintiffs the number of documents added to or deleted from the Corpus and the reason for the addition or deletion. Defendants will also disclose counts of any documents batch-coded as non-responsive without individual review, and the reasons for any such batch-codings.

(a) The Parties’ TAR approach will include the use of a Sample Set to help assure that the Parties have a sufficient level of agreement on what constitutes responsiveness and non-responsiveness.

(b) The Sample Set will be created by drawing a simple random sample of 1,750 documents from the Corpus.

(c) The 3M Defendants will conduct the first review of the Sample Set, and code each document as Responsive or Not Responsive and Privileged or Not Privileged. The 3M Defendants will provide Plaintiffs with access to all NonPrivileged documents in the Sample Set with text files and corresponding native files, images and metadata in a format that complies with the entered ESI protocol, and provide a privilege log for any documents withheld for privilege reasons. Documents coded as Privileged will not be removed from the Sample Set for the purposes of model training. Plaintiffs’ Leadership may designate up to five individuals (plus up to three consultants) to aid in the review (“Plaintiffs’ Designated Sample Set Reviewers”) to be given access to review the Sample Set. This review may take place at such dates and times as the Parties mutually agree (a) at the offices of Kirkland & Ellis, LLP, or at such other location or locations mutually agreed by the Parties, or (b) via a secure web-based viewer. Any documents coded Not Responsive by the 3M Defendants to which Plaintiffs’ Designated Reviewers are provided access as part of this review are provided for the limited and sole purpose of raising and resolving disagreements, if any, regarding the coding calls made by the 3M Defendants. Any such disagreements shall be recorded on a TAR Protocol Classification Dispute Log (the “Log”), which shall be in a form agreed upon by the Parties. Once Plaintiffs’ Designated Reviewers complete their review of the Sample Set, the Parties shall meet and confer to resolve any differences in coding designation. The Parties’ review of the Sample Set, and conferral to resolve any differences in coding designation, shall be completed by July 26, 2019. If resolution cannot be reached, the issue shall be submitted to the Court for resolution.

(d) Plaintiffs’ Designated Reviewers shall not remove from the offices of Kirkland & Ellis, LLP, or other location or locations mutually agreed to by the Parties, any documents to which they are provided access in connection with the Sample Set review process, and shall not copy, record, print, image, photograph, fax, scan, or otherwise capture, transmit or share any document or any information contained therein, in any way, with anyone beyond the other Plaintiffs’ Designated Reviewers conducting the review. Any document in native format accessed via secured web browser and opened in a native application by any Plaintiffs’ Designated Reviewer will be immediately deleted after it has been viewed in the native application.

(a) The 3M Defendants will start the TAR tool’s active learning process with the Parties’ agreed-upon coded documents from the Sample Set, and such other documents as the Parties mutually agree, after which the system will automatically select documents for review based on predicted relevance and also, to allow the TAR tool to learn from all possible definitions of relevance, select for review some nonhighly ranked documents and documents about which the TAR tool is uncertain.

(b) Based on the reviewer coding of other documents, each document will eventually receive a score from 0-100 (the “Classification Score”), with documents with a Classification Score closer to zero being predicted as less likely to be relevant and documents with a Classification Score closer to 100 being predicted as more likely to be relevant.

(c) The predictive model the TAR tool uses will update itself continuously without need for human intervention.

(d) The TAR tool may automatically exclude duplicative content from the review queue in order to avoid delays in training the model. These documents will still receive a Classification Score. Suppressed documents above or equal to the lowest score of any document classified as responsive by a reviewer will be reviewed at the end of the process.

(a) Following initial training, the 3M Defendants will continue to review the prioritized review queue, as prioritized and delivered by the Relativity Active Learning tool, consistent with the coding designations agreed to among the Parties with the Sample Set. After sufficient review of the prioritized review queue such that the 3M Defendants believe that further training is unlikely to yield benefit to the TAR model, the 3M Defendants shall notify Plaintiffs that they believe it is appropriate to freeze training and declare a Classification Cutoff. The Parties shall cooperate in the exchange of information and subsequent conferral to determine whether it is appropriate to set a Classification Cutoff at that point.

(b) Upon reasonable agreement of the Parties, the 3M Defendants will conduct an elusion test using the TAR tool as follows:

(i) The training process shall be frozen.

(ii) The Classification Cutoff will be set at the lowest score of any document manually classified as responsive.

(iii) A simple random sample of 1,750 documents will be drawn from those un-reviewed documents in the Corpus with a Classification Score below the Classification Cutoff (the “Elusion Test Sample”).

(iv) The Elusion Test Sample will be reviewed by the 3M Defendants to determine the responsiveness of the documents in the Elusion Test Sample.

(c) Once the elusion test has been run, the 3M Defendants will report the following to the Plaintiffs:

(i) The Classification Cutoff that was used in connection with drawing the Elusion Test Sample.

(ii) The estimated elusion rate (the number of responsive documents in the Elusion Test Sample divided by the number of documents in the Elusion Test Sample).

(iii) The number of documents in the Elusion Test Sample.

(iv) The number of documents from the Corpus determined by reviewers to be responsive.(v) The number of documents from the Corpus determined by reviewers to be non-responsive.

(vi) The number of un-reviewed documents from the Corpus below the Classification Cutoff.

(vii) The number of un-reviewed documents from the Corpus above the Classification Cutoff, if any.

(viii) The number of responsive documents (as determined by manual review) among the 1,000 documents in the prioritized review queue that were reviewed immediately prior to the freezing of training and proposing the Classification Cutoff.

(d) Plaintiffs’ Designated Reviewers shall have the opportunity to review all of the documents included in the Elusion Test Sample, without any knowledge of how any individual documents were coded by 3M Defendants, in order to perform a blind comparison of the recall estimates provided.

(e) This review may take place (a) at the offices of Kirkland & Ellis, LLP, or at such other location or locations mutually agreed by the Parties, on a date and time to be agreed to by the Parties, or (b) via a secure web-based viewer on a date and time to be agreed to by the Parties. Any documents coded Not Responsive by the 3M Defendants to which Plaintiffs’ Designated Reviewers are provided access as part of this review are provided for the limited and sole purpose of raising and resolving disagreements, if any, regarding the coding calls made by the 3M Defendants. In connection with the Parties conferral, the 3M Defendants shall provide Plaintiffs with the Classification Score for the documents in the Elusion Test Sample. Any such disagreements shall be recorded on a TAR Protocol Classification Dispute Log (the “Log”), which shall be in a form agreed upon by the Parties. Once Plaintiffs’ Designated Reviewers complete their review of the Elusion Test Sample, the Parties shall meet and confer to resolve any differences in coding designation. If resolution cannot be reached, the issue shall be submitted to the Court for resolution.

(f) If the number of responsive documents, or the character or nature of such documents designated as responsive, in the Elusion Test Sample indicates that the TAR tool’s model of responsiveness was too limited, then Plaintiffs and 3M Defendants will discuss potential remedial action to locate an adequate proportion of the remaining relevant documents in the null set by additional rounds of training by the Parties, including selecting a deeper Classification Cutoff or supplemental and prioritized training of the TAR model. 3M Defendants reserve the right to object to such potential remedial action as inappropriate or unnecessary.

(g) If the elusion test confirms that the review has been sufficiently thorough, then 3M Defendants will review all un-reviewed documents with a predicted relevance at or above the Classification Cutoff (or, if 3M Defendants so elect, produce such documents un-reviewed). For the avoidance of doubt, documents identified for manual review in accordance with Section IV of Pre-Trial Order No. 10 (Order Governing Production of Documents and Electronically Stored Information) shall be reviewed (or, if the 3M Defendants so elect, produced unreviewed) without regard to whether the Classification Score assigned to any such document is at or above the Classification Cutoff.

(h) To the extent that the 3M Defendants conduct elusion tests prior to setting the proposed Classification Cutoff, the 3M Defendants shall provide information concerning each such earlier elusion testing, including date/time of the elusion test, the size of the below-cutoff population sampled from, the number of documents in the elusion test sample, and the estimated elusion rate (the number of responsive documents in the elusion test sample divided by the number of documents in the elusion test sample).