Counsel

This Protocol Regarding Validation of Technology-Assisted Review (“TAR”) (hereinafter, “TAR Validation Protocol”) shall govern Plaintiffs and Teva Pharmaceuticals USA, Inc., Teva Pharmaceutical Industries Ltd., Actavis LLC, Actavis Pharma, Inc., and Arrow Pharm (Malta) Ltd. (collectively, the “Teva Defendants”) in the above-captioned matter, in accordance with the Court’s Order filed December 2, 2020 (Document 647), and this Protocol shall be memorialized in a Court Order. This TAR Validation Protocol shall serve as a supplement to Case Management Order No. 8 (Electronic Discovery Protocol – Stipulated) [Dkt. 127], which shall remain in full force, except to the extent modified or superseded herein.

The Teva Defendants do not waive any rights or protections pursuant to privacy, confidentiality, attorney-client privilege, attorney work product, or any other privileges, protections, or objections to discovery that the Teva Defendants may have (individually, “Privilege”; collectively, “Privileges”). The Teva Defendants preserve all such Privileges. The Teva Defendants reserve the right to redact and/or to withhold from any production any document that contains information subject to any appropriate objections, including, without limitation, any Privilege or Privileges, subject to the Confidentiality and Protective Order entered in this matter on June 26, 2019 [Dkt. 139].

I. THE TAR TOOL

The Teva Defendants represent the following. They intend to use Brainspace Continuous Multimodal Learning (“CMML”) (version 6.2.8), which uses a continuous active learning (“CAL”) process to automatically assign a classification score to each document in the review population, and to rank the documents in the collection from most to least likely to be relevant.[1] The predictive models used to generate these scores are developed through a process referred to as supervised machine learning, meaning that they are built based on the coding of humanreviewed documents. To start the TAR review process, CMML is used to select, for human review, a set of documents that are conceptually diverse and representative of the entire dataset. Once the first iteration of classification scores is generated, human reviewers review batches of documents that are deemed the next most likely to be responsive based on their classification scores. The CMML model is updated, iteratively, based on the review of the highest scoring as-of-yetunreviewed documents, continuously improving the predictive model based on the results of the latest human review. This process is repeated until as many responsive documents as reasonably possible have been identified, as determined by the Review Stopping Criteria set forth below in Section VI. The review process includes and is followed by a validation process in order to seek to identify the maximum number of relevant and responsive documents, and derive an estimate of recall, showing the proportion (i.e., percent) of responsive documents in the review population that have been identified through the TAR process, and the estimated proportion and nature (i.e., the uniqueness and importance, or the duplicativeness and marginality) of any documents that have been missed.

II. COOPERATION

For the purposes of avoiding protracted discovery disputes in this proceeding, but without requiring the disclosure of any Privileges, the Teva Defendants have agreed to provide the Plaintiffs with certain non-privileged documents and information set forth below that they would not ordinarily disclose in the course of discovery. Such documents and information shall be subject to the Confidentiality and Protective Order entered in this matter on June 26, 2019 [Dkt. 139]. The Parties agree to meet and confer in good faith to resolve any disputes that may arise in the course of the TAR review process or this TAR Validation Protocol, and if they are unable to do so, they will promptly raise such matters with the Court for resolution.

III. SCOPE OF RELEVANCE

It is important that the Parties agree on the objective of the TAR exercise (i.e., the scope or categories of documents to be included or excluded by the use of CMML). The Parties agree on the scope of relevance, which is governed by both (1) relevance to the claims and defenses in this action, and/or (2) responsiveness to Plaintiffs’ discovery requests (modulo the Teva Defendants’ responses and objections), which is deemed relevant by definition.

IV. TAR REVIEW POPULATION

The review population to which CMML shall be applied (the “TAR Review Population”) shall include the documents of the thirty-six (36) identified custodians (and any others to be added at any point in time) for the time period of January 1, 2012 to present, as previously negotiated and agreed to by the Parties at an in-person conference held on December 11, 2019 and confirmed in the Court’s December 23, 2019 Order [Dkt. 328] (for the applicable custodians), and in the Court’s November 25, 2019 Order [Dkt. 303] confirming the “macro” discovery arguments argued to the Court at the November 20, 2019 Case Management Conference (for the applicable time period). The Teva Defendants represent that the TAR Review Population is reasonably believed to contain information that is (1) relevant to the claims and defenses in this action and/or (2) responsive to Plaintiffs’ requests for production.

The Parties recognize that certain types or categories of documents may not be appropriate for TAR. These include, but are not limited to: photographs or images, chromatograms, mass spectrometry, schematics or drawings, spreadsheets and presentations, audio or video files, documents with handwritten notations, documents with too little text or Optical Character Recognition (“OCR”) quality such as to render the use of TAR ineffective, hard copy documents, contacts, and calendar entries. Such documents will be excluded from the TAR Protocol and will be analyzed for substantive content by human reviewers.

The Teva Defendants shall disclose to Plaintiffs the file types and extensions that are excluded from the TAR work flow and subject to manual analysis. The Teva Defendants may also remove obvious “junk” or non-responsive materials from the TAR Review Population (e.g., identified pools of logos, Amazon.com or ESPN.com emails, etc.) based on sender domain name(s) or address(es), file types, and similar characteristics. The Teva Defendants shall disclose to Plaintiffs the criteria used to identify such obvious “junk” or non-responsive materials.

The Teva Defendants shall not use keywords to cull or otherwise reduce the TAR Review Population, however the TAR Review Population shall be de-duplicated, and email threading shall not be applied to same notwithstanding Case Management Order No. 8 (Electronic Discovery Protocol – Stipulated) [Dkt. 127].

V. TAR TRAINING PROCESS AND REVIEW QUALITY CONTROL (“QC”)

As described in Section I above, following the human review of the initial training set selected by CMML, the TAR tool automatically applies a classification score to each as-of-yetunreviewed document in the collection and ranks the documents from most to least likely to be relevant. The human reviewers are not provided with any information about CMML’s classification scores for the documents when they review them. The coding of each document that is reviewed by a human is ultimately fed back into CMML and the tool’s predictive model iteratively updates as additional documents are reviewed, improving CMML’s ability to distinguish responsive from non-responsive documents. If a document presented for review cannot be reviewed because of a technical difficulty, it is coded “Technical Difficulty.” The Teva Defendants shall maintain a record of all documents coded as “Technical Difficulty” and shall provide a log of same to Plaintiffs. Documents coded “Technical Difficulty” are not fed back into CMML and are no longer considered as part of the TAR Review Population, and must be manually reviewed.

The Teva Defendants shall utilize Teva’s core discovery documents as training examples (with the exception of portions of the ANDA files to be determined), as soon as practicable. Within ten (10) business days of the execution of this TAR Validation Protocol, Plaintiffs may, but are not required to, provide to the Teva Defendants up to one hundred (100) documents for consideration as additional training examples for CMML (e.g. documents produced by Teva, documents produced by other Defendants). Thereafter, Plaintiffs may provide further sets of proposed training examples for CMML, for up to 30 days after this protocol is finalized. Should the Teva Defendants have an objection to any of the proposed training documents, the Parties shall meet and confer and if they are unable to resolve their disagreement within ten (10) business days, they shall promptly raise the issue with the Court for resolution. Within ten (10) business days after each rolling production is made, the Parties shall meet and confer to discuss any concerns the Plaintiffs may have with the progress of the training of CMML up to that point (if any).

To ensure that the human coding decisions are as accurate and consistent as possible, both the Teva Defendants’ vendor and its counsel shall perform continuous Quality Control (“QC”) of the human review process, including but not limited to, review of: (i) documents receiving a high classification score by CMML that are coded non-responsive by a human reviewer, (ii) documents receiving a low classification score by CMML that are coded responsive by a human reviewer; (iii) documents that are or appear to be inconsistently coded by human reviewers; and/or (iv) random samples of documents coded by human reviewers. Any document that was coded nonresponsive in error, or was otherwise not coded responsive in error, shall be re-coded responsive and fed back into CMML with the proper coding.

With each rolling production, Plaintiffs shall be provided: (1) the number of documents, and percentage of documents, for categories (i), (ii), and (iii); (2) the number of documents where a reviewer coded a document as responsive but the document was not produced (except for documents withheld based on assertion of privilege); (3) the percentage of reviewed documents found to be non-responsive; and (4) the percentage of the entire document set, and number of documents, that has/have not been reviewed.

VI. TAR REVIEW STOPPING CRITERIA

The TAR review process for each rolling production will continue until the Teva Defendants have reviewed at least 15% of the entire document set, and can reasonably conclude that further review is unlikely to yield additional responsive documents with sufficient quantity or materiality to justify continuing. The 15% minimum review requirement will permit a credit for the approximately 740,000 documents that Teva represents it has reviewed as of the November 30, 2020 production deadline, but will require a minimum of 460,000 documents of the highest scoring 15% of the entire document set to be newly reviewed (documents not reviewed previously) before consideration can be given to implementing the Stopping Criteria. This will not occur before the last batch of documents identified by CMML classification score and reviewed by humans contains no more than five percent (5%) responsive documents and none of the responsive documents is novel and/or more than marginally relevant.

At that point, the Teva Defendants will conduct an Elusion Sample to estimate the number of responsive documents that have been missed by CMML in that production. The size of the Elusion Sample shall be four hundred (400) documents drawn at random from the “null set,” which is comprised of the documents for which CMML classification scores did not suggest the need for human review (i.e., they received low classification scores). This sample size will provide an estimate of the number of documents missed at the 95% confidence level, with a margin of error of plus or minus five percent (5%). If any of the documents that are found in the Elusion Sample are novel and/or relevant, the TAR process will resume. If the Elusion Sample shows that substantially all relevant documents have been identified (i.e., the recall estimate obtained using that Elusion Sample is 80% or more), the review process shall cease for that particular rolling production. Plaintiffs will be provided the results of the Elusion Sampling, including copies of any responsive documents identified through the Elusion Sampling process, for any rolling production, which shall be promptly provided. Should the Parties disagree that, for any rolling production, the Teva Defendants have chosen a reasonable stopping point, the Parties shall meet and confer in good faith and if they are unable to resolve their disagreement within ten (10) business days, they shall promptly raise the issue with the Court for resolution.

The same Review Stopping Criteria and Elusion Sampling process shall be applied for each rolling production.

Teva’s Review and Elusion Sampling shall be completed, and Plaintiffs will be given Teva’s final list of non-responsive documents on or before January 15, 2021, and February 15, 2021, in connection with each rolling production.

VII. VALIDATION PROTOCOL

Once the final rolling production has been made, the Teva Defendants shall engage in the following Validation Protocol to demonstrate that their production is reasonable, made in good faith, and consistent with their obligations under Fed. R. Civ. P. 26(g).

A. Plaintiffs shall be allotted a budget of five thousand (5,000) documents for the purposes of an audit of the adequacy of the Teva Defendants’ entire production (i.e., consisting of all of the rolling productions). This sample of five thousand (5,000) documents shall be referred to as the “Audit Sample,” and shall be provided to Plaintiffs. Plaintiffs may use their Audit Sample to test whichever aspect(s) of the Teva Defendants’ review process they would like, from the categories set forth below (each an “Audit Category”). By way of example, only, Plaintiffs may request that a random sample of size n documents be drawn from certain of the categories set forth below, or they may request a sample of size n documents be drawn from the documents after application of a search query of their own choosing. At least ten business days prior to Plaintiffs having to select documents and searches for the Audit Categories, the Teva Defendants shall provide Plaintiffs with a hit count report run across the unreviewed set with the currently agreed to search terms. If requested by Plaintiffs the Parties shall meet and confer regarding the potential need for further reviews and hit count reports prior to initiating the Audit process. The total Audit Sample may be allocated among the following Audit Categories as Plaintiffs see fit, provided that the total Audit Sample shall not exceed five thousand (5,000) documents (including family members, if requested).

1. Audit Category 1 – A random sample of the documents that were either produced or withheld on the basis of Privilege [no minimum number of documents]

2. Audit Category 2 – A random sample of the documents selected by CMML, reviewed by a human reviewer, and coded as non-responsive [no minimum number of documents]

3. Audit Category 3 – A random sample of the documents excluded by CMML, not reviewed by a human reviewer, and not selected using any search query per the below Audit Categories [minimum of one thousand (1,000) documents]

4. Audit Category 4 – A sample of the documents excluded from review by CMML, not reviewed by a human reviewer, and selected by using the following search query [ABC] (either all of the “hits” if there are less than the number specified, or a random sample of the “hits” if there are more than the number specified) [no minimum number of documents]

5. Audit Category 5 – A sample of the documents excluded from review by CMML, not reviewed by a human reviewer, and selected by using the following search query [DEF], (either all of the “hits” if there are less than the number specified, or a random sample of the “hits” if there are more than the number specified), but not including any documents identified by the search query [ABC] [no minimum number of documents]

6. Audit Category 6 et seq. – A sample of the documents excluded from review by CMML, not reviewed by human reviewers, and selected by using the following search query [GHI] et seq., (either all of the “hits” if there are less than the number specified, or a random sample of the “hits” if there are more than the number specified), but not including any documents identified by any of the search queries previously employed in any other Audit Category [no minimum number of documents]

B. Plaintiffs shall provide the Teva Defendants with their request for Audit Samples no later than ten (10) business days following the last rolling production on February 15, 2021. The request shall include: (i) specification of the Audit Categories (as described in Paragraph VII.B above); (ii) the search query to be used (if any) for Audit Categories 4 et seq.; and (iii) the number of documents to be included in each Audit Category. The Parties shall meet and confer in good faith to resolve any questions or technical issues raised by the Plaintiffs’ request for Audit Samples.

C. The Teva Defendants shall have their vendor draw the Audit Sample, as specified in Paragraph VII.B by Plaintiffs, and provide the Audit Sample to Plaintiffs within fourteen (14) days after Plaintiffs’ designation. The vendor shall maintain a separate record of which documents were selected for which Audit Categories, and shall place the Audit Sample in a folder that contains no information about the Audit Category or Categories from which the documents came, how the documents were previously coded or whether they were previously produced or withheld and the basis therefor, or the classification score of the documents.

D. The Teva Defendants shall review and code the Audit Sample and shall provide a certification under oath that the reviewer(s) of its Audit Sample had no knowledge of the Audit Category or Categories from which the documents came, how the documents were previously coded or whether they were previously produced or withheld and the basis therefor, or the classification score of the documents.

E. The Plaintiffs shall notify the Teva Defendants of any disagreement with the Teva Defendants’ coding of any document in the Audit Sample. The Parties shall meet and confer in good faith over any such disagreements. Should the Parties be unable to resolve their disagreements on the coding of any document(s) in the Audit Sample, or any other issue stemming from the Audit Process within ten (10) business days, they shall promptly raise the issue with the Court for resolution.

F. Within five (5) business days of final agreement on the coding of the Audit Sample, the Teva Defendants shall complete and provide Plaintiffs with a table indicating for each document in the Audit Sample, the Audit Category or Categories to which it belongs, and whether it was: (i) produced in one of the rolling productions as responsive in its own right; (ii) produced in one of the rolling productions as a family member of a responsive document; (iii) withheld from production as privileged; (iv) withheld from production as non-responsive; (v) not identified as potentially responsive by CMML; or (vi) was not produced for any other reason(s), which reason(s) shall be provided, and the classification score of each document. Along with the table, the Teva Defendants shall provide Plaintiffs with a completed copy of the TAR Worksheet (attached hereto as Exhibit A).

G. The results set forth in the table described in Paragraph VII.G above and in the TAR Worksheet A (including the estimate of recall) should provide the Parties with sufficient information to determine whether the Teva Defendants have made a reasonable, good faith production consistent with their obligations under Fed. R. Civ. P. 26(g). If a problem is identified with respect to the Teva Defendants’ production as a whole, or with respect to any particular Audit Category, the Parties shall meet and confer in good faith to determine a suitable remediation procedure, which may involve re-running CMML again using the responsive document(s) identified during the Audit Process as new training documents, establishment of additional Audit Samples in accordance with the process herein, and reviewing the next-most-likely to be relevant documents suggested by CMML to confirm that there are no additional responsive documents, running some additional keyword searches, etc. If the Parties cannot agree on whether the production is adequate or on an appropriate remediation procedure within ten (10) business days, they shall promptly raise the issue with the Court for resolution.

H. Plaintiffs shall have the right to apply to the Court for further relief within sixty (60) days after the designated non-responsive documents are produced if plaintiffs’ review of the documents demonstrates that more than a minimal amount of materially relevant and non-duplicate or cumulative documents were designated as non-responsive, or if there are any alleged deficiencies in Teva’s ESI production.

EXHIBIT A

TAR WORKSHEET

Audit Category 1: All documents that were either produced or withheld on the basis of Privilege.

Number of documents in category: _____ [no minimum]
Sample size: _____
Number of responsive documents in sample: _____
Estimated number of responsive documents in category (number of responsive documents in sample ÷ sample size × number of documents in category): _____

Audit Category 2: All documents selected by CMML, reviewed by a human review, and coded as non-responsive.

Number of documents in category: __________ [no minimum]
Sample size: __________
Number of responsive documents in sample: __________
Estimated number of responsive documents in category (number of responsive documents in sample ÷ sample size × number of documents in category): __________

Audit Category 3: All documents excluded by CMML, not reviewed by a human reviewer, and not selected using any search query per the below Audit Categories.

Number of documents in category: __________ [minimum of 1,000]
Sample size: __________
Number of responsive documents in sample: __________
Estimated number of responsive documents in category (number of responsive documents in sample ÷ sample size × number of documents in category): __________
_____

[Repeat as necessary]

Audit Category 4: All documents excluded from review by CMML, not reviewed by a human reviewer, and selected by using the following search query [ABC] (either all of the “hits” if there are less than the number specified, or a random sample of the “hits” if there are more than the number specified).

Search query: _________________________________________________________________
Number of documents in category: __________ [no minimum]
Sample size: __________
Number of responsive documents in sample: __________
Estimated number of responsive documents in category (number of responsive documents in sample ÷ sample size × number of documents in category): __________

Audit Category 5: All documents excluded from review by CMML, not reviewed by a human reviewer, and selected by using the following search query [DEF], (either all of the “hits” if there are less than the number specified, or a random sample of the “hits” if there are more than the number specified), but not including any documents identified by the search query [ABC].

Search query: _________________________________________________________________
Number of documents in category: __________ [no minimum]
Sample size: __________
Number of responsive documents in sample: __________
Estimated number of responsive documents in category (number of responsive documents in sample ÷ sample size × number of documents in category): __________

Audit Category 6 et seq.: All documents excluded from review by CMML, not reviewed by human reviewers, and selected by using the following search query [GHI] et seq., (either all of the “hits” if there are less than the number specified, or a random sample of the “hits” if there are more than the number specified), but not including any documents identified by any of the search queries previously employed in any other Audit Category.

Search query: _________________________________________________________________
Number of documents in category: __________ [no minimum]
Sample size: __________
Number of responsive documents in sample: __________
Estimated number of responsive documents in category (number of responsive documents in sample ÷ sample size × number of documents in category): __________

Overall Estimates:
P = Estimated number of responsive documents either produced or withheld on the basis of Privilege: (estimate from Audit Category 1): __________
NP = Estimated number of responsive documents not either produced or withheld on the basis of Privilege: (sum of estimates for Audit Categories 2 et seq.): _________
Estimated Recall: P ÷ (P + NP) × 100% = __________

I certify under penalty of perjury that (i) the Teva Defendants’ Auditor(s) who coded the documents in the Audit Sample had no knowledge of the Audit Category or Categories from which the documents came, how the documents were previously coded or whether they were previously produced or withheld and the basis therefor, or the classification score of the documents, and (ii) that the answers provided in this TAR Worksheet are complete and correct to the best of my knowledge, information, and belief.

Footnotes