CALDERWOOD & HARKNESS LLP
INTERNAL MEMORANDUM
PRIVILEGED AND CONFIDENTIAL — ATTORNEY WORK PRODUCT This memorandum is protected by the attorney-client privilege and the work product doctrine. It is prepared in anticipation of litigation and reflects the mental impressions, conclusions, opinions, and legal theories of attorneys. Do not distribute or disclose without prior authorization from the Responsible Partner.
Date: August 12, 2024
FROM: Peter Vassiliev, Associate, Calderwood & Harkness LLP
TO: Christy Bauer, Lead Associate, Calderwood & Harkness LLP Marcus Delgado, Responsible Partner, Calderwood & Harkness LLP
RE: Technology-Assisted Review Protocol — DataVault Litigation Solutions, LLC Workflow for ESI Processing and Review in Apex Surgical Innovations, Inc. v. Solstice Biomedical, Inc., Case No. 1:24-cv-00587-RGA (D. Del.)
MATTER NO.: 10294.0017
CLASSIFICATION: PRIVILEGED AND CONFIDENTIAL — ATTORNEY WORK PRODUCT
Right-click to update Table of Contents
I. PURPOSE AND BACKGROUND
The purpose of this memorandum is to document in detail the agreed technology-assisted review ("TAR") protocol and workflow between Calderwood & Harkness LLP and DataVault Litigation Solutions, LLC ("DataVault") for the processing, reviewing, and producing of electronically stored information ("ESI") collected from Solstice Biomedical Devices Ltd. ("Solstice" or "SBD") and its wholly owned U.S. subsidiary, Solstice Biomedical, Inc., in connection with the defense of the above-captioned patent infringement action. This memorandum is intended to serve as the authoritative reference for all members of the litigation team — both internal and external — with respect to the end-to-end TAR workflow, from initial data ingestion through quality control and final production to opposing counsel.
DataVault Litigation Solutions, LLC is an e-discovery and litigation technology services provider with offices at 2000 Market Street, Suite 600, Philadelphia, PA 19103. DataVault has been retained to serve as the primary ESI processing and hosting vendor for this engagement. Their selection followed a competitive evaluation of three vendors, and was based on their demonstrated experience in patent litigation document reviews, competitive pricing, and the capabilities of their proprietary continuous active learning platform, DataVault Insight™.
Parties and Counsel. Solstice Biomedical, Inc. is the defendant in this matter. Apex Surgical Innovations, Inc. ("Apex") is the plaintiff, asserting infringement of three United States patents directed to laparoscopic surgical access devices. Apex is represented by Hargrove, Tate & Bennington LLP ("HTB"), located at 900 North King Street, Suite 1200, Wilmington, DE 19801, with Victoria Stanton serving as lead partner and Derek Loomis as the primary associate on the matter. Calderwood & Harkness LLP represents Solstice in all aspects of this litigation, with Marcus Delgado serving as Responsible Partner, Christy Bauer serving as Lead Associate, and the undersigned serving as the working associate responsible for day-to-day management of the discovery process.
Procedural Background. The Court — the United States District Court for the District of Delaware, with the matter assigned to Chief Judge Richard G. Andrews — entered its Scheduling Order on July 2, 2024, following the Initial Scheduling Conference held on June 28, 2024. The Joint discovery Plan was subsequently filed on July 15, 2024. The Joint discovery Plan sets forth the parties' agreed framework for ESI collection, processing, review, and production, including provisions governing production format, metadata fields, and the general acceptability of technology-assisted review methodologies. Both parties expressly agreed in the Joint discovery Plan that the use of TAR is an acceptable and proportionate method for identifying responsive documents, consistent with the requirements of Federal Rule of Civil Procedure 26(b)(1).
discovery Requests. Apex served its First Set of Interrogatories (comprising 18 interrogatories) and its First Requests for Production of Documents (comprising 42 requests) on August 5, 2024. The Requests for Production are broad in scope and encompass, among other things, all documents relating to the design, development, manufacture, testing, regulatory approval, marketing, sale, and distribution of the accused PrecisionEdge™ product lines. Solstice served its own First Set of Interrogatories (15 interrogatories) and First Requests for Production (38 requests) on August 19, 2024. The responsive discovery obligations of both parties are substantial and require a coordinated, efficient approach to document review.
Related Protocols. This memorandum should be read in conjunction with the companion Privilege Log Protocol & Guidelines memorandum dated August 1, 2024, prepared by Christy Bauer. That memorandum governs the identification, coding, and logging of documents subject to attorney-client privilege and work product protection within the production set. The privilege review workflow described in that companion memorandum is integrated into the TAR workflow described herein, as set forth more fully in Section VI.C below.
Production Timeline. Solstice's document production is anticipated to occur on or before September 30, 2024, as required by the Scheduling Order. Apex's production is anticipated on or about October 7, 2024. The timeline set forth in Section X of this memorandum has been developed to ensure that all processing, review, validation, and production formatting activities are completed in advance of the September 30 production deadline.
In sum, this memorandum establishes the complete end-to-end TAR workflow from data ingestion through quality control and final production, including the selection of the TAR methodology, the composition of seed and training sets, iterative training protocols, statistical validation procedures, quality control metrics, production specifications, cost estimates, and defensibility considerations. All members of the litigation team are expected to familiarize themselves with this protocol and to adhere to its requirements throughout the review process.
II. SCOPE OF ESI COLLECTION
A. Data Sources and Custodians
ESI has been collected from the following key custodians at Solstice Biomedical Devices Ltd. and Solstice Biomedical, Inc., as identified in the Litigation Hold Notice issued on May 11, 2024 (three days after service of the Complaint). The custodian list was developed in consultation with Solstice's General Counsel, Eamon Rourke, and was subsequently validated against the document requests served by Apex to ensure completeness. The custodians are as follows:
1. Niamh Cassidy, Vice President, Regulatory & intellectual Property — Ms. Cassidy serves as the primary custodian for patent-related and product development communications. Her files are expected to contain the most significant volume of documents relating to the design and development of the PrecisionEdge™ product lines, including communications with outside patent counsel, internal IP strategy documents, and correspondence with the U.S. Patent and Trademark Office. Ms. Cassidy also maintained extensive correspondence with Solstice's regulatory affairs team regarding FDA submissions for the accused products.
2. Dr. Fiona Tierney, Chief Executive Officer — A limited collection has been made of Dr. Tierney's high-level strategic communications, including board presentations, executive committee meeting materials, and communications with senior leadership regarding the PrecisionEdge™ product lines. The scope of collection from Dr. Tierney's custodial sources was narrowed by agreement with the client to focus on communications referencing the accused products, Apex, or the patents-in-suit.
3. Eamon Rourke, General Counsel — Mr. Rourke's legal department communications are a significant source of potentially privileged materials. His custodial files include communications with outside counsel (including Calderwood & Harkness LLP and prior counsel), internal legal opinions, litigation hold documentation, and contract files related to licensing discussions. Documents from Mr. Rourke's files will be subject to heightened privilege review in accordance with the August 1, 2024 Privilege Log Protocol.
4. Brendan Molloy, Chief Financial Officer — Mr. Molloy's files include financial records related to the accused products, including sales data, margin analysis, revenue projections, cost-of-goods-sold calculations, and pricing information. These documents are relevant to Apex's damages claims and to Solstice's potential defenses regarding the measure of damages.
5. Engineering and Product Development Team Leads for the PrecisionEdge™ Product Lines (PE-5000, PE-12000, and PE-15000 Series) — This group includes Dr. Patrick Hennessy (Lead Design Engineer, PE-5000 Series), Aoife Brennan (Lead Design Engineer, PE-12000 Series), and Tomás Flynn (Lead Design Engineer, PE-15000 Series). Their custodial files include CAD drawings, design specifications, engineering change orders, testing data, prototype documentation, and internal technical correspondence relating to the design and development of the accused products.
6. Regulatory Affairs Personnel Involved in FDA Clearance of Accused Products — This group includes Siobhán Doyle (Director of Regulatory Affairs) and Conor McHugh (Senior Regulatory Specialist). Their files include 510(k) submissions, FDA correspondence, clinical evaluation reports, biocompatibility testing data, and other regulatory submissions related to the PrecisionEdge™ product lines.
Data Sources. ESI has been collected from the following data sources across both Solstice entities:
• Email servers (Strathmore Data Systems / Ashbury Software), including all mailbox data, calendar entries, and attachments for each identified custodian
• Shared network drives, including departmental shared folders for the Engineering, Regulatory, Legal, Finance, and Marketing departments
• Local workstations of key custodians, where identified as containing unique, non-duplicative data
• Cloud storage (Sundial Systems), including project sites for the PrecisionEdge™ product lines and the company's document management repository
• Product design databases, including the company's product lifecycle management ("PLM") system
• ERP and financial systems, including Hollowell & Co. modules for sales, distribution, and cost accounting
In addition, data has been collected from Solstice Biomedical, Inc., the Delaware subsidiary located at 3800 Concord Pike, Suite 410, Wilmington, DE 19803. The subsidiary's data collection focused on U.S. sales and distribution records, marketing materials directed to U.S. healthcare providers, and correspondence with U.S.-based customers and distributors relating to the PrecisionEdge™ product lines.
B. Total Data Volume
The total volume of ESI collected and ingested by DataVault is approximately 1.2 terabytes (TB). This figure represents the raw data volume prior to any processing, de-duplication, or filtering.
After de-duplication (both exact-duplicate and near-duplicate identification), de-NISTing (removal of known system and application files using the National Institute of Standards and Technology's National Software Reference Library hash sets), and file-type filtering (exclusion of non-reviewable file types as described in Section II.C below), the estimated reduction is to approximately 900 GB of reviewable content.
Following full processing — including text extraction, metadata extraction, email threading, and document-family grouping — the estimated total document universe is approximately 312,000 documents. This figure includes both parent documents and their attachments, with email families grouped together for purposes of review. The 312,000-document figure represents the population against which the TAR classifier will be trained and applied.
C. Date Range and Search Parameters
Date Range. The general date range for collection is January 1, 2018 through May 9, 2024, which is the date the Complaint was filed. This date range was selected to encompass the relevant period of accused activity, including the design, manufacture, marketing, and sale of the PrecisionEdge™ product lines, as well as pre-suit communications between the parties and any licensing discussions.
For certain categories of documents, a broader date range has been applied. Specifically, for documents relating to the initial conception and design of the PrecisionEdge™ product lines — including early-stage product development files, industrial design concepts, and engineering feasibility studies — the collection date range extends back to January 1, 2015, which corresponds to the inception of the PrecisionEdge™ product development program at Solstice. This expanded date range is necessary to capture documents that may be relevant to the defense's non-infringement and invalidity theories, particularly with respect to independent development and the prior art landscape.
File Types Included. The following file types have been included in the collection and will be processed for review:
• Emails: .msg, .eml, .pst (Hollowell Systems formats)
• Office documents: .docx, .xlsx, .pptx, .pdf (including both native PDFs and scanned/OCR'd PDFs)
• CAD and engineering files: .dwg (Glenrock Technologies), .step and .stp (STEP files), .iges and .igs (IGES files)
• Image files: .jpg, .png, .tiff (including photographs, screenshots, and scanned documents)
• Database exports: .csv, .xml, and structured data exports from Hollowell & Co. and the PLM system
File Types Excluded. The following file types have been excluded from the reviewable population:
• System files (operating system files, temporary files, log files)
• Executable files (.exe, .dll, .bat, .com)
• Exact duplicates (identified by MD5 hash value matching)
• Near-duplicates (identified using DataVault Insight™'s near-duplicate detection algorithm, with a similarity threshold of 90%)
• NIST library files (files matching hash values in the NIST NSRL, representing known software and system files with no evidentiary value)
III. TAR PLATFORM AND METHODOLOGY SELECTION
A. Platform. DataVault will use its proprietary continuous active learning platform, DataVault Insight™ version 4.2, as the primary technology platform for all TAR-related processing and review activities. DataVault Insight™ is a cloud-hosted, browser-based platform with integrated document review, coding, analytics, and production capabilities. The platform has been validated through use in numerous federal and state court proceedings, and DataVault has provided declarations and testimony regarding the platform's methodology in prior matters before the U.S. District Courts for the District of Delaware, the Southern District of New York, and the Eastern District of Pennsylvania.
DataVault Insight™ supports predictive coding using machine-learning classifiers trained through iterative attorney-reviewed seed and training sets. The platform's classifier engine employs a logistic regression model with support vector machine ("SVM") supplementation, which has been shown to perform well on document populations containing technical and specialized vocabulary, such as those found in patent litigation. The platform assigns a relevance score between 0.00 and 1.00 to each document, with higher scores indicating a higher probability of responsiveness.
In addition to predictive coding, DataVault Insight™ supports the following supplementary analytical tools, which will be used as needed to augment the TAR workflow:
• Keyword search: Full-text Boolean keyword searching, including proximity operators, wildcard characters, and stemming
• Concept clustering: Automated grouping of documents based on semantic similarity, useful for identifying previously unknown categories of responsive documents
• Email threading: Identification and grouping of email conversations, enabling review of only the most inclusive message in each thread (reducing duplicative review effort)
• Near-duplicate detection: Identification of documents with substantially similar content (threshold set at 90% textual similarity), enabling grouped review and consistent coding
B. TAR Methodology: Continuous Active Learning (CAL)
The selected TAR methodology for this engagement is Continuous Active Learning ("CAL"), also commonly referred to as TAR 2.0. CAL has been selected over simple passive learning (TAR 1.0) following careful consideration of the characteristics of the document population in this matter.
The decision to employ CAL rather than TAR 1.0 was driven by several factors specific to this litigation:
• Large volume: The 312,000-document universe is sufficiently large that a static control-set approach (TAR 1.0) would require an impractically large control set to achieve statistically reliable richness estimates, and would not benefit from the iterative refinement that CAL provides.
• Technical complexity: The subject matter of this patent litigation involves highly specialized medical device terminology, including references to locking-seal assemblies, blade-deployment mechanisms, insufflation valve designs, trocar assemblies, and other technical concepts. A CAL workflow allows the classifier to continuously improve its understanding of these specialized terms through ongoing attorney feedback, rather than relying on a single static training set.
• Heterogeneous document types: The document population includes a range document types — emails, engineering drawings, CAD files, financial spreadsheets, regulatory submissions, and marketing materials — each with distinct vocabulary and structure. CAL's iterative learning process is better suited to capturing responsiveness across these diverse document types.
CAL involves iterative rounds of review where the classifier continuously retrains on attorney coding decisions rather than relying on a single static control set. In each round, the system identifies and presents for attorney review those documents about which the classifier is most uncertain — i.e., documents whose relevance scores fall nearest the classification boundary. This approach maximizes the informational value of each reviewed document and accelerates classifier convergence.
The advantages of CAL for this matter include: (1) higher recall for technically specialized documents; (2) reduced overall review time compared to linear manual review or TAR 1.0; (3) a defensible and proportionate methodology consistent with the requirements of Federal Rule of Civil Procedure 26(b)(1); and (4) a well-established body of case law supporting the use of CAL in federal court proceedings.
C. Review Coding Categories
All documents reviewed during the TAR process will be coded using the following standardized coding categories, which have been configured within the DataVault Insight™ platform.
Primary Coding Decisions. Each document must receive exactly one primary coding designation:
• Responsive — The document is responsive to one or more of Apex's 42 Requests for Production. A document is coded as Responsive if it contains information that falls within the scope of any of Apex's production requests, as interpreted in light of the applicable definitions and instructions set forth in the requests.
• Non-Responsive — The document is not responsive to any of Apex's 42 Requests for Production. A document is coded as Non-Responsive if it does not contain information falling within the scope of any production request.
• Privileged — The document is subject to the attorney-client privilege or work product protection and will be withheld from production. Documents coded as Privileged are flagged for inclusion on the privilege log in accordance with the Privilege Log Protocol & Guidelines memorandum dated August 1, 2024.
• Confidential — Attorneys' Eyes Only — Documents designated under the anticipated Confidentiality and Protective Order to be entered by the Court. This designation is applied in addition to the Responsive coding and governs the conditions under which opposing counsel and Apex's representatives may access the produced document.
• Hot Document — Documents of particular strategic significance to Solstice's non-infringement defense, invalidity defense, or damages theories. Hot Documents are flagged for immediate escalation to Christy Bauer and Marcus Delgado for substantive evaluation. This designation is applied in addition to the Responsive or Privileged coding.
Secondary Issue Coding Tags. In addition to the primary coding decision, each responsive document will receive one or more secondary issue coding tags to facilitate downstream case analysis and trial preparation:
Patent-specific tags:
• '237 Patent — U.S. Patent No. 9,814,237
• '884 Patent — U.S. Patent No. 10,271,884
• '105 Patent — U.S. Patent No. 10,932,105
Product-specific tags:
• PE-5000 Series
• PE-12000 Series
• PE-15000 Series
Subject-matter tags:
• Design/Development
• Prior Art
• Licensing
• Sales/Financial
• Regulatory/FDA
• Manufacturing
• Marketing
The issue coding tags are not mutually exclusive; a single document may receive multiple tags across all three categories as appropriate.
IV. SEED SET SELECTION AND INITIAL TRAINING
A. Seed Set Composition
The quality and representativeness of the initial seed set is foundational to the effectiveness of the CAL workflow. Accordingly, the seed set has been designed to capture the full diversity of document types, subject matters, and responsiveness characteristics present in the 312,000-document universe.
The initial seed set will comprise approximately 3,000 documents, composed of the following two components:
1. Stratified random sample: 2,500 documents selected through a stratified random sample from the full 312,000-document universe. The stratification ensures proportional representation across custodians, date ranges, file types, and data sources. DataVault will execute the stratified random sampling using its platform's statistical sampling module, with stratification layers defined as follows: (a) custodian; (b) date range (pre-2018 vs. 2018–2020 vs. 2021–2024); (c) file type (email vs. office document vs. CAD/engineering file vs. other); and (d) data source (email server vs. shared drive vs. Aldermere Systems vs. other).
2. Judgmental sample: Approximately 500 documents identified through targeted keyword searches, as described in Section IV.B below. The judgmental sample is intended to supplement the random sample by ensuring that documents containing key technical terminology — which may be underrepresented in a purely random sample due to the relatively low richness of highly technical documents in the overall population — are included in the seed set.
The total initial seed set of approximately 3,000 documents will be reviewed and coded by Christy Bauer (Lead Associate) and Peter Vassiliev (Working Associate). The decision to have the seed set reviewed exclusively by senior attorneys, rather than by contract reviewers, is deliberate: high-quality training data produced by attorneys with deep familiarity with the legal theories, technical subject matter, and discovery scope of this matter will produce a more accurate and defensible classifier. This approach is consistent with the recommendations of the Sedona Conference and with the methodology endorsed by courts applying the Rio Tinto framework.
The goal in composing and reviewing the seed set is to ensure that it captures the full diversity of both responsive and non-responsive documents, including: technically complex engineering documents (CAD files, design specifications); financial records (sales reports, margin analyses); privileged communications (attorney-client emails, legal memoranda); regulatory submissions (510(k) filings, FDA correspondence); and ordinary business communications that are clearly non-responsive.
B. Initial Keyword Searches for Seed Set Augmentation
The keywords used to identify the 500-document judgmental sample were developed in consultation with two key individuals:
• Niamh Cassidy, VP Regulatory & IP — Ms. Cassidy served as the client's primary subject-matter expert and provided input on product-specific terminology, internal code names, and technical nomenclature used within Solstice's engineering and regulatory departments.
• Dr. Miriam Ostrowski, Professor of Biomedical Engineering, Halford University — Dr. Ostrowski has been retained as Solstice's technical expert (retention date: August 20, 2024) and will provide expert testimony on non-infringement and invalidity. Dr. Ostrowski provided input on industry-standard technical terminology for laparoscopic surgical access devices and identified additional terms of art that may appear in the relevant prior art.
The following keyword categories were developed and applied:
Product Terms:
• "PrecisionEdge," "PE-5000," "PE-12000," "PE-15000," "trocar," "laparoscopic," "surgical access"
Patent Terms:
• "9,814,237," "10,271,884," "10,932,105," "locking seal," "blade deployment," "insufflation valve," "retractable blade"
Inventor and Party Terms:
• "Szabo," "Apex," "Apex Surgical"
Technical Terms:
• "obturator," "cannula seal," "zero-closure valve," "trocar tip," "radially expandable"
Prior Art Terms:
• Specific prior art reference identifiers are to be supplied by Kendrick & Lau LLP (601 Pennsylvania Avenue NW, Suite 820, Washington, DC 20004; Dr. Jonathan Kendrick, Lead Partner) based on their prior art search work, which is ongoing as of the date of this memorandum. Once the prior art reference identifiers are finalized, they will be added to the keyword list and documented in Appendix A.
The above keywords were applied in Boolean combinations with date and custodian filters to identify the supplemental 500-document judgmental sample. For example, the search string (("locking seal" OR "blade deployment") AND ("PrecisionEdge" OR "PE-5000" OR "PE-12000" OR "PE-15000")) was run against the full document population, and responsive hits were sampled for inclusion in the judgmental component of the seed set. The complete keyword search term list, including all Boolean combinations and filters applied, will be documented in Appendix A upon finalization.
C. Seed Set Review Protocol
The seed set review will be conducted in accordance with the following protocol:
1. Each document in the 3,000-document seed set will be reviewed by at least one senior attorney — either Christy Bauer or Peter Vassiliev. The seed set will be divided into approximately equal batches, with each attorney responsible for reviewing approximately 1,500 documents.
2. Documents presenting ambiguous or borderline responsiveness determinations, or documents raising novel privilege issues, will be flagged for dual review. Both Christy Bauer and Peter Vassiliev will independently review such flagged documents and confer to reach a consensus coding decision.
3. Where the two senior attorneys are unable to reach consensus on a coding determination, the matter will be escalated to Marcus Delgado (Responsible Partner) for final resolution. Mr. Delgado's coding decision will be treated as dispositive for purposes of the seed set.
4. Seed set coding is targeted for completion within five (5) business days of ingestion, with a target completion date of August 19, 2024.
5. Upon completion of seed set coding, the coded seed set will be uploaded to DataVault Insight™ to initialize the CAL classifier. DataVault will confirm successful ingestion of the seed set coding data and will generate an initial classifier performance baseline report.
V. TRAINING ROUNDS AND ITERATIVE LEARNING
A. Training Round Structure
Following initialization of the CAL classifier with the coded seed set, DataVault Insight™ will begin prioritizing documents for attorney review based on classifier uncertainty. Specifically, the platform will present for review those documents whose relevance scores fall closest to the classification boundary (i.e., documents for which the classifier has the least confidence in its responsiveness prediction). This uncertainty-based sampling approach maximizes the informational value of each reviewed document and accelerates the convergence of the classifier toward stable performance.
Each training round will present approximately 500 to 1,000 documents for attorney review. The exact number of documents in each round will be determined by DataVault's project management team in consultation with Peter Vassiliev, based on the classifier's convergence trajectory and the available review bandwidth.
The anticipated number of training rounds is 5 to 8 rounds, based on DataVault's experience with similarly sized document collections in patent litigation matters. DataVault has advised that, for collections in the 250,000–400,000 document range involving technical subject matter, classifier stabilization typically occurs after 6 to 7 rounds of training, though the actual number will depend on the richness of the collection and the complexity of the responsiveness determinations.
Each training round will be reviewed by a combination of the following reviewers:
• Senior attorney reviewers (Christy Bauer and Peter Vassiliev): All quality-critical batches — including the first three training rounds and any subsequent rounds involving technically complex documents — will be reviewed exclusively by senior attorneys.
• Contract review attorneys supervised by Calderwood & Harkness: If the volume of documents presented in later training rounds exceeds the capacity of the two senior attorneys, the decision to engage contract review attorneys will be made after the completion of Round 3, based on pace and budget considerations. Any contract reviewers engaged will be supervised by Peter Vassiliev, will receive matter-specific training (including a written review manual and a live training session), and will be subject to the quality control measures described in Section VII.A of this memorandum. The decision to engage contract reviewers requires prior approval from Marcus Delgado and Catherine Bloom (Billing Partner).
B. Classifier Performance Tracking
After each training round, DataVault will generate a classifier performance report and distribute it to Christy Bauer and Peter Vassiliev within one (1) business day of the round's completion. Each classifier performance report will include the following metrics:
• Precision (Positive Predictive Value): The proportion of documents classified as responsive that are actually responsive. Target: ≥ 80%.
• Recall (Sensitivity): The proportion of actually responsive documents that are classified as responsive by the model. Target: ≥ 85%. Recall is the primary performance metric for this engagement. High recall is critical for defensibility, as it directly measures the completeness of the production. A recall target of 85% exceeds the thresholds that courts have accepted as reasonable in prior TAR proceedings and is consistent with the Sedona Conference's recommended best practices.
• F1 Score: The harmonic mean of precision and recall, providing a single balanced metric of classifier performance. Target: ≥ 0.82.
• Richness Estimate: The estimated proportion of responsive documents in the overall collection, calculated based on the classifier's scoring of the full document population. The richness estimate is a useful benchmark for assessing whether the classifier's behavior is consistent with expectations based on the nature of the collection and the scope of the production requests.
• Elbow Curve Analysis: A graphical representation showing the marginal gain in recall achieved with each additional training round. The elbow curve is used to identify the point of diminishing returns, at which further training rounds are unlikely to meaningfully improve classifier performance.
Calderwood & Harkness will review the performance metrics after each training round and will make a determination — in consultation with DataVault's project management team — as to whether additional training is required or the classifier has reached stability. The metrics will be documented and retained as part of the case file for defensibility purposes.
C. Stabilization Criteria
The CAL classifier will be deemed to have reached stability — and no further training rounds will be required — when all of the following criteria are satisfied:
1. Recall exceeds 85% on a statistically valid random sample drawn from the document population, with a confidence level of 95% and a margin of error of ±3%.
2. Precision exceeds 80% on a statistically valid random sample drawn from the documents classified as responsive.
3. Marginal gain in recall from an additional training round is less than 1.0 percentage point, as demonstrated by the elbow curve analysis.
4. Two consecutive training rounds show less than 2% change in overall classifier scores (precision, recall, and F1), indicating that the classifier's performance has converged and is no longer materially changing with additional training data.
Upon confirmation that stabilization criteria have been met, DataVault will apply the final classifier to the entire remaining unreviewed population. All documents receiving a relevance score above the classification threshold will be coded as presumptively responsive and queued for human review (including privilege review). All documents receiving a relevance score below the threshold will be coded as presumptively non-responsive and will be subject to the validation procedures described in Section VI below.
VI. VALIDATION METHODOLOGY
A. Statistical Validation of Recall
Upon completion of the TAR process — including application of the stabilized classifier to the full document population — DataVault will conduct a rigorous statistical validation of recall to confirm that the classifier has achieved the required performance threshold.
DataVault will draw a statistically random validation sample from the documents classified as non-responsive (commonly referred to as the "discard pile"). The sample size will be calculated to achieve a 95% confidence level with a ±2% margin of error. Based on the estimated size of the non-responsive population, the required sample size is estimated at approximately 2,400 documents.
The validation sample will be reviewed by senior attorneys — Christy Bauer and Peter Vassiliev — to identify any responsive documents that were missed by the classifier (i.e., false negatives). Each document in the validation sample will be reviewed independently and coded as either responsive or non-responsive. The proportion of responsive documents identified in the validation sample will be used to estimate the overall recall of the classifier with statistical precision.
If the validation reveals that recall falls below the 85% threshold, the following remediation steps will be taken:
1. The responsive documents identified in the validation sample will be added to the training set and the classifier will be retrained.
2. Additional targeted keyword searches will be run to identify potential pockets of responsive documents that the classifier may have missed.
3. A second validation sample will be drawn and reviewed following retraining.
4. This iterative process will continue until the 85% recall threshold is achieved on a validation sample.
B. Precision Validation
In addition to recall validation, a separate validation of precision will be conducted. DataVault will draw a random sample from the documents classified as responsive by the CAL classifier. The sample size for precision validation will be approximately 1,500 documents, calculated to achieve a 95% confidence level with a ±2.5% margin of error.
The purpose of precision validation is to confirm that documents coded as responsive are in fact responsive to Apex's production requests, and that overinclusion does not create an excessive review burden or, more critically, result in the inadvertent production of privileged or work product materials. A precision rate below 80% would indicate that the classifier is overincluding non-responsive documents, which could waste review resources and increase the risk of inadvertent privilege waiver.
C. Privilege Sweep
All documents classified as responsive will undergo a secondary privilege review pass prior to production, in strict accordance with the Privilege Log Protocol & Guidelines memorandum dated August 1, 2024. The privilege sweep is a critical safeguard and represents the final stage of human review before documents are cleared for production.
The privilege review process will incorporate both human and technological components:
• Technology-assisted privilege identification: DataVault will apply a separate privilege classifier trained on the subset of seed and training documents that were coded as privileged during the TAR review process. This privilege classifier will assign privilege probability scores to all responsive documents, enabling reviewers to prioritize their privilege review efforts on documents with the highest probability of containing privileged content.
• Human privilege review: Notwithstanding the use of the privilege classifier, final privilege determinations will be made exclusively by Calderwood & Harkness attorneys. No algorithmic privilege classification will be treated as a final determination. Every document that the privilege classifier flags as potentially privileged will be reviewed by a senior attorney before a final privilege coding decision is made. Documents coded as privileged will be withheld from production and logged on the privilege log.
• Clawback protections: Calderwood & Harkness will seek entry of a Federal Rule of Evidence 502(d) order from the Court, which will provide that the inadvertent production of privileged materials does not constitute a waiver of the attorney-client privilege or work product protection. The 502(d) order will serve as an additional safeguard against the consequences of any privilege-protected documents that may inadvertently pass through the review process. A proposed order will be submitted to the Court by separate motion. In the interim, the parties have agreed to informal clawback procedures as set forth in the Joint discovery Plan.
VII. QUALITY CONTROL METRICS AND REPORTING
A. Ongoing QC During Review
DataVault will implement the following quality control ("QC") measures throughout the active review period to ensure the accuracy and consistency of attorney coding decisions:
• Random QC sampling: Ten percent (10%) of each reviewer's coded documents will be selected at random and re-reviewed by a senior attorney (Christy Bauer or Peter Vassiliev) on a rolling basis. The QC sample will be drawn from each reviewer's completed work on a daily basis to enable timely identification and correction of coding errors.
• Inter-reviewer agreement rate target: The target inter-reviewer agreement rate is ≥ 90%. This metric measures the degree to which different reviewers reach the same coding decision when reviewing the same document. An agreement rate of 90% or higher is considered an industry benchmark for high-quality document review and is consistent with the standards endorsed by the Sedona Conference.
• Coding discrepancy tracking and reporting: All coding discrepancies identified through QC sampling will be tracked in a discrepancy log maintained by DataVault and reported to Christy Bauer on a weekly basis. The discrepancy log will identify the reviewer, the document, the original coding decision, the QC reviewer's coding decision, and the resolution.
• Reviewer remediation: Reviewers whose inter-reviewer agreement rates fall below 85% on a cumulative basis will be subject to corrective action. In the first instance, the reviewer will receive additional training, including a review of the coding manual and a discussion of the specific types of coding errors identified. If the reviewer's agreement rate does not improve to 85% or above within one additional week of review, the reviewer will be removed from the review team.
B. Weekly Status Reports
DataVault will provide weekly status reports to Calderwood & Harkness during the active review period. Reports will be distributed each Monday by 12:00 PM ET and will be directed to Peter Vassiliev with copies to Christy Bauer. Each weekly status report will contain the following information:
• Total documents reviewed to date (cumulative count)
• Responsiveness rates, broken down by coding category (responsive, non-responsive, privileged, hot document)
• Classifier performance metrics (precision, recall, F1 score) as of the most recent training round
• Estimated number of documents remaining for review
• Budget burn rate (dollars spent to date as a percentage of the total engagement budget) and projected total cost based on current pace
• Any technical issues, data anomalies, or processing errors encountered during the reporting period, including a description of the issue and the steps taken to resolve it
C. Final QC and Production Report
Upon completion of all review, validation, and privilege sweep activities, and prior to the production of documents to Apex, DataVault will prepare a comprehensive final production report. This report will serve as the definitive record of the TAR process and will be retained in the case file for defensibility purposes. The final production report will document the following:
• Total documents processed: Target of 312,000 documents
• Total documents produced: Anticipated Bates range of SBD-000001 through the final Bates number, with production scheduled for September 30, 2024
• Recall achieved: The validated recall rate, including the confidence level and margin of error
• Precision achieved: The validated precision rate, including the confidence level and margin of error
• Number of documents withheld on privilege grounds: Including a summary of privilege log entries by custodian and privilege type (attorney-client privilege vs. work product)
• Total data volume processed: 1.2 TB (raw) / 900 GB (after de-duplication and filtering)
• Total cost summary: Itemized breakdown of DataVault charges against the engagement budget (current budget estimate: $141,200)
• Technical methodology summary: A description of the TAR methodology, classifier parameters, training rounds, and validation procedures, suitable for inclusion as an exhibit in any motion practice
VIII. PRODUCTION FORMAT AND SPECIFICATIONS
A. Production Format
All documents will be produced in accordance with the ESI protocol agreed by the parties in the Joint discovery Plan filed on July 15, 2024. The production format specifications are as follows:
• Native format production: Spreadsheets (.xlsx) and databases will be produced in native format, preserving formulas, pivot tables, and embedded data. Native files will be accompanied by a slip sheet bearing the Bates number assigned to the document.
• TIFF image production: All other file types — including emails, word processing documents, presentations, PDFs, and image files — will be produced as single-page TIFF images with extracted text (.txt files) and metadata. TIFF images will be produced at 300 DPI resolution in black and white, with color TIFF production available upon request for documents where color is material (e.g., marketing materials, annotated diagrams).