Facebook, Inc. v. OnlineNIC Inc.
Facebook, Inc. v. OnlineNIC Inc.
Case No. 19-cv-07071-SI (SVK) (N.D. Cal. 2021)
July 12, 2021

Howe, Thomas P.,  Special Master

Failure to Produce
Search Terms
Failure to Preserve
Special Master
Form of Production
Spoliation
ESI Protocol
Manner of Production
Scope of Preservation
Native Format
Forensic Examination
Cloud Computing
Download PDF
To Cite List
Summary
The defendants failed to preserve and produce ESI to the plaintiffs, instead deleting and obfuscating the data. The Special Master identified 11,059,388 deleted records, with an estimated 30% of those records being responsive to the agreed past discovery protocol. The defendants also withheld and deleted ticket attachments, with 76.7% of all attachments being deleted. The Special Master concluded that the defendants' behavior was intentional and the plaintiffs suffered irreparable harm due to the permanently deleted responsive database records and attachment files.
Additional Decisions
FACEBOOK, INC., et al., Plaintiffs,
v.
ONLINENIC INC, et al., Defendants
Case No. 19-cv-07071-SI (SVK)
United States District Court, N.D. California, San Jose Division
Filed July 12, 2021
Howe, Thomas P., Special Master

SPECIAL DISCOVERY MASTER'S DATA DESTROYED OR WITHHELD REPORT

EXECUTIVE SUMMARY

 For this Data Destroyed or Withheld Report (hereinafter “Report”), the Special Discovery Master (hereinafter “Special Master”) was charged with determining the adequacy of Defendants’ past productions to Plaintiffs and whether Defendants destroyed or withheld data from their Kayako Ticket Database (hereinafter “Ticket Database”).

In reaching the conclusions below, Special Master collected electronically stored information (hereinafter “ESI”) with a cumulative data size of 184 GB, a total of 545,013 files, and 442,680,623 database records. To forensically analyze this large data set, most information was separately ingested into a “Consolidated Database,” which is defined below. See Exhibit 1. Of particular importance to the analysis in this Report are Defendants’ support tickets and other ticket-related database tables. This ticket-related data is located in the following six main tables: swtickets, swticketposts, swticketnotes, swattachments, swticketmergelog, and swauditlogs.

In this case, there are two discovery protocols between the parties. The parties’ first discovery protocol was limited to the July 1, 2015 – July 15, 2020, date range, and had 47 search terms. However, the parties’ new discovery protocol was not similarly limited, and instead covers all dates, and has 194 search terms. This Report covers destroyed and withheld ESI that was responsive to the parties’ expanded new discovery protocol with no date limits but does not take into account the expanded search term list as it was received only one business day before submission to the parties. As much as possible, this Report also details destroyed and withheld ESI that was responsive to the old discovery protocol’s limited date range.

As described in detail below, there is ample evidence that Defendants failed to preserve responsive ESI, deleted ESI, and withheld ESI. Sometimes Defendants deleted data using the Kayako SupportSuite Application, and other times, they deleted records directly from the live Ticket Database. Overall, Special Master identified that Defendants deleted over one-half (52.35%) of the ticket-related database records in the entire database (11,059,388 records). Of these deleted records, Special Master estimates 30% of those records (based on the percent of responsive records from July 1, 2015 – July 15, 2020), or 3,317,816 records, were responsive to the agreed past discovery protocol. However, because the new discovery protocol agreement of the parties includes all dates and more search terms, presumably the missing responsive tickets and ticket-related records will be higher and cannot be produced to Plaintiffs.

 This Report narrative covers the most relevant examples of record types that Defendants deleted and withheld. With this Report, the parties received accompanying exhibits, a supporting database, and analytics spreadsheets. Collectively, these items detail the analysis Special Master applied to Defendants’ productions and reveal instances of known deleted and withheld data.

The most relevant instances of known deleted records involve tickets, ticket posts, ticket notes, and attachment files. This latter group, attachment files, is particularly noteworthy because they frequently contain the most material information in a ticket. In this case, over 432,033 attachment records once existed in the Ticket Database. However, Defendants deleted 331,390 of those attachments (76.7%). Of the 100,643 surviving attachments, Special Master found 23,464 (23.31%) responsive attachment files. Assuming the same responsive rate and applying it to the deleted attachments, Plaintiffs will never receive an estimated 77,247 responsive attachments files. Furthermore, if the search term only appeared in the attachment (and not the database records), then Plaintiffs would have received additional related ticket database records had the attachments not been deleted.

 In addition to deleting records, Defendants withheld responsive ESI that still resided in the database. For example, Defendants used overly restrictive searches that excluded entire years’ worth of records and excluded entire ticket-related tables. Below is information about tickets, ticket posts, and ticket attachments not provided to Plaintiffs.

To compound matters, Defendants produced to Plaintiffs a significant amount of unresponsive data in their productions. In a review of a subset of all produced records, 5,096 responsive records were obscured by 27,823,240 non-responsive records.

Finally, while some harm to Plaintiffs is mitigated because Special Master recovered a subset of Defendants’ deleted data, the attachment files and other ticket-related records that are not recoverable are lost forever to Plaintiffs.

I. SCOPE OF REPORT

The Order Appointing Special Discovery Master (hereinafter “Order”) appointing Thomas Howe, on March 3, 2021, contained three primary responsibilities:

A. Supervise and complete Defendants’ collection, search, and production to Plaintiffs’ counsel of the Ticket Database;

B. Produce a result set of responsive data to the parties in native database format, based on an agreed new discovery protocol, with all related database tables (hereinafter “Result Set”); and

C. Determine whether data was destroyed or withheld from Defendants’ Ticket Database, and the adequacy of the productions to Plaintiffs.

This Data Destroyed or Withheld Report (hereinafter “Report”) is in response to the third point above – determining the adequacy of Defendants’ past productions to Plaintiffs and whether data from the Ticket Database was destroyed or withheld. The details below discuss Special Master’s collection, analysis, findings, and conclusions

II. DISCOVERY AGREEMENT BY THE PARTIES

The parties had a limited discovery protocol agreement for past productions. Typically, a discovery protocol delineates the parameters of the search for responsive items by defining such terms as date filters; data types; and the method of production. Although there was no formal written discovery protocol or discovery agreement for the past productions, in this case, the parties did discuss the discovery they needed and exchanged emails with lists of search terms. Special Master worked with each parties’ counsel to recreate their discovery protocol, based on their understanding at the time. This understanding is captured in the “Discovery Protocol for Past Productions.” See Exhibit 2. That discovery agreement provides a benchmark against which to measure the adequacy of Defendants’ past productions. However, it is important to note that several important specifics about the parties’ past discovery protocol remained unidentified. For example:

A. Should the search terms be applied to the entire database, or only to certain tables and table columns?

B. Should the search terms be applied to the attachment files or just the database records?

C. What was the method of production for legal review? After Defendants’ initial production to Plaintiffs, the parties discussed and agreed on a preferred method of production for subsequent productions.

To analyze the productions provided to Plaintiffs per the agreed discovery protocol for past productions, Special Master provided the parties a new database that contained responsive records based on the 47 search terms in the past production protocol. As mentioned above, this Report uses all dates per the new discovery protocol, but does not include an analysis based on the expanded 194 search terms in the parties’ new discovery protocol, because it was received one business day before providing this Report to the parties. See Exhibit 42.

III. DATA SOURCES COLLECTED

Special Master relied on Defendants’ previous productions and collected data directly from Defendants to complete the analysis and findings for this Report. Below is a description of the hardware devices and online services relevant to this Report.

A. Domainwhois-verification.com Server (hereinafter “Domain Who Is Server”).

B. Kayako_support.onlinenic.com Server (hereinafter “Kayako Ticket Server”)

C. Dsktop-6h8msks Developer Workstation (hereinafter “Developer Workstation”). Defendants report that there was only one developer workstation used for programming and productions of the Kayako Ticket Database.

D. Amazon AWS online storage. Special Master requested any Ticket Database related files on Defendant’s Amazon AWS Storage drive but Defendants state that “nothing relating to the Ticketing software/database information was/is on AWS.”

E. SVN or GIT version control software or online storage. Special Master located GIT files and SVN files on the Developer Workstation and requested them, but Defendants stated, “There was no version control for the files located in E:\phpstudy\WWW\onlinenic\1.1 Code\script as located on the Developer Workstation. There were no such logs.”

F. Additionally, the database, backups, and files that Special Master requested from Defendants included:

1. Kayako Ticket Database

2. Kayako Ticket Database Backups

3. Ticket Attachment Files

4. PHP and SQL Script Files

5. Past Productions to Plaintiffs

6. Defendants’ System and Log Files

7. Directory Lists (text files) of Servers and Developer Workstation Listed above.

G. Special Master did not collect or analyze Defendants’ Zoho Support Ticket Database for this Report because its use did not begin until October or November 2020, per Defendants. The responsive ticket information from the Zoho Support Ticket Database will be provided to the parties soon based on the new production protocol provided to the Special Master on July 2, 2021.
According to Defendants, not all devices listed above contained responsive data. Overall, Special Master collected information with a cumulative data size of 184 GB, 545,013 files, and a total of 442,680,623 database records. See Exhibit 3.

IV. KAYAKO TICKET DATABASE
To manage support tickets, Defendants used a relational database known as the Kayako Ticket Database (hereinafter “Ticket Database”). A “ticket” is created by Defendants’ staff and customers (e.g., resellers and users) any time someone requests support for a domain or other issue.
In a relational database, complete information about an individual support ticket requires consolidating related data found across multiple tables. When a user creates a ticket, it is held in a parent table, “swtickets,” and is automatically assigned a unique numerical identifier (e.g., 1234) in the “ticketid” column. The ticket is also assigned a unique alphanumeric identifier, which is stored in the “ticketmaskid” column. Additional ticket-related information is contained in related child tables, such as “swticketposts” (messages about a ticket) and “swattachments” (attached images, spreadsheets, or documents that are linked to a ticket or ticket post). The parent and child tables rely on “ticketid” to relate to each other. Therefore, complete information for a responsive ticket necessitates compiling information from multiple tables.
There are 18 tables in the Ticket Database with ticket-related data.
• swattachments
• swauditlogs
• swescalationpaths
• swparserlogdata
• swparserlogs
• swticketdrafts
• swticketemails
• swticketlabellinks
• swticketlabels
• swticketmergelog
 • swticketmessageids
• swticketpostindex
• swticketpostlocks
• swticketposts
• swticketrecipients
• swtickets
• swtickettimetrack
• swticketwords
A. Kayako Ticket Audit Log
The audit log gathers logged Kayako SupportSuite application actions performed by users, the system, or a staff user for a ticket. The ticket audit log provides information including, but not limited to: ticket created; ticket deleted; ticket post deleted; and ticket note deleted. See Kayako SupportSuite User Manual Version 3.60, Revision 13.
B. Defendants’ Staff Members with Permission to Delete Tickets
Only users that are specifically authorized within the Ticket Database can delete ticket-related information. In this case, Defendants’ staff members with permissions to delete ticket-related information include:
• Le**@onlinenic.com - Last Visit 3/17/2021
• Ra****@onlinenic.com - Last Visit 3/8/2021
• Lu**@onlinenic.com - Last Visit 9/16/2020
• Wa****@onlinenic.com - Last Visit 3/11/2021
V. COLLECTION, PROCESSING, AND PRODUCTION METHODOLOGY
This section provides a detailed overview of Special Master’s analysis and workflow.
A. Collection of all Data Sources
First, Special Master collected all possible electronically stored information (hereinafter “ESI”) from all data sources needed to analyze this Report. The collection began March 12, 2021, and was completed June 17, 2021; Defendants’ have produced to Special Master a total of 184 GBs of data. Overall, the ESI fell into the following high-level categories:
• Databases and Database Backups
• PHP & SQL Script Files
• Directory List Text Files
• Ticket Attachment Files
• HTML Files
• Miscellaneous Files
The collection was an arduous process involving numerous email requests, Skype conference calls, and Skype remote sessions with Defendants’ staff and their counsel, Perry Narancic. Throughout the process, Mr. Narancic was cooperative, responsive, and provided timely responses to Special Master’s voluminous requests. For example, he arranged and participated in late-night phone calls on weekdays and weekends, as well as remote computer sessions. Furthermore, he assisted with language and communication challenges between Special Master and Defendants in China. In particular, he encouraged his clients to comply with all information requests and to cooperate with Special Master. Accordingly, Mr. Narancic’s professionalism facilitated the collection of the ESI required by Special Master for this Report.
B. Backup Policies, Procedures, and Schedules
Special Master also requested copies of backup policies, schedules, and procedures from Defendants for both servers and the Kayako Ticket Database. The Defendants did not produce any of this information and stated there are no backups of either server or any workstation referenced in this Report. Also, Defendants stated several times there were no database backups of the Kayako Ticket Database before December 16, 2020. In total, Defendants produced to Special Master four databases dated: December 16, 2020; March 13, 2021; March 17, 2021; and March 23, 2021.
During the analysis, Special Master discovered a 2013 Ticket database on Defendants’ server that was not previously disclosed or provided by Defendants. This database was restored and analyzed for this Report.
C. Past Productions to Plaintiffs
Special Master requested from Defendants all past productions delivered to Plaintiffs. Defendants stated they no longer had the productions, so Special Master requested the productions from Plaintiffs. Eventually, Defendants stated they had found four of the five productions and produced them to Special Master. Defendants could not find and produce to Special Master the November 27, 2020 production to Plaintiffs. However, Plaintiffs were able to provide the November 27, 2020, production to the Special Master.
The following is the list of all past productions Defendants produced to Plaintiffs:
• July 15, 2020
• November 27, 2020
• December 26, 2020
• February 4, 2021
• February 5, 2021
D. Pre-Processing and Directory Organization
Next, Special Master organized and cataloged all the collections into a pre-processing directory. The root folder was named “\Original Source Files” and all sub-directories were organized based on file type: databases; directory listings; ticket attachment files; HTML productions; programmer scripts; and miscellaneous files.
“Data Pre-Processing” involves preparing ESI for processing pre-analysis. Preprocessing methods vary based on the data type(s) involved. For example, for the database productions, pre-processing involved ingesting and consolidating all sources into the MySQL “Consolidated Database” (described below). Other pre-processing methods are described below.
1. Database Files
The database MySQL Script and database backup files were organized into folders (directories).
• “\Pre-Processing\Databases\Kayako Ticket Server\2013-05-16”
• “\Pre-Processing\Databases\Kayako Ticket Server\2020-12-16”
• “\Pre-Processing\Databases\Kayako Ticket Server\2021-03-11”
• “\Pre-Processing\Databases\Kayako Ticket Server\2021-03-14”
• “\Pre-Processing\Databases\Kayako Ticket Server\2021-03-17”
• “\Pre-Processing\Databases\Kayako Ticket Server\2021-03-23”
• “\Pre-Processing\Databases\Produced by Defendant\2021-01-28”
• “\Pre-Processing\Databases\Produced by Defendant\2021-02-04”
• “\Pre-Processing\Databases\Produced by Defendant\2021-02-05”
• “\Pre-Processing\Databases\Produced by Plaintiff\2020-11-27”
• “\Pre-Processing\Databases\Produced by Plaintiff\2020-12-26”
• “\Pre-Processing\Databases\Produced by Plaintiff\2021-02-04”
• “\Pre-Processing\Databases\Produced by Plaintiff \2021-02-05”
2. Attachment Files
Each directory of attachment files was stored in a directory and cataloged. The attachment files were files that were part of the attachments to emails that were received by the Kayako Ticket Server and parsed into tickets, ticket posts, ticket emails, ticket recipients, and ticket attachments. Each attachment was renamed, encoded, and cataloged by the Kayako Ticket Server parser software. Special Master organized these files into directories based on when they were received, as many of these directories and files were identical and there were many duplicates. Special Master pre-processed and prepared each directory to perform destruction and omission analysis.
3. Directory Listings
Defendants, at the request of Special Master, produced directory listings for the Kayako Ticket Server, Domain Who-Is Server, and the Developer Workstation named “desktop-6h8msks” using a variety of tools. For the Linux servers, Defendants downloaded, installed, and configured a utility named “Zabbix-Apache” and produced directory listings to Special Master on more than one occasion.
Listings of files in directories produced to Special Master from the Kayako Ticket Server, the Domain Who-Is Server, and the Developer Workstation were placed in the directories listed in Exhibit 4.
Each directory listing was processed by a software program created by Special Master to convert each of these listed files into a Comma Separated File (CSV). As CSV files, Special Master was able to import them into MySQL and create the new databases named here:
• “files_kayako_support_2021_03_16”
• “files_kayako_support_2021_04_06”
• “files_kayako_support_2021_04_13”
• “files_domainwhois_2021_03_16”
• “files_domainwhois_2021_04_06”
• “files_domainwhois_2021_06_04”
• “files_desktop_6h8msks_2021_05_03”
Special Master then created SQL Scripts to move the contents of directory listings from these databases into the Consolidated Database (described below). Special Master then used these records to create scripts and file requests of Defendants.
4. HTML Productions
One production to Plaintiffs and one production to Special Master contained ticket information in HTML file format. The HTML information in both productions included the Ticket Number, Subject, Email Address, Contents, and Date columns from the ticket (‘swtickets’) and ticket posts (‘swticketposts’) tables in the database tables.
To pre-process these files, Special Master first wrote custom software to convert these HTML files into CSV files to facilitate importing them into the Consolidated Database. However, due to the nature of the information contained in the “contents” column, the files were always corrupt. So, Special Master modified the software to instead insert these HTML tickets directly into the Consolidated Database for processing and production. HTML directories are listed in Exhibit 5.
5. Programming Scripts
Special Master requested Defendants to provide the programming script files they used for productions. Defendants provided some but not all. The files received were organized in the directories listed in Exhibit 6 for pre-processing.
6. Miscellaneous Files
From time to time, Special Master received files that did not fit into any of the above categories, and they were pre-processed from the directories listed in Exhibit 7.
E. Consolidated Database
Processing, analyzing, and producing information spread across multiple databases is particularly challenging and time-consuming. To ensure accuracy and efficiency, Special Master centralized all data into a single database. Thus, searching, producing, and cataloging information became considerably faster.
The newly consolidated database for production was named “kayakodball” (hereinafter “Consolidated Database”). In creating the Consolidated Database, over 250 SQL scripts were created by Special Master to populate the tables in the Consolidated Database with all the records in the source databases. SQL Scripts were also generated to compare record counts to verify that no records were dropped when consolidating the database records. Special Master imported each set of produced data sources into the “Consolidated Database” and assigned a “Database ID” to distinctly identify each set of files. See Exhibit 8.
In the consolidated database, Special Master found 3,010,020 ticket records and 403,601,954 ticket-related records from all database versions reviewed. Thus, each ticket had an average of 134 ticket-related records. Specifically, there were:
 
Record Type  Total Records
Tickets (swtickets, tmp_data) 3,025,281
Ticket-Related Records 403,601,954
Ticket Posts (swticketposts, temp_data, htmltickets, kayako_searchterms tables) Sub-posts or messages related to a ticket. 12,389,604
Attachments (swattachments table) Records such as images, spreadsheets, or documents that are linked to a ticket or ticket post. 877,211
Table 1. Overview of tickets and ticket-related records
Collectively, there are 22 database versions, from all collected data sources, with a cumulative file size of 82 GB and a total of 442,680,623 records across all versions of the database.
F. Data Processing
To facilitate responsive and privilege searches, Special Master created scripts to add the following columns to each table in the Consolidated Database that had been consolidated from the source databases:
• “batesid” - This is a unique identification number or Bates Number generated for each record in each table in the database. These numbers are unique within each table, as each table is treated as a separate production file.
• “searchtermsfound” - This column contains a list of any of the responsive search terms that were found for the record based on the production protocol.
• “ProducedBefore” - This column is set to “Yes” or “No” based on whether the record was ever part of a previous production to the Plaintiffs.
• “Responsive” - This column is set to “Yes” if the record was responsive based on the agreed protocol for the past collections and needed to be produced, and “No” or blank if it was not responsive.
• “Related” - This column is set to “yes” if it is related to a ticket marked responsive in a related table anywhere in the database.
Special Master then created scripts to populate the “ProducedBefore” column for each record in the Consolidated Database that had been produced by Defendants to Plaintiffs, and SQL scripts were generated to verify the records counts between the source databases and the Consolidated Database. See Exhibit 9.
Special Master then created scripts to populate the responsive column for each table and table records based upon the search terms in the agreed discovery protocol for past productions. See Exhibit 10.
Special Master then created a table in the Consolidated Database named “responsivetickets” (hereinafter “Responsive Tickets Table”). This table contained the Ticket ID or Ticket Mask ID (Ticket Number) for any ticket that was marked as responsive in the system. Special Master then created and executed SQL Scripts that populated the Responsive Tickets Table with all the responsive ticket information from each of the responsive rows in the source tables.
For records not marked as responsive, Special Master created scripts to mark records as “related” in the database if the record was a related part of the responsive ticket. This allowed Special Master to produce every column in every table for any information that is contained in each responsive ticket. Thus, all responsive ticket records will be marked “Yes” in either the related or responsive columns. Conversely, a non-responsive ticket will have the column marked with a “No”.
G. Data Analysis
Special Master applied a variety of analysis methods to arrive at the conclusions in this Report. Briefly, a few primary methods are described below.
One method used to determine if data was deleted, was to analyze index numbers in each table. Tables assign index numbers automatically to each new record in sequential order. This is known as auto-indexing. For example, the first record will be assigned the index ID of 1, and subsequent records are assigned the next sequential number. This can reveal the maximum number of records in the table, as well as any gaps in sequential numbering created by deleted records. Special Master applied this analysis to all the database tables.
Another method included comparing temporary records tables to live database records. Programmers commonly create temporary tables in a database to test queries, test programming code, update or delete data, and analyze data in a table. Temporary tables should reflect the records in the live database. When a temporary table has records that no longer exist in the live database, this indicates they were used to test deletion scripts, which were then applied against the live Database, and resulted in deleted records from the database.
In addition to the methods listed above, Special Master also reviewed and analyzed directory listings from the file servers and Developer Workstation, reviewed PHP programming scripts, reviewed attachment file directories, produced, and looked for missing database records and attachments, compared previous production records to the live database to identify missing or deleted records, and analyzed and compared 22 different databases created from source information, among other things.
H. Database Production for Report
 Accompanying this Report, Special Master provided both parties a MySQL database with all the responsive records (per the search terms in the past agreed discovery protocol), responsive attachment files, and Excel spreadsheets containing the analytics data used for working data to provide the information, findings, and conclusions in this Report.
Special Master produced responsive database records and attachments in the Kayako Ticket Server from the Consolidated Database. The database provided to the parties with this Report had a file size of 26 GB.
To produce the ticket attachment files, Special Master used custom software and an industry-standard dtSearch engine to perform full-text indexing and searching for responsive attachment files. This software assigns its own Bates Number to each attachment, creates a spreadsheet with reference information, and includes the responsive records flagged with matching responsive keywords. Hyperlink columns in the spreadsheet allow the parties to quickly open PDF and Text files with the information from content fields in the database that had voluminous text content
To produce the production database, Special Master created a database named “kayakodball_prod” (referred to as the “Consolidated” database in this report) to serve as the production database for this Report. and then created scripts for each table. Next, he copied and indexed any records that were marked responsive or related.   Special Master then scripted and ran table maintenance to optimize the indexes and remove any disk space that was no longer needed for this database.
Special Master then backed up each of these new databases and compressed and archived the database backups. Special Master then created scripts to compare the record counts in each table in each of these production databases and compared the results. See Exhibit 11. The results were identical, and Special Master will use the second method whenever producing records in the future.
The final production database is a relational database that has been reindexed to optimize it for reporting and legal review by the parties. Most of the original indexing of the tables was done to optimize it for being a live database, and these unnecessary indexes have been removed to save disk space.
VI. DEFENDANTS FAILED TO PRESERVE ESI
Defendants failed to preserve potentially relevant ESI for this matter pre-litigation, post complaint filing, during discovery, and even after the appointment of Special Master. Defendants have sophisticated IT skills, requisite infrastructure, and have had ample time to implement backup management software and procedures. Despite these circumstances, they failed to preserve and produce relevant ESI. The exact extent of Defendants’ backups is unknown as Defendants obfuscated their backup management systems for servers, files, and databases.
A. Defendants’ Have the Requisite IT Skills and Software Developer Knowledge to Backup and Preserve ESI
Notably, Defendants resell backup services to their clients. To preserve and produce data, a company must have backup systems and procedures in place. Through their publicly available website, found at https://onlinenic.com/en/, Defendants offer, market, and resell backup services. A copy of OnlineNIC Inc.’s webpage is attached to this Report as Exhibit 12. Thus, Defendants have demonstrated they have the capacity and infrastructure to backup and preserve responsive ESI.
Secondly, Defendants have advanced software developer skills. To produce or delete data from a database, a company must have sufficiently advanced software developer skills. In this case, Defendants created PHP script files (text files with programming code) to query the Ticket Database and produce responsive materials to Plaintiffs and Special Master. In producing responsive ESI to Special Master on March 25, 2021, April 15, 2021, and April 26, 2021, Defendants used 28 PHP script files. See Exhibit 13. Their third production even included three additional PHP script files. See Exhibit 14. Furthermore, Defendants used a SQL statement file (programming code used to copy and delete database records and attachment files) and likely used these scripts or similar scripts to retrieve responsive information for the productions to Plaintiffs. See Exhibit 15. Defendants’ repeated employment of PHP scripts and SQL statements demonstrates their advanced level of software developer skills. Therefore, Special Master concludes Defendants possess the requisite competence to backup, preserve, produce, and even delete responsive records from the Ticket Database.
B. Defendants Have Inadequate Backup Processes Despite Having Since At Least October 2019 to Create Comprehensive Backup Procedures
Defendants’ backup processes are inadequate. The federal rules of civil procedure require a party to take reasonable steps to preserve ESI from the moment they can reasonably anticipate litigation. Fed. R. Civ. P. 37(e). In this case, Plaintiffs filed the complaint on October 28, 2019. Dkt. No. 1. At a minimum, Defendants’ duty to preserve arose at least by the time served with Plaintiffs’ complaint. Special Master was appointed on March 3, 2021. Dkt. No. 72. By the time of Special Master’s appointment Defendants had over one year to address backup procedures and data retention policies in accordance with their preservation obligations. However, despite this extensive period for implementing comprehensive backup systems such as basic backup management software, Defendants failed to do so.
C. Alternatively, Defendants Have a Robust Backup Management System but Have Obscured or Hidden Backups
Defendants obfuscate their backup processes. Generally, data backup for an online services company is crucial, not only for litigation, but also for business security, stability, compliance, and protection of client and intellectual property. Hence, companies typically act accordingly to institute clear backup policies and procedures. However, in this case, Defendants have not identified or defined their backup management system, and in fact, have been untruthful.
For example, during discovery, Defendants claimed no backups existed; however, this was not true. There are multiple instances of backups existing in seemingly random places. First, SQL backup files and attachment files were found on both servers in multiple locations and archive files. See Exhibit 16. In addition, Special Master found 11 unique backup scripts for the databases. Exhibit 17 shows the 11 backup scripts with the metadata. Furthermore, some of the productions Defendants made to Plaintiffs would have required creating backups, which were created on developer workstations but were not disclosed. Id. Further still, even though Defendants claimed that no programming files were on the servers, the files were present in their third production to the Special Master on April 26, 2021. See Exhibit 18. Finally, around May 2021, Special Master found a fifth database on Defendants’ server they had failed to provide or disclose. Thus, Defendants have a history of being untruthful or misleading regarding the nature and extent of their backups and backup systems and procedures.
Therefore, given Defendants’ sophisticated IT skills, a business need for backups, and a history of untruthfulness, Special Master believes it is likely there are additional files, including developer script files, on undisclosed or unlocated developer workstations or servers.
VII. DEFENDANTS DESTROYED DATA
Not only did Defendants fail to preserve ESI, but they also deleted significant amounts of records. In fact, Defendants deleted 52.35% of the ticket-related database records (11,059,388 records) in the Ticket database. Of these deleted records, Special Master estimates 30% of those records (based on the percent of responsive records from July 1, 2015 – July 15, 2020), or 3,317,816 were responsive to the agreed past discovery protocol. This ticket-related data is located in the following six tables: swtickets; swticketposts; swticketnotes; swattachments; swticketmergelog; and swauditlogs. Broadly speaking, Defendants employed two primary methods to delete data: 1) deletion using the Kayako SupportSuite Application; and 2) deletion directly from the Ticket Database using PHP and SQL scripts.
Special Master utilized a variety of analysis methods to identify instances of deleted records. This Report narrative covers the most relevant examples of the record types that Defendants deleted, including ticket posts, ticket notes, attachment files, and HTML files.
A. Defendants Deleted Records Using the Kayako SupportSuite Application and Obscured User Activity by Deleting Audit Logs
For earlier ticket records, Defendants deleted records using the Kayako SupportSuite Application (hereinafter “Application”). When using the Application, user activity is recorded in the audit logs. An example of this is when records are deleted from the ticket and ticketrelated tables. According to the Application’s audit logs, Defendants deleted: 4,106,268 tickets; 85 ticket posts; 8,991 ticket notes; and 25,253 e-mail recipients. This is a total of 4,140,597 records. As summarized in Table 2 below, 93% (3,863,451 records) of the deleted records were created in 2008, 2009, and 2010, while newer records constitute a much smaller portion of the total.
Year Created  Number of Deleted Database Records
2020 302
2019 22,348
2018 3,209
2017 2,926
2016 548
2015 16,850
2014 1,204
2013 25,859
2012 52,596
2011 151,304
2010 809,668
2009 1,782,491
2008 1,271,292
Total 4,140,597
Table 2. Total records deleted using the Kayako SupportSuite Application (2008 – 2020), organized by record creation date.
However, not only did Defendants delete potentially responsive records using the Application, but they also deleted the Application’s audit logs. Audit logs provide some information about when a user deletes records. In this case, there were over 15 million audit log records. Defendants deleted approximately 7.5 million of those records (roughly onehalf). Special Master was able to recover 459,226 audit log records from a May 16, 2013 database. For this reason, the records in Table 2 are organized by the records’ creation date data, as it was the only information consistently available. This made analyzing user activity considerably more challenging and thereby limits Special Master’s ability to determine a precise timeline of when Defendants deleted these records, or even when “most” of the records were deleted
Furthermore, users require special permission to access the database and delete the audit logs. Because audit logs are so small and require little storage, the primary benefit to deleting audit logs is to hide user activity (such as deleting tickets). Therefore, Special Master concludes that the audit logs were deleted by Defendants to hide their deletion activity.
B. Defendants Deleted Records Directly from the Ticket Database After Production
In more recent years, Defendants began deleting records directly from the live Ticket Database using PHP and SQL scripts. Based on the creation date of PHP and SQL scripts used for deletion, Special Master found Defendants began using this technique in more recent years.
It is considered best practice, and certainly a common practice, to never delete data in a database because it can create problems with the database’s referential integrity. For example, when deleting tickets directly in the database, a programmer can leave orphan records (e.g., a ticket record is deleted, but associated ticket post and ticket attachment records remain). Here, Defendants deleted records in some tables but not all related tables, thereby creating orphan records in multiple places. If Defendants wished to make the deleted tickets inactive, they should have closed tickets in the Application by setting the ticket status field to “Closed.” Instead, Defendants’ deleted records directly from the database and left a lot of orphaned records behind as markers of deletion activity.
Further indicators of data deletion directly from the Ticket Database existed in a comparison between Defendants’ productions and to the live Ticket Database. In fact, this showed that Defendants even deleted database records in the live database after initially producing the records to Plaintiffs. During multiple productions from Defendants, some records were contained in one production but were missing from subsequent productions and from the live Ticket Database itself.
For example, on March 23, 2021, Special Master collected Defendants’ Ticket Database and compared it to Defendants’ production dated November 27, 2020. By relying on the Ticket Mask Id, Special Master found that 208 of the 20,615 distinct tickets produced November 27, 2020, were missing from the Ticket Database on and after December 16, 2020. See Exhibit 19.
In another example, on or around March 12, 2021, Defendants’ programmer created a table named ‘tmp_data’. See Exhibit 20. This temporary table contains a subset of the columns found in the live Ticket Database table named ‘swtickets’ and ‘swticketposts’. See Exhibits 21 and 22. In fact, there are 3,716 matching ticket records between these two tables (76% match). Id. However, the tmp_data table has 1,146 ticket posts that were since deleted from the live Ticket Database table. Id.
Similarly, on or around March 12, 2021, Defendants created a table named ‘temp_data’. See Exhibit 23. This table contains a subset of the columns found in the live Kayako Ticket Database table, “swticketposts”. Id. There are 1,278,354 matching ticket posts between the two tables. Id. However, there are 89,294 ticket posts in the ‘temp_data’ table that were since deleted from the live Ticket Database table. Id.
In addition to the records deleted between productions to Plaintiffs, Special Master found that Defendants deleted 1,281 unique tickets and 445 related ticket posts from the live Ticket Database after these tickets and posts were initially produced to Special Master. These records are reproducible using the Consolidated Database created by Special Master and will be provided to the parties in a new production set.
For records deleted directly from the database, Special Master cannot determine when the records were deleted. When records are deleted directly from the database, as opposed to via the Application, there are no audit logs; a record created 10 years ago could have been deleted 8 years ago or 8 days ago. Had Defendants provided regular backups, Special Master could have provided a timeline for deleted records. In this case, Defendants did not maintain or provide regular backups, and therefore he cannot determine when records were deleted.
However, Special Master did discover a 2013 database. By recovering the 2013 database, Special Master gleaned some context behind Defendants’ deletion activities. In particular, Special Master noted that post-2013, Defendants used programming scripts to delete more records from the Ticket Word table (swticketwords) and Post Index table (swticketpostindex) than any other table including tickets, ticket posts, attachments, and audit logs. Special Master believes Defendants were removing word index records so that search terms related to litigation would not be linked to specific tickets or ticket-related records. The Special Master found that the Defendants deleted 2,919,130 database records between May 16, 2013, and December 16, 2020. The Special Master also found an additional 4,102,283 database records that were deleted and missing in the live ticket database as of March 23, 2021. This results in a total number of 7,021,413 orphaned records which means that there is an incomplete set of records for the tickets.
Below is a summary of the surviving responsive tickets:
• No date filter: 65,810 tickets
• Filtered for July 1, 2015 – July 14, 2020: 32,906 tickets
• Defendants previously produced to Plaintiffs: 32,054 tickets
• Additional tickets produced by Special Master to Plaintiffs: 852 (2.7% more)
C. Defendants Deleted Ticket Posts
Defendants deleted ticket post records from the live Ticket Database and thereby created orphaned ticket post index records. When a database record is deleted from a child table, such as a subsequent ticket post, but not delete the corresponding record from the parent table (based on the unique identifier), the result is an “orphaned record”. In this case, Special Master found orphaned records by analyzing and comparing the unique Ticket ID fields in all 14 ticket-related tables. Specifically, Special Master found 987,862 orphaned ticket post index records, which is 3.6% of the total. See Exhibit 24. This means that the actual ticket posts, that originally referred to these orphaned records, have been deleted. It is also noteworthy that only 158 of these unique posts are also missing from the “temp_data” table of posts and is a further indication of Defendants deletion from the database.
Below is a summary of the surviving responsive tickets posts:
• No date filter: 253,506 ticket posts
• Filtered for July 1, 2015 – July 14, 2020: 93,669 ticket posts
• Defendants previously produced to Plaintiffs: 91,053 ticket posts
• Additional tickets posts produced by Special Master to Plaintiffs: 2,616 (2.9% more)
D. Defendants Deleted Ticket Notes
Defendants deleted ticket note records from the Ticket Database. Database indexes commonly assign unique numerical identifiers in consecutive order to new records automatically. This is known as auto-incrementing. Gaps in the sequential numbering of an index table are evidence that records were deleted. Special Master applied this analysis to the ‘swticketnotes’ table, where ticket notes are stored. He identified 612,729 ticket note records were deleted, representing 83% of the total ticket notes that were in the system. See Exhibit 25. Audit logs show that 8,991 of these ticket notes were deleted via the Application, and the rest were deleted directly from the live ticket database.
E. Defendants Deleted Attachment Database Records
Not only did Defendants delete attachment files, as discussed above, they also deleted attachment database records. Many of the attachment database records were responsive to Plaintiffs’ discovery requests and the agreed past discovery protocol of the parties
As discussed above, Special Master determined Defendants had a total of 432,033 attachment database records. By the time of Special Master’s final collection on March 23, 2021, Defendants had deleted 296,349 attachment database records – 68.59% of the total – leaving only 135,684 attachment records. Of the remaining 135,684 attachment records, 24,719 were responsive (18.22%). Id. Accordingly, of the 296,349 deleted attachment database records, an estimated 53,995 attachments may have been responsive (18.22%).
Additionally, Defendants created orphaned attachment files when they deleted ticket records, such as tickets or ticket posts, that had attachments, and failed to delete the referenced attachment files. By comparing the attachment file directory to the live Ticket Database, Special Master found 8,165 orphaned attachment files representing 8.22% of the total. The 8,165 orphaned attachment files indicate Defendants deleted the corresponding ticket records in the database. Because any given ticket record could have more than one attachment, Special Master is unable to conclude how many ticket records were deleted.
Furthermore, of the 8,165 orphaned attachment files, Special Master used the agreed past discovery protocol and determined that 97 are responsive. However, because they were orphaned, they are no longer linked to a specific ticket or ticket post, and therefore contextual information is missing for those attachment files. See Exhibit 28. Exhibit 28 shows the responsive files categorized and totaled by file type.
Special Master noted that originally the total number of attachment database records and the total number of attachment files was 432,033 each. Defendants deleted more attachment files in the file system (331,390) than attachment database records (296,349). A user must have special permissions, and more importantly, expertise, to delete database records or attachment files either directly from the database or the file system. The deleted attachment files and records cannot have happened by accident. Therefore, Special Master concludes it must have been intentional.
Below is a summary of the surviving responsive database attachment records:
• No date filter: 44,004 attachment database records
• Filtered for July 1, 2015 – July 14, 2020: 23,464 attachment database records
• Defendants previously produced to Plaintiffs: 22,223 attachment database records
• Additional attachment database records produced by Special Master to Plaintiffs: 1,241 (5.6% more)
F. Defendants Deleted Attachment Files
Defendants deleted attachment database records from the Ticket Database, as well as attachment files from the file system. The Ticket Database stores an attachment file’s location, original file name, and stored file name as a database record. See Exhibit 26. The actual attachment files, however, are stored in the file system on the server, outside the database. Id. Attachment files consist of various file types such as images; documents; and spreadsheets. Critically, attachments may be the most important part of a ticket, much like an email attachment may be more important than the email message itself. In this case, attachments included copyright notifications; email conversations; legal demands; account summaries; complaints; letters; legal information; financial information; and more.
Special Master determined that over time, Defendants had a total of 432,033 attachment files in the file system. While a ticket can have more than one attachment file, each attachment file has an associated database record in a one-to-one relationship. Therefore, since there were 432,033 attachment files in the file system there were also 432,033 attachment records in the database.
However, by the time of Special Master’s final collection on March 23, 2021, Defendants had deleted 331,390 attachment files – 76.70% of the total. Of the surviving 100,643 attachment files, 23,464 were responsive (23.31%) to the search terms in the agreed past production protocol. See Exhibit 27. To determine how many deleted attachment files would have been responsive, Special Master used the 23.31%, responsive rate (the only number available). Under this methodology, approximately 77,247 deleted attachment files would have been responsive. Furthermore, the responsive rate for both the surviving and deleted attachment files would likely be higher using the expanded list of search terms (194 instead of 42) in the new discovery protocol.
Finally, it is important to note that while some attachment files are a single file like a PDF, others are compressed archive files like zip, tar, or gz files, and contain many individual files. In other words, while the number of deleted attachment files is 331,390 there are likely many more deleted individual files within those attachment files.
G. Defendants Deleted HTML Files, Tickets, and Ticket Post Records
Defendants deleted tickets after initially producing them to Special Master. On March 12, 2021, Defendants produced to Special Master a backup of the live Ticket Database. The backup contained HTML files that were named in the same format as the original HTML production Defendants made to Plaintiffs on July 15, 2020. Database record comparison revealed they also deleted 1,073 tickets after initially producing them to Special Master in the backup of the live Ticket Database on March 12, 2021.
Based on a group of search terms, Special Master requested Defendants produce 251 matching HTML files that were located on the Kayako Ticket Server as they were not included in any of the previous productions. Defendants produced them to Special Master on April 26, 2021. These newly produced HTML files contained 334,400 partial ticket post records and 8,313 partial ticket records. A subset of 190 of these tickets were responsive across all ticket-related tables.
Of the 190 responsive tickets, 31 of the tickets and 11,700 related ticket posts were deleted from the Ticket Database. In addition, the HTML files contained 1,073 tickets that Defendants since deleted and no longer exist in the Ticket Database. Of these 1,073 tickets, 14 tickets were responsive and had been produced to Plaintiffs before. Id. Finally, the PHP Script code used to produce these HTML files is not located on the servers. This is additional evidence that Defendants deleted records from the Ticket database during discovery
All HTML files discussed above are named with a prefix consisting of a search term from the parties’ past discovery protocol. “JenryHaris” is one of the search terms in the past discovery protocol. When sorted alphabetically, Special Master saw that all files, beginning and ending, were named with the JenryHaris prefix. See Exhibit 29. This indicates that there were most likely other files with prefix names for other search terms that were deleted or removed after they were created on March 12, 2021, and were not available for subsequent productions. Relatedly, Defendants did not produce any HTML files or records related to JenryHaris in the July 15, 2020, HTML production to Plaintiffs. See Exhibit 30.
H. SQL Files and SQL Backups of Productions
Defendants deleted SQL files and databases they previously provided Plaintiffs. Generally, to produce a database, a programmer writes and executes SQL scripts to define the scope of the produced database. Defendants produced MySQL Database Script Files containing records for MySQL Database productions to Plaintiffs in December of 2020 through February of 2021. However, neither of these databases, or any associated programming or SQL backup scripts, are located on any of the servers or workstations. This poses two concerns. First, Special Master is unable to conclude how these production files and backups were created to evaluate their adequacy at producing responsive records. Second, the question remains of whether they were deleted from one of the two servers, whether they were produced on the Developer Workstation and then deleted, or whether they were produced on an undisclosed developer workstation.
I. Defendants Used PHP and SQL Deletions Scripts to Delete Ticket Database Records and Attachment Files
Defendants' programmers created specific PHP and SQL scripts to delete database records and attachment files. Scripts can be used to produce responsive records as well as to delete records from a database. Special Master found 28 PHP scripts and one SQL script that used record deletion language. See Exhibit 13. Here is a sample of some of those scripts:
• delete from swattachments where ticketid...
• delete from swauditlogs where ticketid...
• delete from swescalationpaths where ticketid...
• delete from swparserlogs where ticketmaskid...
• delete from swticketlocks where ticketid...
• delete from swticketmergelog where oldticketmaskid...
• delete from swticketmessageids where ticketid...
• delete from swticketpostindex where ticketpostid...
• delete from swticketpostlocks where ticketid...
• delete from swticketposts where ticketid...
• delete from swticketrecipients where ticketid...
Special Master can only conclude that Defendants programmatically deleted files using PHP and SQL scripts, and tried to hide evidence of their use by deleting the scripts themselves.
J. Text File Created by Defendants’ PHP Script Files Reveals Deleted Attachment Files and Attachment Database Records
Defendants deleted 472 physical attachment files using PHP scripts. Defendants demonstrate their ability to copy or delete files to and from servers with PHP Programming Scripts, and to work with remote and local databases. On March 17, 2021, Defendants created two PHP script files named ‘zenghy_tmp_20210317.php’ and ‘zengy_attachment_copy_file.php.’ The latter PHP script generated a text file named ‘/root/duo_files.txt.’ See Exhibit 31. This file was located on the Kayako Ticket Server, was requested by Special Master, and was delivered on April 26, 2021. The file references 472 unique attachment files by their file paths. See Exhibit 31. However, when Special Master cross referenced the directory lists from both servers and developer workstation, the 472 files were not listed, indicating those attachment files had been deleted.
In addition, Special Master found that a subset of the 472 attachment files were missing corresponding database records. Special Master cross-referenced all 472 files against the ‘swattachments’ table to look for matching database records. Notably, 34 of the listed attachment files did not have matching database records in the cross-referenced table. See Exhibit 32. The fact that the 34 records were no longer included in the ‘swattachments’ table indicates Defendants deleted those attachments from the database with the PHP script file. Notably, this deletion occurred just before Defendants’ first production to Special Master on or about March 17, 2021.
Conspicuously, the ‘/root/duo_files.txt’ file also revealed other actions. For example, the file was used to copy files to a directory, '/home/jumpol/script/ticket_20210106/files/', which itself no longer exists. Furthermore, and most notably, the text file references database servers, other than the live database server, indicating that there are undisclosed servers.
VIII. DEFENDANTS WITHHELD DATA FROM PLAINTIFFS
There are multiple indicators of withheld records within Defendants’ productions to Plaintiffs including using an incorrect date filter; applying restrictive searches; providing incomplete database records; excluding attachment files; and more.
A. Defendants Excluded Two Years’ Worth of ESI with Restrictive Date Filters
Defendants withheld multiple years’ worth of ESI. The parties agreed to a date filter of July 1, 2015, to July 14, 2020, for their past productions. See Exhibit 2. However, for each production made to Plaintiffs, Defendants used different date ranges. For example, one PHP script Defendants employed to query the Ticket Database for responsive records requested data back to 2015, while a subsequent production only went back to 2017. See Exhibits 33 and 34. The varied date ranges were not consistent with the agreed past production protocol and resulted in withholding responsive data from the Plaintiffs.
B. Defendants Withheld ESI by Executing Incomplete Database Searches
Defendants withheld responsive ESI found in ticket-related tables. The parties’ discovery protocol for past productions included a list of search terms. See Exhibit 2. The protocol did not specify which database tables or columns to search for the responsive records and did not specify if search terms should apply to ticket attachments. However, it is common and a best practice to apply search terms to the entire unit of an item. For example, in an email production, search terms should be applied against the entire email by including the header, content, metadata fields, and all attachments to an email. Applying this same logic, Defendants should have applied the search terms to the ticket database table, and all ticketrelated database tables.
In the present case, Defendants executed search terms inconsistently. In some instances, Defendants’ responsive searches were restricted to the email, subject, and contents columns of the tickets or ticket posts table. See Exhibits 35 and 36. In other scripts, search terms were applied in some but not all tables. See Exhibit 37. Therefore, Defendants withheld responsive ticket-related ESI.
C. Defendants did not Search Attachment Files
The agreed past discovery protocol contained search terms. However, the search terms were not applied against the full text of each ticket attachment file to determine responsiveness. All these attachments were subsequently not produced, and some were in fact deleted, as discussed in Section VII. Special Master verified this by creating a full-text search index of all the attachment files to search the attachment files for each search term.
D. Defendants Withheld Responsive Ticket-Related Tables and Table Columns in Early Productions
Defendants did not produce complete ticket records to Plaintiffs. As discussed previously, complete responsive ticket data from the relational Ticket Database requires consolidating information across multiple related tables. However, Defendants only included a subset of all ticket-related database tables in their productions. See Exhibit 38. Specifically, Defendants withheld records from most of the ticket and ticket post-related tables described in productions conducted before December of 2020.
Furthermore, Defendants’ productions not only excluded entire database tables, as described above, but their productions also failed to include all columns within the tables that they did produce. See Exhibit 39. As an analogy, imagine a party producing responsive Excel spreadsheets, but then deleting certain columns in the spreadsheet. Therefore, by not providing all ticket-related tables, and all columns in each database table, Defendants withheld responsive ticket-related information from Plaintiffs.
E. Defendants Produced Limited Attachment Files that were not in a Reasonably Useable Format
Defendants produced 21,455 ticket attachments to Plaintiffs in rather unusable format. Ticket attachments to responsive tickets include many file types such as PDFs; emails; HTML documents; spreadsheets; Word documents; PHP files; compressed files (e.g., rar, gz, and zip files); Text files; and ISO. As discussed previously, attachment content can be the most important part of a ticket, much like email attachments to an email. Special Master reviewed numerous attachments and found the attachments were particularly important to provide context and information about the tickets. In fact, among the withheld attachments were: copyright notifications; email conversations; legal demands; account summaries; complaints; letters; legal information; financial information, and more.
In this case, Defendants produced 21,455 attachments files to Plaintiffs out of a total of 100,643 surviving attachment files. However, Defendants produced a greater number of responsive attachment database records, 21,577, on February 05, 2021. This is unusual because attachment files and attachment database records exist in a 1:1 relationship. Thus, the discrepancy further confirms that Defendants deleted records directly from the database and broke referential integrity.
Below is a summary of the surviving attachment files:
• Defendants previously produced plaintiffs: 21,455 attachment files
• Additional attachment files produced by Special Master to Plaintiffs: 2,122 (9.9% more)
Furthermore, the attachment files Defendants did produce, were not in a reasonably useable format for Plaintiffs. When Defendants produced attachments, there were two problems: 1) the attachments were missing file extensions; and 2) they contained virus files.
First, because the files were missing file extensions, they were not easily opened or viewable. As mentioned, attachment files are stored outside the database. The database stores the file name and file extension of each attachment file. The Kayako SupportSuite Application automatically looks up the file extension and opens the attachment file. Although Defendants provided the files as they exist in their file system, Plaintiffs, do not have Defendants’ Kayako SupportSuite Application, and therefore were unable to open and view the files. Therefore, to provide files in a reasonably useable format for Plaintiffs, Defendants should have at a minimum supplied the file extensions to each attachment file, provided a license to use the Application to view the attachments, and/or provided PDFs of all text-based attachments.
Second, the group of attachment files provided to Plaintiffs contained viruses. It is common best practice to run anti-virus software before a file production to opposing party. However, Defendants reported to Special Master that its servers and workstations do not have anti-virus software. In this case, there were over 150 virus files among the attachment files provided by Defendants. Failing to run anti-virus software against files being produced to Plaintiffs posed a risk to Plaintiffs’ computer systems. Therefore, of the limited group of attachment files provided, they were not provided to Plaintiffs in a reasonably useable format.
F. SQL Backup File Requested but Withheld
Special Master requested all database files and backup files in initial and subsequent production requests. Defendants produced files from developer workstations, but many requested files were omitted or incomplete. In fact, there was a specific file Special Master identified in a directory list. See Exhibit 40. The Special Master on May 9, 2021, requested that specific file named “/home/kayako/mysql.12.dump.gz”, which is a Ticket Database backup. Defendants responded to the request on May 10, 2021, that the file does “not exist”. However, the file did exist as shown on the directory listing provided by Defendants on April 06, 2021. When questioned further, Defendants claimed they deleted this file to free up space on their server on April 11, 2021, in an email dated June 02, 2021, which contained this text:
The server (whois.onlinenic.com) [sic] running out of space around Apr 11. As a server mainly for shared hosting servers’ backup and sending out notification [sic], it was a temporary station to place some processing files in this legal case, for example to analyze so it would not affect stableness of ticket server, or to upload files from it, etc. When the server was running out of space, the first move was to clean those temporary files. I think server’s [sic] admin cleaned a couple of temporary users (folders) to leave room for SHARED hosting servers.
As shown in the email above, Defendants claimed the file had been deleted from the “temporary station to place some processing files in this legal case.” Special Master asked Defendants by email, “Before deleting files to make room on the server, did your clients backup or copy the files that were deleted to another computer, hard drive, storage device, or cloud backup service? If so, where are those files now?” Defendants answered, “Unfortunately, the server admin did not keep a copy of the files when cleaning server.”
On June 7, 2021, the Defendants produced a new server directory list requested by the Special Master. The Special Master analyzed the directory list and confirmed the Ticket Database backup file, “/home/kayako/mysql.12.dump.gz”, was no longer on the server. The requested Ticket Database backup file, “/home/kayako/mysql.12.dump.gz,” was never provided to the Special Master.
Defendants had other options to free up space and did not need to delete the requested, directly responsive, backup file. Not only could Defendants have moved (or copied) the files to another server or purchased and used an external hard drive (less than $100 USD), they could have instead deleted older versions of much larger files. For example, Defendants elected to preserve 56 larger files created in 2017 and 2020 instead of preserving the file named “mysql.12.dump.gz” directly responsive to the case. See Exhibit 41. Furthermore, there were triplicate files that existed, meaning Defendants could have deleted older versions of the same exceptionally large file. Id. Instead, they deleted a directly responsive, relatively small file directly responsive to the case under pretense of freeing up space.
IX. DEFENDANTS CONDUCTED “DATA DUMPING”
Defendants created a burden for Plaintiffs by producing a significant amount of ESI that was not responsive to the agreed past discovery protocol (often referred to as “data dumping”). Data dumping is the practice of deliberately producing a significant amount of non-responsive information and is often discouraged by courts. Generally, the intent is to make it more difficult and expensive for the opposing party to find important evidence and thereby receive a strategic advantage. By reviewing a subset of 27,823,240 records that Defendants produced to Plaintiffs, only 5,096 records were responsive. These examples are shown here:
Table Total Records Responsive Records Responsive Percent
swcommentdata 163,502 3,892 2.38%
swcountryinfo 79,440 0 0.00%
swticketnotes 128,770 5 0.004%
swticketpostindex 26,288,906 0 0.00%
swticketpostindex 529,528 1,046 0.19%
swusers 633,094 153 0.28%
Table 3. Sample of productions comparing number of responsive records to total records provided
X. RECORDS DELETED & WITHHELD FROM SPECIAL MASTER
Special Master made a great effort to collect databases and other information from Defendants via email, skype, phone calls, and remote computer sessions for information needed for this Report. Sometimes, Special Master requested information that was not provided by Defendants. After discovering information withheld, Defendants sometimes produced the specified items to the Special Master after a special request. In one example, Special Master found that Defendants’ PHP scripts generate four text files (ticketid.txt, ticketmaskid.txt, email.txt, and postid.txt). Despite a specific request for the files, Defendants never produced them. Other times, Special Master had to make multiple requests for information before Defendants eventually complied. In a few instances, Defendants stated the information had been deleted and was no longer available. Thus, on multiple occasions, Defendants withheld and deleted records from Special Master, even though the Special Master has documentation in server directory lists to show the files did exist on the server within the past two months.
XI. REPORT EXHIBITS AND PRODUCTION TO THE PARTIES
This Report is derived from a tremendous amount of data and information. To fully understand the information in this Report, the parties were provided the following supplemental information:
• The “Special Discovery Master’s Data Destroyed or Withheld Report” and all exhibits (2 files, 13MB);
• MySQL database with all the responsive records, across all databases collected by Special Master, per the past agreed discovery protocol (24 GB);
• Attachment files with all the responsive records per the past agreed discovery protocol (23,464 files, 2.6 GB); and
• Excel spreadsheets containing the analytics data used to provide the information, findings, and conclusions in this Report (33 files, 29 MB).
The cumulative file size of this Report and all supplemental information is over 26 GB.
XII. PARTIES’ COMMENTS TO DRAFT REPORT
The parties were provided two draft copies of this Report filed on July 12, 2021. The parties received the first draft on June 9, 2021, and the second draft on July 7, 2021. Both parties had an opportunity to provide comments. Those comments are included with this Report as Exhibits 43 and 44 and are discussed below.
A. Plaintiffs’ Comments
Plaintiffs’ comments sought two additions. One, to clarify that based on Defendants’ own admissions, they deleted at least one database backup during discovery. See Exhibit 43. Two, Plaintiffs sought to have Defendants’ employees who provided information for this Report be individually named. Id.
On Plaintiffs’ first point regarding a known deleted backup, Special Master believes Defendants have done several database backups and file copy backups of the database (see references to the 2013 database in the Report). However, the database backups and file copy backups that existed on servers and workstations have been deleted. On Plaintiffs’ second point regarding employee identification, Special Master included the email address, partially obfuscated, to reference but not identify employee names who are not officers of Defendants.
B. Defendants’ Comments
Defendants’ comments were more extensive and are included in Exhibit 44. Defendants' comments are summarized here: 1) Defendants regularly deleted records to improve the aging Ticket Database’s functionality; 2) Defendants’ expert performed a waterfall analysis to show 44 non-recoverable tickets and posed hypotheses of what happened to them; 3) the Application itself may be responsible for deleted audit log files; 4) backup files discovered by Special Master were not in fact backups, but rather files that were created for Special Master; 5) PHP files referenced in the Report do not contain deletion language and created for responding to discovery in this case; and 6) date ranges in Defendants’ PHP scripts written for production are irrelevant and seeks clarification that Defendants produced records going back to 2015 or not.
Special Master carefully reviewed Defendants’ comments. In many ways, Defendants’ responses support the most important concerns Special Master raised in the Report. In fact, Defendants do not dispute deleting records, they instead try to justify why. Defendants deleted data post case filing, during discovery, after productions, and after Special Master’s appointment.
Special Master highlights the following points:
• Defendants did not state one step, method, process, or system they used for data preservation.
• Servers and workstations were not backed up.
• Defendants failed to create and maintain regular backups of their database, even post case filing.
• Defendants failed to use anti-spam/anti-virus software on their servers or workstations.
• Defendants allege database performance issues caused some problems. Defendants’ Kayako Ticketing Database was approximately 7.7 GB. Special Master did not notice performance issues with the larger Consolidated Database (24 GB).
• Defendants continue to use the Kayako Ticketing Database that they allege was unstable, poor performing, spam-riddled, and un-backedup, even after purchasing the new Zoho Ticketing Database in October 2020.
• Defendants speculate that some audit logs were never written due to “glitches” in the system resulting from “several concurrent deletion requests getting submitted to the system.” Special Master strongly disputes this assertion.
• Defendants do not explain why they would delete records directly from the database (causing orphaned records) instead of via the Application.
• Defendants failed to address why so many attachment files and database records were deleted, why surviving attachments files were not searched, or why they were provided to Plaintiffs with viruses.
• Defendants did not explain why they data-dumped 27,818,144 nonresponsive records on Plaintiffs.
Finally, Defendants’ experts’ “waterfall analysis” is flawed.
• Defendants selected only those tables that supported their desired conclusion. The waterfall analysis does not include the many ticket-related tables that contained large numbers of orphaned records, such as: swattachments; swticketrecipients; or swticketposts.
• Defendants’ analysis is based on the swticket table outward. Conversely, Special Master’s analysis is based on an outward analysis from all ticket and ticket-related tables. Special Masters’ more expansive analysis was necessitated by the high number of orphaned records across many ticket and ticket-related tables.
• The analysis did not explain deleted ticket database records and attachment files.
• Defendants justify data exists because it is in “temp_data,” even though the table was never produced to Plaintiffs.
• The waterfall analysis refers to a date range beginning “July 1, 2015”, but fails to specify what end date was applied.
• The analysis focuses on deleted records but is silent concerning the issue of withheld records to the Plaintiffs. There was no explanation for why searches for responsive records were not done in ticket-related tables, certain columns in tables, or attachment files.
EXPERT CONCLUSIONS
Briefly put, Defendants did not do what they should have done (preserve and produce responsive ESI) yet did do what they should not have done (delete and obfuscate). Based on the sum of the evidence, Special Master concludes Defendants’ behavior was intentional.
First, Defendants’ failure to preserve ESI and deletion activities were widespread. In total, Special Master identified 11,059,388 deleted records. Deletion activities were so pervasive that they included many database tables (tickets, ticket posts, ticket notes, ticket words, audit logs, etc.), and attachment files. Of the deleted records, Special Master estimates 30% of those records (based on the percent of responsive records from July 1, 2015 – July 15, 2020), or 3,317,816 were responsive to the agreed past discovery protocol. Special Master even obtained some of the very PHP and SQL scripts Defendants used to programmatically delete data directly from the database. This behavior is more outrageous when considering the type of ESI Defendants deleted and withheld. Most egregious was Defendants’ deletion and withholding of ticket attachments. As discussed, attachments arguably contained the most responsive ESI, yet Defendants deleted 76.7% of all attachments. Defendants’ conduct was consistent and continuous. Defendants began deleting data before the complaint was filed, continued deleting responsive records during the discovery process, and even after Special Master’s appointment.
Admittedly, for many of the deleted records, Special Master is unable to determine when they were deleted. This is because Defendants deleted audit logs and failed to provide regular backups of Ticket Databases and attachment file directories. However, what is clear, is that Defendants deleted data during discovery after previous productions to both Plaintiffs and Special Master.
Second, throughout the discovery process Defendants’ conduct included obfuscation and ESI withholding. From the beginning, Defendants made material misrepresentations regarding their backups and other ESI items to Special Master. For example, on multiple occasions, Defendants denied items existed until they were discovered by Special Master. Furthermore, Defendants used inaccurate date filters and did not search all ticket-related tables and columns within those tables, resulting in more withheld records. Finally, Defendants over-produced 27,823,240 non-responsive records (aka, data dumped) to obscure 5,096 responsive records. Consequently, Defendants’ conduct throughout discovery resulted in Plaintiffs not receiving all available responsive information.
Third, Plaintiffs suffered irreparable harm. Only some of the withheld and deleted records were recoverable. While a subset of deleted records is recoverable from Defendants’ earlier productions, other responsive ESI will be unavailable because the data was destroyed and no longer exists. In fact, of the most critical evidence type, attachments, 76.7% were deleted. Furthermore, Special Master cannot indicate whether or how much of the nonrecoverable ESI was responsive or not; a record that cannot be seen, cannot be searched. Therefore, Defendants caused irreparable harm to Plaintiffs through permanently deleted responsive database records and attachment files, that they will never see or know the contents.
Fourth, Plaintiffs will continue to suffer harm with the new production. Based on the new discovery protocol (with an expanded date range and more search terms), there will be a greater number of missing responsive database records and attachments. Even though Special Master has recovered all possible data, not all deleted responsive ESI is available. Due to Defendants’ data destruction efforts, the new production will be inadequate because it will not include all responsive ESI that should have been produced to Plaintiffs.
In summary, based on Defendants’ widespread activities involving ESI deletion, information withholding, and data dumping there is no other conclusion than Defendants acted intentionally to avoid producing responsive ESI to Plaintiffs.
DATED: July 12, 2021

Howe Law Firm
By: /s/  Thomas P. Howe
Thomas P. Howe,
Special Discovery Master