EXECUTIVE SUMMARY
For this Data Destroyed or Withheld Report (hereinafter “Report”), the Special
Discovery Master (hereinafter “Special Master”) was charged with determining the adequacy
of Defendants’ past productions to Plaintiffs and whether Defendants destroyed or withheld
data from their Kayako Ticket Database (hereinafter “Ticket Database”).
In reaching the conclusions below, Special Master collected electronically stored
information (hereinafter “ESI”) with a cumulative data size of 184 GB, a total of 545,013
files, and 442,680,623 database records. To forensically analyze this large data set, most
information was separately ingested into a “Consolidated Database,” which is defined below.
See Exhibit 1. Of particular importance to the analysis in this Report are Defendants’ support
tickets and other ticket-related database tables. This ticket-related data is located in the
following six main tables: swtickets, swticketposts, swticketnotes, swattachments,
swticketmergelog, and swauditlogs.
In this case, there are two discovery protocols between the parties. The parties’ first
discovery protocol was limited to the July 1, 2015 – July 15, 2020, date range, and had 47
search terms. However, the parties’ new discovery protocol was not similarly limited, and
instead covers all dates, and has 194 search terms. This Report covers destroyed and withheld
ESI that was responsive to the parties’ expanded new discovery protocol with no date limits
but does not take into account the expanded search term list as it was received only one
business day before submission to the parties. As much as possible, this Report also details
destroyed and withheld ESI that was responsive to the old discovery protocol’s limited date
range.
As described in detail below, there is ample evidence that Defendants failed to
preserve responsive ESI, deleted ESI, and withheld ESI. Sometimes Defendants deleted data
using the Kayako SupportSuite Application, and other times, they deleted records directly
from the live Ticket Database. Overall, Special Master identified that Defendants deleted over
one-half (52.35%) of the ticket-related database records in the entire database (11,059,388
records). Of these deleted records, Special Master estimates 30% of those records (based on the percent of responsive records from July 1, 2015 – July 15, 2020), or 3,317,816 records,
were responsive to the agreed past discovery protocol. However, because the new discovery
protocol agreement of the parties includes all dates and more search terms, presumably the
missing responsive tickets and ticket-related records will be higher and cannot be produced to
Plaintiffs.
This Report narrative covers the most relevant examples of record types that
Defendants deleted and withheld. With this Report, the parties received accompanying
exhibits, a supporting database, and analytics spreadsheets. Collectively, these items detail the
analysis Special Master applied to Defendants’ productions and reveal instances of known
deleted and withheld data.
The most relevant instances of known deleted records involve tickets, ticket posts,
ticket notes, and attachment files. This latter group, attachment files, is particularly
noteworthy because they frequently contain the most material information in a ticket. In this
case, over 432,033 attachment records once existed in the Ticket Database. However,
Defendants deleted 331,390 of those attachments (76.7%). Of the 100,643 surviving
attachments, Special Master found 23,464 (23.31%) responsive attachment files. Assuming
the same responsive rate and applying it to the deleted attachments, Plaintiffs will never
receive an estimated 77,247 responsive attachments files. Furthermore, if the search term only
appeared in the attachment (and not the database records), then Plaintiffs would have received
additional related ticket database records had the attachments not been deleted.
In addition to deleting records, Defendants withheld responsive ESI that still resided in
the database. For example, Defendants used overly restrictive searches that excluded entire
years’ worth of records and excluded entire ticket-related tables. Below is information about
tickets, ticket posts, and ticket attachments not provided to Plaintiffs.
To compound matters, Defendants produced to Plaintiffs a significant amount of
unresponsive data in their productions. In a review of a subset of all produced records, 5,096
responsive records were obscured by 27,823,240 non-responsive records.
Finally, while some harm to Plaintiffs is mitigated because Special Master recovered a
subset of Defendants’ deleted data, the attachment files and other ticket-related records that
are not recoverable are lost forever to Plaintiffs.
I. SCOPE OF REPORT
The Order Appointing Special Discovery Master (hereinafter “Order”) appointing
Thomas Howe, on March 3, 2021, contained three primary responsibilities:
A. Supervise and complete Defendants’ collection, search, and production to
Plaintiffs’ counsel of the Ticket Database;
B. Produce a result set of responsive data to the parties in native database
format, based on an agreed new discovery protocol, with all related
database tables (hereinafter “Result Set”); and
C. Determine whether data was destroyed or withheld from Defendants’
Ticket Database, and the adequacy of the productions to Plaintiffs.
This Data Destroyed or Withheld Report (hereinafter “Report”) is in response to the
third point above – determining the adequacy of Defendants’ past productions to Plaintiffs
and whether data from the Ticket Database was destroyed or withheld. The details below
discuss Special Master’s collection, analysis, findings, and conclusions
II. DISCOVERY AGREEMENT BY THE PARTIES
The parties had a limited discovery protocol agreement for past productions.
Typically, a discovery protocol delineates the parameters of the search for responsive items
by defining such terms as date filters; data types; and the method of production. Although
there was no formal written discovery protocol or discovery agreement for the past
productions, in this case, the parties did discuss the discovery they needed and exchanged
emails with lists of search terms. Special Master worked with each parties’ counsel to recreate
their discovery protocol, based on their understanding at the time. This understanding is
captured in the “Discovery Protocol for Past Productions.” See Exhibit 2. That discovery
agreement provides a benchmark against which to measure the adequacy of Defendants’ past productions. However, it is important to note that several important specifics about the
parties’ past discovery protocol remained unidentified. For example:
A. Should the search terms be applied to the entire database, or only to certain
tables and table columns?
B. Should the search terms be applied to the attachment files or just the
database records?
C. What was the method of production for legal review? After Defendants’
initial production to Plaintiffs, the parties discussed and agreed on a
preferred method of production for subsequent productions.
To analyze the productions provided to Plaintiffs per the agreed discovery protocol for
past productions, Special Master provided the parties a new database that contained
responsive records based on the 47 search terms in the past production protocol. As
mentioned above, this Report uses all dates per the new discovery protocol, but does not
include an analysis based on the expanded 194 search terms in the parties’ new discovery
protocol, because it was received one business day before providing this Report to the parties.
See Exhibit 42.
III. DATA SOURCES COLLECTED
Special Master relied on Defendants’ previous productions and collected data directly
from Defendants to complete the analysis and findings for this Report. Below is a description
of the hardware devices and online services relevant to this Report.
A. Domainwhois-verification.com Server (hereinafter “Domain Who Is Server”).
B. Kayako_support.onlinenic.com Server (hereinafter “Kayako Ticket Server”)
C. Dsktop-6h8msks Developer Workstation (hereinafter “Developer
Workstation”). Defendants report that there was only one developer
workstation used for programming and productions of the Kayako Ticket
Database.
D. Amazon AWS online storage. Special Master requested any Ticket Database
related files on Defendant’s Amazon AWS Storage drive but Defendants state that “nothing relating to the Ticketing software/database information was/is on
AWS.”
E. SVN or GIT version control software or online storage. Special Master located
GIT files and SVN files on the Developer Workstation and requested them, but
Defendants stated, “There was no version control for the files located in
E:\phpstudy\WWW\onlinenic\1.1 Code\script as located on the Developer
Workstation. There were no such logs.”
F. Additionally, the database, backups, and files that Special Master requested
from Defendants included:
1. Kayako Ticket Database
2. Kayako Ticket Database Backups
3. Ticket Attachment Files
4. PHP and SQL Script Files
5. Past Productions to Plaintiffs
6. Defendants’ System and Log Files
7. Directory Lists (text files) of Servers and Developer Workstation Listed
above.
G. Special Master did not collect or analyze Defendants’ Zoho Support Ticket
Database for this Report because its use did not begin until October or
November 2020, per Defendants. The responsive ticket information from the
Zoho Support Ticket Database will be provided to the parties soon based on
the new production protocol provided to the Special Master on July 2, 2021.
According to Defendants, not all devices listed above contained responsive data.
Overall, Special Master collected information with a cumulative data size of 184 GB, 545,013
files, and a total of 442,680,623 database records. See Exhibit 3.
IV. KAYAKO TICKET DATABASE
To manage support tickets, Defendants used a relational database known as the
Kayako Ticket Database (hereinafter “Ticket Database”). A “ticket” is created by Defendants’ staff and customers (e.g., resellers and users) any time someone requests support for a domain
or other issue.
In a relational database, complete information about an individual support ticket
requires consolidating related data found across multiple tables. When a user creates a ticket,
it is held in a parent table, “swtickets,” and is automatically assigned a unique numerical
identifier (e.g., 1234) in the “ticketid” column. The ticket is also assigned a unique alphanumeric identifier, which is stored in the “ticketmaskid” column. Additional ticket-related
information is contained in related child tables, such as “swticketposts” (messages about a
ticket) and “swattachments” (attached images, spreadsheets, or documents that are linked to a
ticket or ticket post). The parent and child tables rely on “ticketid” to relate to each other.
Therefore, complete information for a responsive ticket necessitates compiling information
from multiple tables.
There are 18 tables in the Ticket Database with ticket-related data.
• swattachments
• swauditlogs
• swescalationpaths
• swparserlogdata
• swparserlogs
• swticketdrafts
• swticketemails
• swticketlabellinks
• swticketlabels
• swticketmergelog
• swticketmessageids
• swticketpostindex
• swticketpostlocks
• swticketposts
• swticketrecipients
• swtickets
• swtickettimetrack
• swticketwords
A. Kayako Ticket Audit Log
The audit log gathers logged Kayako SupportSuite application actions performed by
users, the system, or a staff user for a ticket. The ticket audit log provides information
including, but not limited to: ticket created; ticket deleted; ticket post deleted; and ticket note
deleted. See Kayako SupportSuite User Manual Version 3.60, Revision 13.
B. Defendants’ Staff Members with Permission to Delete Tickets
Only users that are specifically authorized within the Ticket Database can delete
ticket-related information. In this case, Defendants’ staff members with permissions to delete
ticket-related information include:
• Le**@onlinenic.com - Last Visit 3/17/2021
• Ra****@onlinenic.com - Last Visit 3/8/2021
• Lu**@onlinenic.com - Last Visit 9/16/2020
• Wa****@onlinenic.com - Last Visit 3/11/2021
V. COLLECTION, PROCESSING, AND PRODUCTION METHODOLOGY
This section provides a detailed overview of Special Master’s analysis and workflow.
A. Collection of all Data Sources
First, Special Master collected all possible electronically stored information
(hereinafter “ESI”) from all data sources needed to analyze this Report. The collection began
March 12, 2021, and was completed June 17, 2021; Defendants’ have produced to Special
Master a total of 184 GBs of data. Overall, the ESI fell into the following high-level
categories:
• Databases and Database Backups
• PHP & SQL Script Files
• Directory List Text Files
• Ticket Attachment Files
• HTML Files
• Miscellaneous Files
The collection was an arduous process involving numerous email requests, Skype
conference calls, and Skype remote sessions with Defendants’ staff and their counsel, Perry
Narancic. Throughout the process, Mr. Narancic was cooperative, responsive, and provided
timely responses to Special Master’s voluminous requests. For example, he arranged and
participated in late-night phone calls on weekdays and weekends, as well as remote computer
sessions. Furthermore, he assisted with language and communication challenges between
Special Master and Defendants in China. In particular, he encouraged his clients to comply
with all information requests and to cooperate with Special Master. Accordingly, Mr.
Narancic’s professionalism facilitated the collection of the ESI required by Special Master for
this Report.
B. Backup Policies, Procedures, and Schedules
Special Master also requested copies of backup policies, schedules, and procedures
from Defendants for both servers and the Kayako Ticket Database. The Defendants did not
produce any of this information and stated there are no backups of either server or any
workstation referenced in this Report. Also, Defendants stated several times there were no
database backups of the Kayako Ticket Database before December 16, 2020. In total,
Defendants produced to Special Master four databases dated: December 16, 2020; March 13,
2021; March 17, 2021; and March 23, 2021.
During the analysis, Special Master discovered a 2013 Ticket database on Defendants’
server that was not previously disclosed or provided by Defendants. This database was
restored and analyzed for this Report.
C. Past Productions to Plaintiffs
Special Master requested from Defendants all past productions delivered to Plaintiffs.
Defendants stated they no longer had the productions, so Special Master requested the
productions from Plaintiffs. Eventually, Defendants stated they had found four of the five productions and produced them to Special Master. Defendants could not find and produce to
Special Master the November 27, 2020 production to Plaintiffs. However, Plaintiffs were able
to provide the November 27, 2020, production to the Special Master.
The following is the list of all past productions Defendants produced to Plaintiffs:
• July 15, 2020
• November 27, 2020
• December 26, 2020
• February 4, 2021
• February 5, 2021
D. Pre-Processing and Directory Organization
Next, Special Master organized and cataloged all the collections into a pre-processing
directory. The root folder was named “\Original Source Files” and all sub-directories were
organized based on file type: databases; directory listings; ticket attachment files; HTML
productions; programmer scripts; and miscellaneous files.
“Data Pre-Processing” involves preparing ESI for processing pre-analysis. Preprocessing methods vary based on the data type(s) involved. For example, for the database
productions, pre-processing involved ingesting and consolidating all sources into the MySQL
“Consolidated Database” (described below). Other pre-processing methods are described
below.
1. Database Files
The database MySQL Script and database backup files were organized into folders
(directories).
• “\Pre-Processing\Databases\Kayako Ticket Server\2013-05-16”
• “\Pre-Processing\Databases\Kayako Ticket Server\2020-12-16”
• “\Pre-Processing\Databases\Kayako Ticket Server\2021-03-11”
• “\Pre-Processing\Databases\Kayako Ticket Server\2021-03-14”
• “\Pre-Processing\Databases\Kayako Ticket Server\2021-03-17”
• “\Pre-Processing\Databases\Kayako Ticket Server\2021-03-23”
• “\Pre-Processing\Databases\Produced by Defendant\2021-01-28”
• “\Pre-Processing\Databases\Produced by Defendant\2021-02-04”
• “\Pre-Processing\Databases\Produced by Defendant\2021-02-05”
• “\Pre-Processing\Databases\Produced by Plaintiff\2020-11-27”
• “\Pre-Processing\Databases\Produced by Plaintiff\2020-12-26”
• “\Pre-Processing\Databases\Produced by Plaintiff\2021-02-04”
• “\Pre-Processing\Databases\Produced by Plaintiff \2021-02-05”
2. Attachment Files
Each directory of attachment files was stored in a directory and cataloged. The
attachment files were files that were part of the attachments to emails that were received by
the Kayako Ticket Server and parsed into tickets, ticket posts, ticket emails, ticket recipients,
and ticket attachments. Each attachment was renamed, encoded, and cataloged by the Kayako
Ticket Server parser software. Special Master organized these files into directories based on
when they were received, as many of these directories and files were identical and there were
many duplicates. Special Master pre-processed and prepared each directory to perform
destruction and omission analysis.
3. Directory Listings
Defendants, at the request of Special Master, produced directory listings for the
Kayako Ticket Server, Domain Who-Is Server, and the Developer Workstation named
“desktop-6h8msks” using a variety of tools. For the Linux servers, Defendants downloaded,
installed, and configured a utility named “Zabbix-Apache” and produced directory listings to
Special Master on more than one occasion.
Listings of files in directories produced to Special Master from the Kayako Ticket
Server, the Domain Who-Is Server, and the Developer Workstation were placed in the
directories listed in Exhibit 4.
Each directory listing was processed by a software program created by Special Master
to convert each of these listed files into a Comma Separated File (CSV). As CSV files, Special Master was able to import them into MySQL and create the new databases named
here:
• “files_kayako_support_2021_03_16”
• “files_kayako_support_2021_04_06”
• “files_kayako_support_2021_04_13”
• “files_domainwhois_2021_03_16”
• “files_domainwhois_2021_04_06”
• “files_domainwhois_2021_06_04”
• “files_desktop_6h8msks_2021_05_03”
Special Master then created SQL Scripts to move the contents of directory listings
from these databases into the Consolidated Database (described below). Special Master then
used these records to create scripts and file requests of Defendants.
4. HTML Productions
One production to Plaintiffs and one production to Special Master contained ticket
information in HTML file format. The HTML information in both productions included the
Ticket Number, Subject, Email Address, Contents, and Date columns from the ticket
(‘swtickets’) and ticket posts (‘swticketposts’) tables in the database tables.
To pre-process these files, Special Master first wrote custom software to convert these
HTML files into CSV files to facilitate importing them into the Consolidated Database.
However, due to the nature of the information contained in the “contents” column, the files
were always corrupt. So, Special Master modified the software to instead insert these HTML
tickets directly into the Consolidated Database for processing and production. HTML
directories are listed in Exhibit 5.
5. Programming Scripts
Special Master requested Defendants to provide the programming script files they
used for productions. Defendants provided some but not all. The files received were organized
in the directories listed in Exhibit 6 for pre-processing.
6. Miscellaneous Files
From time to time, Special Master received files that did not fit into any of the above
categories, and they were pre-processed from the directories listed in Exhibit 7.
E. Consolidated Database
Processing, analyzing, and producing information spread across multiple databases is
particularly challenging and time-consuming. To ensure accuracy and efficiency, Special
Master centralized all data into a single database. Thus, searching, producing, and cataloging
information became considerably faster.
The newly consolidated database for production was named “kayakodball”
(hereinafter “Consolidated Database”). In creating the Consolidated Database, over 250 SQL
scripts were created by Special Master to populate the tables in the Consolidated Database
with all the records in the source databases. SQL Scripts were also generated to compare
record counts to verify that no records were dropped when consolidating the database records.
Special Master imported each set of produced data sources into the “Consolidated Database”
and assigned a “Database ID” to distinctly identify each set of files. See Exhibit 8.
In the consolidated database, Special Master found 3,010,020 ticket records and
403,601,954 ticket-related records from all database versions reviewed. Thus, each ticket had
an average of 134 ticket-related records. Specifically, there were:
Record Type Total Records
Tickets (swtickets, tmp_data) 3,025,281
Ticket-Related Records 403,601,954
Ticket Posts (swticketposts,
temp_data, htmltickets,
kayako_searchterms tables)
Sub-posts or messages related to a ticket.
12,389,604
Attachments (swattachments table)
Records such as images, spreadsheets, or
documents that are linked to a ticket or
ticket post.
877,211
Table 1. Overview of tickets and ticket-related records
Collectively, there are 22 database versions, from all collected data sources, with a
cumulative file size of 82 GB and a total of 442,680,623 records across all versions of the
database.
F. Data Processing
To facilitate responsive and privilege searches, Special Master created scripts to add
the following columns to each table in the Consolidated Database that had been consolidated
from the source databases:
• “batesid” - This is a unique identification number or Bates Number generated
for each record in each table in the database. These numbers are unique within
each table, as each table is treated as a separate production file.
• “searchtermsfound” - This column contains a list of any of the responsive
search terms that were found for the record based on the production protocol.
• “ProducedBefore” - This column is set to “Yes” or “No” based on whether the
record was ever part of a previous production to the Plaintiffs.
• “Responsive” - This column is set to “Yes” if the record was responsive based
on the agreed protocol for the past collections and needed to be produced, and
“No” or blank if it was not responsive.
• “Related” - This column is set to “yes” if it is related to a ticket marked
responsive in a related table anywhere in the database.
Special Master then created scripts to populate the “ProducedBefore” column for each
record in the Consolidated Database that had been produced by Defendants to Plaintiffs, and
SQL scripts were generated to verify the records counts between the source databases and the
Consolidated Database. See Exhibit 9.
Special Master then created scripts to populate the responsive column for each table
and table records based upon the search terms in the agreed discovery protocol for past
productions. See Exhibit 10.
Special Master then created a table in the Consolidated Database named
“responsivetickets” (hereinafter “Responsive Tickets Table”). This table contained the Ticket
ID or Ticket Mask ID (Ticket Number) for any ticket that was marked as responsive in the
system. Special Master then created and executed SQL Scripts that populated the Responsive
Tickets Table with all the responsive ticket information from each of the responsive rows in
the source tables.
For records not marked as responsive, Special Master created scripts to mark records
as “related” in the database if the record was a related part of the responsive ticket. This
allowed Special Master to produce every column in every table for any information that is
contained in each responsive ticket. Thus, all responsive ticket records will be marked “Yes”
in either the related or responsive columns. Conversely, a non-responsive ticket will have the
column marked with a “No”.
G. Data Analysis
Special Master applied a variety of analysis methods to arrive at the conclusions in
this Report. Briefly, a few primary methods are described below.
One method used to determine if data was deleted, was to analyze index numbers in
each table. Tables assign index numbers automatically to each new record in sequential order.
This is known as auto-indexing. For example, the first record will be assigned the index ID of
1, and subsequent records are assigned the next sequential number. This can reveal the
maximum number of records in the table, as well as any gaps in sequential numbering created
by deleted records. Special Master applied this analysis to all the database tables.
Another method included comparing temporary records tables to live database records.
Programmers commonly create temporary tables in a database to test queries, test
programming code, update or delete data, and analyze data in a table. Temporary tables
should reflect the records in the live database. When a temporary table has records that no longer exist in the live database, this indicates they were used to test deletion scripts, which
were then applied against the live Database, and resulted in deleted records from the database.
In addition to the methods listed above, Special Master also reviewed and analyzed
directory listings from the file servers and Developer Workstation, reviewed PHP
programming scripts, reviewed attachment file directories, produced, and looked for missing
database records and attachments, compared previous production records to the live database
to identify missing or deleted records, and analyzed and compared 22 different databases
created from source information, among other things.
H. Database Production for Report
Accompanying this Report, Special Master provided both parties a MySQL database
with all the responsive records (per the search terms in the past agreed discovery protocol),
responsive attachment files, and Excel spreadsheets containing the analytics data used for
working data to provide the information, findings, and conclusions in this Report.
Special Master produced responsive database records and attachments in the Kayako
Ticket Server from the Consolidated Database. The database provided to the parties with this
Report had a file size of 26 GB.
To produce the ticket attachment files, Special Master used custom software and an
industry-standard dtSearch engine to perform full-text indexing and searching for responsive
attachment files. This software assigns its own Bates Number to each attachment, creates a
spreadsheet with reference information, and includes the responsive records flagged with
matching responsive keywords. Hyperlink columns in the spreadsheet allow the parties to
quickly open PDF and Text files with the information from content fields in the database that
had voluminous text content
To produce the production database, Special Master created
a database named “kayakodball_prod” (referred to as the “Consolidated” database in this
report) to serve as the production database for this Report. and then created scripts for each
table. Next, he copied and indexed any records that were marked responsive or related. Special Master then scripted and ran table maintenance to optimize the indexes and remove
any disk space that was no longer needed for this database.
Special Master then backed up each of these new databases and compressed and
archived the database backups. Special Master then created scripts to compare the record
counts in each table in each of these production databases and compared the results. See
Exhibit 11. The results were identical, and Special Master will use the second method
whenever producing records in the future.
The final production database is a relational database that has been reindexed to
optimize it for reporting and legal review by the parties. Most of the original indexing of the
tables was done to optimize it for being a live database, and these unnecessary indexes have
been removed to save disk space.
VI. DEFENDANTS FAILED TO PRESERVE ESI
Defendants failed to preserve potentially relevant ESI for this matter pre-litigation,
post complaint filing, during discovery, and even after the appointment of Special Master.
Defendants have sophisticated IT skills, requisite infrastructure, and have had ample time to
implement backup management software and procedures. Despite these circumstances, they
failed to preserve and produce relevant ESI. The exact extent of Defendants’ backups is
unknown as Defendants obfuscated their backup management systems for servers, files, and
databases.
A. Defendants’ Have the Requisite IT Skills and Software Developer Knowledge
to Backup and Preserve ESI
Notably, Defendants resell backup services to their clients. To preserve and produce
data, a company must have backup systems and procedures in place. Through their publicly
available website, found at https://onlinenic.com/en/, Defendants offer, market, and resell
backup services. A copy of OnlineNIC Inc.’s webpage is attached to this Report as Exhibit
12. Thus, Defendants have demonstrated they have the capacity and infrastructure to backup
and preserve responsive ESI.
Secondly, Defendants have advanced software developer skills. To produce or delete
data from a database, a company must have sufficiently advanced software developer skills.
In this case, Defendants created PHP script files (text files with programming code) to query
the Ticket Database and produce responsive materials to Plaintiffs and Special Master. In
producing responsive ESI to Special Master on March 25, 2021, April 15, 2021, and April 26,
2021, Defendants used 28 PHP script files. See Exhibit 13. Their third production even
included three additional PHP script files. See Exhibit 14. Furthermore, Defendants used a
SQL statement file (programming code used to copy and delete database records and
attachment files) and likely used these scripts or similar scripts to retrieve responsive
information for the productions to Plaintiffs. See Exhibit 15. Defendants’ repeated
employment of PHP scripts and SQL statements demonstrates their advanced level of
software developer skills. Therefore, Special Master concludes Defendants possess the
requisite competence to backup, preserve, produce, and even delete responsive records from
the Ticket Database.
B. Defendants Have Inadequate Backup Processes Despite Having Since At
Least October 2019 to Create Comprehensive Backup Procedures
Defendants’ backup processes are inadequate. The federal rules of civil procedure
require a party to take reasonable steps to preserve ESI from the moment they can reasonably
anticipate litigation. Fed. R. Civ. P. 37(e). In this case, Plaintiffs filed the complaint on
October 28, 2019. Dkt. No. 1. At a minimum, Defendants’ duty to preserve arose at least by
the time served with Plaintiffs’ complaint. Special Master was appointed on March 3, 2021.
Dkt. No. 72. By the time of Special Master’s appointment Defendants had over one year to
address backup procedures and data retention policies in accordance with their preservation
obligations. However, despite this extensive period for implementing comprehensive backup
systems such as basic backup management software, Defendants failed to do so.
C. Alternatively, Defendants Have a Robust Backup Management System but
Have Obscured or Hidden Backups
Defendants obfuscate their backup processes. Generally, data backup for an online
services company is crucial, not only for litigation, but also for business security, stability,
compliance, and protection of client and intellectual property. Hence, companies typically act
accordingly to institute clear backup policies and procedures. However, in this case,
Defendants have not identified or defined their backup management system, and in fact, have
been untruthful.
For example, during discovery, Defendants claimed no backups existed; however, this
was not true. There are multiple instances of backups existing in seemingly random places.
First, SQL backup files and attachment files were found on both servers in multiple locations
and archive files. See Exhibit 16. In addition, Special Master found 11 unique backup scripts
for the databases. Exhibit 17 shows the 11 backup scripts with the metadata. Furthermore,
some of the productions Defendants made to Plaintiffs would have required creating backups,
which were created on developer workstations but were not disclosed. Id. Further still, even
though Defendants claimed that no programming files were on the servers, the files were
present in their third production to the Special Master on April 26, 2021. See Exhibit 18.
Finally, around May 2021, Special Master found a fifth database on Defendants’ server they
had failed to provide or disclose. Thus, Defendants have a history of being untruthful or
misleading regarding the nature and extent of their backups and backup systems and
procedures.
Therefore, given Defendants’ sophisticated IT skills, a business need for backups, and
a history of untruthfulness, Special Master believes it is likely there are additional files,
including developer script files, on undisclosed or unlocated developer workstations or
servers.
VII. DEFENDANTS DESTROYED DATA
Not only did Defendants fail to preserve ESI, but they also deleted significant amounts
of records. In fact, Defendants deleted 52.35% of the ticket-related database records (11,059,388 records) in the Ticket database. Of these deleted records, Special Master
estimates 30% of those records (based on the percent of responsive records from July 1, 2015
– July 15, 2020), or 3,317,816 were responsive to the agreed past discovery protocol. This
ticket-related data is located in the following six tables: swtickets; swticketposts;
swticketnotes; swattachments; swticketmergelog; and swauditlogs. Broadly speaking,
Defendants employed two primary methods to delete data: 1) deletion using the Kayako
SupportSuite Application; and 2) deletion directly from the Ticket Database using PHP and
SQL scripts.
Special Master utilized a variety of analysis methods to identify instances of deleted
records. This Report narrative covers the most relevant examples of the record types that
Defendants deleted, including ticket posts, ticket notes, attachment files, and HTML files.
A. Defendants Deleted Records Using the Kayako SupportSuite Application and
Obscured User Activity by Deleting Audit Logs
For earlier ticket records, Defendants deleted records using the Kayako SupportSuite
Application (hereinafter “Application”). When using the Application, user activity is recorded
in the audit logs. An example of this is when records are deleted from the ticket and ticketrelated tables. According to the Application’s audit logs, Defendants deleted: 4,106,268
tickets; 85 ticket posts; 8,991 ticket notes; and 25,253 e-mail recipients. This is a total of
4,140,597 records. As summarized in Table 2 below, 93% (3,863,451 records) of the deleted
records were created in 2008, 2009, and 2010, while newer records constitute a much smaller
portion of the total.
Year Created Number of Deleted Database Records
2020 302
2019 22,348
2018 3,209
2017 2,926
2016 548
2015 16,850
2014 1,204
2013 25,859
2012 52,596
2011 151,304
2010 809,668
2009 1,782,491
2008 1,271,292
Total 4,140,597
Table 2. Total records deleted using the Kayako SupportSuite Application (2008 –
2020), organized by record creation date.
However, not only did Defendants delete potentially responsive records using the
Application, but they also deleted the Application’s audit logs. Audit logs provide some
information about when a user deletes records. In this case, there were over 15 million audit
log records. Defendants deleted approximately 7.5 million of those records (roughly onehalf). Special Master was able to recover 459,226 audit log records from a May 16, 2013
database. For this reason, the records in Table 2 are organized by the records’ creation date
data, as it was the only information consistently available. This made analyzing user activity
considerably more challenging and thereby limits Special Master’s ability to determine a
precise timeline of when Defendants deleted these records, or even when “most” of the
records were deleted
Furthermore, users require special permission to access the database and delete the
audit logs. Because audit logs are so small and require little storage, the primary benefit to
deleting audit logs is to hide user activity (such as deleting tickets). Therefore, Special Master
concludes that the audit logs were deleted by Defendants to hide their deletion activity.
B. Defendants Deleted Records Directly from the Ticket Database After
Production
In more recent years, Defendants began deleting records directly from the live Ticket
Database using PHP and SQL scripts. Based on the creation date of PHP and SQL scripts used for deletion, Special Master found Defendants began using this technique in more recent
years.
It is considered best practice, and certainly a common practice, to never delete data in
a database because it can create problems with the database’s referential integrity. For
example, when deleting tickets directly in the database, a programmer can leave orphan
records (e.g., a ticket record is deleted, but associated ticket post and ticket attachment records
remain). Here, Defendants deleted records in some tables but not all related tables, thereby
creating orphan records in multiple places. If Defendants wished to make the deleted tickets
inactive, they should have closed tickets in the Application by setting the ticket status field to
“Closed.” Instead, Defendants’ deleted records directly from the database and left a lot of
orphaned records behind as markers of deletion activity.
Further indicators of data deletion directly from the Ticket Database existed in a
comparison between Defendants’ productions and to the live Ticket Database. In fact, this
showed that Defendants even deleted database records in the live database after initially
producing the records to Plaintiffs. During multiple productions from Defendants, some
records were contained in one production but were missing from subsequent productions and
from the live Ticket Database itself.
For example, on March 23, 2021, Special Master collected Defendants’ Ticket
Database and compared it to Defendants’ production dated November 27, 2020. By relying on
the Ticket Mask Id, Special Master found that 208 of the 20,615 distinct tickets produced
November 27, 2020, were missing from the Ticket Database on and after December 16, 2020.
See Exhibit 19.
In another example, on or around March 12, 2021, Defendants’ programmer created a
table named ‘tmp_data’. See Exhibit 20. This temporary table contains a subset of the
columns found in the live Ticket Database table named ‘swtickets’ and ‘swticketposts’. See
Exhibits 21 and 22. In fact, there are 3,716 matching ticket records between these two tables
(76% match). Id. However, the tmp_data table has 1,146 ticket posts that were since deleted
from the live Ticket Database table. Id.
Similarly, on or around March 12, 2021, Defendants created a table named
‘temp_data’. See Exhibit 23. This table contains a subset of the columns found in the live
Kayako Ticket Database table, “swticketposts”. Id. There are 1,278,354 matching ticket posts
between the two tables. Id. However, there are 89,294 ticket posts in the ‘temp_data’ table
that were since deleted from the live Ticket Database table. Id.
In addition to the records deleted between productions to Plaintiffs, Special Master
found that Defendants deleted 1,281 unique tickets and 445 related ticket posts from the live
Ticket Database after these tickets and posts were initially produced to Special Master. These
records are reproducible using the Consolidated Database created by Special Master and will
be provided to the parties in a new production set.
For records deleted directly from the database, Special Master cannot determine when
the records were deleted. When records are deleted directly from the database, as opposed to
via the Application, there are no audit logs; a record created 10 years ago could have been
deleted 8 years ago or 8 days ago. Had Defendants provided regular backups, Special Master
could have provided a timeline for deleted records. In this case, Defendants did not maintain
or provide regular backups, and therefore he cannot determine when records were deleted.
However, Special Master did discover a 2013 database. By recovering the 2013
database, Special Master gleaned some context behind Defendants’ deletion activities. In
particular, Special Master noted that post-2013, Defendants used programming scripts to
delete more records from the Ticket Word table (swticketwords) and Post Index table
(swticketpostindex) than any other table including tickets, ticket posts, attachments, and audit
logs. Special Master believes Defendants were removing word index records so that search
terms related to litigation would not be linked to specific tickets or ticket-related records. The
Special Master found that the Defendants deleted 2,919,130 database records between May
16, 2013, and December 16, 2020. The Special Master also found an additional 4,102,283
database records that were deleted and missing in the live ticket database as of March 23,
2021. This results in a total number of 7,021,413 orphaned records which means that there is
an incomplete set of records for the tickets.
Below is a summary of the surviving responsive tickets:
• No date filter: 65,810 tickets
• Filtered for July 1, 2015 – July 14, 2020: 32,906 tickets
• Defendants previously produced to Plaintiffs: 32,054 tickets
• Additional tickets produced by Special Master to Plaintiffs: 852 (2.7% more)
C. Defendants Deleted Ticket Posts
Defendants deleted ticket post records from the live Ticket Database and thereby
created orphaned ticket post index records. When a database record is deleted from a child
table, such as a subsequent ticket post, but not delete the corresponding record from the parent
table (based on the unique identifier), the result is an “orphaned record”. In this case, Special
Master found orphaned records by analyzing and comparing the unique Ticket ID fields in all
14 ticket-related tables. Specifically, Special Master found 987,862 orphaned ticket post index
records, which is 3.6% of the total. See Exhibit 24. This means that the actual ticket posts, that
originally referred to these orphaned records, have been deleted. It is also noteworthy that
only 158 of these unique posts are also missing from the “temp_data” table of posts and is a
further indication of Defendants deletion from the database.
Below is a summary of the surviving responsive tickets posts:
• No date filter: 253,506 ticket posts
• Filtered for July 1, 2015 – July 14, 2020: 93,669 ticket posts
• Defendants previously produced to Plaintiffs: 91,053 ticket posts
• Additional tickets posts produced by Special Master to Plaintiffs: 2,616 (2.9%
more)
D. Defendants Deleted Ticket Notes
Defendants deleted ticket note records from the Ticket Database. Database indexes
commonly assign unique numerical identifiers in consecutive order to new records
automatically. This is known as auto-incrementing. Gaps in the sequential numbering of an
index table are evidence that records were deleted. Special Master applied this analysis to the
‘swticketnotes’ table, where ticket notes are stored. He identified 612,729 ticket note records were deleted, representing 83% of the total ticket notes that were in the system. See Exhibit
25. Audit logs show that 8,991 of these ticket notes were deleted via the Application, and the
rest were deleted directly from the live ticket database.
E. Defendants Deleted Attachment Database Records
Not only did Defendants delete attachment files, as discussed above, they also deleted
attachment database records. Many of the attachment database records were responsive to
Plaintiffs’ discovery requests and the agreed past discovery protocol of the parties
As discussed above, Special Master determined Defendants had a total of 432,033
attachment database records. By the time of Special Master’s final collection on March 23,
2021, Defendants had deleted 296,349 attachment database records – 68.59% of the total –
leaving only 135,684 attachment records. Of the remaining 135,684 attachment records,
24,719 were responsive (18.22%). Id. Accordingly, of the 296,349 deleted attachment
database records, an estimated 53,995 attachments may have been responsive (18.22%).
Additionally, Defendants created orphaned attachment files when they deleted ticket
records, such as tickets or ticket posts, that had attachments, and failed to delete the
referenced attachment files. By comparing the attachment file directory to the live Ticket
Database, Special Master found 8,165 orphaned attachment files representing 8.22% of the
total. The 8,165 orphaned attachment files indicate Defendants deleted the corresponding
ticket records in the database. Because any given ticket record could have more than one
attachment, Special Master is unable to conclude how many ticket records were deleted.
Furthermore, of the 8,165 orphaned attachment files, Special Master used the agreed
past discovery protocol and determined that 97 are responsive. However, because they were
orphaned, they are no longer linked to a specific ticket or ticket post, and therefore contextual
information is missing for those attachment files. See Exhibit 28. Exhibit 28 shows the
responsive files categorized and totaled by file type.
Special Master noted that originally the total number of attachment database records
and the total number of attachment files was 432,033 each. Defendants deleted more
attachment files in the file system (331,390) than attachment database records (296,349). A user must have special permissions, and more importantly, expertise, to delete database
records or attachment files either directly from the database or the file system. The deleted
attachment files and records cannot have happened by accident. Therefore, Special Master
concludes it must have been intentional.
Below is a summary of the surviving responsive database attachment records:
• No date filter: 44,004 attachment database records
• Filtered for July 1, 2015 – July 14, 2020: 23,464 attachment database records
• Defendants previously produced to Plaintiffs: 22,223 attachment database
records
• Additional attachment database records produced by Special Master to
Plaintiffs: 1,241 (5.6% more)
F. Defendants Deleted Attachment Files
Defendants deleted attachment database records from the Ticket Database, as well as
attachment files from the file system. The Ticket Database stores an attachment file’s
location, original file name, and stored file name as a database record. See Exhibit 26. The
actual attachment files, however, are stored in the file system on the server, outside the
database. Id. Attachment files consist of various file types such as images; documents; and
spreadsheets. Critically, attachments may be the most important part of a ticket, much like an
email attachment may be more important than the email message itself. In this case,
attachments included copyright notifications; email conversations; legal demands; account
summaries; complaints; letters; legal information; financial information; and more.
Special Master determined that over time, Defendants had a total of 432,033
attachment files in the file system. While a ticket can have more than one attachment file,
each attachment file has an associated database record in a one-to-one relationship. Therefore,
since there were 432,033 attachment files in the file system there were also 432,033
attachment records in the database.
However, by the time of Special Master’s final collection on March 23, 2021,
Defendants had deleted 331,390 attachment files – 76.70% of the total. Of the surviving 100,643 attachment files, 23,464 were responsive (23.31%) to the search terms in the agreed
past production protocol. See Exhibit 27. To determine how many deleted attachment files
would have been responsive, Special Master used the 23.31%, responsive rate (the only
number available). Under this methodology, approximately 77,247 deleted attachment files
would have been responsive. Furthermore, the responsive rate for both the surviving and
deleted attachment files would likely be higher using the expanded list of search terms (194
instead of 42) in the new discovery protocol.
Finally, it is important to note that while some attachment files are a single file like a
PDF, others are compressed archive files like zip, tar, or gz files, and contain many individual
files. In other words, while the number of deleted attachment files is 331,390 there are likely
many more deleted individual files within those attachment files.
G. Defendants Deleted HTML Files, Tickets, and Ticket Post Records
Defendants deleted tickets after initially producing them to Special Master. On March
12, 2021, Defendants produced to Special Master a backup of the live Ticket Database. The
backup contained HTML files that were named in the same format as the original HTML
production Defendants made to Plaintiffs on July 15, 2020. Database record comparison
revealed they also deleted 1,073 tickets after initially producing them to Special Master in the
backup of the live Ticket Database on March 12, 2021.
Based on a group of search terms, Special Master requested Defendants produce 251
matching HTML files that were located on the Kayako Ticket Server as they were not
included in any of the previous productions. Defendants produced them to Special Master on
April 26, 2021. These newly produced HTML files contained 334,400 partial ticket post
records and 8,313 partial ticket records. A subset of 190 of these tickets were responsive
across all ticket-related tables.
Of the 190 responsive tickets, 31 of the tickets and 11,700 related ticket posts were
deleted from the Ticket Database. In addition, the HTML files contained 1,073 tickets that
Defendants since deleted and no longer exist in the Ticket Database. Of these 1,073 tickets,
14 tickets were responsive and had been produced to Plaintiffs before. Id. Finally, the PHP Script code used to produce these HTML files is not located on the servers. This is additional
evidence that Defendants deleted records from the Ticket database during discovery
All HTML files discussed above are named with a prefix consisting of a search term
from the parties’ past discovery protocol. “JenryHaris” is one of the search terms in the past
discovery protocol. When sorted alphabetically, Special Master saw that all files, beginning
and ending, were named with the JenryHaris prefix. See Exhibit 29. This indicates that there
were most likely other files with prefix names for other search terms that were deleted or
removed after they were created on March 12, 2021, and were not available for subsequent
productions. Relatedly, Defendants did not produce any HTML files or records related to
JenryHaris in the July 15, 2020, HTML production to Plaintiffs. See Exhibit 30.
H. SQL Files and SQL Backups of Productions
Defendants deleted SQL files and databases they previously provided Plaintiffs.
Generally, to produce a database, a programmer writes and executes SQL scripts to define the
scope of the produced database. Defendants produced MySQL Database Script Files
containing records for MySQL Database productions to Plaintiffs in December of 2020
through February of 2021. However, neither of these databases, or any associated
programming or SQL backup scripts, are located on any of the servers or workstations. This
poses two concerns. First, Special Master is unable to conclude how these production files
and backups were created to evaluate their adequacy at producing responsive records. Second,
the question remains of whether they were deleted from one of the two servers, whether they
were produced on the Developer Workstation and then deleted, or whether they were
produced on an undisclosed developer workstation.
I. Defendants Used PHP and SQL Deletions Scripts to Delete Ticket Database
Records and Attachment Files
Defendants' programmers created specific PHP and SQL scripts to delete database
records and attachment files. Scripts can be used to produce responsive records as well as to
delete records from a database. Special Master found 28 PHP scripts and one SQL script that
used record deletion language. See Exhibit 13. Here is a sample of some of those scripts:
• delete from swattachments where ticketid...
• delete from swauditlogs where ticketid...
• delete from swescalationpaths where ticketid...
• delete from swparserlogs where ticketmaskid...
• delete from swticketlocks where ticketid...
• delete from swticketmergelog where oldticketmaskid...
• delete from swticketmessageids where ticketid...
• delete from swticketpostindex where ticketpostid...
• delete from swticketpostlocks where ticketid...
• delete from swticketposts where ticketid...
• delete from swticketrecipients where ticketid...
Special Master can only conclude that Defendants programmatically deleted files
using PHP and SQL scripts, and tried to hide evidence of their use by deleting the scripts
themselves.
J. Text File Created by Defendants’ PHP Script Files Reveals Deleted
Attachment Files and Attachment Database Records
Defendants deleted 472 physical attachment files using PHP scripts. Defendants
demonstrate their ability to copy or delete files to and from servers with PHP Programming
Scripts, and to work with remote and local databases. On March 17, 2021, Defendants created
two PHP script files named ‘zenghy_tmp_20210317.php’ and
‘zengy_attachment_copy_file.php.’ The latter PHP script generated a text file named
‘/root/duo_files.txt.’ See Exhibit 31. This file was located on the Kayako Ticket Server, was
requested by Special Master, and was delivered on April 26, 2021. The file references 472
unique attachment files by their file paths. See Exhibit 31. However, when Special Master
cross referenced the directory lists from both servers and developer workstation, the 472 files
were not listed, indicating those attachment files had been deleted.
In addition, Special Master found that a subset of the 472 attachment files were
missing corresponding database records. Special Master cross-referenced all 472 files against the ‘swattachments’ table to look for matching database records. Notably, 34 of the listed
attachment files did not have matching database records in the cross-referenced table. See
Exhibit 32. The fact that the 34 records were no longer included in the ‘swattachments’ table
indicates Defendants deleted those attachments from the database with the PHP script file.
Notably, this deletion occurred just before Defendants’ first production to Special Master on
or about March 17, 2021.
Conspicuously, the ‘/root/duo_files.txt’ file also revealed other actions. For example,
the file was used to copy files to a directory, '/home/jumpol/script/ticket_20210106/files/',
which itself no longer exists. Furthermore, and most notably, the text file references database
servers, other than the live database server, indicating that there are undisclosed servers.
VIII. DEFENDANTS WITHHELD DATA FROM PLAINTIFFS
There are multiple indicators of withheld records within Defendants’ productions to
Plaintiffs including using an incorrect date filter; applying restrictive searches; providing
incomplete database records; excluding attachment files; and more.
A. Defendants Excluded Two Years’ Worth of ESI with Restrictive Date
Filters
Defendants withheld multiple years’ worth of ESI. The parties agreed to a date filter of
July 1, 2015, to July 14, 2020, for their past productions. See Exhibit 2. However, for each
production made to Plaintiffs, Defendants used different date ranges. For example, one PHP
script Defendants employed to query the Ticket Database for responsive records requested
data back to 2015, while a subsequent production only went back to 2017. See Exhibits 33
and 34. The varied date ranges were not consistent with the agreed past production protocol
and resulted in withholding responsive data from the Plaintiffs.
B. Defendants Withheld ESI by Executing Incomplete Database Searches
Defendants withheld responsive ESI found in ticket-related tables. The parties’
discovery protocol for past productions included a list of search terms. See Exhibit 2. The
protocol did not specify which database tables or columns to search for the responsive records
and did not specify if search terms should apply to ticket attachments. However, it is common and a best practice to apply search terms to the entire unit of an item. For example, in an email
production, search terms should be applied against the entire email by including the header,
content, metadata fields, and all attachments to an email. Applying this same logic,
Defendants should have applied the search terms to the ticket database table, and all ticketrelated database tables.
In the present case, Defendants executed search terms inconsistently. In some
instances, Defendants’ responsive searches were restricted to the email, subject, and contents
columns of the tickets or ticket posts table. See Exhibits 35 and 36. In other scripts, search
terms were applied in some but not all tables. See Exhibit 37. Therefore, Defendants withheld
responsive ticket-related ESI.
C. Defendants did not Search Attachment Files
The agreed past discovery protocol contained search terms. However, the search terms
were not applied against the full text of each ticket attachment file to determine
responsiveness. All these attachments were subsequently not produced, and some were in fact
deleted, as discussed in Section VII. Special Master verified this by creating a full-text search
index of all the attachment files to search the attachment files for each search term.
D. Defendants Withheld Responsive Ticket-Related Tables and Table Columns
in Early Productions
Defendants did not produce complete ticket records to Plaintiffs. As discussed
previously, complete responsive ticket data from the relational Ticket Database requires
consolidating information across multiple related tables. However, Defendants only included
a subset of all ticket-related database tables in their productions. See Exhibit 38. Specifically,
Defendants withheld records from most of the ticket and ticket post-related tables described in
productions conducted before December of 2020.
Furthermore, Defendants’ productions not only excluded entire database tables, as
described above, but their productions also failed to include all columns within the tables that
they did produce. See Exhibit 39. As an analogy, imagine a party producing responsive Excel
spreadsheets, but then deleting certain columns in the spreadsheet. Therefore, by not providing all ticket-related tables, and all columns in each database table, Defendants
withheld responsive ticket-related information from Plaintiffs.
E. Defendants Produced Limited Attachment Files that were not in a
Reasonably Useable Format
Defendants produced 21,455 ticket attachments to Plaintiffs in rather unusable format.
Ticket attachments to responsive tickets include many file types such as PDFs; emails; HTML
documents; spreadsheets; Word documents; PHP files; compressed files (e.g., rar, gz, and zip
files); Text files; and ISO. As discussed previously, attachment content can be the most
important part of a ticket, much like email attachments to an email. Special Master reviewed
numerous attachments and found the attachments were particularly important to provide
context and information about the tickets. In fact, among the withheld attachments were:
copyright notifications; email conversations; legal demands; account summaries; complaints;
letters; legal information; financial information, and more.
In this case, Defendants produced 21,455 attachments files to Plaintiffs out of a total
of 100,643 surviving attachment files. However, Defendants produced a greater number of
responsive attachment database records, 21,577, on February 05, 2021. This is unusual
because attachment files and attachment database records exist in a 1:1 relationship. Thus, the
discrepancy further confirms that Defendants deleted records directly from the database and
broke referential integrity.
Below is a summary of the surviving attachment files:
• Defendants previously produced plaintiffs: 21,455 attachment files
• Additional attachment files produced by Special Master to Plaintiffs: 2,122
(9.9% more)
Furthermore, the attachment files Defendants did produce, were not in a reasonably
useable format for Plaintiffs. When Defendants produced attachments, there were two
problems: 1) the attachments were missing file extensions; and 2) they contained virus files.
First, because the files were missing file extensions, they were not easily opened or
viewable. As mentioned, attachment files are stored outside the database. The database stores the file name and file extension of each attachment file. The Kayako SupportSuite
Application automatically looks up the file extension and opens the attachment file. Although
Defendants provided the files as they exist in their file system, Plaintiffs, do not have
Defendants’ Kayako SupportSuite Application, and therefore were unable to open and view
the files. Therefore, to provide files in a reasonably useable format for Plaintiffs, Defendants
should have at a minimum supplied the file extensions to each attachment file, provided a
license to use the Application to view the attachments, and/or provided PDFs of all text-based
attachments.
Second, the group of attachment files provided to Plaintiffs contained viruses. It is
common best practice to run anti-virus software before a file production to opposing party.
However, Defendants reported to Special Master that its servers and workstations do not have
anti-virus software. In this case, there were over 150 virus files among the attachment files
provided by Defendants. Failing to run anti-virus software against files being produced to
Plaintiffs posed a risk to Plaintiffs’ computer systems. Therefore, of the limited group of
attachment files provided, they were not provided to Plaintiffs in a reasonably useable format.
F. SQL Backup File Requested but Withheld
Special Master requested all database files and backup files in initial and subsequent
production requests. Defendants produced files from developer workstations, but many
requested files were omitted or incomplete. In fact, there was a specific file Special Master
identified in a directory list. See Exhibit 40. The Special Master on May 9, 2021, requested
that specific file named “/home/kayako/mysql.12.dump.gz”, which is a Ticket Database
backup. Defendants responded to the request on May 10, 2021, that the file does “not exist”.
However, the file did exist as shown on the directory listing provided by Defendants on April
06, 2021. When questioned further, Defendants claimed they deleted this file to free up space
on their server on April 11, 2021, in an email dated June 02, 2021, which contained this text:
The server (whois.onlinenic.com) [sic] running out of space around Apr 11.
As a server mainly for shared hosting servers’ backup and sending out
notification [sic], it was a temporary station to place some processing files in this legal case, for example to analyze so it would not affect stableness of
ticket server, or to upload files from it, etc. When the server was running out
of space, the first move was to clean those temporary files. I think server’s
[sic] admin cleaned a couple of temporary users (folders) to leave room for
SHARED hosting servers.
As shown in the email above, Defendants claimed the file had been deleted from the
“temporary station to place some processing files in this legal case.” Special Master asked
Defendants by email, “Before deleting files to make room on the server, did your clients
backup or copy the files that were deleted to another computer, hard drive, storage device, or
cloud backup service? If so, where are those files now?” Defendants answered,
“Unfortunately, the server admin did not keep a copy of the files when cleaning server.”
On June 7, 2021, the Defendants produced a new server directory list requested by the
Special Master. The Special Master analyzed the directory list and confirmed the Ticket
Database backup file, “/home/kayako/mysql.12.dump.gz”, was no longer on the server. The
requested Ticket Database backup file, “/home/kayako/mysql.12.dump.gz,” was never
provided to the Special Master.
Defendants had other options to free up space and did not need to delete the requested,
directly responsive, backup file. Not only could Defendants have moved (or copied) the files
to another server or purchased and used an external hard drive (less than $100 USD), they
could have instead deleted older versions of much larger files. For example, Defendants
elected to preserve 56 larger files created in 2017 and 2020 instead of preserving the file
named “mysql.12.dump.gz” directly responsive to the case. See Exhibit 41. Furthermore,
there were triplicate files that existed, meaning Defendants could have deleted older versions
of the same exceptionally large file. Id. Instead, they deleted a directly responsive, relatively
small file directly responsive to the case under pretense of freeing up space.
IX. DEFENDANTS CONDUCTED “DATA DUMPING”
Defendants created a burden for Plaintiffs by producing a significant amount of ESI
that was not responsive to the agreed past discovery protocol (often referred to as “data dumping”). Data dumping is the practice of deliberately producing a significant amount of
non-responsive information and is often discouraged by courts. Generally, the intent is to
make it more difficult and expensive for the opposing party to find important evidence and
thereby receive a strategic advantage. By reviewing a subset of 27,823,240 records that
Defendants produced to Plaintiffs, only 5,096 records were responsive. These examples are
shown here:
Table Total Records Responsive
Records Responsive
Percent
swcommentdata 163,502 3,892 2.38%
swcountryinfo 79,440 0 0.00%
swticketnotes 128,770 5 0.004%
swticketpostindex 26,288,906 0 0.00%
swticketpostindex 529,528 1,046 0.19%
swusers 633,094 153 0.28%
Table 3. Sample of productions comparing number of responsive records to total
records provided
X. RECORDS DELETED & WITHHELD FROM SPECIAL MASTER
Special Master made a great effort to collect databases and other information from
Defendants via email, skype, phone calls, and remote computer sessions for information
needed for this Report. Sometimes, Special Master requested information that was not
provided by Defendants. After discovering information withheld, Defendants sometimes
produced the specified items to the Special Master after a special request. In one example,
Special Master found that Defendants’ PHP scripts generate four text files (ticketid.txt,
ticketmaskid.txt, email.txt, and postid.txt). Despite a specific request for the files, Defendants
never produced them. Other times, Special Master had to make multiple requests for
information before Defendants eventually complied. In a few instances, Defendants stated the
information had been deleted and was no longer available. Thus, on multiple occasions,
Defendants withheld and deleted records from Special Master, even though the Special Master has documentation in server directory lists to show the files did exist on the server
within the past two months.
XI. REPORT EXHIBITS AND PRODUCTION TO THE PARTIES
This Report is derived from a tremendous amount of data and information. To fully
understand the information in this Report, the parties were provided the following
supplemental information:
• The “Special Discovery Master’s Data Destroyed or Withheld Report” and all
exhibits (2 files, 13MB);
• MySQL database with all the responsive records, across all databases collected
by Special Master, per the past agreed discovery protocol (24 GB);
• Attachment files with all the responsive records per the past agreed discovery
protocol (23,464 files, 2.6 GB); and
• Excel spreadsheets containing the analytics data used to provide the
information, findings, and conclusions in this Report (33 files, 29 MB).
The cumulative file size of this Report and all supplemental information is over 26
GB.
XII. PARTIES’ COMMENTS TO DRAFT REPORT
The parties were provided two draft copies of this Report filed on July 12, 2021. The
parties received the first draft on June 9, 2021, and the second draft on July 7, 2021. Both
parties had an opportunity to provide comments. Those comments are included with this
Report as Exhibits 43 and 44 and are discussed below.
A. Plaintiffs’ Comments
Plaintiffs’ comments sought two additions. One, to clarify that based on Defendants’
own admissions, they deleted at least one database backup during discovery. See Exhibit 43.
Two, Plaintiffs sought to have Defendants’ employees who provided information for this
Report be individually named. Id.
On Plaintiffs’ first point regarding a known deleted backup, Special Master believes
Defendants have done several database backups and file copy backups of the database (see references to the 2013 database in the Report). However, the database backups and file copy
backups that existed on servers and workstations have been deleted. On Plaintiffs’ second
point regarding employee identification, Special Master included the email address, partially
obfuscated, to reference but not identify employee names who are not officers of Defendants.
B. Defendants’ Comments
Defendants’ comments were more extensive and are included in Exhibit 44.
Defendants' comments are summarized here: 1) Defendants regularly deleted records to
improve the aging Ticket Database’s functionality; 2) Defendants’ expert performed a
waterfall analysis to show 44 non-recoverable tickets and posed hypotheses of what happened
to them; 3) the Application itself may be responsible for deleted audit log files; 4) backup
files discovered by Special Master were not in fact backups, but rather files that were created
for Special Master; 5) PHP files referenced in the Report do not contain deletion language and
created for responding to discovery in this case; and 6) date ranges in Defendants’ PHP scripts
written for production are irrelevant and seeks clarification that Defendants produced records
going back to 2015 or not.
Special Master carefully reviewed Defendants’ comments. In many ways, Defendants’
responses support the most important concerns Special Master raised in the Report. In fact,
Defendants do not dispute deleting records, they instead try to justify why. Defendants deleted
data post case filing, during discovery, after productions, and after Special Master’s
appointment.
Special Master highlights the following points:
• Defendants did not state one step, method, process, or system they used for
data preservation.
• Servers and workstations were not backed up.
• Defendants failed to create and maintain regular backups of their database,
even post case filing.
• Defendants failed to use anti-spam/anti-virus software on their servers or
workstations.
• Defendants allege database performance issues caused some problems.
Defendants’ Kayako Ticketing Database was approximately 7.7 GB.
Special Master did not notice performance issues with the larger
Consolidated Database (24 GB).
• Defendants continue to use the Kayako Ticketing Database that they allege
was unstable, poor performing, spam-riddled, and un-backedup, even after
purchasing the new Zoho Ticketing Database in October 2020.
• Defendants speculate that some audit logs were never written due to
“glitches” in the system resulting from “several concurrent deletion
requests getting submitted to the system.” Special Master strongly disputes
this assertion.
• Defendants do not explain why they would delete records directly from the
database (causing orphaned records) instead of via the Application.
• Defendants failed to address why so many attachment files and database
records were deleted, why surviving attachments files were not searched,
or why they were provided to Plaintiffs with viruses.
• Defendants did not explain why they data-dumped 27,818,144 nonresponsive records on Plaintiffs.
Finally, Defendants’ experts’ “waterfall analysis” is flawed.
• Defendants selected only those tables that supported their desired
conclusion. The waterfall analysis does not include the many ticket-related
tables that contained large numbers of orphaned records, such as:
swattachments; swticketrecipients; or swticketposts.
• Defendants’ analysis is based on the swticket table outward. Conversely,
Special Master’s analysis is based on an outward analysis from all ticket
and ticket-related tables. Special Masters’ more expansive analysis was
necessitated by the high number of orphaned records across many ticket
and ticket-related tables.
• The analysis did not explain deleted ticket database records and attachment
files.
• Defendants justify data exists because it is in “temp_data,” even though the
table was never produced to Plaintiffs.
• The waterfall analysis refers to a date range beginning “July 1, 2015”, but
fails to specify what end date was applied.
• The analysis focuses on deleted records but is silent concerning the issue of
withheld records to the Plaintiffs. There was no explanation for why
searches for responsive records were not done in ticket-related tables,
certain columns in tables, or attachment files.
EXPERT CONCLUSIONS
Briefly put, Defendants did not do what they should have done (preserve and produce
responsive ESI) yet did do what they should not have done (delete and obfuscate). Based on
the sum of the evidence, Special Master concludes Defendants’ behavior was intentional.
First, Defendants’ failure to preserve ESI and deletion activities were widespread. In
total, Special Master identified 11,059,388 deleted records. Deletion activities were so
pervasive that they included many database tables (tickets, ticket posts, ticket notes, ticket
words, audit logs, etc.), and attachment files. Of the deleted records, Special Master estimates
30% of those records (based on the percent of responsive records from July 1, 2015 – July 15,
2020), or 3,317,816 were responsive to the agreed past discovery protocol. Special Master
even obtained some of the very PHP and SQL scripts Defendants used to programmatically
delete data directly from the database. This behavior is more outrageous when considering the
type of ESI Defendants deleted and withheld. Most egregious was Defendants’ deletion and
withholding of ticket attachments. As discussed, attachments arguably contained the most
responsive ESI, yet Defendants deleted 76.7% of all attachments. Defendants’ conduct was
consistent and continuous. Defendants began deleting data before the complaint was filed,
continued deleting responsive records during the discovery process, and even after Special
Master’s appointment.
Admittedly, for many of the deleted records, Special Master is unable to determine
when they were deleted. This is because Defendants deleted audit logs and failed to provide
regular backups of Ticket Databases and attachment file directories. However, what is clear,
is that Defendants deleted data during discovery after previous productions to both Plaintiffs
and Special Master.
Second, throughout the discovery process Defendants’ conduct included obfuscation
and ESI withholding. From the beginning, Defendants made material misrepresentations
regarding their backups and other ESI items to Special Master. For example, on multiple
occasions, Defendants denied items existed until they were discovered by Special Master.
Furthermore, Defendants used inaccurate date filters and did not search all ticket-related
tables and columns within those tables, resulting in more withheld records. Finally,
Defendants over-produced 27,823,240 non-responsive records (aka, data dumped) to obscure
5,096 responsive records. Consequently, Defendants’ conduct throughout discovery resulted
in Plaintiffs not receiving all available responsive information.
Third, Plaintiffs suffered irreparable harm. Only some of the withheld and deleted
records were recoverable. While a subset of deleted records is recoverable from Defendants’
earlier productions, other responsive ESI will be unavailable because the data was destroyed
and no longer exists. In fact, of the most critical evidence type, attachments, 76.7% were
deleted. Furthermore, Special Master cannot indicate whether or how much of the nonrecoverable ESI was responsive or not; a record that cannot be seen, cannot be searched.
Therefore, Defendants caused irreparable harm to Plaintiffs through permanently deleted
responsive database records and attachment files, that they will never see or know the
contents.
Fourth, Plaintiffs will continue to suffer harm with the new production. Based on the
new discovery protocol (with an expanded date range and more search terms), there will be a
greater number of missing responsive database records and attachments. Even though Special
Master has recovered all possible data, not all deleted responsive ESI is available. Due to Defendants’ data destruction efforts, the new production will be inadequate because it will not
include all responsive ESI that should have been produced to Plaintiffs.
In summary, based on Defendants’ widespread activities involving ESI deletion,
information withholding, and data dumping there is no other conclusion than Defendants
acted intentionally to avoid producing responsive ESI to Plaintiffs.
DATED: July 12, 2021
Howe Law Firm
By: /s/ Thomas P. Howe
Thomas P. Howe,
Special Discovery Master