The Use of Third-Party Surveys in Litigation
“Courts have excluded third-party surveys when the questions posed did not match the facts of the case well, or the methodologies were deemed unreliable for lack of adequate support.”
Surveys conducted independent of litigation have had mixed success in court. These surveys have been offered as evidence of customer confusion in false advertising cases, intellectual property value in patent cases, consumer behavior in antitrust cases, and plaintiff identification in class actions. In some cases, non-litigation surveys have been admitted as useful evidence on important questions for which data are scarce; in others, they have been excluded as irrelevant or unreliable.
This article examines the use of non-litigation surveys in recent cases, highlighting key considerations courts have articulated in determining their admissibility. On the one hand, surveys conducted by third parties and/or in the ordinary course of business can be particularly persuasive, precisely because they were generated independent of the dispute at hand. Such evidence may seem more objective and be viewed with less suspicion than surveys designed, fielded, and interpreted by litigants’ experts. On the other hand, non-litigation-based surveys may be deemed irrelevant if not tied closely enough to the facts of the case, unreliable if their methodology is unclear or unsupported by a survey expert, or both.
In the sections that follow, we will set out some broad principles regarding survey admissibility, review some cases in which non-litigation surveys have been both admitted and rejected, and conclude with some guidance as to best practices in this area.
Admissibility Standards
In evaluating non-litigation-based surveys, courts use factors similar to those for litigation-based surveys. According to the Federal Rules of Evidence, a qualified expert may offer testimony (including either type of survey) if “(a) the expert’s scientific, technical, or other specialized knowledge will help the trier of fact to understand the evidence or to determine a fact in issue; (b) the testimony is based on sufficient facts or data; (c) the testimony is the product of reliable principles and methods; and (d) the expert has reliably applied the principles and methods to the facts of the case.” Furthermore, “[a]n expert may base an opinion on facts or data in the case that the expert has been made aware of or personally observed,” even if those supporting facts or data are otherwise inadmissible hearsay evidence. Similarly, whether offered via expert testimony or directly by a litigating party, a non-litigation-based survey must be deemed i) sufficiently tied to the facts of the case, and b) methodologically sound to be admissible.
Illustrative Examples
Broadly speaking, surveys, whether specific to the litigation in question or not, must be demonstrably reliable and relevant for the purpose(s) for which they are being offered in order to admitted. A string of cases in recent years illustrates the how courts have applied these admissibility standards. While these standards are conceptually clear, their application is often nuanced, so it would behoove litigants to be familiar these cases when considering their survey-related evidentiary strategy.
Successfully admitted surveys
In the first cases discussed below, the courts admitted non-litigation-based surveys because they were directly relevant to key questions in the cases and deemed reliable either on their own or in conjunction with expert testimony.
- In Counts v. General Motors (2022), a false advertising case, Plaintiffs’ marketing expert successfully relied on two surveys conducted in the ordinary course of GM’s business as evidence that consumers overpaid for automobiles marketed as “clean diesel” because they were misled into thinking that those vehicles emitted less pollution than gasoline-powered alternatives. One of the surveys demonstrated the effectiveness of GM’s advertising, while the other showed the importance to consumers of the at-issue diesel badge. The court admitted Plaintiffs’ marketing expert’s opinions that relied in part on these studies because the expert “analyzed [the latter] survey in the context of GM’s advertising and applied his knowledge and experience to conclude that (1) the badge was prominently featured on the vehicle and in the advertising; and (2) the badge was meant to convey the themes consistent with the advertising, such as environmental friendliness.”
- In Optolum, Inc v. Cree, Inc (2021), a patent case involving lighting technology, Plaintiff’s damages expert successfully relied on third-party survey evidence as a basis for reasonable royalty opinions. Despite excluding other portions of the expert’s testimony, the court admitted his reliance on a journal article reporting royalty rates paid for patents based on their innovativeness gleaned from a survey of hundreds of licensing executives across a range of industries. It found that the rates reported for “major improvement” were sufficiently tied to the facts of the case, because Defendant’s internal documents described the technology at issue as “game changing.” Similarly, the court allowed the expert’s reliance on a journal article summarizing royalty rates by industry from 28 years of licensing agreements relevant because the industry categories covered included the at-issue technology/product space. In doing so, the court noted that “[t]he ‘degree of comparability’ of the license agreement [is] ‘[a] factual issue [ ] best addressed by cross examination and not by exclusion.’”
- In Epic Games v. Apple (2021), an antitrust case regarding policies governing digital mobile gaming transactions, the court relied on non-litigation surveys proffered by Apple in its findings of fact. One of those surveys “shows that 64% of iOS users would not switch to Android simply because they ‘prefer iOS,’ which is the number one reason for not switching”; another “shows that users who do switch from Android to iOS do so because they like the speed and reliability provided by iPhones”; and “[o]ther surveys show high rates of satisfaction with iOS devices.” The court found this ordinary-course survey evidence “significant not only because it was not litigation driven, but because Epic Games does not provide its own consumers survey to show that users fail to switch even when they are dissatisfied with app price, quality, or availability.” While the court concluded that these ordinary-course surveys “strongly suggests that low switching between operating systems stems from overall satisfaction with existing devices, rather than any ‘lock in,’” it was less favorably disposed to a survey conducted on behalf of Epic specifically for the case, noting that it suffered “several methodological flaws” and finding that its sponsoring expert’s “trial testimony revealed that he was more interested in a result which would assist his client’s case than in providing any objective ground to assist the Court in its decision making.”
Non-admitted surveys
In other cases, however, courts have excluded third-party surveys when the questions posed did not match the facts of the case well, or the methodologies were deemed unreliable for lack of adequate support.
- In Sgouros v. Trans Union (2022), a putative class action regarding Defendant’s VantageScore credit scoring product, Plaintiffs’ industry expert unsuccessfully proffered a pair of surveys Trans Union conducted in the ordinary course of its business as evidence of consumer confusion about different credit scoring methods. One of the surveys found that only 41% of respondents understood that a consumer’s credit score depends on the credit scoring scale being used. The other found that 70% of consumers know little about the differences between various credit scoring scales. The court struck all portions of the expert’s report citing these surveys and all her opinions reliant on them for two reasons. First, while it recognized the expert’s qualifications in consumer finance, it deemed her unqualified to opine on consumer behavior or the interpretation of consumer confusion surveys. Second, while noting the general reliability of survey evidence under Rule 702 (contingent on reliable survey design), the court found it had “no way to evaluate the reliability of [these] surveys themselves,” because the expert had merely cited a press release and webpage containing the surveys’ results without doing any evaluation of their methodology or underlying data. In doing so, the court explained that “an expert must do more than simply parrot the conclusions of other experts,” and that “when an expert relies on data they did not personally collect, their use must meet the ‘same standards of intellectual rigor that are demanded in their professional work.’”
- In Engram v. GSK Consumer Healthcare Holdings (US) Inc. (2021), another false advertising case, the court found Plaintiff’s reliance on a third-party survey insufficient to support its claim of consumer confusion. Specifically, Plaintiff claimed that a reasonable consumer would misconstrue the proximate (but graphically distinct) phrases “8 HOUR MOISTURE” and “SPF 15” on the front of Chapstick packaging as meaning that the product provides eight hours of sun protection, regardless of directions on the back of the packaging to “apply liberally before sun exposure” and “reapply at least every 2 hours.” Plaintiff buttressed this claim with a “neutral third-party” survey showing that 64.4% of respondents shown the front of the packaging “believed that the Product provided 8 hours of sun protection based on the [front] packaging.” In granting Defendant’s motion to dismiss, the court stressed that “context is crucial” in deceptive marketing cases, particularly when the alleged misstatements fall in the “ambiguous” range between “absolutely true” and “outright false,” as it deemed they did in this case. As such, the court found the survey unpersuasive because respondents “were differently situated from the ‘reasonable consumer’” by virtue of not being shown the back packaging as well.
- In Hoekman v. Educ. Minn (2020), a dispute over union fee requirements, Plaintiffs’ economic expert’s opinions on class certification requirements were excluded in part due to reliance on what the court deemed insufficient survey evidence. Specifically, the court found a National Education Association (NEA) survey finding that 69% of union member respondents who “‘woke up tomorrow’ and discovered they could still have union representation without paying anything would choose that option” unpersuasive because “the question posed in the NEA survey at issue offers no data on the number of union members who ‘reluctantly joined the union because of a lack of information about constitutional rights, or as a result of a ‘cost-benefit’ analysis’ between union dues and ‘fair share’ fees,’” which were key questions in the case. The court was also critical of the fact that the expert’s consideration of the [national] survey does not account for any specific factors, whatever they may be, “unique to Minnesota public employees in this state.” In its denial of class certification, the court also rejected the expert’s assertion that he could design his own survey to identify members of the “reluctant” and “reluctant-uninformed” subclasses because he had failed to provide any details regarding how he would do so.
- Finally, in Hawyuan Yu v. Dr Pepper Snapple Group (2020), a class action alleging misleading use of the terms “natural” and/or “all natural ingredients” in advertisements of applesauce containing trace amounts of pesticides, the court found two third-party surveys added to Plaintiff’s amended complaint insufficient to defeat Defendant’s motion to dismiss. Regarding a 2015 Consumer Reports Survey claiming that “63 percent of respondents think that the natural label on packaged and processed food means that ‘no toxic pesticides were used,’” the court rejected Plaintiff’s expansive interpretation of “toxic” as meaning harmful to the environment rather than to humans. The court found that a 2019 study conducted for the Corn Refiners Association “stat[ing] that 68.1 percent of consumers would consider food produced from crops sprayed with synthetic pesticides not natural” was only “tangentially related to Plaintiff’s claims, at best,” and insufficient in a context where “[o]ther courts in the district have held surveys specifically commissioned for the case insufficient as a matter of law when the pleadings had previously been dismissed for failing to state a claim.” The court ultimately granted the Defendant’s motion to dismiss the case in full, without leave to amend.
A Cautionary Note
In formulating survey-related strategies, litigants should consider all possible interpretations of all the survey data and results before deciding whether to proffer it and how heavily to rely on it. For example, in dismissing the Hawyuan Yu v. Dr. Pepper Snapple Group case discussed above, the court noted a different portion of one of the surveys relied on by Plaintiff that “arguably undermine[d] rather than support[ed], Plaintiff’s argument about the reasonable consumer’s interpretation of the word ‘natural.’” Similarly, in Dent v. NFL (2021), the court drew a conclusion from a prior third-party survey of retired NFL players that was diametrically opposed to the one Plaintiff had intended. Specifically, Plaintiff argued that the survey documenting players’ use of opioids and injuries sustained while in the NFL was evidence of the NFL’s culpability for players’ injuries. Instead, in its order denying class certification, the court critically noted that the survey documented that significant numbers of survey respondents reported 1) not using prescription opioids during their NFL careers, and 2) receiving opioids from sources other than their NFL clubs.
Steps to Survey Success
Survey evidence must be sufficiently tied to the facts of the case and methodologically sound to be admissible. As we have shown here, these guidelines apply equally to third-party/non-litigation-based surveys as they do to those designed and executed by experts for a specific matter. In some cases, having a survey, marketing, consumer-behavior, or similarly qualified expert opine on how non-litigation surveys meet these criteria can improve the likelihood of admissibility of this type of evidence. Adherence to these principles and familiarity with the examples above will aid litigants’ efforts in this area.
ACKNOWLEDGEMENTS: The authors would like to thank Genna Liu, formerly of Analysis Group, Inc., for her contributions to this article.
Hana Dai
Hana Dai is a Senior Analyst with Analysis Group, Inc. Ms. Dai has done work in the antitrust, commercial disputes, and intellectual property practices with an emphasis on the use […see more]
Peter Hess
Peter Hess is a Vice President with Analysis Group, Inc. Dr. Hess specializes in the application of microeconomic theory and analytical methods in litigation, business, and policy contexts. He has […see more]
Jacob Judd
Jacob Judd is a Senior Analyst with Analysis Group, Inc. Mr. Judd has worked in the antitrust, intellectual property, and data security practices with a focus on the use of […see more]
