There is little question that it is really important for quantitative BG researchers that definition A of the assumption is true: it is essential to being able to interpret heritability estimates causally, as Lewontin noted in 1974. If this assumption is invalid, no functional conclusions can be drawn regarding the trait being analyzed.\cite{LEWONTIN_2006} As Oftedal explains, critics argue that this is because of the inherently local nature of heritability estimates if the additivity assumption is not met: "In situations of additivity, heritability estimates are no longer just local. The result from one environment can be extrapolated to other environments."\cite{Oftedal_2005} (p. 702) Similarly, Lynch (2016) has recently noted that the heritability statistic's validity "...relies on the assumption that V_G and V_E act additively, so that there is no interaction or correlation between the two terms."\cite{Lynch_2016}

But there is also a second definition of the additivity assumption in regards to BG: namely, the assumption that the effects of genetic loci on variation in a complex trait are independent of each other and of the environment, so that you can add their individual effects together to get the total genetic effect. As Wahlsten (1994) wrote in describing the method of heritability analysis, "the effect of all polymorphic loci affecting a behaviour are combined by adding them to yield the total G_i for an individual, which assumes genotype at one locus does not influence the action of genes at other loci." \cite{Wahlsten_1994} (p. 245)

Thus, I will refer to the assumption that most genetic variance in complex traits, behavioral or otherwise, is additive in nature as "definition B" of the additivity assumption.

Defining heritability

Next, it should be noted that there are two types of heritability: narrow-sense heritability (h_B²) and broad-sense heritability (h²). The difference between the two is that h_B²is only based on additive genetic variance, whereas h² is based on both additive and non-additive (i.e. total) genetic variance.\cite{edition} h_B², rather than h², is the value that agricultural breeders care about, because it is used to predict what the fastest way will be to maximize a desired trait in the organism of interest through a specific selective breeding strategy.\cite{Feldman1975}

It also needs to be explained just what heritability estimation actually is: as Oftedal has noted, it is "a statistical method based on a linear analysis of variance".\cite{Oftedal_2005} (p. 700)

So first we will consider the first definition ("definition A") of the additivity assumption: that variation in a phenotype = genetically caused + environmentally caused + maybe a small interaction term. Lynch (2016) has recently highlighted the two ways that this assumption can be violated: gene-environment interaction (G x E) and gene-environment correlation (the latter also called gene-environment covariance, abbreviated G-E covariance).\cite{Lynch_2016}

In addition, Wahlsten noted in a 1990 paper that

"Additivity is often tested by examining the interaction effect in a two-way analysis of variance (ANOVA) or its equivalent multiple regression model. If this effect is not statistically significant at the α = 0.05 level, it is common practice in certain fields (e.g., human behavior genetics) to conclude that the two factors really are additive and then to use linear models, which assume additivity."

But he reported in the same paper that

"...ANOVA often fails to detect nonadditivity because it has much less power in tests of interaction than in tests of main effects. Likewise, the sample sizes needed to detect real interactions are substantially greater than those needed to detect main effects."\cite{Wahlsten_1990}

In a subsequent paper, Wahlsten noted that the issue of non-additivity is often brushed aside by human BG researchers:

"...it is asserted that the measured score (or phenotype) of an individual on a psychological test (Y_i) is the sum of only two components, G_i determined by the genes and E_i specified by the person's environment; that is, G and E must not interact."\cite{Wahlsten_1994} (p. 245)

He argues that there are fundamental biological reasons to believe that the assumption of additivity will almost always be false:

"The additive model is not biologically realistic. There are so many instances where the response of an organism to a change in environment depends on its genotype or where the consequences of a genetic defect depend strongly upon the environment, that genuine additivity of the two factors is very likely the rare exception."\cite{Wahlsten_1994} (p. 249)

And elsewhere, he contends that human BG faces unique obstacles in controlling for non-additivity that animal researchers (such as himself) do not have as much of a problem with:

"To test interaction between genotype and environment, there must be many individuals with the same genotype who are reared in different environments. This is easily achieved with standard laboratory strains but not with humans. For our species, there is no valid test of gene x environment interaction, no matter what the sample size, unless distinct alleles of a specific gene in question can be identified...Because the additivity assumption cannot be tested empirically, the whole edifice of path models must be accepted on faith, if it is to be accepted at all."\cite{Wahlsten_2000}(p. 50)

Many critics of BG have argued that definition A of the additivity assumption is untenable, and that the way in which genes and environments actually interact to produce phenotypes is just that--interactive, not additive. Thus this criticism alleges that heritability calculations are uninterpretable (at least in terms of the relative roles of genes vs. environment in causing phenotypic variation), because definition A is simply false. This criticism is well summed up in a paper by Vreeke: "The core of the critique of behavior genetics, as far as it relies on the analysis of variance, is thus that it conceptualises the relation between genes and the environment as (mainly) additive, whereas in fact development is interactive."\cite{Vreeke_2000} (p. 37) The same paper notes, "Experimental animal research shows that interaction between genotype and the environment occurs often. And if genes and the environment interact, it is not possible to separately weigh the effect of one of those factors: they depend on each other. There is no reason to expect that humans are different in this respect. An analysis of variance ignores those effects, so cannot provide a true account of the causes of behavior."\cite{Vreeke_2000} (p. 37)

Locality and causality

As noted above, critics of heritability analysis argue that the additivity assumption is false, and that heritability estimates are really just local. But which assumption is it that the critics claim is false? To some extent it is both, but definition A seems to be a more common target of such criticisms. Lewontin makes it clear that he considers the locality of heritability estimates to prevent them from allowing causal conclusions to be drawn: "There is one circumstance in which the analysis of variance can, in fact, estimate functional relationships...It is not surprising that the assumption of additivity is so often made, since this assumption is necessary to make the analysis of variance anything more than a local description."\cite{LEWONTIN_2006} This criticism is referred to by Oftedal as the "locality objection".\cite{Oftedal_2005} (p. 702)

So what we have here are two BG responses to argument 5: 1) Actually, most (if not all) genetic variance is additive, so this assumption is going to be at least mostly correct, and 2) to the extent that the assumption of additivity is false, there are plenty of ways that we can successfully account for it already, thank you very much! I will now focus a bit more on the first of these responses: that most variation in the traits behavior geneticists are studying is actually additive, meaning that the assumption Charney criticizes so harshly is actually pretty accurate. One frequently cited study by those making this claim is that of Hill et al. (2008), entitled "Data and Theory Point to Mainly Additive Genetic Variance for Complex Traits". I noted above that Neiderhiser et al. (2017) cited this paper to justify their claim that genetic variation in complex traits is mostly additive. But is this conclusion really justified by the evidence presented by Hill et al. (2008)?

Zuk et al. (2012) don't seem to think so: they argue that "...mistakenly assuming that a trait is additive can seriously distort inferences about missing heritability. From a biological standpoint, there is no a priori reason to expect that traits should be additive. Biology is filled with nonlinearity: The saturation of enzymes with substrate concentration and receptors with ligand concentration yields sigmoid response curves; cooperative binding of proteins gives rise to sharp transitions; the outputs of pathways are constrained by rate-limiting inputs; and genetic networks exhibit bistable states."\cite{Zuk_2012}

In their supplementary information (p. 45), Zuk et al. go into more detail about why they consider Hill et al.'s claims not to stand up to scrutiny. First, Zuk et al. explain two key arguments made by Hill et al.: "(a) most variants in a large population will have extremely low minor allele frequency and (b) traits caused by low-frequency alleles will not have substantial variance due to interactions." But Zuk et al. don't find these arguments the least bit convincing:

"Their claim is wrong, because the LP [linear pathway] models (a) can have substantial variance due to interactions (indeed, the majority) and yet (b) can involve any class of allele frequencies. (Specifically, LP models are defined as the minimum value of a set of traits, each of which is additive and normally distributed. There is no constraint on the allele frequencies of the variants that sum to yield these additive and normally distributed traits.)"

And on the next page:

"In effect, Hill et al.’s theory thus actually describes what happens for rare traits caused by a few rare variants. Not surprisingly, interactions account for a small proportion of the variance for such traits. Hill et al.’s model, however, is not pertinent to common traits. The interesting complex traits are those that have significant genetic variance in the population: these traits necessarily have higher allele frequencies (assuming they depend only on a few, e.g. two loci) and thus, under Hill et al.’s analysis, can involve larger interaction variance and a higher ratio VAA/VG." (Note: VAA = interaction variance and VG = total genetic variance.)

Behavior geneticists respond

To the extent that BG heritability-estimation researchers have defended their practice against the charge that it inaccurately assumes additivity, they have made such arguments as this one, made by Michael Rutter in 2003: "Critics of behavior genetics are fond of attacking it on the grounds of the unwarranted presumption of additivity. However, behavior geneticists are well aware of this issue, and it is commonplace nowadays to make explicit tests for dominance or epistatic effects. Moreover, it is perfectly straightforward to include these in any overall model. There is a need to consider such effects, but their likely existence for some traits is not a justifiable reason for doubting behavior genetics."\cite{michael2003}

So how exactly do behavior geneticists take the (non)existence of additivity/presence of non-additive effects into account? Rutter makes it sound really easy, but just how do they do it, and are their procedures for doing so adequate? It is important to keep in mind that many critics of BG argue that the techniques researchers in the field use to try to test and account for genetic non-additivity are woefully inadequate. In fact, such arguments have been made since at least 1973(!), when Willis Overton wrote, "...it does not change the situation any to maintain that this position does consider interactions by introducing an interaction term into the analysis of variance...As discussed by Overton and Reese [1972], such interaction effects, ‘are themselves linear, since they are defined as population cell means minus the sum of main effects (plus the population base rate)’ (p. 84). In fact, the very use of the term ‘interaction’ within this paradigm indicates that definitions of terms are not model independent".\cite{Overton_1973}

Just fix the model!

More recently, Partridge has argued that "Although these advances in GxE transactional models represent a substantial step forward for quantitative behavioral genetics models, there are inherent structural limitations to their analytic foundations...the nature of GxE transactions go much deeper than statistical interactionist models can accommodate. If structural sequences in the genome were isomorphic to genetic function and, more important, to protein function, then the inferred genetic variability assumed by behavioral genetic models might be more instrumental. However, genes, rather than being static structural entities, are dynamic processes."\cite{Partridge_2011}